[ale] rsync exclude list

[Michael B. Trausch]

On Wed, 2010-06-30 at 13:01 -0400, Paul Cartwright wrote:
>  ok, so just in case my HD decides to get too hot & crashes, I decided
> to take 
>  my OLD dell desktop & make a copy of my current setup, with logins
> for me & 
>  wife..
>  Debian Lenny. No big deal, got it setup, setup openssh-server, ran
> rsync.. 
>  oops, I don't have near enough space on old driver, so I need to
> EXCLUDE some 
>  STUFF.. I have a folder under Documents called software. Here is my
> rsync 
>  script:
>  
>  rsync -varpltzo --exclude-from=/home/pbc/bin/dontrsync 
>  192.168.10.2:/home/pbc/Documents/ /home/pbc/Documents/
>  
>  heres the /home/pbc/bin/dontrsync file:
>  /home/pbc/Documents/software/*
> software/
> /software/
> /software/*
>  
> 
> I've also tried:
> 
> rsync -varpltzo --exclude=/home/pbc/Documents/software/* 
> 192.168.10.2:/home/pbc/Documents/ /home/pbc/Documents/
> 
>  
>  it still copies software.. what am I doing wrong... 

According to the rsync man page, the following options are likely to be
relevant:

        -C, --cvs-exclude           auto-ignore files in the same
                                    way CVS does
            --exclude=PATTERN       exclude files matching PATTERN
            --exclude-from=FILE     read exclude patterns from FILE

Now, if you use -C (--cvs-exclude), the man page has this to say:

       -C, --cvs-exclude
              This  is  a useful shorthand for excluding a broad
              range of files that you often don’t want to trans‐
              fer  between  systems. It uses a similar algorithm
              to CVS to determine if a file should be ignored.

              The exclude list is  initialized  to  exclude  the
              following items (these initial items are marked as
              perishable — see the FILTER RULES section):

                     RCS SCCS CVS CVS.adm RCSLOG  cvslog.*  tags
                     TAGS  .make.state .nse_depinfo *~ #* .#* ,*
                     _$*  *$  *.old  *.bak  *.BAK  *.orig  *.rej
                     .del-*  *.a  *.olb *.o *.obj *.so *.exe *.Z
                     *.elc *.ln core .svn/ .git/ .bzr/

              then, files listed in a $HOME/.cvsignore are added
              to  the list and any files listed in the CVSIGNORE
              environment  variable  (all  cvsignore  names  are
              delimited by whitespace).

              Finally,  any file is ignored if it is in the same
              directory as a .cvsignore file and matches one  of
              the  patterns listed therein.  Unlike rsync’s fil‐
              ter/exclude files, these  patterns  are  split  on
              whitespace.  See the cvs(1) manual for more infor‐
              mation.

              If you’re combining  -C  with  your  own  --filter
              rules, you should note that these CVS excludes are
              appended at the end of your own rules,  regardless
              of  where  the  -C was placed on the command-line.
              This makes them a lower priority  than  any  rules
              you  specified explicitly.  If you want to control
              where these CVS excludes get  inserted  into  your
              filter rules, you should omit the -C as a command-
              line option and use a combination  of  --filter=:C
              and --filter=-C (either on your command-line or by
              putting the ":C" and "-C" rules into a filter file
              with your other rules).  The first option turns on
              the  per-directory  scanning  for  the  .cvsignore
              file.  The second option does a one-time import of
              the CVS excludes mentioned above.

This might be the most useful option if you have lots of files scattered
throughout the filesystem.  You can also maintain such a list in a
single exclude file, though that would be, IMHO, less ideal than using a
per-directory exclusion file.  Reading this section of the man page,
though, led me to look into the "--filter" option that is discussed here
(it's funny, I use tools like rsync every day and I _still_ find that
there is so much to learn about them!).

So, I read about -f (--filter=RULE) first, which shows that you can "add
rules to selectively exclude certain files from the list of files to be
transferred."  It then says to see the "FILTER RULES" section.  Alright,
but first, there's the -F option listed next in the man page.  It is
some shorthand (I'll leave you to read the details of it in your local
copy of the man page), but essentially, the first time you use -F, this
tells rsync to look for a file ".rsync-filter" in each directory and use
rules found from there to filter the transfer.  (Again, we have to wait
until we reach FILTER RULES before we learn how to do that.)  The second
time -F is used on the command line, it says to also filter out
the .rsync-filter files from the transfer (and although I can see where
that might be useful, I can't see using it all the time).

Okay, so on to FILTER RULES, which says... well, it says a lot.  I'd
suggest reading it directly.  But essentially you can use these files
to, say, exclude all files but particular ones, or include all files but
particular ones, other filter files to read in, what _not_ to delete
when a delete option is specified, and so forth.  It is quite powerful,
but I think -F is probably all that you would really need, and then just
manage the things per-directory.  That seems reasonable to me... and in
fact, I am going to adjust my script to do just that.  I've been using
'--exclude' to just include a few select files in my transfers, but now
I think that I will use these files because that is far more flexible.
For example, one of my exclude rules is '*.iso', but then I have to
manually sync the ISO images that *do* matter, so... yep, time to switch
to this thing now that I have learned about it.

	--- Mike