[ale] rsync with big files (.5GB and above)

[Jeff Lightner]

Check out the --append option.

man rsync

by the way even ftp will append to files on resends.

-----Original Message-----
From: ale-bounces at ale.org [mailto:ale-bounces at ale.org] On Behalf Of Greg
To: ale at ale.org
Freemyer
Sent: Monday, April 30, 2007 4:08 PM
To: Atlanta Linux Enthusiasts
Subject: Re: [ale] rsync with big files (.5GB and above)

Jeff,

If I've sent 500 MB of a 700MB file when it times out, how can I
resume that file?  I have not noticed a rsync option to do that.

It appears my trouble is I have about 50 500MB+ files.  (All pretty
static.)  I'm getting one or  two across to the backup server each
night, but that is it.  In a month or two they should all be there if
I just leave things the way they are :(

Worse, any small files that may have changed recently are not getting
updated either because of the initial timeout.

Greg


On 4/30/07, Jeff Lightner <jlightner at water.com> wrote:
> I don't think it's a generic rsync issue.   At a prior job we used to
> run all day rsync scripts to transfer databases - of course that was
all
> done within the same VLAN.
>
> Perhaps if you script it so it just resumes where it left off if it
> drops the connection?
>
> -----Original Message-----
> From: ale-bounces at ale.org [mailto:ale-bounces at ale.org] On Behalf Of
Greg
> Freemyer
> Sent: Monday, April 30, 2007 3:42 PM
> To: Atlanta Linux Enthusiasts
> Subject: [ale] rsync with big files (.5GB and above)
>
> All,
>
> I setup a dreamhost account like I think I mentioned before.  I've
> been rsync'ing my encrypted backups there for a few weeks.
>
> As part of my implementation process I started only sending small
> files (via the --max-size option of rsync).
>
> When I was working with files below .5 GB I had the occasional
> time-out, but in general it worked most of the time.
>
> I set my max to 2 GB about a week ago.  Since then I get a few
> additional files copied each attempt, but I don't think I have sent
> more than 2 files in any one rsync instance.
>
> Does anyone know if this is a generic issue with rsync, or am I just
> seeing more timeouts because my discrete rsync instances are running
> longer and dreamhost is having issues.  (I've never run more than 3 or
> 4 hours without a timeout).
>
> FYI: I have --timeout set to 5 minutes in my backup script, but I've
> been manually running rsync during the day with timeouts as long as 2
> hours.  Longer timeouts seem more reliable, but rsync still fails
> after a couple of files.  And I think the timeout is per i/o, not per
> file.
>
> Thanks
> Greg
> --
> Greg Freemyer
> The Norcross Group
> Forensics for the 21st Century
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> http://www.ale.org/mailman/listinfo/ale
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> http://www.ale.org/mailman/listinfo/ale
>


-- 
Greg Freemyer
The Norcross Group
Forensics for the 21st Century
_______________________________________________
Ale mailing list
Ale at ale.org
http://www.ale.org/mailman/listinfo/ale