[ale] SHA1SUM question
Greg Freemyer
greg.freemyer at gmail.com
Wed Dec 6 17:21:43 EST 2006
Jerry,
That is typically what rsync is used for. ie. Efficiently making 2
files the same that start out close to being the same but having
random differences in the middle somewhere..
Let's say you have redundant webservers handling different parts of
the world and a R&D webserver you work on. From the webservers you
are distributing some big files / pdfs / etc.
You would work on the R&D server until your happy with your big files,
then use rsync to update the production sites.
rsync takes the big files and breaks them into chunks (I don't know
the chunk size). For each chunk it calculates the md5 on both the
source and destination ends. If they are the same it moves on to the
next chunk, If they're different it breaks the chunks into smaller
chunks and compares those md5s. It continues the process until it
finds the minimal amount of data that is different. It then sends the
differing data from the source to the destination and replaces that
part of the file. Then it continues the process until the 2 files are
in "sync".
I think it is also smart enough to look for data insertions, but I'm
not sure about that.
As you can see the main purpose of rsync is to synchronize large files
that are on opposite ends of a slow data connection.
Obviously rsync can also handle small files too, but all the chunk
logic is wasted in that situation.
Greg
On 12/6/06, Jerry Yu <jjj863 at gmail.com> wrote:
> I knew it can do files as well as directories, but I just didn't know rsync
> has intelligence to fetch the 'diff' between the source and a corrupted
> download. I assume that's what you meant, right?
>
> On 12/6/06, James P. Kinney III <jkinney at localnetsolutions.com> wrote:
> > Jerry,
> > If you just specify a file, that's all it will do. I've been using it
> > lately to spread around config file updates to a list of servers
> >
> > for server in `cat serverlist`
> > do
> > rsync -e ssh foo $server:foo
> > done
> >
> > With ssh-keys installed, life is smooth. :)
> >
> > On Wed, 2006-12-06 at 10:20 -0500, Jerry Yu wrote:
> > > Thanks, James. I usually use rsync to sync up directories, but didn't
> > > know it can efficiently sync up partial/corrupted single file as
> > > well.
> > >
> > > John, my torrent session takes three hours to finish the last 1%, not
> > > exactly sure why.
> > >
> > > On 12/6/06, James P. Kinney III < jkinney at localnetsolutions.com> wrote:
> > > It just a basic rsync process to the server. The caveat is the
> > > server
> > > must support rsync. "rsync <options> <master-file>
> > > <copy-file>". I have
> > > used this on the Fedora mirrors that support rsync with great
> > > success.
> > >
> > > The nice thing is if the connection croaks, just run it again.
> > >
> > > On Wed, 2006-12-06 at 08:39 -0500, Jerry Yu wrote:
> > > > John, I downloaded a month back and my sha1sum does match.
> > > > James, I'd like to learn how to 'clean-up' a corrupted
> > > download of a
> > > > single file using rsync? Occasionally a large download gets
> > > corrupted
> > > > and it's pia to restart from scratch...
> > > >
> > > > On 12/5/06, James P. Kinney III
> > > <jkinney at localnetsolutions.com> wrote:
> > > > John,
> > > >
> > > > The sha1sum works just like the md5sum but uses a
> > > different
> > > > algorithm.
> > > > If the checksum is bad either the download is bad or
> > > the
> > > > checksum is
> > > > bad. I have had good luck using rsync to "cleanup" a
> > > bad
> > > > download.
> > > >
> > > > On Tue, 2006-12-05 at 21:48 -0500, John Mills wrote:
> > > > > ALErs -
> > > > >
> > > > > I downloaded the FC6 i386 DVD image, and also a
> > > file of SHA1
> > > > sums for the
> > > > > various FC6 *.iso files. I ran 'sha1sum' on my
> > > file and it
> > > > wasn't even
> > > > > close.
> > > > > That's a lot of downloading for garbage results!
> > > > >
> > > > > Is there a second check on this in case I made a
> > > mistake the
> > > > first time?
> > > > > It's the first time I use sha1sum so I'ld love to
> > > be wrong,
> > > > but it looked
> > > > > pretty simple going in!
> > > > >
> > > > > $%^&%#@!!
> > > > >
> > > > > - Mills
> > > > >
> > > > >
> _______________________________________________
> > > > > Ale mailing list
> > > > > Ale at ale.org
> > > > >
> http://www.ale.org/mailman/listinfo/ale
> > > > --
> > > > James P. Kinney III
> > > > CEO & Director of Engineering
> > > > Local Net Solutions,LLC
> > > > 770-493-8244
> > > > http://www.localnetsolutions.com
> > > >
> > > > GPG ID: 829C6CA7 James P. Kinney III ( M.S. Physics)
> > > > <jkinney at localnetsolutions.com >
> > > > Fingerprint = 3C9E 6366 54FC A3FE BA4D 0659 6190
> > > ADC3 829C
> > > > 6CA7
> > > >
> > > >
> > > >
> _______________________________________________
> > > > Ale mailing list
> > > > Ale at ale.org
> > > >
> http://www.ale.org/mailman/listinfo/ale
> > > >
> > > >
> > > >
> > > >
> _______________________________________________
> > > > Ale mailing list
> > > > Ale at ale.org
> > > > http://www.ale.org/mailman/listinfo/ale
> > > --
> > > James P. Kinney III
> > > CEO & Director of Engineering
> > > Local Net Solutions,LLC
> > > 770-493-8244
> > > http://www.localnetsolutions.com
> > >
> > > GPG ID: 829C6CA7 James P. Kinney III ( M.S. Physics)
> > > <jkinney at localnetsolutions.com>
> > > Fingerprint = 3C9E 6366 54FC A3FE BA4D 0659 6190 ADC3 829C
> > > 6CA7
> > >
> > >
> > > _______________________________________________
> > > Ale mailing list
> > > Ale at ale.org
> > > http://www.ale.org/mailman/listinfo/ale
> > >
> > >
> > >
> > > _______________________________________________
> > > Ale mailing list
> > > Ale at ale.org
> > > http://www.ale.org/mailman/listinfo/ale
> > --
> > James P. Kinney III
> > CEO & Director of Engineering
> > Local Net Solutions,LLC
> > 770-493-8244
> > http://www.localnetsolutions.com
> >
> > GPG ID: 829C6CA7 James P. Kinney III (M.S. Physics)
> > <jkinney at localnetsolutions.com>
> > Fingerprint = 3C9E 6366 54FC A3FE BA4D 0659 6190 ADC3 829C 6CA7
> >
> >
> > _______________________________________________
> > Ale mailing list
> > Ale at ale.org
> > http://www.ale.org/mailman/listinfo/ale
> >
> >
> >
>
>
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> http://www.ale.org/mailman/listinfo/ale
>
>
--
Greg Freemyer
The Norcross Group
Forensics for the 21st Century
More information about the Ale
mailing list