[ale] SHA1SUM question
Jerry Yu
jjj863 at gmail.com
Wed Dec 6 18:05:57 EST 2006
ah, thanks for the detailed explannation, Greg. Learn something new
everyday, even on some oldest tools in my bag!
I'll gladly stop wasting the chunking magic/logic.
On 12/6/06, Greg Freemyer <greg.freemyer at gmail.com> wrote:
>
> Jerry,
>
> That is typically what rsync is used for. ie. Efficiently making 2
> files the same that start out close to being the same but having
> random differences in the middle somewhere..
>
> Let's say you have redundant webservers handling different parts of
> the world and a R&D webserver you work on. From the webservers you
> are distributing some big files / pdfs / etc.
>
> You would work on the R&D server until your happy with your big files,
> then use rsync to update the production sites.
>
> rsync takes the big files and breaks them into chunks (I don't know
> the chunk size). For each chunk it calculates the md5 on both the
> source and destination ends. If they are the same it moves on to the
> next chunk, If they're different it breaks the chunks into smaller
> chunks and compares those md5s. It continues the process until it
> finds the minimal amount of data that is different. It then sends the
> differing data from the source to the destination and replaces that
> part of the file. Then it continues the process until the 2 files are
> in "sync".
>
> I think it is also smart enough to look for data insertions, but I'm
> not sure about that.
>
> As you can see the main purpose of rsync is to synchronize large files
> that are on opposite ends of a slow data connection.
>
> Obviously rsync can also handle small files too, but all the chunk
> logic is wasted in that situation.
>
> Greg
>
> On 12/6/06, Jerry Yu <jjj863 at gmail.com> wrote:
> > I knew it can do files as well as directories, but I just didn't know
> rsync
> > has intelligence to fetch the 'diff' between the source and a corrupted
> > download. I assume that's what you meant, right?
> >
> > On 12/6/06, James P. Kinney III <jkinney at localnetsolutions.com> wrote:
> > > Jerry,
> > > If you just specify a file, that's all it will do. I've been using it
> > > lately to spread around config file updates to a list of servers
> > >
> > > for server in `cat serverlist`
> > > do
> > > rsync -e ssh foo $server:foo
> > > done
> > >
> > > With ssh-keys installed, life is smooth. :)
> > >
> > > On Wed, 2006-12-06 at 10:20 -0500, Jerry Yu wrote:
> > > > Thanks, James. I usually use rsync to sync up directories, but
> didn't
> > > > know it can efficiently sync up partial/corrupted single file as
> > > > well.
> > > >
> > > > John, my torrent session takes three hours to finish the last 1%,
> not
> > > > exactly sure why.
> > > >
> > > > On 12/6/06, James P. Kinney III < jkinney at localnetsolutions.com>
> wrote:
> > > > It just a basic rsync process to the server. The caveat is
> the
> > > > server
> > > > must support rsync. "rsync <options> <master-file>
> > > > <copy-file>". I have
> > > > used this on the Fedora mirrors that support rsync with
> great
> > > > success.
> > > >
> > > > The nice thing is if the connection croaks, just run it
> again.
> > > >
> > > > On Wed, 2006-12-06 at 08:39 -0500, Jerry Yu wrote:
> > > > > John, I downloaded a month back and my sha1sum does
> match.
> > > > > James, I'd like to learn how to 'clean-up' a corrupted
> > > > download of a
> > > > > single file using rsync? Occasionally a large download
> gets
> > > > corrupted
> > > > > and it's pia to restart from scratch...
> > > > >
> > > > > On 12/5/06, James P. Kinney III
> > > > <jkinney at localnetsolutions.com> wrote:
> > > > > John,
> > > > >
> > > > > The sha1sum works just like the md5sum but uses a
> > > > different
> > > > > algorithm.
> > > > > If the checksum is bad either the download is bad
> or
> > > > the
> > > > > checksum is
> > > > > bad. I have had good luck using rsync to "cleanup"
> a
> > > > bad
> > > > > download.
> > > > >
> > > > > On Tue, 2006-12-05 at 21:48 -0500, John Mills
> wrote:
> > > > > > ALErs -
> > > > > >
> > > > > > I downloaded the FC6 i386 DVD image, and also a
> > > > file of SHA1
> > > > > sums for the
> > > > > > various FC6 *.iso files. I ran 'sha1sum' on my
> > > > file and it
> > > > > wasn't even
> > > > > > close.
> > > > > > That's a lot of downloading for garbage results!
> > > > > >
> > > > > > Is there a second check on this in case I made a
> > > > mistake the
> > > > > first time?
> > > > > > It's the first time I use sha1sum so I'ld love
> to
> > > > be wrong,
> > > > > but it looked
> > > > > > pretty simple going in!
> > > > > >
> > > > > > $%^&%#@!!
> > > > > >
> > > > > > - Mills
> > > > > >
> > > > > >
> > _______________________________________________
> > > > > > Ale mailing list
> > > > > > Ale at ale.org
> > > > > >
> > http://www.ale.org/mailman/listinfo/ale
> > > > > --
> > > > > James P. Kinney III
> > > > > CEO & Director of Engineering
> > > > > Local Net Solutions,LLC
> > > > > 770-493-8244
> > > > > http://www.localnetsolutions.com
> > > > >
> > > > > GPG ID: 829C6CA7 James P. Kinney III ( M.S.
> Physics)
> > > > > <jkinney at localnetsolutions.com >
> > > > > Fingerprint = 3C9E 6366 54FC A3FE BA4D 0659 6190
> > > > ADC3 829C
> > > > > 6CA7
> > > > >
> > > > >
> > > > >
> > _______________________________________________
> > > > > Ale mailing list
> > > > > Ale at ale.org
> > > > >
> > http://www.ale.org/mailman/listinfo/ale
> > > > >
> > > > >
> > > > >
> > > > >
> > _______________________________________________
> > > > > Ale mailing list
> > > > > Ale at ale.org
> > > > > http://www.ale.org/mailman/listinfo/ale
> > > > --
> > > > James P. Kinney III
> > > > CEO & Director of Engineering
> > > > Local Net Solutions,LLC
> > > > 770-493-8244
> > > > http://www.localnetsolutions.com
> > > >
> > > > GPG ID: 829C6CA7 James P. Kinney III ( M.S. Physics)
> > > > <jkinney at localnetsolutions.com>
> > > > Fingerprint = 3C9E 6366 54FC A3FE BA4D 0659 6190 ADC3 829C
> > > > 6CA7
> > > >
> > > >
> > > > _______________________________________________
> > > > Ale mailing list
> > > > Ale at ale.org
> > > > http://www.ale.org/mailman/listinfo/ale
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > Ale mailing list
> > > > Ale at ale.org
> > > > http://www.ale.org/mailman/listinfo/ale
> > > --
> > > James P. Kinney III
> > > CEO & Director of Engineering
> > > Local Net Solutions,LLC
> > > 770-493-8244
> > > http://www.localnetsolutions.com
> > >
> > > GPG ID: 829C6CA7 James P. Kinney III (M.S. Physics)
> > > <jkinney at localnetsolutions.com>
> > > Fingerprint = 3C9E 6366 54FC A3FE BA4D 0659 6190 ADC3 829C 6CA7
> > >
> > >
> > > _______________________________________________
> > > Ale mailing list
> > > Ale at ale.org
> > > http://www.ale.org/mailman/listinfo/ale
> > >
> > >
> > >
> >
> >
> > _______________________________________________
> > Ale mailing list
> > Ale at ale.org
> > http://www.ale.org/mailman/listinfo/ale
> >
> >
>
>
> --
> Greg Freemyer
> The Norcross Group
> Forensics for the 21st Century
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> http://www.ale.org/mailman/listinfo/ale
>
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Ale
mailing list