[ale] Online Backup

[Greg Freemyer]

On Mon, Aug 25, 2008 at 3:31 PM, Jim Popovitch <yahoo at jimpop.com> wrote:
> On Mon, Aug 25, 2008 at 12:22 PM, Greg Freemyer <greg.freemyer at gmail.com> wrote:
>>
>> Will S3 let me use rsync in a similar way?  If so, I may look into that.
>
> Nope, not yet.  There are a bunch of "workarounds".... but I haven't
> run into one yet that works the way I would think it would.  Here's a
> good list of what's available:
> http://jeremy.zawodny.com/blog/archives/007641.html
>
>> FYI: I encrypt my data before sending it offsite. I do so by holding
>> my backup archive within encfs encrypted folders.  I use the
>> unencrypted virtual folders, but rsync out the raw encrypted folders.
>
> I, for one, would like to know more about this.   Can this solution be
> automated?  How is the encryption achieved?
>

I have it automated now.  The hardest part was getting rsync to work
well in the presence of network errors.  (ie. timeouts, etc.)

The way I have done it the encryption password is in plain text in my
script, but I can live with that.  (It may be possible to avoid doing
so.  I did not care. I'm just trying to keep the admins on the remote
server from seeing my data.)

So in my script I have (with some newly added comments):

# attempt to mount the unencrypted dir
if [ ! -d "/backup/data-rdiff/rdiff-backup-data" ] ; then
        encfs -S "/backup/data-enc-raw" "/backup/data-rdiff" << EOF
<plain_text_pasword>
EOF

# verify the unencrypted dir is mounted
    if [ ! -d "/backup/data-rdiff/rdiff-backup-data" ] ; then
        mailx Greg.Freemyer at gmail.com -s "Backup Failure" << EOF
/backup/rdiff-backup-data directory NOT available
EOF
        exit 0
    fi


Then I update my unencrypted backup archive via:

       rdiff-backup -v5 --print-statistics /data "/backup/data-rdiff"

Then I unmount the unencrypted dir by:

    fusermount -uz /backup/data-rdiff

Then I rsync it with a bash function perform_rsync_backups() I wrote.

perform_rsync_backups()
{
date; echo rsync data starting

START=`date +%s`
MAX_TIME='14400'    # 4 hours

# This loop says to keep retrying rsyncs for up to 4 hours.  Then give
up.  We will try again tomorrow night.
for (( DELTA=0 ; DELTA < MAX_TIME ; DELTA = NOW - START))
do
       #while building initial repository use --max-size to limit the
amount of data being sent.  Start very small, then grow it daily.
        #rsync -avh --stats --links
--partial-dir=/home/forensic1000/forensic1000/transfer
--max-size=3700M --timeout=1800 /$BACKUP_PATH/data-enc-raw
forensic1000 at norcrossgrop.dreamhosters.com:forensic1000/

       # once archive is complete, comment out above and use below to
rsync the whole repository nightly.
        rsync -avh --stats --links
--partial-dir=/home/forensic1000/forensic1000/transfer --timeout=1800
/$BACKUP_PATH/data-enc-raw
forensic1000 at norcrossgrop.dreamhosters.com:forensic1000/

# return code of 30 is a timeout (iirc)
if [ $? != 30 ]
then
   break
fi
NOW=`date +%s`
done
...
date; echo All rsync\'s done
}

Hope that helps.

Any questions, just let me know.

Greg
-- 
Greg Freemyer
Litigation Triage Solutions Specialist
http://www.linkedin.com/in/gregfreemyer
First 99 Days Litigation White Paper -
http://www.norcrossgroup.com/forms/whitepapers/99%20Days%20whitepaper.pdf

The Norcross Group
The Intersection of Evidence & Technology
http://www.norcrossgroup.com