[ale] Duplicity very slow

[James Sumners]

I use the script included in this email to run backups. It's really just a
call to duplicity after some sanity checks. Run on a system that hasn't
been backed up before, where that system is about 1.7GB of data, it takes
about 8 minutes to run.

````
#!/bin/bash

# Pre-execution check for bsfl
# Set options afterward
if [ ! -f /etc/bsfl ]; then
  echo "Backup script requires bsfl (https://code.google.com/p/bsfl/)."
  exit 1
fi
source /etc/bsfl

### Options ###

# Set to the desired logfile path and name
LOG_FILE="$(dirname $0)/logs/runbackup-$(date +'%m-%d-%Y').log"

# Set to the file that contains backup exclusions (format = line separated
paths)
EXCLUDES="$(dirname $0)/excludes"

# Set to the NFS mount point
# Be sure to configure /etc/fstab appropriately
NFS_DIR="$(dirname $0)/remote"

# Set to test string for testing NFS mount success
NFS_MOUNT_TEST="^//bigspace.*/this_system_dn"

# Set to the remote backup container directory
# Backups will be stored in subdirectories of this directory
BACKUP_DIR="${NFS_DIR}"

# Set to the email address that will receive notifications
# of backup failures
ERROR_EMAIL_ADDR="admin at example.com"

# Set to the list of switches you want passed to Duplicity
# Add '--time-separator .' if using an old version of Duplicity
DUPLICITY_SWITCHES="-v4 --no-encryption --exclude-globbing-filelist
${EXCLUDES} --exclude-other-filesystems --volsize 150 --full-if-older-than
30D"


### Begin actual script ###

function notify {
  mail -s "Backup failure on $(hostname)" ${ERROR_EMAIL_ADDR} < ${LOG_FILE}
}

# Turn on bsfl logging support
LOG_ENABLED="yes"

# We need to be root to 1) read all files and 2) mount the NFS
USER=$(whoami)
if [ "${USER}" != "root" ]; then
  notify
  die 2 "Backup must be run as root."
fi

log "Mounting NFS"
mount ${NFS_DIR}

NFS_MOUNTED=$(cat /proc/mounts | grep ${NFS_MOUNT_TEST})
if [ ! $? -eq 0 ]; then
  notify
  umount ${NFS_DIR}
  die 3 "Could not mount remote share."
fi

# Let's make sure we have enough room on the remote system
STAT_INFO=$(stat -f --format='%b %a %S' ${NFS_DIR})
TOTAL_BLOCKS=$(echo ${STAT_INFO} | awk '{print $1}')
FREE_BLOCKS=$(echo ${STAT_INFO} | awk '{print $2}')
BLOCK_SIZE=$(echo ${STAT_INFO} | awk '{print $3}')
# (1024bytes * 1024kilobytes) / (x bytes) = (1 megabyte [in bytes]) / (x
bytes)
# => number of blocks in 1 megabyte = y
REMOTE_FREE_BYTES=$(echo "${FREE_BLOCKS} / (1048576 / ${BLOCK_SIZE})" | bc
-l)
log "Remote free bytes = ${REMOTE_FREE_BYTES}"

STAT_INFO=$(stat -f --format='%b %a %S' /)
TOTAL_BLOCKS=$(echo ${STAT_INFO} | awk '{print $1}')
FREE_BLOCKS=$(echo ${STAT_INFO} | awk '{print $2}')
BLOCK_SIZE=$(echo ${STAT_INFO} | awk '{print $3}')
LOCAL_USED_BYTES=$(echo "(${TOTAL_BLOCKS} - ${FREE_BLOCKS}) / (1048576 /
${BLOCK_SIZE})" | bc -l)
log "Local used bytes = ${LOCAL_USED_BYTES}"

REMOTE_HAS_ROOM=$(echo "${REMOTE_FREE_BYTES} > ${LOCAL_USED_BYTES}" | bc -l)
if [ ${REMOTE_HAS_ROOM} -eq 0 ]; then
  log_error "Remote system does not have enough free space for the backup."
  notify
  umount ${NFS_DIR}
  die 4 "Remote system does not have enough free space for the backup."
else
  log "Remote system has enough room. Proceeding with backup."
  log "===== ===== ===== ====="
  log ""
fi

# Send today's delta over
duplicity ${DUPLICITY_SWITCHES} / file://${BACKUP_DIR} 2>&1 1>>${LOG_FILE}

# Remove unnecessary data over 6 days old
duplicity remove-older-than 6D --force ${DUPLICITY_SWITCHES}
file://${BACKUP_DIR} 2>&1 1>>${LOG_FILE}

log ""
log "===== ===== ===== ====="
log "Backup complete."

umount ${NFS_DIR}
````

On Tue, Apr 21, 2015 at 1:21 PM, Edward Holcroft <eholcroft at mkainc.com>
wrote:

> I'm trying without encryption. It still seems the same, not constant hard
> drive i/o going on. Will let it run for a while and see what happens.
>
> ed
>


-- 
James Sumners
http://james.sumners.info/ (technical profile)
http://jrfom.com/ (personal site)
http://haplo.bandcamp.com/ (band page)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ale.org/pipermail/ale/attachments/20150421/63154090/attachment.html>