[ale] Duplicity very slow

[Edward Holcroft]

This is what I got local drive to local drive test (no USB involved). This
seems impossibly slow.

StartTime 1429713308.46 (Wed Apr 22 10:35:08 2015)
EndTime 1429728281.54 (Wed Apr 22 14:44:41 2015)
ElapsedTime 14973.08 (4 hours 9 minutes 33.08 seconds)
SourceFiles 52736
SourceFileSize 98274663358 (91.5 GB)
NewFiles 52736
NewFileSize 98274663358 (91.5 GB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 52736
RawDeltaSize 98251115454 (91.5 GB)
TotalDestinationSizeChange 92760516730 (86.4 GB)
Errors 0


On Wed, Apr 22, 2015 at 9:53 AM, James Sumners <james.sumners at gmail.com>
wrote:

> Try it to another internal volume or across your network. After
> discovering my oversight yesterday, I ran a backup to pickup the missed
> data. This was the result:
>
> ````
> Import of duplicity.backends.dpbxbackend Failed: No module named dropbox
> Reading globbing filelist ./excludes
> Local and Remote metadata are synchronized, no sync needed.
> Last full backup date: Wed Apr 15 15:18:59 2015
> --------------[ Backup Statistics ]--------------
> StartTime 1429648992.82 (Tue Apr 21 16:43:12 2015)
> EndTime 1429649857.73 (Tue Apr 21 16:57:37 2015)
> ElapsedTime 864.91 (14 minutes 24.91 seconds)
> SourceFiles 192730
> SourceFileSize 9120190895 (8.49 GB)
> NewFiles 112321
> NewFileSize 6508225863 (6.06 GB)
> DeletedFiles 4
> ChangedFiles 8
> ChangedFileSize 13665 (13.3 KB)
> ChangedDeltaSize 0 (0 bytes)
> DeltaEntries 112333
> RawDeltaSize 6410062775 (5.97 GB)
> TotalDestinationSizeChange 4983679911 (4.64 GB)
> Errors 0
> -------------------------------------------------
> ````
>
> Notice that I added almost 6GB to my backup set and it only took 14.5
> minutes to run the backup. This is from a VM to a physical server across my
> network via CIFS.
>
> Duplicity will use the bandwidth available to it. Unless you renice it,
> duplicity runs full tilt and will impact performance until it is done.
> There's something else going on with your setup.
>
> On Wed, Apr 22, 2015 at 8:36 AM, Edward Holcroft <eholcroft at mkainc.com>
> wrote:
>
>> I tested this on my production box with all the identical pieces of kit.
>> A full rsync of the 1.5 TB or so of data on this server takes maybe 10 -11
>> hours. A full backup using Duplicity to the same USB drive on the same
>> server was less than half that - I gave up on 26% after some 16 hours. I'd
>> expect some added overhead for the encrypting and tarring that's going on,
>> but this is nuts. I have to be doign something wrong or nobody would be
>> using duplicity. Here's the command I use, which is pretty stock as far as
>> I know:
>>
>> duplicity /data --exclude /data/ret --exclude /data/rsynclogs
>> file:///mnt/daily/MONDAY >> /data/rsynclogs/daily-`date +%Y%m%d_%H%M%S`.log
>>
>> I also tried with --no-encyrption and dropped the logging with no
>> noticeable difference.
>>
>> ed
>>
>> --
> James Sumners
> http://james.sumners.info/ (technical profile)
> http://jrfom.com/ (personal site)
> http://haplo.bandcamp.com/ (band page)
>
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> http://mail.ale.org/mailman/listinfo/ale
> See JOBS, ANNOUNCE and SCHOOLS lists at
> http://mail.ale.org/mailman/listinfo
>
>


-- 
Edward Holcroft | Madsen Kneppers & Associates Inc.
11695 Johns Creek Parkway, Suite 250 | Johns Creek, GA 30097
O (770) 446-9606 | M (770) 630-0949

-- 
MADSEN, KNEPPERS & ASSOCIATES USA, MKA Canada Inc. WARNING/CONFIDENTIALITY 
NOTICE: This message may be confidential and/or privileged. If you are not 
the intended recipient, please notify the sender immediately then delete it 
- you should not copy or use it for any purpose or disclose its content to 
any other person. Internet communications are not secure. You should scan 
this message and any attachments for viruses. Any unauthorized use or 
interception of this e-mail is illegal.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ale.org/pipermail/ale/attachments/20150422/a272a585/attachment.html>