[ale] Duplicity very slow

[Jim Kinney]

Have you checked the raid array health? A raid 5 with a bad drive will CRAWL.

On April 22, 2015 3:08:39 PM EDT, Edward Holcroft <eholcroft at mkainc.com> wrote:
>This is what I got local drive to local drive test (no USB involved).
>This
>seems impossibly slow.
>
>StartTime 1429713308.46 (Wed Apr 22 10:35:08 2015)
>EndTime 1429728281.54 (Wed Apr 22 14:44:41 2015)
>ElapsedTime 14973.08 (4 hours 9 minutes 33.08 seconds)
>SourceFiles 52736
>SourceFileSize 98274663358 (91.5 GB)
>NewFiles 52736
>NewFileSize 98274663358 (91.5 GB)
>DeletedFiles 0
>ChangedFiles 0
>ChangedFileSize 0 (0 bytes)
>ChangedDeltaSize 0 (0 bytes)
>DeltaEntries 52736
>RawDeltaSize 98251115454 (91.5 GB)
>TotalDestinationSizeChange 92760516730 (86.4 GB)
>Errors 0
>
>
>On Wed, Apr 22, 2015 at 9:53 AM, James Sumners
><james.sumners at gmail.com>
>wrote:
>
>> Try it to another internal volume or across your network. After
>> discovering my oversight yesterday, I ran a backup to pickup the
>missed
>> data. This was the result:
>>
>> ````
>> Import of duplicity.backends.dpbxbackend Failed: No module named
>dropbox
>> Reading globbing filelist ./excludes
>> Local and Remote metadata are synchronized, no sync needed.
>> Last full backup date: Wed Apr 15 15:18:59 2015
>> --------------[ Backup Statistics ]--------------
>> StartTime 1429648992.82 (Tue Apr 21 16:43:12 2015)
>> EndTime 1429649857.73 (Tue Apr 21 16:57:37 2015)
>> ElapsedTime 864.91 (14 minutes 24.91 seconds)
>> SourceFiles 192730
>> SourceFileSize 9120190895 (8.49 GB)
>> NewFiles 112321
>> NewFileSize 6508225863 (6.06 GB)
>> DeletedFiles 4
>> ChangedFiles 8
>> ChangedFileSize 13665 (13.3 KB)
>> ChangedDeltaSize 0 (0 bytes)
>> DeltaEntries 112333
>> RawDeltaSize 6410062775 (5.97 GB)
>> TotalDestinationSizeChange 4983679911 (4.64 GB)
>> Errors 0
>> -------------------------------------------------
>> ````
>>
>> Notice that I added almost 6GB to my backup set and it only took 14.5
>> minutes to run the backup. This is from a VM to a physical server
>across my
>> network via CIFS.
>>
>> Duplicity will use the bandwidth available to it. Unless you renice
>it,
>> duplicity runs full tilt and will impact performance until it is
>done.
>> There's something else going on with your setup.
>>
>> On Wed, Apr 22, 2015 at 8:36 AM, Edward Holcroft
><eholcroft at mkainc.com>
>> wrote:
>>
>>> I tested this on my production box with all the identical pieces of
>kit.
>>> A full rsync of the 1.5 TB or so of data on this server takes maybe
>10 -11
>>> hours. A full backup using Duplicity to the same USB drive on the
>same
>>> server was less than half that - I gave up on 26% after some 16
>hours. I'd
>>> expect some added overhead for the encrypting and tarring that's
>going on,
>>> but this is nuts. I have to be doign something wrong or nobody would
>be
>>> using duplicity. Here's the command I use, which is pretty stock as
>far as
>>> I know:
>>>
>>> duplicity /data --exclude /data/ret --exclude /data/rsynclogs
>>> file:///mnt/daily/MONDAY >> /data/rsynclogs/daily-`date
>+%Y%m%d_%H%M%S`.log
>>>
>>> I also tried with --no-encyrption and dropped the logging with no
>>> noticeable difference.
>>>
>>> ed
>>>
>>> --
>> James Sumners
>> http://james.sumners.info/ (technical profile)
>> http://jrfom.com/ (personal site)
>> http://haplo.bandcamp.com/ (band page)
>>
>> _______________________________________________
>> Ale mailing list
>> Ale at ale.org
>> http://mail.ale.org/mailman/listinfo/ale
>> See JOBS, ANNOUNCE and SCHOOLS lists at
>> http://mail.ale.org/mailman/listinfo
>>
>>
>
>
>-- 
>Edward Holcroft | Madsen Kneppers & Associates Inc.
>11695 Johns Creek Parkway, Suite 250 | Johns Creek, GA 30097
>O (770) 446-9606 | M (770) 630-0949
>
>-- 
>MADSEN, KNEPPERS & ASSOCIATES USA, MKA Canada Inc.
>WARNING/CONFIDENTIALITY 
>NOTICE: This message may be confidential and/or privileged. If you are
>not 
>the intended recipient, please notify the sender immediately then
>delete it 
>- you should not copy or use it for any purpose or disclose its content
>to 
>any other person. Internet communications are not secure. You should
>scan 
>this message and any attachments for viruses. Any unauthorized use or 
>interception of this e-mail is illegal.
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Ale mailing list
>Ale at ale.org
>http://mail.ale.org/mailman/listinfo/ale
>See JOBS, ANNOUNCE and SCHOOLS lists at
>http://mail.ale.org/mailman/listinfo

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ale.org/pipermail/ale/attachments/20150422/cd3d80af/attachment.html>