duplicity 0.7 slowness

Hej,

we recently upgraded a couple of machines to 0.7.07.1 and it feels like everything is wayy slower than on
0.6. We use duply and run `duply $profile status` thru icinga/nrpe. On 0.6.x it took <30sec to return the
status info, on 0.7 it takes around 5-10 minutes. 
I also have a feeling that backups take longer, but that’s hard to measure because the file server is
pretty busy at times and there are too many things that could slow that down, but collection-status
shouldn’t do much network stuff anyway, should it?
Did something significantly change on 0.7?
I saw something fixed in 0.7.07.1 about an fsync issue, but that doesn’t seem to make a real difference.

Cheers,

Philip
_______________________________________________
Duplicity-talk mailing list
Duplicity-talk <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

Detecting renames

Hi

I am considering adding a flag that looks for files that have been
renamed or hard-linked. I want this so that I can back up my emails more
efficiently; I frequently move files to different folders, but I never
change the contents of the files. I am curious as to whether this has
been tried before and whether anyone else would use it.

I imagine that I would maintain a mapping from inode and
hash-of-file-contents to a set of file names that currently point to the
inode. When duplicity is run with this flag, it would first walk the
filesystem and mark what has changed. I presently only really care about
renames, not multiple hard links to the same file, so I think I would
just update the globals.rename dict (as the --rename flag does) for
simple renames; hard links would still be treated as copies.

If there's a good reason why this won't work, maybe I won't do it, and
if other people think they would use it, maybe I will do it.

Tom

Compression Levels

Currently I am doing a encrypted backup and cant find any documentation on how to set compression level.  I have seens mention of a COMPRESSION_LEVEL arable and --gpg-options but no details.

The drive I am backing up is around 17.4GB and the, backup around 14 so I guess I am getting around 25% but wondering if there is a way to save more.

Ben
--
Ben Edwards, Video Editor and Cameraman
web:www.nkytwig.com mobile:07773 02 44 82 skype:funkytwig twitter: <at> funkytwig
iContact Community Video | Bristol Community Channel
_______________________________________________
Duplicity-talk mailing list
Duplicity-talk <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

(no subject)

Hi there,

so I am back on version 1.9.1-1 and the exception down below is gone (globbing file warning is back though). So it seems that the combination I tried down below produces the exceptions. Anyone has a clue why or if it has any relevance for later versions?

Regards
  Christian

 

On 11.06.2016 23:31, list-christian--- via Duplicity-talk wrote:

Hi there,

I switched to the duply version out of the unstable branch and now had 1.11-1 0 runnig. However at regular backup time I got the error below. Not sure, if just the version switch did this, so just went back to the old version, to see what happens. Will come back as soon as I know.

For reference again the other versions:
duplicity 0.7.07.1
python-boto 2.40.0-1

Regards
  Christian

Log:

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1539, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1533, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1371, in main
    action = commandline.ProcessCommandLine(sys.argv[1:])
  File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py", line 1116, in ProcessCommandLine
    backup, local_pathname = set_backend(args[0], args[1])
  File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py", line 1005, in set_backend
    globals.backend = backend.get_backend(bend)
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 223, in get_backend
    obj = get_backend_object(url_string)
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 209, in get_backend_object
    return factory(pu)
  File "/usr/lib/python2.7/dist-packages/duplicity/backends/_boto_single.py", line 161, in __init__
    self.resetConnection()
  File "/usr/lib/python2.7/dist-packages/duplicity/backends/_boto_single.py", line 183, in resetConnection
    self.conn = get_connection(self.scheme, self.parsed_url, self.storage_uri)
  File "/usr/lib/python2.7/dist-packages/duplicity/backends/_boto_single.py", line 99, in get_connection
    is_secure=(not globals.s3_unencrypted_connection))
  File "/usr/lib/python2.7/dist-packages/boto/storage_uri.py", line 117, in connect
    **connection_args)
  File "/usr/lib/python2.7/dist-packages/boto/s3/connection.py", line 191, in __init__
    validate_certs=validate_certs, profile_name=profile_name)
  File "/usr/lib/python2.7/dist-packages/boto/connection.py", line 569, in __init__
    host, config, self.provider, self._required_auth_capability())
  File "/usr/lib/python2.7/dist-packages/boto/auth.py", line 989, in get_auth_handler
    'Check your credentials' % (len(names), str(names)))
NoAuthHandlerFound: No handler was ready to authenticate. 1 handlers were checked. ['S3HmacAuthV4Handler'] Check your crede$
03:30:03.659 Task 'BKP' failed with exit code '30'.
 
 
On 10.06.2016 12:34, edgar.soldin--- via Duplicity-talk wrote:
On 10.06.2016 12:09, Christian via Duplicity-talk wrote:
Hi, for the versions: duplicity 0.7.07.1 duply 1.5.10-1 <- Yes, I am using duply sorry for confusing it. Is there an extra mailing list?
nope, duply has no own ml
The command line I generate is: duplicity --name 'duply_profile' --encrypt-key 40XXXXXX --sign-key 40XXXXXX --verbosity '4' --include-filelist /etc/duply/profile/filelist.txt --s3-use-new-style --s3-european-buckets --full-if-older-than 6M --volsize 100 --exclude-globbing-filelist '/etc/duply/profile/exclude' '/' 's3://s3-eu-central-1.amazonaws.com/xxx' You can see, that it uses the --exclude-globbing-filelist. However it is not activated in the config.
it's activated by default, even when empty. simply update to latest greatest duply 1.11.3 and you should be set. ..ede/duply.net _______________________________________________ Duplicity-talk mailing list Duplicity-talk <at> nongnu.org https://lists.nongnu.org/mailman/listinfo/duplicity-talk

 
      _______________________________________________ Duplicity-talk mailing list Duplicity-talk <at> nongnu.org https://lists.nongnu.org/mailman/listinfo/duplicity-talk

 
_______________________________________________
Duplicity-talk mailing list
Duplicity-talk <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

How often should I verify

Verify seem to be a lot slower than the actually backup.  I was planing on doing one for each backup but am thinking doing it once a day at night.  Is this advisable?

Ben
--
Ben Edwards, Video Editor and Cameraman
web:www.nkytwig.com mobile:07773 02 44 82 skype:funkytwig twitter: <at> funkytwig
iContact Community Video | Bristol Community Channel
_______________________________________________
Duplicity-talk mailing list
Duplicity-talk <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

Six monthly roiling backups

Hi, ime new here and have setup a duplicity backup for a owncloud server.  so far I have done a full backup and an ruining incremental backups every 30 minutes.  I guess whatever I do I will at some point have to have two full backups but would like to avid even this if possible.  

So I guess I need to do a full every six months.  I think then I need to tell duplicity to delete anything over 6 months old.  Or is there a better way of doing it.  I am backing up onto a NAS (we have two buildings on the LAN so  we are off siting a couple of doors down).  The owncloud is sized as 1TB and the backup drive is 3TB (all RAID 1).

I am a little worry about what happens if one of the incrementals gets corrupted. Will the verify command tell me?  If it is corrupted will I loose everything or is there a way of recovering some of the backup? 

Any help with command like captions would be great.  I think I can tell duplicity to delete everything older than 6 months on the incremental backup command line but will have to schedule the full backups with cron.  Will duplicity delete the incremental and full backups for me?

Regards,
Ben

--
Ben Edwards, Video Editor and Cameraman
web:www.nkytwig.com mobile:07773 02 44 82 skype:funkytwig twitter: <at> funkytwig
iContact Community Video | Bristol Community Channel
_______________________________________________
Duplicity-talk mailing list
Duplicity-talk <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

Re: [Duplicity-team] Python 2.7+ for 0.8 series?

On 14.06.2016 11:46, Aaron wrote:
> Hello all,
> 
> Does anybody have any objections to dropping Python 2.6 support in the 0.8 series and making the
requirements 2.7+? Even Ubuntu Precise (12.04) is running Python 2.7.3.

sounds reasonable, at least for 0.8.. ede/duply.net

S3 - Connection reset by peer failed upload

Hi guys,

I had an initial full backup running last Friday. 

However, for some reason it was not successful. This is the error message -- https://bpaste.net/show/989567ddb593

When I checked the destination folder, I've seen few uploaded backups on the bucket and there was no "duplicity-full-signatures.20160610T125704Z.sigtar.gz".

I found similar case from this discussion -- https://answers.launchpad.net/duplicity/+question/253609.

Question(s):

1) Does it mean that the files were uploaded on the bucket are completely backup but only file duplicity-full-signatures.20160610T125704Z.sigtar.gz which I need to re-upload to s3 bucket? 

I found two files from cache folder stated below. Should I rename them to remove ".part"  and upload to s3?

total 7.6G
7.6G -rw------- 1 7.6G Jun 11 18:35 duplicity-full-signatures.20160610T125704Z.sigtar.part
296K -rw------- 1 289K Jun 11 18:35 duplicity-full.20160610T125704Z.manifest.part

2) Would this problem be solved by adding "--s3-use-multiprocessing" to  DUPL_PARAMS ?


Thank you.
James
_______________________________________________
Duplicity-talk mailing list
Duplicity-talk <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

0.6 Series Support EOL

Hi,

This is to announce that the 0.6 series has reached End-Of-Life.  No further updates or upgrades will be made to it.  The long term support folks can easily upgrade to the tarball version, the Daily PPA, or the Stable PPA, depending on their needs.

The 0.7 series will continue while we work on 0.8 with an eye to ever larger backups and split signature files.

We would love to work with the distros to figure out how to assist in their long term support needs, but currently we are at odds with how development is done. We do not have time to maintain dozens of different versions dependent on the vagaries of each of the distros, while at the same time adding new features.  Heck, we don't even have the time to maintain the half-dozen or so versions that Ubuntu would need.  So, progress will be made, but if you want or need to stick with the distribution repository for whatever reason, we'll try to help, but the most common response will be to upgrade to one of the above.

...Thanks,
...Ken

_______________________________________________
Duplicity-talk mailing list
Duplicity-talk <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

Globbing filelist warning

Hi there,

I have two short questions.
1. I am getting the warning about the globbing file list. However I can't find any option where I am using this. I only use "--include-filelist". Any clue where I can turn it off?

Warning: Option --exclude-globbing-filelist is pending deprecation and will be removed in a future release.
--include-filelist and --exclude-filelist now accept globbing characters and should be used instead.

2. Is there a way to simulate the backup and get an overview on how big it will be?

Regards
  Christian
_______________________________________________
Duplicity-talk mailing list
Duplicity-talk <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

Re: BackendException: could not load /etc/ssh/ssh_known_hosts, maybe corrupt?

On 08.06.2016 13:20, Florent B via Duplicity-talk wrote:
> On 06/08/2016 11:40 AM, edgar.soldin--- via Duplicity-talk wrote:
>> On 08.06.2016 11:35, Florent B wrote:
>>> On 06/06/2016 09:40 PM, edgar.soldin--- via Duplicity-talk wrote:
>>>> On 06.06.2016 18:20, Florent B via Duplicity-talk wrote:
>>>>> Hi list,
>>>>>
>>>>> Since Debian upgrade to Jessie, my Duplicity (last stable version)
>>>>> stopped working with this error :
>>>>>
>>>>> BackendException: could not load /etc/ssh/ssh_known_hosts, maybe corrupt?
>>>>>
>>>>> /etc/ssh/ssh_known_hosts didn't change, content is the same as before
>>>>> upgrade.
>>>>>
>>>>> What could be the problem ?
>>>> the backend seems to swallow the underlying exception. would you be willing to patch your
>>>>  duplicity/backends/ssh_paramiko_backend.py
>>>> ?
>>>>
>>>> what's your paramiko version?
>>>>
>>>> ..ede/duply.net
>>>>
>>> Hi,
>>>
>>> Yes I can patch ssh_paramiko_backend.py to test :)
>>>
>>> pip show paramiko
>>> ---
>>> Metadata-Version: 2.0
>>> Name: paramiko
>>> Version: 2.0.0
>>> Summary: SSH2 protocol library
>>> Home-page: https://github.com/paramiko/paramiko/
>>> Author: Jeff Forcier
>>> Author-email: jeff <at> bitprophet.org
>>> Installer: pip
>>> License: LGPL
>>> Location: /usr/local/lib/python2.7/dist-packages
>>> Requires: pyasn1, cryptography
>>> Classifiers:
>>>   Development Status :: 5 - Production/Stable
>>>   Intended Audience :: Developers
>>>   License :: OSI Approved :: GNU Library or Lesser General Public
>>> License (LGPL)
>>>   Operating System :: OS Independent
>>>   Topic :: Internet
>>>   Topic :: Security :: Cryptography
>>>   Programming Language :: Python
>>>   Programming Language :: Python :: 2
>>>   Programming Language :: Python :: 2.6
>>>   Programming Language :: Python :: 2.7
>>>   Programming Language :: Python :: 3
>>>   Programming Language :: Python :: 3.2
>>>   Programming Language :: Python :: 3.3
>>>   Programming Language :: Python :: 3.4
>>>   Programming Language :: Python :: 3.5
>>>
>>>
>>> But I also have 2.0.0 version of Paramiko on Wheezy and it works well..
>>>
>> for starters, can you send me the file privately? as it only contains a known public key/host name list
this shouldn't be a security concern.
>>
>> just want to check the format.
>>

ok, looks good. just to make sure can you 'ls -la /etc/ssh/ssh_known_hosts' and show me that the
permissions are correct?

and then please patch (keep a copy of course ;) your duplicity/backends/ssh_paramiko_backend.py as such

- remove 2x the try/except's around the known hosts loading routine in lines 1340- 148

rerun your previous duplicity command and see if the thrown exception is more detailed.

..ede/duply.net

Gmane