Thomas Hartmann | 29 Mar 11:18 2015

does duplicity check for missing difftar files on backup?

hi there,
i am running duplicity 0.6.24.
i do backup to a remote location. lets say, one of the difftar files 
gets deleted there, does duplicity handle this situation on the next 
backup run?

thanks a lot,
Norbert Kéri | 28 Mar 13:13 2015

Encrypt without the private key?


I'm trying to set up an unattended backup to S3, with the following command:

duplicity --progress --name mystuff --full-if-older-than 6M --s3-unencrypted-connection --encrypt-key A6ACD7BF ./myfolder s3://

However, if I rerun the above command, I get:

Local and Remote metadata are synchronized, no sync needed.
Last inc backup left a partial set, restarting.
Last full backup date: Sun Mar 22 16:54:42 2015

Then it pops up a pinentry dialog, asking for the passphrase for my private key. This surprised me, because I was expecting it to only ask for a passphrase when I restore files from the backup. Even more, if I just cancel the pinentry password dialog, it successfully finishes the backup, so it's not even using the key?

So what's happening here? Does duplicity need to decrypt some parts of the previous backup, is that why it's asking for a key? Why does it continue if I cancel the dialog then? I was thinking maybe it's trying to sign the backups, but I'm not using any of the signing switches, and it doesn't do that by default?

Is this still a problem?
Duplicity-talk mailing list
Duplicity-talk <at>
Philip Jocks | 13 Mar 17:47 2015

another issue with duply and GnuPG 2.1


GnuPG's --export-secret-keys command isn't supplied with the passphrase, which is now mandatory in
GnuPG 2.1. 
The attached patch seems to fix the issue, which occurs only when you run a new duply profile for the first
time, since duply is then extracting the secret key to the profile directory.
Hope that helps someone.


Attachment (duply_gpg21.patch): application/octet-stream, 661 bytes
Duplicity-talk mailing list
Duplicity-talk <at>
Aaron Whitehouse | 13 Mar 16:53 2015

Single vs double asterisk

Hello all,

I've been having a play around with single and double asterisks, to see what the current behaviour is.

The manual says:
'As in a normal shell, * can be expanded to any string of characters not containing "/", ? expands to any character except "/", and [...] expands to a single character of those characters specified (ranges are acceptable). The new special pattern, **, expands to any string of characters whether or not it contains "/".'

I understood this to mean that "parent/*" should include files within the folder "parent", but not any subdirectories, whereas "parent/**" would include files in "parent" but also include subfolders. Is that how other people understand it?

My preliminary tests suggest it isn't working like that and a single * seems to also include subfolders.

The test:

    def test_single_and_double_asterisks(self):
        """This compares a backup using --include-globbing-filelist with a single and double *."""
        with open("testfiles/filelist.txt", 'w') as f:
            f.write("+ testfiles/select2/*\n"
                    "- testfiles/select")
        self.backup("full", "testfiles/", options=["--include-globbing-filelist=testfiles/filelist.txt"])
        restore_dir = 'testfiles/restore_out'
        restored = self.directory_tree_to_list_of_lists(restore_dir + "/select2")
        with open("testfiles/filelist2.txt", 'w') as f:
            f.write("+ testfiles/select2/**\n"
                    "- testfiles/select")
        self.backup("full", "testfiles/", options=["--include-globbing-filelist=testfiles/filelist2.txt"])
        restore_dir = 'testfiles/restore_out'
        restored2 = self.directory_tree_to_list_of_lists(restore_dir + "/select2")
        self.assertEqual(restored, restored2)

currently passes (suggesting, at least in this set-up, * is equivalent to **) and "select2/*" is including all subfolders of select2. I get the same result using the (non-filelist) --include=testfiles/select2/* or --include=testfiles/select2/** options.

I would really appreciate it if people who use either * or ** in their filelists or in --include/-excludes could have a quick try with the other and see if it makes any difference. Any other thoughts would of course be appreciated.

Kind regards,

Duplicity-talk mailing list
Duplicity-talk <at>
Rob Browning | 13 Mar 03:30 2015

collection-status crashes, given a a non-ASCII path element

Assuming I'm understanding the situation correctly, given a directory
with a non-ASCII character like this:


Debian 0.6.24-1 collection-status crashes here like this:

  Traceback (most recent call last):
    File "/usr/lib/python2.7/logging/", line 859, in emit
      msg = self.format(record)
    File "/usr/lib/python2.7/logging/", line 732, in format
      return fmt.format(record)
    File "/usr/lib/python2.7/dist-packages/duplicity/", line 372, in format
      s += ('\n' + record.message).replace('\n', '\n. ')
  UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 66: ordinal not in range(128)

Hope this helps.

Rob Browning
rlb  <at> and  <at>
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Duplicity-talk mailing list
Duplicity-talk <at>
ST | 12 Mar 10:26 2015


I want to make a backup using duplicity (which uses paramiko) and upload
files to a remote host R2, the problem is, that I can access R2(sftp
access) only from R1(ssh access). How can I tunnel duplicity through R1
to R2? I defined (in .ssh/config) Host R2 and ProxyCommand ssh -p XXXX
-qW %h:%p user <at> R1 and I can actually do `sftp user <at> R2` but when I run
`duplicity Dir sftp://user <at> R2/` there is a delay and I get
"BackendException: ssh connection to R2:22 failed: timed out". Is there
a way to tell paramiko to use my .ssh/config? Or is there a way to
tunnel through another host in paramiko?

My duplicity version 0.6.18, seems not to have --ssh-backend option, so
I cann't choose pexpect... or am I wrong?

Thank you.
Kenneth Loafman | 11 Mar 13:55 2015

Duplicity 0.7.02 Released

Hello All,

We have a lot of fixes in this release.  Thanks to all the contributors!

Gory details of the release and the tarball can be found at Milestone 0.7.02



Duplicity-talk mailing list
Duplicity-talk <at>
Kenneth Loafman | 9 Mar 19:36 2015

Re: SCP Issue With duplicity 0.7.01 and

OK, this problem is easily fixed.  The code was trying to do:

$ test -d 'foo/bar/zot' || mkdir -p 'foo/bar/zot'

on the remote system, when all that was needed was

$ mkdir -p 'foo/bar/zot'

The -p option means that mkdir does not care if the directories already exist, thus the test is not needed.

I've attached a patch file to fix the backend under 0.7.01.  I'll have it on trunk shortly.


On Mon, Mar 9, 2015 at 12:42 PM, Kenneth Loafman <kenneth <at>> wrote:
Actually, anyone that might know.


On Mon, Mar 9, 2015 at 8:58 AM, Edgar Soldin <edgar <at>> wrote:
On 09.03.2015 14:51, Kenneth Loafman wrote:
> Actually, I do.  I'll take a look this afternoon.

ok, will still have to update the manpage. remove '--use-scp' refernces and document the way scp is selected now.

> What is the advantage of scp over sftp anyway?

none ;).. you're asking him right? ..ede

Duplicity-talk mailing list
Duplicity-talk <at>
Simon Watson | 9 Mar 03:29 2015

SCP Issue With duplicity 0.7.01 and


Just letting you know I came across an issue with using duplicity 0.7.01 and my account.

When I used duplicity 0.7.01 with the scp protocol to connect to my account it would fail with:
BackendException: scp mkdir failed(2): test: target/remote/directory: unexpected operator

Thanks to Kenneth Loafman's assistance I was able to resolve this issue.
If I changed to using the sftp protocol instead it would work fine. Also if I downgraded to duplicity 0.7.0 or lower it would also work fine. So it seems like a remote command that duplicity 0.7.01 uses with the scp method won't work on accounts.

Is anyone else able to replicate or confirm this issue?

Here is the full traceback for the exception I was getting:
Backend error detail: Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1497, in <module>
  File "/usr/bin/duplicity", line 1491, in with_tempdir
  File "/usr/bin/duplicity", line 1324, in main
    action = commandline.ProcessCommandLine(sys.argv[1:])
  File "/usr/lib/python2.7/dist-packages/duplicity/", line 1048, in ProcessCommandLine
    globals.backend = backend.get_backend(args[0])
  File "/usr/lib/python2.7/dist-packages/duplicity/", line 221, in get_backend
    obj = get_backend_object(url_string)
  File "/usr/lib/python2.7/dist-packages/duplicity/", line 207, in get_backend_object
    return factory(pu)
  File "/usr/lib/python2.7/dist-packages/duplicity/backends/", line 230, in __init__
    self.runremote("test -d '%s' || mkdir -p '%s'" % (self.remote_dir, self.remote_dir), False, "scp mkdir ")
  File "/usr/lib/python2.7/dist-packages/duplicity/backends/", line 359, in runremote
    raise BackendException("%sfailed(%d): %s" % (errorprefix, res, chan.recv_stderr(4096)))
BackendException: scp mkdir failed(2): test: duplicity/pavlics: unexpected operator

Moon Buggy Media
m. 0410 190 553
p. 03 9913 1715
Duplicity-talk mailing list
Duplicity-talk <at>
Mikko Ohtamaa | 6 Mar 11:23 2015

Understanding Duplicity run report

Hi all,

I have the following report from the duplicity from the incremental backup run. Does RawDelta and TotalDestinationSize actually mean that this was the delta amount required to upload and store in the backup?

Is this so? In my case something is off as, as from 70 GB set almost 69.5 GB would have changed. This cannot be true, as I know that the most of the files are static.

Using SFTP backend.

What could cause this behavior?

I also run the duplicity in verbose mode and it seems to be adding (A) files to the backup which I know have not changed - does this happen for every backed up file or only for new/delta files?


--------------[ Backup Statistics ]--------------
StartTime 1425596471.39 (Fri Mar  6 00:01:11 2015)
EndTime 1425609804.32 (Fri Mar  6 03:43:24 2015)
ElapsedTime 13332.93 (3 hours 42 minutes 12.93 seconds)
SourceFiles 924693
SourceFileSize 75405362645 (70.2 GB)
NewFiles 924693
NewFileSize 75405361109 (70.2 GB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 924693
RawDeltaSize 74641311673 (69.5 GB)
TotalDestinationSizeChange 65357662979 (60.9 GB)
Errors 0

Duplicity-talk mailing list
Duplicity-talk <at>
Aaron Whitehouse | 18 Feb 18:13 2015

Use of --include-filelist-stdin and --exclude-filelist-stdin

Hello all,

Does anybody on the list use --include-filelist-stdin or

If so, would you mind please giving me an example of how you use them?
It isn't a feature that I use, but I want to make sure that I don't
break the behaviour with some changes I am making in the select module.
I could just make something up, but it would be nice to know that my
test at least represented one person's use-case.

A little more info is here:

Many thanks,