Martin Pool | 23 Jan 17:43 2015
Picon

new librsync release

Hi,

I've just made a new librsync release <https://github.com/librsync/librsync/releases/tag/v1.0.0>.

This includes a fix for a security bug reported by therealmik that's relevant to Duplicity, <https://github.com/librsync/librsync/issues/5>.

Unfortunately the fix for this necessitates a change in signature file format, so probably also a new version of the Duplicity archive format. librsync can read the old format but writing it is deprecated.

Regards,
--
Martin
_______________________________________________
Duplicity-talk mailing list
Duplicity-talk <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Josh Triplett | 20 Jan 16:37 2015

Support for current Google Drive API?

[Please CC me on replies.]

The current Duplicity "gdocs" backend uses the deprecated "Google
Documents List Data API"
(https://developers.google.com/google-apps/documents-list/), which says:

> Warning: The deprecation period for Version 3 of the Google Documents
> List API is nearly at an end. On April 20, 2015, we will discontinue
> service for this API. This means that service calls to the API are no
> longer supported, and features implemented using this API will not
> function after April 20, 2015. You must migrate to the Drive API as
> soon as possible to avoid disruptions to your application.

Any plans to migrate to the Drive API?

Also, switching to the new API would avoid the need to supply a Google
username and password; instead, Duplicity could obtain an access token
with the appropriate permission (drive.file) to access only files
created with that token.

- Josh Triplett
Duplicity Mailing List | 13 Jan 00:21 2015
Picon

Par2 not fixing | Hubic review?

This is a two part message.

//FIRST PART//

How do I get par2 to repair? Upon hitting a corrupted difftar file,
Duplicity throws:-

>Backtrace of previous error: Traceback (innermost last):
>  File "/usr/local/lib/python2.7/dist-packages/duplicity/backend.py",
>line 361, in inner_retry
>    return fn(self, *args)
>  File "/usr/local/lib/python2.7/dist-packages/duplicity/backend.py",
>line 534, in get
>    self.backend._get(remote_filename, local_path)
>  File
"/usr/local/lib/python2.7/dist-packages/duplicity/backends/par2backend.py",
line 119, in get
>    par2volumes = filter(re.compile((r'%s\.vol[\d+]*\.par2' %
>remote_filename).match,
> AttributeError: 'str' object has no attribute 'match'

This is version 0.7.1 of Duplicity, running on a Debian Wheezy install.

//END OF FIRST PART//

//SECOND PART - Slightly off topic//

Has anyone here used Hubic seriously? I recently gave it a try
considering they offer 25GB for free and Duplicity just got support for
it, I uploaded 5GB to their service then tried to restore it. Within the
hour my files were up on their service my files were already corrupted
(Granted, it was one of the last files (But not the very last file), at
~4.9GB through my backup), this is less than impressive. After looking
through their forums it seems like they have lots of issues with very
few replies from administrators, and when there is a reply, it's nearly
always "PM us" with no public discussion. The combination of both the
failure to keep my files, even for one hour, secure and the terrible
support on their site pushes me away from their services, although, I am
interested in what Duplicity users think, being only €1/TB/month, I
might be worth going with them even if they do corrupt *some* data.

//END OF SECOND PART//

_______________________________________________
Duplicity-talk mailing list
Duplicity-talk <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Kenneth Loafman | 11 Jan 18:13 2015

Duplicity 0.7.01 Released

Hello Everyone,

This is the second release of the 0.7 series.  We have a lot of fixes in this release, plus two new backends: Microsoft OneDrive and Azure Blob Storage.  Thanks to all the contributors!

Gory details of the release and the tarball can be found at Milestone 0.7.01

Note that there are still no RPM files.  If someone would like to volunteer to maintain the RPM files, that would be appreciated.  I really do not want to revisit that learning curve again!

...Thanks,
...Ken

_______________________________________________
Duplicity-talk mailing list
Duplicity-talk <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Aaron Whitehouse | 9 Jan 18:47 2015
Picon

Why do we have --include-filelist (as opposed to just globbing)?

Hi all,

We've had a couple of bugs recently that only affect
--include/exclude-filelist but not --include/exclude-globbing-filelist.
One reason for this is that the code paths for the two different types
of filelists are separate in selection.py. I have been contemplating
trying to simplify this file a bit to keep things together until the
globs are interpreted. Before I do that, I wanted to check that there is
actually a use case for filelists that *aren't* globbing. I have often
used globbing filelists for normal paths without issues. The only
advantage that I can see in a non-globbing filelist would be if you had
filenames that had globbing characters in them, but we probably want a
way to escape such characters in globbing filelists anyway (if there
isn't already), as otherwise people with such files couldn't take
advantage of the benefits of globbing filelists.

I'm wondering if the more sensible answer is to essentially get rid of
include and exclude filelists altogether (we could obviously redirect
the option for the short term etc. and may want to rename globbing
filelists to just be filelists if there aren't any others).

I would be interested in your thoughts.

Kind regards,

Aaron
Christian Saga | 8 Jan 21:45 2015
Picon

Re: Duplicity error with s3-eu-central-1.amazonaws.com

Hi Edgar, Hi Nicolas,
thanks for your response

 <at> Edgar:
1. does the bucket exist prior to running the backup?
>Tried both, didn't work

2. can you try to use the latest boto?
>How do I do that? 

3. what's your duplicity version?
>0.6.23-1ubuntu4.1

 <at> Nicolas:
>I tried what you described, however the error still persists.
>Below some logging:

Start duply v1.5.10, time is 2015-01-08 21:38:12.
Using profile '/home/christian/.duply/Christian-ThinkPad'.
Using installed duplicity version 0.6.23, python 2.7.6, gpg 1.4.16 (Home: ~/.gnupg), awk 'GNU Awk 4.0.1'.
Autoset found secret key of first GPG_KEY entry 'XXXXXXXX' for signing.
Test - Encrypt to XXXXXXXX & Sign with XXXXXXXX (OK)
Test - Decrypt (OK)
Test - Compare (OK)
Cleanup - Delete '/tmp/duply.24702.1420749492_*'(OK)

--- Start running command PRE at 21:38:13.037 ---
Running '/home/christian/.duply/Christian-ThinkPad/pre' - OK
--- Finished state OK at 21:38:13.055 - Runtime 00:00:00.018 ---

--- Start running command BKP at 21:38:13.069 ---
Benutze Archiv-Verzeichnis: /home/christian/.cache/duplicity/duply_Christian-ThinkPad
Benutze Sicherungs-Namen: duply_Christian-ThinkPad
Import of duplicity.backends.localbackend Succeeded
Import of duplicity.backends.megabackend Succeeded
Import of duplicity.backends.botobackend Succeeded
Import of duplicity.backends.webdavbackend Succeeded
Import of duplicity.backends.u1backend Succeeded
Import of duplicity.backends.rsyncbackend Succeeded
Import of duplicity.backends.dpbxbackend Succeeded
Import of duplicity.backends.hsibackend Succeeded
Import of duplicity.backends.imapbackend Succeeded
Import of duplicity.backends.cfbackend Succeeded
Import of duplicity.backends.sshbackend Succeeded
Import of duplicity.backends.gdocsbackend Succeeded
Import of duplicity.backends.swiftbackend Succeeded
Import of duplicity.backends.ftpsbackend Succeeded
Import of duplicity.backends.ftpbackend Succeeded
Import of duplicity.backends.tahoebackend Succeeded
Lese ausdruckbasierte Dateiliste /home/christian/.duply/Christian-ThinkPad/filelist.txt
Lese ausdruckbasierte Dateiliste /home/christian/.duply/Christian-ThinkPad/exclude
Hauptaufgabe: inc
================================================================================
duplicity 0.6.23 (January 24, 2014)
Args: /usr/bin/duplicity --name duply_Christian-ThinkPad --encrypt-key XXXXXXXX --sign-key
XXXXXXXX --verbosity 9 --include-globbing-filelist
/home/christian/.duply/Christian-ThinkPad/filelist.txt --s3-use-new-style
--s3-european-buckets --full-if-older-than 1M --volsize 50 --exclude-globbing-filelist
/home/christian/.duply/Christian-ThinkPad/exclude / s3://s3-eu-central-1.amazonaws.com/XX/backup
Linux Christian-ThinkPad 3.13.0-37-generic #64-Ubuntu SMP Mon Sep 22 21:28:38 UTC 2014 x86_64 x86_64
/usr/bin/python 2.7.6 (default, Mar 22 2014, 22:59:56) 
[GCC 4.8.2]
================================================================================
Benutze temporäres Verzeichnis /tmp/duplicity-Vamu0I-tempdir
Registriere (mkstemp) temporäre Datei /tmp/duplicity-Vamu0I-tempdir/mkstemp-rxuJjJ-1
3781853184 temporärer Speicherplatz ist verfügbar, das Backup wird ungefähr 68157440 nutzen.
Releasing lockfile <lockfile.LinkFileLock instance at 0x7f299927b8c0>
Entferne temporäre Datei /tmp/duplicity-Vamu0I-tempdir/mkstemp-rxuJjJ-1
Backend Fehler Detail: Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1494, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1488, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1337, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1366, in do_backup
    sync_archive(decrypt)
  File "/usr/bin/duplicity", line 1099, in sync_archive
    remlist = globals.backend.list()
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 430, in list
    return map(tobytes, self._list())
  File "/usr/lib/python2.7/dist-packages/duplicity/backends/_boto_single.py", line 270, in _list
    raise BackendException("No connection to backend")
BackendException: No connection to backend

--- Finished state FAILED 'code 23' at 21:38:13.432 - Runtime 00:00:00.363 ---

--- Start running command POST at 21:38:13.445 ---
Skipping n/a script '/home/christian/.duply/Christian-ThinkPad/post'.
--- Finished state OK at 21:38:13.460 - Runtime 00:00:00.015 ---

Regards
  Christian

> Hello
> 
> On Wed, Jan 7, 2015 at 2:48 PM, Edgar Soldin wrote:
> 
> > On 07.01.2015 00:11, Christian Saga wrote:
> > >
> > >
> > > Hi there,
> > > I wanted to switch to the new S3 datacenter in Frankfurt, however when
> > > doing so, I receive a 403 error. Similar to what was described in a
> > > question asked on launchpad:
> > > https://answers.launchpad.net/duplicity/+question/256418
> > >
> > > However no one is picking this up? Does anyone here have a solution or
> > > idea what to do?
> > >
> >
> >
> I encountered the same issue. Here is how to make it work: before using
> duplicity, set the following environment variable
> S3_USE_SIGV4="True"
> 
> and use the full url to access your bucket, e.g.
> duplicity [your options here] s3://
> s3-eu-central-1.amazonaws.com/my_own_bucket
> 
> 
> check this http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region
> and
> http://stackoverflow.com/questions/26744712/s3-using-boto-and-sigv4-missing-host-parameter
> (among other links)
> 
> Regards
> Nicolas

_______________________________________________
Duplicity-talk mailing list
Duplicity-talk <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Christian Saga | 7 Jan 00:11 2015
Picon

Duplicity error with s3-eu-central-1.amazonaws.com

Hi there,
I wanted to switch to the new S3 datacenter in Frankfurt, however when doing so, I receive a 403 error. Similar to what was described in a question asked on launchpad: https://answers.launchpad.net/duplicity/+question/256418

However no one is picking this up? Does anyone here have a solution or idea what to do?

Regards
  Chris

_______________________________________________
Duplicity-talk mailing list
Duplicity-talk <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Tobias Vollmer | 6 Jan 20:28 2015
Picon

duplicity with s3 and glacier, lifecycling might result in old manifests not being deleted

Hi,

tl;dr: Moving archives results in not deleted manifests of full backups.
Bug?

I started to play around with duplicity. I would like to create a setup
where I have one full backup (and incrementals) in S3, and a second in
glacier.

To find out the proper setup, I played around with local file storage
and moved "old" backups to a different folder.

I run this command every Minute
> duplicity --encrypt-key $KEYID  --volsize 100 --full-if-older-than 5m --file-prefix-archive
archive. $DATAPATH file://$BACKUPPATH

Additionally I run the following command after the backup to clear old
Backups
> duplicity  remove-all-but-n-full 1 --force --file-prefix-archive archive. file://$BACKUPPATH
and move the archive.* files to a second location.

If I skip the moving part, everything works as expected, if I move the
archive files, old manifests are not deleted.

My setup is a Debian Box with the back ported 0.6.24 Version of duplicity.

I added the following line to the definition of the delete function in
/usr/share/pyshared/duplicity/collections.py, line 147:
> log.Notice("Deleting files:" + str(rfn))
to get some debugging information.

The first listing shows which files are send to the backend to delete,
if I do _not_ move the archives manually:

> Local and Remote metadata are synchronized, no sync needed.
> Last full backup date: Tue Jan  6 18:09:01 2015
> Deleting backup chain at time:
> Tue Jan  6 18:08:01 2015
> Deleting complete signature chain Tue Jan  6 18:08:01 2015
> Deleting complete signature chain Tue Jan  6 18:08:01 2015
> Deleting complete backup chain Tue Jan  6 18:08:01 2015
> Deleting files:['duplicity-inc.20150106T170701Z.to.20150106T170801Z.manifest.gpg', 'archive.duplicity-inc.20150106T170701Z.to.20150106T170801Z.vol1.difftar.gpg']
> Deleting files:['duplicity-inc.20150106T170602Z.to.20150106T170701Z.manifest.gpg', 'archive.duplicity-inc.20150106T170602Z.to.20150106T170701Z.vol1.difftar.gpg']
> Deleting files:['duplicity-inc.20150106T170501Z.to.20150106T170602Z.manifest.gpg', 'archive.duplicity-inc.20150106T170501Z.to.20150106T170602Z.vol1.difftar.gpg']
> Deleting files:['duplicity-inc.20150106T170401Z.to.20150106T170501Z.manifest.gpg', 'archive.duplicity-inc.20150106T170401Z.to.20150106T170501Z.vol1.difftar.gpg']
> Deleting files:['duplicity-inc.20150106T170301Z.to.20150106T170401Z.manifest.gpg', 'archive.duplicity-inc.20150106T170301Z.to.20150106T170401Z.vol1.difftar.gpg']
> Deleting files:['duplicity-full.20150106T170301Z.manifest.gpg',
'archive.duplicity-full.20150106T170301Z.vol2.difftar.gpg', 'archive.duplicity-full.20150106T170301Z.vol1.difftar.gpg']

The second listing is from a run after the archives are moved manually:

> Local and Remote metadata are synchronized, no sync needed.
> Last full backup date: Tue Jan  6 18:15:01 2015
> Deleting backup chain at time:
> Tue Jan  6 18:14:02 2015
> Deleting complete signature chain Tue Jan  6 18:14:02 2015
> Deleting complete signature chain Tue Jan  6 18:14:02 2015
> Deleting complete backup chain Tue Jan  6 18:14:02 2015
> Deleting files:['duplicity-inc.20150106T171301Z.to.20150106T171402Z.manifest.gpg']
> Deleting files:['duplicity-inc.20150106T171201Z.to.20150106T171301Z.manifest.gpg']
> Deleting files:['duplicity-inc.20150106T171101Z.to.20150106T171201Z.manifest.gpg']
> Deleting files:['duplicity-inc.20150106T171002Z.to.20150106T171101Z.manifest.gpg']
> Deleting files:['duplicity-inc.20150106T170901Z.to.20150106T171002Z.manifest.gpg']

As you can see, the incrementals are completely removed, but the full
backup is missing.

After the removal, the directory listing shows the additional manifest:

> ls -a
> total 2848
> .
> ..
> backup.log # <- Expected, logfile :-)
> duplicity-full.20150106T170901Z.manifest.gpg
> duplicity-full.20150106T171501Z.manifest.gpg
> duplicity-full-signatures.20150106T171501Z.sigtar.gpg

This might be a problem, as the number of manifests will grow over time
on a long running backup system.

Could this be a bug? Or is it expected behaviour?

Cheers,

Tobias
--

-- 
Please watch your privacy! Encrypt your emails!
Achten Sie auf Ihre Privatsphäre! Verschlüsseln Sie Ihre Emails!
PGP-Key: 8AE12233 (http://key.tvollmer.de/)
75D8 DA44 69B5 9621 1262 A41D 1952 B22F 8AE1 2233

_______________________________________________
Duplicity-talk mailing list
Duplicity-talk <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Jacob Godserv | 2 Jan 19:05 2015
Picon

Thank you for Duplicity

Hey guys,


I just did a complete reinstall of Funtoo Linux. I was able to successfully restore my package manager configuration, home directory, and local package overlay, all with a few restore commands. Despite there being network access, it was local and unencrypted, so recovery time was surprisingly fast.

I wanted to thank the volunteers of this project who continue to maintain this software. I use it hourly both at home and at work, and it's an amazing feeling to be able to recover just about anything without having to wrestle overly complicated backup software configurations and filesystem requirements to get it to work the way I want.

tldr: I'm back up and running now, with all my projects and configurations intact. Thanks everyone for your hard work. I appreciate it. :)
_______________________________________________
Duplicity-talk mailing list
Duplicity-talk <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Norman Goldstein | 2 Jan 18:49 2015
Picon

Re: Fwd: Re: Oddity using --include-globbing-filelist

Here is an example that illustrates the problem:

The file hierarchy for the demo of the "bug"

ls -R src/
src/:
fileA  fileB


Here is the globbing filelist file (contains only the one line):

- **/fileB


Here are the two commands to verify the "bug"

duplicity --include-globbing-filelist /home/save/systems/Duplicity/TestBug/globlist.txt /home/save/systems/Duplicity/TestBug/src file:///home/save/systems/Duplicity/TestBug/dest

duplicity list-current-files file:///home/save/systems/Duplicity/TestBug/dest


The intended listing of the dest backup is:

Last full backup date: Fri Jan  2 09:37:44 2015
Fri Jan  2 09:35:00 2015 .
Fri Jan  2 09:30:28 2015 fileA

However, if the globbing filelist is edited to put a space after "fileB", the listing is

Last full backup date: Fri Jan  2 09:41:18 2015
Fri Jan  2 09:35:00 2015 .
Fri Jan  2 09:30:28 2015 fileA
Fri Jan  2 09:35:00 2015 fileB


Cheers,
Norm



On 12/30/2014 06:49 AM, Aaron Whitehouse wrote:
Hi Norman,

I sent the below to the list, but it occurred to me that you may not subscribe to it, so forwarding the email on in case.

Kind regards (and apologies if nobody else replied to you for so long),

Aaron


-------- Original Message -------- Subject: Date: From: Reply-To: To:
Re: [Duplicity-talk] Oddity using --include-globbing-filelist
Thu, 25 Dec 2014 16:02:30 +0000
Aaron Whitehouse <lists <at> whitehouse.kiwi.nz>
Discussion of the backup program duplicity <duplicity-talk <at> nongnu.org>
duplicity-talk <at> nongnu.org


Hello Norman, On 18/09/14 04:53, Norman Goldstein wrote: > [...] > > So, it seems that duplicity is doing two passes > of the lines in the globbing filelist file. The > first pass vets the file names, ignoring trailing > white space. However, the 2nd pass, which actually > uses the file names, includes the trailing white > space, and does not complain if the resulting > file does not exist. > > I can attach the example, it you would find it useful. That's interesting thanks. Sorry for the delay in replying. I've noticed that white space at the end of a globbing file list caused things to break, but I hadn't delved into it in any detail. I'm not one of the main duplicity developers, but I've been meaning to look into this issue in a bit more detail. It would therefore be helpful to me if you could please send your example (if you still have it). Kind regards, Aaron _______________________________________________ Duplicity-talk mailing list Duplicity-talk <at> nongnu.org https://lists.nongnu.org/mailman/listinfo/duplicity-talk


_______________________________________________
Duplicity-talk mailing list
Duplicity-talk <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Yves Goergen | 24 Dec 10:02 2014
Picon

GPG key size and performance

Hi,

I'm wondering whether a GPG key size of 2048 or 4096 bit will make a 
difference on the encryption/signing or decryption/verifying performance 
in duplicity. I'm used to much shorter backup and restore times without 
using an asymmetric key in duplicity, i.e. only a password. Then I 
started with a 4096 bit key and added signing and things got much 
slower, especially for the restore process (4 to 30 minutes).

So I redid it all and used a 2048 bit key. But today as I try that, it's 
even a lot slower. I believe that remote storage (SFTP) performance may 
have a greater effect on this, so my test method won't work.

Maybe SFTP is also slower to transfer than FTP which I used before, 
without the GPG key.

Does anybody have experience with key size vs. backup/restore speed? In 
general, if I can get higher security, I'd like to use it. But if it 
makes a full restore take double the time, I might reconsider this.

--

-- 
Yves Goergen
http://unclassified.de
http://dev.unclassified.de

Gmane