David Ehrmann | 20 Aug 19:21 2014
Picon

Migrating to a different data store URL

It looks like migrating to a new data store (from file:// to s3, or s3 to ssh) is as simple as moving the files, and optionally renaming the ~/.cache/duplicity entry to the md5 of the new destination, though omitting the last step will just cause Duplicity to fetch the metadata. Obviously, all the prefix flags will have to match, but otherwise, it looks like migrating is straightforward.

Am I missing anything? My plan is to do an initial backup to a local disk, use AWS Import to move the data to S3, then do incremental backups off of that.
_______________________________________________
Duplicity-talk mailing list
Duplicity-talk <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
a.grandi@gmail.com | 15 Aug 18:10 2014
Picon

Skylable backend ready to be merged and relased

Hi!

As you already know, in these days I was working to the Skylable
backend for Duplicity and you can find it here:
https://github.com/andreagrandi/duplicity-sx/blob/master/duplicity/backends/sxbackend.py

People from Skylable have tested heavily in the past days and they
consider it quite stable (it's been written using the devel/0.7
version of Duplicity).

What is the official procedure to have this integrated in the
mainstream Duplicity code?

Thank you so much!

Cheers.

--

-- 
Andrea Grandi -  Software Engineer / Qt Ambassador / Nokia Developer Champion
website: http://www.andreagrandi.it
Jelle de Jong | 13 Aug 16:58 2014
Picon

problem: duplicity is not backing up the my files


Hello everyone,

After some investigation I found that duplicity does not have the same
files in its collection as the files in the location that should be
backuped. I think it has to do that the files may be changed or
removed after the duplicity command has been started. (since the
backup takes a few days, and some files rotate every day).

See the attached debug file for more information.

However I also never get any warning or error that it failed to backup
anything. It will finish saying it went all okay.

Is this normal behaviour? Should I change my backup systems?

Kind regards,

Jelle de Jong
root <at> backup:~# duplicity --version
duplicity 0.6.24

root <at> backup:~# duplicity collection-status --archive-dir /var/cache/backupninja/duplicity
--name ca76476632badf50cdf688a006875374 scp://user-backup <at> server.somewhere.nl/backups/libvirt-guests
duplicity list-current-files --archive-dir /var/cache/backupninja/duplicity --name
ca76476632badf50cdf688a006875374 scp://user-backup <at> server.somewhere.nl/backups/libvirt-guests
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Fri Aug  8 19:02:18 2014
Collection Status
-----------------
Connecting with backend: SSHParamikoBackend
Archive dir: /var/cache/backupninja/duplicity/ca76476632badf50cdf688a006875374

Found 3 secondary backup chains.
Secondary chain 1 of 3:
-------------------------
Chain start time: Fri Jul 18 19:04:27 2014
Chain end time: Fri Jul 18 19:04:27 2014
Number of contained backup sets: 1
Total number of contained volumes: 7123
 Type of backup set:                            Time:      Num volumes:
                Full         Fri Jul 18 19:04:27 2014              7123
-------------------------

Secondary chain 2 of 3:
-------------------------
Chain start time: Fri Jul 25 19:02:40 2014
Chain end time: Fri Jul 25 19:02:40 2014
Number of contained backup sets: 1
Total number of contained volumes: 7572
 Type of backup set:                            Time:      Num volumes:
                Full         Fri Jul 25 19:02:40 2014              7572
-------------------------

Secondary chain 3 of 3:
-------------------------
Chain start time: Fri Aug  1 19:20:09 2014
Chain end time: Fri Aug  1 19:20:09 2014
Number of contained backup sets: 1
Total number of contained volumes: 8068
 Type of backup set:                            Time:      Num volumes:
                Full         Fri Aug  1 19:20:09 2014              8068
-------------------------

Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Fri Aug  8 19:02:18 2014
Chain end time: Fri Aug  8 19:02:18 2014
Number of contained backup sets: 1
Total number of contained volumes: 8071
 Type of backup set:                            Time:      Num volumes:
                Full         Fri Aug  8 19:02:18 2014              8071
-------------------------
No orphaned or incomplete backup sets found.
root <at> backup:~# duplicity list-current-files --archive-dir /var/cache/backupninja/duplicity
--name ca76476632badf50cdf688a006875374 scp://user-backup <at> server.somewhere.nl/backups/libvirt-guests
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Fri Aug  8 19:02:18 2014
Sat Aug  2 13:21:08 2014 .
Fri Apr 11 11:35:37 2014 srv
Mon Aug 19 12:24:22 2013 srv/storage
Fri Aug  8 18:56:33 2014 srv/storage/backups
Fri Aug  8 17:45:01 2014 srv/storage/backups/libvirt-guests
Fri Aug  8 18:56:30 2014 srv/storage/backups/libvirt-guests/latest
Mon Aug  4 18:11:55 2014 srv/storage/backups/libvirt-guests/latest/charibase1-disk1-snapshot-2014-08-04-1407167107.dd.disk.gz
Mon Aug  4 18:31:54 2014 srv/storage/backups/libvirt-guests/latest/charibase1-disk2-snapshot-2014-08-04-1407167107.dd.disk.gz
Mon Aug  4 18:04:10 2014 srv/storage/backups/libvirt-guests/latest/charibase1-disk3-snapshot-2014-08-04-1407167107.dd.disk.gz
Fri Aug  8 18:45:40 2014 srv/storage/backups/libvirt-guests/latest/somewhere-disk1-snapshot-2014-08-08-1407512704.dd.disk.gz
Fri Aug  8 01:29:15 2014 srv/storage/backups/libvirt-guests/latest/somewhere-disk2-snapshot-2014-08-07-1407426307.dd.disk.gz
Tue Aug  5 18:26:55 2014 srv/storage/backups/libvirt-guests/latest/horoscope-disk-snapshot-2014-08-05-1407253505.dd.disk.gz
Fri Aug  8 18:03:49 2014 srv/storage/backups/libvirt-guests/latest/kvm01-disk-snapshot-2014-08-08-1407512704.dd.disk.gz
Fri Aug  8 18:56:09 2014 srv/storage/backups/libvirt-guests/latest/ldapproxy1-disk-snapshot-2014-08-08-1407513839.dd.disk.gz
Wed Aug  6 19:32:56 2014 srv/storage/backups/libvirt-guests/latest/maintenance1-disk-snapshot-2014-08-06-1407344350.dd.disk.gz
Thu Aug  7 21:46:44 2014 srv/storage/backups/libvirt-guests/latest/nginx1-disk-snapshot-2014-08-07-1407439270.dd.disk.gz
Thu Aug  7 21:53:09 2014 srv/storage/backups/libvirt-guests/latest/ns0-disk-snapshot-2014-08-07-1407440130.dd.disk.gz
Fri Aug  8 07:19:28 2014 srv/storage/backups/libvirt-guests/latest/opsi-disk-snapshot-2014-08-07-1407440836.dd.disk.gz
Sun Aug  3 23:40:55 2014 srv/storage/backups/libvirt-guests/latest/osiris-at-disk-snapshot-2014-08-03-1407095682.dd.disk.gz
Sun Aug  3 23:52:39 2014 srv/storage/backups/libvirt-guests/latest/osiris-ot-disk-snapshot-2014-08-03-1407095853.dd.disk.gz
Mon Aug  4 03:31:41 2014 srv/storage/backups/libvirt-guests/latest/relax1-disk1-snapshot-2014-08-04-1407107877.dd.disk.gz
Mon Aug  4 03:37:24 2014 srv/storage/backups/libvirt-guests/latest/socks1-disk-snapshot-2014-08-04-1407115215.dd.disk.gz
Fri Aug  8 11:31:05 2014 srv/storage/backups/libvirt-guests/latest/sr5-disk2-snapshot-2014-08-08-1407477324.dd.disk.gz
Fri Aug  8 13:50:33 2014 srv/storage/backups/libvirt-guests/latest/webserver1-disk2-snapshot-2014-08-08-1407480987.dd.disk.gz
Fri Aug  8 09:44:03 2014 srv/storage/backups/libvirt-guests/latest/zabbix2-disk-snapshot-2014-08-08-1407481493.dd.disk.gz
root <at> backup:~# ls -hal /srv/storage/backups/libvirt-guests/latest/
total 330G
drwxr-xr-x  2 root root 4.0K Aug 13 12:12 .
drwxr-x--- 37 root root 4.0K Aug 12 23:56 ..
-rw-r--r--  1 root root 2.6G Aug 11 18:26 charibase1-disk1-snapshot-2014-08-11-1407772806.dd.disk.gz
-rw-r--r--  1 root root 4.1G Aug 11 18:47 charibase1-disk2-snapshot-2014-08-11-1407772806.dd.disk.gz
-rw-r--r--  1 root root 1.6G Aug 11 18:19 charibase1-disk3-snapshot-2014-08-11-1407772806.dd.disk.gz
-rw-r--r--  1 root root 4.4G Aug 12 18:45 somewhere-disk1-snapshot-2014-08-12-1407858305.dd.disk.gz
-rw-r--r--  1 root root  20G Aug 13 01:20 somewhere-disk2-snapshot-2014-08-12-1407858305.dd.disk.gz
-rw-r--r--  1 root root 4.3G Aug 12 18:26 horoscope-disk-snapshot-2014-08-12-1407858305.dd.disk.gz
-rw-r--r--  1 root root 1.7G Aug 12 18:46 kvm01-disk-snapshot-2014-08-12-1407860819.dd.disk.gz
-rw-r--r--  1 root root 5.5G Aug 12 19:38 ldapproxy1-disk-snapshot-2014-08-12-1407861988.dd.disk.gz
-rw-r--r--  1 root root  15G Aug 12 21:29 liferay1-disk1-snapshot-2014-08-12-1407861988.dd.disk.gz
-rw-r--r--  1 root root 3.4G Aug  6 19:32 maintenance1-disk-snapshot-2014-08-06-1407344350.dd.disk.gz
-rw-r--r--  1 root root  13G Aug 12 22:01 media1-disk3-snapshot-2014-08-12-1407865141.dd.disk.gz
-rw-r--r--  1 root root 2.5G Aug 10 21:52 nginx1-disk-snapshot-2014-08-10-1407698737.dd.disk.gz
-rw-r--r--  1 root root 1.5G Aug 10 21:56 ns0-disk-snapshot-2014-08-10-1407699464.dd.disk.gz
-rw-r--r--  1 root root  57G Aug 13 06:54 opsi-disk-snapshot-2014-08-12-1407871944.dd.disk.gz
-rw-r--r--  1 root root 7.9G Aug 10 23:36 osiris-at-disk-snapshot-2014-08-10-1407700383.dd.disk.gz
-rw-r--r--  1 root root 9.7G Aug 10 23:58 osiris-ot-disk-snapshot-2014-08-10-1407700588.dd.disk.gz
-rw-r--r--  1 root root 8.8G Aug 12 23:56 osiris-training2-disk-snapshot-2014-08-12-1407873831.dd.disk.gz
-rw-r--r--  1 root root 7.4G Aug 13 01:29 owncloud1-disk1-snapshot-2014-08-12-1407880682.dd.disk.gz
-rw-r--r--  1 root root  12G Aug 13 03:36 portal1-disk1-snapshot-2014-08-13-1407885830.dd.disk.gz
-rw-r--r--  1 root root  11G Aug 11 03:12 relax1-disk1-snapshot-2014-08-11-1407712305.dd.disk.gz
-rw-r--r--  1 root root 5.3G Aug 13 02:51 reporting-disk-snapshot-2014-08-13-1407886264.dd.disk.gz
-rw-r--r--  1 root root 4.0G Aug 13 03:45 sec1-disk1-snapshot-2014-08-13-1407891153.dd.disk.gz
-rw-r--r--  1 root root 1.3G Aug 11 03:11 socks1-disk-snapshot-2014-08-11-1407718475.dd.disk.gz
-rw-r--r--  1 root root 6.0G Aug 13 05:00 sr1-disk1-snapshot-2014-08-13-1407893897.dd.disk.gz
-rw-r--r--  1 root root 610M Aug 13 03:54 sr1-disk2-snapshot-2014-08-13-1407894397.dd.disk.gz
-rw-r--r--  1 root root  15G Aug 13 06:35 sr2-disk1-snapshot-2014-08-13-1407894904.dd.disk.gz
-rw-r--r--  1 root root 1.1G Aug 13 05:16 sr2-disk3-snapshot-2014-08-13-1407898880.dd.disk.gz
-rw-r--r--  1 root root  21G Aug 13 08:53 sr4-disk1-snapshot-2014-08-13-1407899777.dd.disk.gz
-rw-r--r--  1 root root  13G Aug 13 08:40 sr4-disk2-snapshot-2014-08-13-1407904622.dd.disk.gz
-rw-r--r--  1 root root 7.7G Aug 13 08:36 sr5-disk1-snapshot-2014-08-13-1407906095.dd.disk.gz
-rw-r--r--  1 root root  23G Aug 13 12:10 sr5-disk2-snapshot-2014-08-13-1407911854.dd.disk.gz
-rw-r--r--  1 root root 6.2G Aug 13 09:40 webserver1-disk-snapshot-2014-08-13-1407912157.dd.disk.gz
-rw-r--r--  1 root root  36G Aug 12 14:18 webserver1-disk2-snapshot-2014-08-12-1407828382.dd.disk.gz
-rw-r--r--  1 root root 2.6G Aug 11 09:05 zabbix2-disk-snapshot-2014-08-11-1407738226.dd.disk.gz
root <at> backup:~# ls -hal /etc/backup.d/
10.sys                               90.system.dup                        91.libvirt-guests.dup.disabled       92.libvirt-filesystems.dup.disabled  93.somewhere-zimbra.dup.disabled
root <at> backup:~# grep -i include /etc/backup.d/91.libvirt-guests.dup.disabled
# NB: neither quote this, nor should it include any quotes
# A few notes about includes and excludes:
# 1. include, exclude and vsinclude statements support globbing with '*'
# 2. Symlinks are not dereferenced. Moreover, an include line whose path
#      include = /home/user/Mail
#      include = /mnt/crypt/home/user/Mail
# 3. All the excludes come after all the includes. The order is not otherwise
# files to include in the backup
include = /srv/storage/backups/libvirt-guests/latest/
# vsinclude = <path>
# vsinclude = <path>
# Any path specified in vsinclude is added to the include list for each vserver
# For example, vsinclude = /home will backup the /home directory in every
# vsinclude will add to the include list /vservers/foo/home, /vservers/bar/home
_______________________________________________
Duplicity-talk mailing list
Duplicity-talk <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
boytruc | 12 Aug 00:26 2014
Picon

git and duplicity

Hello 

My question will be strange but, is it possible to get any problem with git folder who are backup by
duplicity. I have 2 VPS and 1 raspberry who are all using duplicity script under debian (threw
duplicity-backup script) and I'm getting sometimes (3 times this year) some weird problem with git folder.

Folder with project 'git cloned' getting empty (except unrecognized .git folder) or having problem
without any changes on files (example for dot-files). I'm not doing lot's of stuff on this machine except
usual update. 

It's not probably related, but I was wondering.

Thanks in advance for your help.

Ben
Alex | 5 Aug 09:35 2014
Picon

Problem with file list

Hi there. I ran duplicity in this manner:

duplicity --progress -v8 --full-if-older-than 1M
--include-globbing-filelist /root/file_list --sign-key [...] /
ftp://$FTP_USER <at> $FTP_HOST/backup

With file_list containing the following:

+ /usr/local/content
+ /etc/nginx
- /**

Why does duplicity insist on backing up /bin, /sys, /proc, all of /etc,
even though I specifically told it not to with "- /**"?
Nate Eldredge | 3 Aug 02:03 2014

Incremental backup of files with changed data but unchanged timestamp

I am using duplicity to make incremental backups of my system.  I have 
some files whose data has changed since the last backup, but whose mtime 
stayed the same.  It looks like `duplicity incremental' ignores files 
whose timestamp has not changed, so it doesn't back up the new data.  Is 
there a way to force duplicity to compare the file with a stored checksum, 
or even to use rdiff unconditionally?  I'd prefer not to have to do a new 
full backup.

I'd consider hacking duplicity myself but it would be helpful to know 
where in the code I should look.

(Before you accuse me of abusing timestamps: it isn't my fault!  I 
crossgraded this Ubuntu system from 32-bit to 64-bit.  It appears that 
some Ubuntu packages have the same timestamps on corresponding files in 
the 32-bit and 64-bit versions.  Presumably the packages were generated at 
the same time, and coincidentally those files were compiled during the 
same second.  So when I replaced the 32-bit package with the 64-bit 
package, I get a different file with the same timestamp.)

I'm using duplicity 0.6.23 (latest from the PPA) on Ubuntu 14.04.

Thanks!

--

-- 
Nate Eldredge
nate <at> thatsmathematics.com
Ian Chard | 31 Jul 16:13 2014

cleanup process is 13GB and takes >16 hours at 100% CPU

Hi,

I'm running a cleanup of a large target, and the process I get is 13GB
in size and uses 100% CPU for many hours.  So far it's been running for
over 16 hours, and apart from reading a signatures file every few hours
it's giving no indication of progress.  It's spinning at 100% on the
CPU, so this isn't the remote end being slow.  I'm having to run it on a
spare machine because it's using so much memory.

My command line is:

duplicity -v9 --archive-dir /data/duplicity-archive/ --gpg-options
--homedir=/data/gpg-home/ --encrypt-key xxxxxxxx --asynchronous-upload
--full-if-older-than 10D --allow-source-mismatch --num-retries 1 cleanup
--force cf+http://my.target.name

I'm using duplicity 0.6.24, python 2.7.3, and Debian Wheezy.

Is this a scalability problem of duplicity, or more likely to be a bug?

I've logged a Launchpad bug
(https://bugs.launchpad.net/duplicity/+bug/1350404), but I see there are
many bugs in 'New' status, so I thought I'd try the mailing list too.

Thanks for any help
- Ian

--

-- 
Ian Chard   <ian <at> mysociety.org>
mySociety systems administrator   http://www.mysociety.org/
a.grandi@gmail.com | 25 Jul 11:19 2014
Picon

ImportError: cannot import name _librsync

Hi,

I'm trying to re-create the development environment for Duplicity on
two different machines and I always get the same error when I try to
run duplicity.

I've installed all the requirements, checked the source code and finally I do:

python setup.py build

to build the libs. Everything run without any error. Then I do:

PYTHONPATH=. ./bin/duplicity

but I get this error:

Traceback (most recent call last):
  File "./bin/duplicity", line 41, in <module>
    from duplicity import collections
  File "/home/andrea/Documents/development/duplicity-sx/duplicity/collections.py",
line 32, in <module>
    from duplicity import path
  File "/home/andrea/Documents/development/duplicity-sx/duplicity/path.py",
line 38, in <module>
    from duplicity import librsync
  File "/home/andrea/Documents/development/duplicity-sx/duplicity/librsync.py",
line 29, in <module>
    from . import _librsync
ImportError: cannot import name _librsync

Infact _librsync is not under the source folder, nor in the /bin/
folder. The only _librsync I could find was this one:

build/lib.linux-x86_64-2.7/duplicity/_librsync.so

What am I doing wrong that duplicity can't see that lib? I remember
that even at home I was executing duplicity using this command:
PYTHONPATH=. ./bin/duplicity

So I don't understand why on these other two machines is not working.

Thank you so much for your help!

Cheers.

--

-- 
Andrea Grandi -  Software Engineer / Qt Ambassador / Nokia Developer Champion
website: http://www.andreagrandi.it
Sergei G | 24 Jul 10:02 2014
Picon

FYI: pip freeze of my duplicity on mac and installation of duplicity with it

Quick installation instructions for OS X and not only OS X.

If you have virtualenvwrapper configured then:

mkproj MyProjNameLikeBackup

This will create virtual environment with Python 2.7.
then run

pip install -r requirements.txt

the content of requirements.txt is the following block of lines:

duplicity==0.6.24
ecdsa==0.11
lockfile==0.9.1
paramiko==1.14.0
pycrypto==2.6.1
wsgiref==0.1.2

Install Pretty Good Privacy.

Install other protocol specific dependencies, for example ncftp.

I think it is worth including requirements file with duplicity source code.
Sergei G | 24 Jul 09:47 2014
Picon

Feedback based on restore experience

Background: My old Mac Mini failed, so I had to restore data from 
duplicity backup. Unfortunately, I experienced a set of mostly unrelated 
issues with hardware and FreeBSD 10 handling of external USB drives. 
These issues made me to try restore multiple times. My experience ended 
up focused on sftp and later ftp.

Absolutely every attempt was delayed by a single type of error message:
     CollectionsError: No backup chains found
I provide a full stack trace at the end.

The problem with this error is that it reports a consequence and not an 
issue. Duplicity failed to get to the desired source directory, but it 
fails to report the preceding error.

For example, If FTP user's root is /home/userName, but backup is at 
/mnt/extufs/, FTP client will fail to cd to the new directory, while 
interactive console will have no problem.  FTP failure to change path is 
not properly reported and that's the preceding error.

 From what I remember the absolute form of sftp was one of the reasons 
for the same error message. I think sftp properly reports failure to cd 
to the remote directory. In my case, I was using 
sftp://user <at> host/abs/path/to/dir, but I should've used 
sftp://user <at> host//abs/path/to/dir. RTFM, but reporting a hint would be nice.

I usually had to find answers using other programs to questions:

1. What's my login remote directory? Is it the same as in other program 
I used?
2. What's my current remote directory when error occurred?

Answers to both questions from duplicity itself would make my life 
easier when troubleshooting.

Another set of problems was connected to incomplete error reporting when 
calling command line programs. I had a missing gpg command and I could 
not figure out its name, but its arguments were reported in the error 
message. I actually had to add print statements to source code to see 
what's the name of the missing application.

Here is a full stack trace for the typical FTP error:

NcFTP version is 3.2.5
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
Traceback (most recent call last):
   File "/Users/sergei/.py-virtualenvs/duplicity/bin/duplicity", line 
1502, in <module>
     with_tempdir(main)
   File "/Users/sergei/.py-virtualenvs/duplicity/bin/duplicity", line 
1496, in with_tempdir
     fn()
   File "/Users/sergei/.py-virtualenvs/duplicity/bin/duplicity", line 
1345, in main
     do_backup(action)
   File "/Users/sergei/.py-virtualenvs/duplicity/bin/duplicity", line 
1430, in do_backup
     restore(col_stats)
   File "/Users/sergei/.py-virtualenvs/duplicity/bin/duplicity", line 
693, in restore
     restore_get_patched_rop_iter(col_stats)):
   File "/Users/sergei/.py-virtualenvs/duplicity/bin/duplicity", line 
715, in restore_get_patched_rop_iter
     backup_chain = col_stats.get_backup_chain_at_time(time)
   File

"/Users/sergei/.py-virtualenvs/duplicity/lib/python2.7/site-packages/duplicity/collections.py", 
line 952, in get_backup_chain_at_time
     raise CollectionsError("No backup chains found")
CollectionsError: No backup chains found
Rubin Abdi | 23 Jul 03:57 2014
Picon

Re: Backing up desktop virtual machines?

Laurence Perkins (OE) wrote on 2014-07-22 09:52:
> My attempts to reply to the list don't seem to be getting through, so
> I'll send you my best guess as to the problem directly and you can
> forward it on to the list if I am correct.

Maybe the list admin is around and will see this.

> Duplicity only backs up changed parts of the file, but must scan the
> entire file to determine which parts have changed.  If your .VDIs for
> your VMs are large, this will take a while.  Try snapshotting the VMs.
> This will cause all changes to the disks to be written to a separate,
> potentially much smaller file.  If you snapshot after each incremental,
> then all the changes for the VM will be in their own, small file that
> duplicity can just pick up and add to the archive without having to scan
> the big, unchanged root VDI.

Thanks to both you and Edgar for pointing that out. I had no idea.
That's kind of awesome. It now makes sense that the slow down is simply
from Duplicity having to deal with scanning a large file in order to
find diffs.

> Then, before running your next full backup, merge all the snapshots back
> into the main image.

I'll try using Virtualbox with snapshots and see how that goes. I've
only been using Duplicity for a month. How often should I do a full
backup if I'm running it for my whole laptop?

Thanks!

--

-- 
Rubin
rubin <at> starset.net

_______________________________________________
Duplicity-talk mailing list
Duplicity-talk <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

Gmane