Roberto Bampi | 21 Apr 10:25 2014
Picon

Issue with verify and S3

Hello,
I recently started using duplicity to back-up some machines on amazon s3.
The way I made it work was by using a cronjob to perform backups and a sensu client check to periodically verify if the backup was up-to-date.
This particular check called duplicity verify.

Yesterday I discovered I have over 100$ in transfer fees from S3 which is strange, given that I uploaded only about 10 gigabytes in this billing period.
Is it possible that verify is downloading all the data every time to check if the backup is up-to-date ? Has anything like this ever happened to anyone before?

I include this pastebin which shows the total usage of traffic: http://pastebin.com/SP2PEJk5
foobar is the backup bucket and the numbers are in gigabytes (except for bytesHour which has to be divided by 24 to get the actual usage).

Thank you,
Roberto

_______________________________________________
Duplicity-talk mailing list
Duplicity-talk <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Adam Gold | 18 Apr 16:51 2014
Picon

Re: duply list

>> On 17.04.2014 21:52, Adam Gold wrote:
>> I have two backup sets I run, let's call them A and B. Sometime in >> the
>> last month or so (I think), when I run duply A list it returns the >>
same
>> result as if I'd run duply status rather than providing a list of all
>> the files. I have tried this on two different machines, one running
>> Fedora 19 and one running Ubuntu 14.04 beta 2. I'm using duply 1.60
>>
>> However with backup set B, I get an individual list of files as >>
would be expected.
>>
>> Could anyone give me suggestions on how I can try and track down why
>> this is happening? Thanks.

> first update to the latest release 1.7.3. you can simply download the
> tarball from http://duply.net/ and replace your local ./duply with
>the on contained in the archive. alternatively place it somewhere else
>and simply run it from there.

I tried with the latest version but still the same (repo A only, repo B
still works)

> secondly.. see if the error persists and if so.. run duply with the
>--preview option to see the generated command lines and post the out
>put here..

Here go you:

==============================================
Start duply v1.7.3, time is 2014-04-17 22:19:37.
Using profile '[ ]'.
Using installed duplicity version 0.6.23, python 2.7.6, gpg 1.4.16
(Home: ~/.gnupg), awk 'mawk 1.3.3 Nov 1996, Copyright (C) Michael D.
Brennan', bash '4.3.8(1)-release (x86_64-pc-linux-gnu)'.
Use configured key '[ ]' as signing key.
-- Run cmd 'Test - Encrypt to [ ] & Sign with [ ]' --
echo 'passphrase' | /usr/bin/gpg --sign --default-key [ ]
--passphrase-fd 0 --batch -r [ ] --status-fd 1 --compress-algo=bzip2
--bzip2-compress-level=9 --trust-model always -o
/tmp/duply.30257.1397769577_ENC -e ./duply 2>&1
-- Run cmd 'Test - Decrypt' --
echo 'passphrase' | /usr/bin/gpg --passphrase-fd 0 --batch
--compress-algo=bzip2 --bzip2-compress-level=9 --trust-model always -o
/tmp/duply.30257.1397769577_DEC -d /tmp/duply.30257.1397769577_ENC 2>&1
-- Run cmd 'Test - Compare' --
test "$(cat './duply')" = "$(cat '/tmp/duply.30257.1397769577_DEC')" 2>&1
Cleanup - Delete '/tmp/duply.30257.1397769577_*'(FAILED)

--- Start running command LIST at 22:19:37.798 ---
TMPDIR='/tmp' PASSPHRASE='passphrase' SIGN_PASSPHRASE='passphrase'
GS_ACCESS_KEY_ID='[ ]' GS_SECRET_ACCESS_KEY='[ ]' duplicity
list-current-files --archive-dir 'cache' --name duply_data --encrypt-key
[ ] --sign-key [ ] --verbosity '8' --gpg-options
'--compress-algo=bzip2 --bzip2-compress-level=9 --trust-model always'
--full-if-older-than 6W --volsize 25 --allow-source-mismatch
--asynchronous-upload -t now 'gs://bucket/folders/'
--- Finished state OK at 22:19:37.873 - Runtime 00:00:00.074 ---
==============================================

One other thing which I just had a look at. This backup is the
recreation of a previous backup now using GS as the backend (previously
I was using some remote drives with sftp). The total number of volumes
is approximately the same with this latest full backup vs a full backup
of the old version. However I had a look at the sig file in the local
cache: for the old version of this backup, the file was appx 400MB but
this time around it's 209MB. I think at this point, I'm going to run a new backup and I'll report the results.‎

_______________________________________________
Duplicity-talk mailing list
Duplicity-talk <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Adam Gold | 17 Apr 21:52 2014
Picon

duply list

I have two backup sets I run, let's call them A and B. Sometime in the
last month or so (I think), when I run duply A list it returns the same
result as if I'd run duply status rather than providing a list of all
the files.  I have tried this on two different machines, one running
Fedora 19 and one running Ubuntu 14.04 beta 2.  I'm using duply 1.60

However with backup set B, I get an individual list of files as would be
expected.

Could anyone give me suggestions on how I can try and track down why
this is happening?  Thanks.
Jeremy Doran | 16 Apr 20:00 2014

Recovery possible without signatures/sigtar and manifest files?


I'm trying to find out if it's possible to restore a backup that, for some reason, does not contain the duplicity-full-signatures.*.sigtar.gpg and duplicity-full.*.manifest.gpg files.

I'm not sure how this particular state happened, but I now have 2300+ duplicity-full.*.vol*.difftar.gpg files.

I don't care about extracting individual files from them. I'd like to just pull out everything.

Thanks.
_______________________________________________
Duplicity-talk mailing list
Duplicity-talk <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Michael Terry | 16 Apr 14:52 2014

Re: list-all-files command

I imagine another thing useful for such a use case would be asking duplicity to only consider a particular directory, rather than giving all files in the entire backup (which could be a lot of data, especially if it gives you all versions of those files).


On 16 April 2014 04:49, Tristan Hill <tristan <at> saticed.me.uk> wrote:
Hi,

I want to make a restore interface that allows browsing through the
backed up data, including different versions of files and deleted
files.

I'm thinking that I want duplicity to print a list of all versions of
files stored and when those files were deleted in an output similar to
list-current-files.  I can then parse this to create a view showing
the data.

To aid this I'm playing with adding a list-all-files command.  I've
made some initial changes on a branch,
https://code.launchpad.net/~stan/duplicity/list-all-files.

Currently I'm looking over all the signature chains and printing the
file objects in each.  However, I'm not really sure what's in the
signature chain.  For example, in the reasonable sized backup I'm
testing there are 4 entries for my vimrc:

Sun Mar 23 10:08:27 2014 home/stan/.vimrc
Sun Mar 23 10:08:27 2014 home/stan/.vimrc
Sun Mar 23 10:08:27 2014 home/stan/.vimrc
Sun Mar 23 10:08:27 2014 home/stan/.vimrc

Can anyone explain?

Thanks
Tristan

_______________________________________________
Duplicity-talk mailing list
Duplicity-talk <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

_______________________________________________
Duplicity-talk mailing list
Duplicity-talk <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Tristan Hill | 16 Apr 10:49 2014
Picon

list-all-files command

Hi,

I want to make a restore interface that allows browsing through the
backed up data, including different versions of files and deleted
files.

I'm thinking that I want duplicity to print a list of all versions of
files stored and when those files were deleted in an output similar to
list-current-files.  I can then parse this to create a view showing
the data.

To aid this I'm playing with adding a list-all-files command.  I've
made some initial changes on a branch,
https://code.launchpad.net/~stan/duplicity/list-all-files.

Currently I'm looking over all the signature chains and printing the
file objects in each.  However, I'm not really sure what's in the
signature chain.  For example, in the reasonable sized backup I'm
testing there are 4 entries for my vimrc:

Sun Mar 23 10:08:27 2014 home/stan/.vimrc
Sun Mar 23 10:08:27 2014 home/stan/.vimrc
Sun Mar 23 10:08:27 2014 home/stan/.vimrc
Sun Mar 23 10:08:27 2014 home/stan/.vimrc

Can anyone explain?

Thanks
Tristan
Carlos Chavez | 10 Apr 23:41 2014

Missing full backup...

     I seem to be having a problem.  I have several servers backing up 
to a single server.  All except one are working perfectly and I have 
been able to restore files from all of them.  All servers do a FULL 
backup on Sunday and incremental backups the rest of the week. One of 
the servers (used for email) misses the FULL backup on Sunday some of 
the times and keeps going with incremental backups.  Here is the listing:

Last full backup date: Sun Mar 30 04:39:30 2014
Collection Status
-----------------
Connecting with backend: SSHBackend
Archive dir: /root/.cache/duplicity/d0e2f4f0aec36ed03a5b14adcbd2eec4

Found 3 secondary backup chains.
Secondary chain 1 of 3:
-------------------------
Chain start time: Sat Feb  8 04:03:08 2014
Chain end time: Sat Feb 22 04:02:59 2014
Number of contained backup sets: 6
Total number of contained volumes: 28522
  Type of backup set:                            Time:      Num volumes:
                 Full         Sat Feb  8 04:03:08 2014 28350
          Incremental         Tue Feb 18 22:15:42 2014 110
          Incremental         Wed Feb 19 04:09:47 2014                 2
          Incremental         Thu Feb 20 04:03:01 2014 20
          Incremental         Fri Feb 21 04:04:13 2014 28
          Incremental         Sat Feb 22 04:02:59 2014 12
-------------------------

Secondary chain 2 of 3:
-------------------------
Chain start time: Tue Mar 25 13:42:25 2014
Chain end time: Tue Mar 25 13:42:25 2014
Number of contained backup sets: 1
Total number of contained volumes: 5852
  Type of backup set:                            Time:      Num volumes:
                 Full         Tue Mar 25 13:42:25 2014 5852
-------------------------

Secondary chain 3 of 3:
-------------------------
Chain start time: Sun Feb 23 04:24:34 2014
Chain end time: Sun Mar 30 04:03:49 2014
Number of contained backup sets: 32
Total number of contained volumes: 6214
  Type of backup set:                            Time:      Num volumes:
                 Full         Sun Feb 23 04:24:34 2014 5717
          Incremental         Mon Feb 24 04:03:25 2014                 5
          Incremental         Tue Feb 25 04:03:01 2014 17
          Incremental         Wed Feb 26 04:03:03 2014 17
          Incremental         Thu Feb 27 04:03:01 2014 20
          Incremental         Fri Feb 28 04:02:58 2014 20
          Incremental         Sat Mar  1 04:03:55 2014 17
          Incremental         Sun Mar  2 04:03:01 2014                 6
          Incremental         Sun Mar  2 04:38:44 2014                 1
          Incremental         Mon Mar  3 04:02:49 2014                 4
          Incremental         Tue Mar  4 04:03:07 2014 12
          Incremental         Wed Mar  5 04:03:13 2014 30
          Incremental         Thu Mar  6 04:02:56 2014 15
          Incremental         Fri Mar  7 04:02:56 2014 14
          Incremental         Sat Mar  8 04:04:09 2014 16
          Incremental         Sun Mar  9 04:03:09 2014                 5
          Incremental         Sun Mar  9 04:41:11 2014                 1
          Incremental         Mon Mar 10 04:03:11 2014                 7
          Incremental         Tue Mar 11 04:03:09 2014 18
          Incremental         Wed Mar 12 04:03:32 2014 25
          Incremental         Thu Mar 13 04:03:06 2014 39
          Incremental         Fri Mar 14 04:03:02 2014 17
          Incremental         Sat Mar 15 04:02:59 2014 21
          Incremental         Sun Mar 16 04:03:31 2014                 5
          Incremental         Sun Mar 16 04:41:58 2014                 1
          Incremental         Mon Mar 17 04:03:00 2014                 4
          Incremental         Tue Mar 18 04:03:13 2014 10
          Incremental         Wed Mar 26 04:08:34 2014 87
          Incremental         Thu Mar 27 04:03:14 2014 14
          Incremental         Fri Mar 28 04:03:01 2014 28
          Incremental         Sat Mar 29 04:03:02 2014 14
          Incremental         Sun Mar 30 04:03:49 2014                 7
-------------------------

Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Sun Mar 30 04:39:30 2014
Chain end time: Thu Apr 10 04:03:30 2014
Number of contained backup sets: 13
Total number of contained volumes: 6018
  Type of backup set:                            Time:      Num volumes:
                 Full         Sun Mar 30 04:39:30 2014 5848
          Incremental         Mon Mar 31 04:03:03 2014                 4
          Incremental         Tue Apr  1 04:04:00 2014 22
          Incremental         Wed Apr  2 04:03:54 2014 14
          Incremental         Thu Apr  3 04:04:20 2014 26
          Incremental         Fri Apr  4 04:03:03 2014 16
          Incremental         Sat Apr  5 04:05:03 2014 18
          Incremental         Sun Apr  6 04:03:39 2014                 5
          Incremental         Sun Apr  6 04:43:11 2014                 1
          Incremental         Mon Apr  7 04:03:14 2014                 5
          Incremental         Tue Apr  8 04:03:22 2014 13
          Incremental         Wed Apr  9 04:03:32 2014 18
          Incremental         Thu Apr 10 04:03:30 2014 28
-------------------------
No orphaned or incomplete backup sets found.

     I have no idea why this particular server will sometimes fail to do 
the FULL backup on Sunday.  There seem to be two incremental backups for 
Sunday.  I run the incremental backup from cron.daily and the full 
backup from cron.weekly.  All servers have exactly the same cron jobs.  
We are using Duplicity 0.6.13 which is the latest version we can compile 
on out CentOS 5.10 servers.

--

-- 
Telecomunicaciones Abiertas de México S.A. de C.V.
Carlos Chávez
+52 (55)9116-91161
Chris Vanden Berghe | 3 Apr 10:06 2014

Google Drive vs. Google Cloud Storage

Hi all,

I understand that Duplicity supports both Google Drive and Google Cloud
Storage through gdocs:// and gs:// respectively. Does anyone have a view
on the (dis)advantages of using GD vs GCS as a Duplicity backend in
terms of speed, reliability or other technical aspects? I understand
that the pricing model is different (tiers vs. actual usage, network
charges).

Because I'm moving from LAN to cloud backups, I'm also rethinking my
backup schedule. I used to make a full backup once per week with daily
incrementals while keeping at least 2 full backups. This might no longer
be practical as a full backup to GCS takes ~24h (for ~200GB). What is
the main disadvantage of less frequent full backups? Slower retrieval,
increasing total storage requirement or something else?

Thanks,
Chris.
Barnet Wagman | 31 Mar 21:54 2014

Encryption without GnuPG?

Is it possible to use something other than GnuPG for encryption in 
duplicity?  E.g. I often write programs that use the Bouncy Castle 
library, and I'd like to write a little one to use with duplicity (if it 
doesn't require modifying  duplicity itself).

thanks
Tristan Hill | 30 Mar 15:46 2014
Picon

backup restore web interface

Hi,

I would like a tool similar to box backup explorer[1] for duplicity.
Basically I would describe this as a web based file restoration tool
with the following features:
* browse directory listing of backup (including deleted and old
versions of files)
* download selected files to an archive

It seems to achieve such a thing with duplicity I would need to parse
the list-current-files and collection-status command outputs.  Any
tips on whether this is the best approach?

Thanks
Tristan

[1] http://www.joonis.de/en/software/boxbackup-explorer
Grant | 30 Mar 05:00 2014
Picon

Automated restore

Can I run an automated restore of my backups to verify that they're in
working order and not tampered with?

- Grant

Gmane