Remy van Elst | 25 May 11:04 2015
Picon

Duplicity using 1.5 TB storage and loosing incremental backups?


Duplicity seems to loose it's status when trying to do a incremental
backup set over 14 days. The backup has been running for a while now
but it is using about 1.5 TB of backend storage since 2 weeks. It is
going to an Openstack swift object store as backend.

This is the command line:

     duplicity --asynchronous-upload --verbosity 9 --log-file
/var/log/duplicity.log --volsize 25 --tempdir="/tmp"
--file-prefix="vps1.sparklingclouds.nl."
--name="vps1.sparklingclouds.nl." --exclude-device-files
--exclude-globbing-filelist=/etc/duplicity-backup/exclude.conf
--full-if-older-than="14D" --no-encryption  / swift://duplicity-backup

This is what caught my eye in the verbose log file:

    File
vps1.sparklingclouds.nl.duplicity-full.20150430T034939Z.vol166.difftar.g
z
is part of known set
    File
vps1.sparklingclouds.nl.duplicity-full.20150430T034939Z.vol1660.difftar.
gz
is part of known set
    Found backup chain [Tue Mar 24 14:42:44 2015]-[Tue Mar 24 14:42:44
2015]
    Last full backup is too old, forcing full backup

The last full backup is from yesterday, and it should only do a full
(Continue reading)

Tiernan OToole | 20 May 13:57 2015
Picon

HubiC progress not showing properly


morning all.

Im using Duplicity and HubiC together to backup some systems, and
noticed that if i put --progress at the end, it shows 0kb/s and
stalled and then when the upload finishes it jumps to 100%... Any
ideas what needs to be done here?

Thanks.

--Tiernan
Roman Yepishev | 16 May 15:20 2015

Recreates and uploads already uploaded volume upon restart?

Hello,

Using duplicity 0.6.25 on Fedora 21.

I am currently working on MediaFire backend[0] and I found that when
backup fails to upload, duplicity re-creates and re-uploads an already
uploaded volume. For large volumes this increases the backup time. Since
the new archive is not exactly the same as the one already uploaded (I
suspect file order/metadata differs) it does not match the checksum on
the server, so the server can't just tell the client "yeah, I already
have that, no need to reupload".

More logs are available at [1], where I reproduced the same issue with
file backend, but the interesting lines are:

---------------------------------------------------------------------

Writing /home/rye/tmp/duplicity/1/duplicity-full.20150516T130024Z.vol1.difftar.gpg
AsyncScheduler: task completed successfully
Processed volume 1
...
AsyncScheduler: running task synchronously (asynchronicity disabled)
Writing /home/rye/tmp/duplicity/1/duplicity-full.20150516T130024Z.vol2.difftar.gpg
Processed volume 2
----------------------------------------------------------------------

I press CTRL+C - simulating backend failure to upload, as it sometimes
happens with MediaFire.

----------------------------------------------------------------------
(Continue reading)

Kenneth Loafman | 11 May 15:10 2015

Duplicity 0.7.03 Released

Hello All,

We have a lot of good stuff in this release.  Thanks to all the contributors!

Full details of the release and the tarball can be found at Milestone 0.7.03

...Thanks,
...Ken

_______________________________________________
Duplicity-talk mailing list
Duplicity-talk <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Kenneth Loafman | 6 May 18:48 2015

Duplicity 0.6.26 Released

Hello All,

This release is primarily due due to bug 1416344. This is a security bug with such a low chance of being used as to be nonexistent, but the distros have decided to break the world anyway, so here goes, needless work to fix a nonexistent problem.

We have minimal other fixes in this release.  Thanks to all the contributors!

Gory details of the release and the tarball can be found at Milestone 0.6.26

...Thanks,
...Ken

_______________________________________________
Duplicity-talk mailing list
Duplicity-talk <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
ewering@gmail.com | 6 May 10:27 2015
Picon

Duplicity wiped out my server

OK folks, this software is truly DANGEROUS. The command line is incredibly obscure and counter-intuitive, and just trying to SIMPLY restore a single file resulted in duplicity WIPING OUT the entire root file system. What the hell?

I can't say what I typed exactly because as I said the server is gone and the commandlines are friggin complicated and long, however, I pretty damn well remember that I included the "file-to-restore /path/to/blabla.ext" option, and why the FUCK can this result in the software deleting everything? The instruction I gave it is clear: RESTORE A SINGLE DAMN FILE. It was NOT: "ERASE ALL FILES"!

Now if this were paid software I would at least ask for a refund, however, obviously I'll stop using it and warn everyone I'll ever talk to about even considering its use. It's no surprise that mostly everyone I know, even big corporations, still simply make fucking tar archives.
_______________________________________________
Duplicity-talk mailing list
Duplicity-talk <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Dirk-Willem van Gulik | 3 May 18:37 2015

duplicity 0.7.02, freebsd and sftp backend -- how to further debug.

------------ ------------ ------------ ------------ ------------ ——————

On a (new) 0.7.02 upgrade/install; which works fine with a file:/// backend to /tmp as a test — we’re
finding that it hangs ‘forever’ on ‘sftp’.

sftp itself works fine; and when we place a few files there - we can see in the output that they are nicely seen;
and ignored (as is the .ssh directory with the right authorized_keys magic).

Below is the command used and the output. Any suggestion as to where/how to start debugging this ?

The main change with prior version/use is a sharply increased volume and going to version 0.7.02 (as python
needed to be upgraded).

Thanks,

Dw.

# duplicity --version
duplicity 0.7.02

# cat backup.sh

#!/bin/sh
unset SSH_AUTH_SOCK
export GNUPGHOME=/XXXXXXX
export PASSPHRASE=""
export SIGN_PASSPHRASE=“”

# Removed: --gpg-options "--yes --lock-never --no-random-seed-file  --no-greeting
--no-secmem-warning --no-auto-check-trustd -q " 

/usr/local/bin/duplicity \
	--ssh-options "-oIdentityFile=XXXXX  \
	--archive-dir /var/db/duplicity \
	--hidden-encrypt-key XXX \
	--hidden-encrypt-key XXX \
	--hidden-encrypt-key XXX \
	--sign-key XXX \
	--include /XXX \
	..snipped directories XXXXX ...
	--exclude '**' \
	/ sftp://XXX <at> XXX.XXX

# sh backup.sh 
Using archive dir: /var/db/duplicity/be3234cf4cbafbdf41099030093b6bbe
Using backup name: be3234cf4cbafbdf41099030093b6bbe
Import of duplicity.backends.azurebackend Succeeded
Import of duplicity.backends.botobackend Succeeded
Import of duplicity.backends.cfbackend Succeeded
Import of duplicity.backends.copycombackend Succeeded
Import of duplicity.backends.dpbxbackend Succeeded
Import of duplicity.backends.ftpbackend Succeeded
Import of duplicity.backends.ftpsbackend Succeeded
Import of duplicity.backends.gdocsbackend Succeeded
Import of duplicity.backends.giobackend Succeeded
Import of duplicity.backends.hsibackend Succeeded
Import of duplicity.backends.hubicbackend Succeeded
Import of duplicity.backends.imapbackend Succeeded
Import of duplicity.backends.lftpbackend Failed: the scheme ftp already has a backend associated with it
Import of duplicity.backends.localbackend Succeeded
Import of duplicity.backends.megabackend Succeeded
Import of duplicity.backends.ncftpbackend Succeeded
Import of duplicity.backends.onedrivebackend Failed: No module named requests
Import of duplicity.backends.par2backend Succeeded
Import of duplicity.backends.pydrivebackend Succeeded
Import of duplicity.backends.rsyncbackend Succeeded
Import of duplicity.backends.ssh_paramiko_backend Succeeded
Import of duplicity.backends.ssh_pexpect_backend Succeeded
Import of duplicity.backends.sshbackend Failed: 'module' object has no attribute 'ssh_backend'
Import of duplicity.backends.swiftbackend Succeeded
Import of duplicity.backends.sxbackend Succeeded
Import of duplicity.backends.tahoebackend Succeeded
Import of duplicity.backends.webdavbackend Succeeded
Import of duplicity.backends.~par2wrapperbackend Succeeded
ssh: starting thread (client mode): 0x816c750L
ssh: Connected (version 2.0, client mod_sftp/0.9.9)
ssh: kex algos:[u'ecdh-sha2-nistp256', u'ecdh-sha2-nistp384', u'ecdh-sha2-nistp521',
u'diffie-hellman-group-exchange-sha256', u'diffie-hellman-group-exchange-sha1',
u'diffie-hellman-group14-sha1', u'diffie-hellman-group1-sha1', u'rsa1024-sha1'] server
key:[u'ssh-rsa', u'ssh-dss'] client encrypt:[u'aes256-ctr', u'aes192-ctr', u'aes128-ctr',
u'aes256-cbc', u'aes192-cbc', u'aes128-cbc', u'blowfish-ctr', u'blowfish-cbc', u'cast128-cbc',
u'arcfour256', u'arcfour128', u'3des-ctr', u'3des-cbc'] server encrypt:[u'aes256-ctr',
u'aes192-ctr', u'aes128-ctr', u'aes256-cbc', u'aes192-cbc', u'aes128-cbc', u'blowfish-ctr',
u'blowfish-cbc', u'cast128-cbc', u'arcfour256', u'arcfour128', u'3des-ctr', u'3des-cbc'] client
mac:[u'hmac-sha2-256', u'hmac-sha2-512', u'hmac-sha1', u'hmac-sha1-96', u'hmac-md5',
u'hmac-md5-96', u'hmac-ripemd160', u'umac-64 <at> openssh.com'] server mac:[u'hmac-sha2-256',
u'hmac-sha2-512', u'hmac-sha1', u'hmac-sha1-96', u'hmac-md5', u'hmac-md5-96',
u'hmac-ripemd160', u'umac-64 <at> openssh.com'] client compress:[u'none'] server compress:[u'none']
client lang:[u''] server lang:[u''] kex follows?False
ssh: Ciphers agreed: local=aes128-ctr, remote=aes128-ctr
ssh: using kex diffie-hellman-group1-sha1; server key type ssh-rsa; cipher: local aes128-ctr, remote
aes128-ctr; mac: local hmac-sha1, remote hmac-sha1; compression: local none, remote none
ssh: Switch to new keys ...
ssh: Trying key ef4214c5fdba6d3c519c737ed1bf6358 from XXXXX
ssh: userauth is OK
ssh: Authentication (publickey) successful!
ssh: [chan 1] Max packet in: 34816 bytes
ssh: [chan 1] Max packet out: 0 bytes
ssh: Secsh channel 1 opened.
ssh: [chan 1] Sesch channel 1 request ok
ssh: [chan 1] Opened sftp connection (server version 3)
ssh: [chan 1] stat('.')
ssh: [chan 1] stat('.')
ssh: [chan 1] normalize('.')
Main action: inc
================================================================================
duplicity 0.7.02 (March 11, 2015)
Args: /usr/local/bin/duplicity --verbosity debug --ssh-options -oIdentityFile=XXXXX
--archive-dir /var/db/duplicity --hidden-encrypt-key XXX --hidden-encrypt-key XXX
--hidden-encrypt-key XXXX --sign-key XXX --include /XXX XXXXXX  --exclude ** / sftp://XXX <at> XXXX.XXX
FreeBSD XXXXXXX 10.1-RELEASE-p9 FreeBSD 10.1-RELEASE-p9 #0: Tue Apr  7 01:09:46 UTC 2015    
root <at> amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 amd64
/usr/local/bin/python 2.7.9 (default, Apr  8 2015, 15:12:41) 
[GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)]
================================================================================
Using temporary directory /tmp/duplicity-MSIr4U-tempdir
Registering (mkstemp) temporary file /tmp/duplicity-MSIr4U-tempdir/mkstemp-xUwUpW-1
Temp has 13190230286848 available, backup will use approx 34078720.
ssh: [chan 1] listdir('/.')
Local and Remote metadata are synchronized, no sync needed.
ssh: [chan 1] listdir('/.')
1 file exists on backend
4 files exist in cache
Extracting backup chains from list of files:
[u'duplicity-full-signatures.20150502T140646Z.sigtar.part',
u'duplicity-full.20150502T140646Z.manifest.part', u'.ssh']
File duplicity-full-signatures.20150502T140646Z.sigtar.part is not part of a known set; creating new set
Ignoring file (rejected by backup set) 'duplicity-full-signatures.20150502T140646Z.sigtar.part'
File duplicity-full.20150502T140646Z.manifest.part is not part of a known set; creating new set
File .ssh is not part of a known set; creating new set
Ignoring file (rejected by backup set) '.ssh'
Found backup chain [Sat May  2 16:06:46 2015]-[Sat May  2 16:06:46 2015]
Last full backup left a partial set, restarting.
Last full backup date: Sat May  2 16:06:46 2015
Collection Status
-----------------
Connecting with backend: BackendWrapper
Archive dir: /var/db/duplicity/be3234cf4cbafbdf41099030093b6bbe

Found 0 secondary backup chains.

Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Sat May  2 16:06:46 2015
Chain end time: Sat May  2 16:06:46 2015
Number of contained backup sets: 1
Total number of contained volumes: 0
 Type of backup set:                            Time:      Num volumes:
-------------------------
No orphaned or incomplete backup sets found.
RESTART: The first volume failed to upload before termination.
         Restart is impossible...starting backup from beginning.
Deleting /var/db/duplicity/be3234cf4cbafbdf41099030093b6bbe/duplicity-full-signatures.20150502T140646Z.sigtar.part
Deleting /var/db/duplicity/be3234cf4cbafbdf41099030093b6bbe/duplicity-full.20150502T140646Z.manifest.part
Releasing lockfile <LinkLockFile:
'/var/db/duplicity/be3234cf4cbafbdf41099030093b6bbe/XXXXXX.650626177224228893873130' -- '/var/db/duplicity/be3234cf4cbafbdf41099030093b6bbe/lockfile'>
Using archive dir: /var/db/duplicity/be3234cf4cbafbdf41099030093b6bbe
Using backup name: be3234cf4cbafbdf41099030093b6bbe
Import of duplicity.backends.azurebackend Succeeded
Import of duplicity.backends.botobackend Succeeded
Import of duplicity.backends.cfbackend Succeeded
Import of duplicity.backends.copycombackend Succeeded
Import of duplicity.backends.dpbxbackend Succeeded
Import of duplicity.backends.ftpbackend Succeeded
Import of duplicity.backends.ftpsbackend Succeeded
Import of duplicity.backends.gdocsbackend Succeeded
Import of duplicity.backends.giobackend Succeeded
Import of duplicity.backends.hsibackend Succeeded
Import of duplicity.backends.hubicbackend Succeeded
Import of duplicity.backends.imapbackend Succeeded
Import of duplicity.backends.lftpbackend Failed: the scheme ftp already has a backend associated with it
Import of duplicity.backends.localbackend Succeeded
Import of duplicity.backends.megabackend Succeeded
Import of duplicity.backends.ncftpbackend Succeeded
Import of duplicity.backends.onedrivebackend Failed: No module named requests
Import of duplicity.backends.par2backend Succeeded
Import of duplicity.backends.pydrivebackend Succeeded
Import of duplicity.backends.rsyncbackend Succeeded
Import of duplicity.backends.ssh_paramiko_backend Succeeded
Import of duplicity.backends.ssh_pexpect_backend Succeeded
Import of duplicity.backends.sshbackend Failed: 'module' object has no attribute 'ssh_backend'
Import of duplicity.backends.swiftbackend Succeeded
Import of duplicity.backends.sxbackend Succeeded
Import of duplicity.backends.tahoebackend Succeeded
Import of duplicity.backends.webdavbackend Succeeded
Import of duplicity.backends.~par2wrapperbackend Succeeded
ssh: starting thread (client mode): 0x816c750L
ssh: Connected (version 2.0, client mod_sftp/0.9.9)
ssh: kex algos:[u'ecdh-sha2-nistp256', u'ecdh-sha2-nistp384', u'ecdh-sha2-nistp521',
u'diffie-hellman-group-exchange-sha256', u'diffie-hellman-group-exchange-sha1',
u'diffie-hellman-group14-sha1', u'diffie-hellman-group1-sha1', u'rsa1024-sha1'] server
key:[u'ssh-rsa', u'ssh-dss'] client encrypt:[u'aes256-ctr', u'aes192-ctr', u'aes128-ctr',
u'aes256-cbc', u'aes192-cbc', u'aes128-cbc', u'blowfish-ctr', u'blowfish-cbc', u'cast128-cbc',
u'arcfour256', u'arcfour128', u'3des-ctr', u'3des-cbc'] server encrypt:[u'aes256-ctr',
u'aes192-ctr', u'aes128-ctr', u'aes256-cbc', u'aes192-cbc', u'aes128-cbc', u'blowfish-ctr',
u'blowfish-cbc', u'cast128-cbc', u'arcfour256', u'arcfour128', u'3des-ctr', u'3des-cbc'] client
mac:[u'hmac-sha2-256', u'hmac-sha2-512', u'hmac-sha1', u'hmac-sha1-96', u'hmac-md5',
u'hmac-md5-96', u'hmac-ripemd160', u'umac-64 <at> openssh.com'] server mac:[u'hmac-sha2-256',
u'hmac-sha2-512', u'hmac-sha1', u'hmac-sha1-96', u'hmac-md5', u'hmac-md5-96',
u'hmac-ripemd160', u'umac-64 <at> openssh.com'] client compress:[u'none'] server compress:[u'none']
client lang:[u''] server lang:[u''] kex follows?False
ssh: Ciphers agreed: local=aes128-ctr, remote=aes128-ctr
ssh: using kex diffie-hellman-group1-sha1; server key type ssh-rsa; cipher: local aes128-ctr, remote
aes128-ctr; mac: local hmac-sha1, remote hmac-sha1; compression: local none, remote none
ssh: Switch to new keys ...
ssh: Trying key ef4214c5fdba6d3c519c737ed1bf6358 from /XXXXX
ssh: userauth is OK
ssh: Authentication (publickey) successful!
ssh: [chan 1] Max packet in: 34816 bytes
ssh: [chan 1] Max packet out: 0 bytes
ssh: Secsh channel 1 opened.
ssh: [chan 1] Sesch channel 1 request ok
ssh: [chan 1] Opened sftp connection (server version 3)
ssh: [chan 1] stat('.')
ssh: [chan 1] stat('.')
ssh: [chan 1] normalize('.')
Main action: inc
================================================================================
duplicity 0.7.02 (March 11, 2015)
Args: /usr/local/bin/duplicity --verbosity debug --ssh-options -oIdentityFile=//XXXX
--archive-dir /var/db/duplicity --hidden-encrypt-key XXXX --hidden-encrypt-key XXX
--hidden-encrypt-key XXX --sign-key XXXX --include /XXX XXXXXX-exclude ** / sftp://XXX <at> XXXX.XX
FreeBSD XXXXXX 10.1-RELEASE-p9 FreeBSD 10.1-RELEASE-p9 #0: Tue Apr  7 01:09:46 UTC 2015    
root <at> amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 amd64
/usr/local/bin/python 2.7.9 (default, Apr  8 2015, 15:12:41) 
[GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)]
================================================================================
Using temporary directory /tmp/duplicity-ny_w0E-tempdir
Registering (mkstemp) temporary file /tmp/duplicity-ny_w0E-tempdir/mkstemp-hiD03y-1
Temp has 13190230286848 available, backup will use approx 34078720.
ssh: [chan 1] listdir('/.')
Local and Remote metadata are synchronized, no sync needed.
ssh: [chan 1] listdir('/.')
1 file exists on backend
2 files exist in cache
Extracting backup chains from list of files: [u'.ssh']
File .ssh is not part of a known set; creating new set
Ignoring file (rejected by backup set) '.ssh'
Last full backup date: none
Collection Status
-----------------
Connecting with backend: BackendWrapper
Archive dir: /var/db/duplicity/be3234cf4cbafbdf41099030093b6bbe

Found 0 secondary backup chains.
No backup chains with active signatures found
No orphaned or incomplete backup sets found.
No signatures found, switching to full backup.
Using temporary directory /var/db/duplicity/be3234cf4cbafbdf41099030093b6bbe/duplicity-uGRFgC-tempdir
Registering (mktemp) temporary file /var/db/duplicity/be3234cf4cbafbdf41099030093b6bbe/duplicity-uGRFgC-tempdir/mktemp-v8RrI4-1
Using temporary directory /var/db/duplicity/be3234cf4cbafbdf41099030093b6bbe/duplicity-Ek5eEY-tempdir
Registering (mktemp) temporary file /var/db/duplicity/be3234cf4cbafbdf41099030093b6bbe/duplicity-Ek5eEY-tempdir/mktemp-gWTVYe-1
AsyncScheduler: instantiating at concurrency 0
Registering (mktemp) temporary file /tmp/duplicity-ny_w0E-tempdir/mktemp-cOWPt7-2
Selecting /
Comparing . and None
Getting delta of (. dir) and None
A .
Selecting /XXXX
Comparing XXXX and None
Getting delta of (XXX dir) and None
A XXX

…. some 350k lines snipped ….

Removing still remembered temporary file /var/db/duplicity/be3234cf4cbafbdf41099030093b6bbe/duplicity-uGRFgC-tempdir/mktemp-v8RrI4-1
Removing still remembered temporary file /var/db/duplicity/be3234cf4cbafbdf41099030093b6bbe/duplicity-Ek5eEY-tempdir/mktemp-gWTVYe-1
AsyncScheduler: running task synchronously (asynchronicity disabled)
Writing duplicity-full.20150503T112752Z.vol1.difftar.gpg
ssh: [chan 1] open('/duplicity-full.20150503T112752Z.vol1.difftar.gpg', 'wb')
ssh: [chan 1] open('/duplicity-full.20150503T112752Z.vol1.difftar.gpg', 'wb') -> 61623031376465316137393162356235
ssh: Sending global request "keepalive <at> lag.net"
ssh: Sending global request "keepalive <at> lag.net"
ssh: Sending global request "keepalive <at> lag.net"
ssh: Sending global request "keepalive <at> lag.net”
… repeated few 100 times ...
ssh: Sending global request "keepalive <at> lag.net"
ssh: Sending global request "keepalive <at> lag.net"
ssh: Sending global request "keepalive <at> lag.net"
ssh: Disconnect (code 11): data stalled timeout reached
ssh: [chan 1] close(61623031376465316137393162356235)
Backtrace of previous error: Traceback (innermost last):
  File "/usr/local/lib/python2.7/site-packages/duplicity/backend.py", line 365, in inner_retry
    return fn(self, *args)
  File "/usr/local/lib/python2.7/site-packages/duplicity/backend.py", line 515, in put
    self.__do_put(source_path, remote_filename)
  File "/usr/local/lib/python2.7/site-packages/duplicity/backend.py", line 501, in __do_put
    self.backend._put(source_path, remote_filename)
  File "/usr/local/lib/python2.7/site-packages/duplicity/backends/ssh_paramiko_backend.py",
line 288, in _put
    self.sftp.put(source_path.name, remote_filename)
  File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 587, in put
    return self.putfo(fl, remotepath, os.stat(localpath).st_size, callback, confirm)
  File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 546, in putfo
    fr.write(data)
  File "/usr/local/lib/python2.7/site-packages/paramiko/file.py", line 330, in write
    self._write_all(data)
  File "/usr/local/lib/python2.7/site-packages/paramiko/file.py", line 447, in _write_all
    count = self._write(data)
  File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_file.py", line 176, in _write
    self._reqs.append(self.sftp._async_request(type(None), CMD_WRITE, self.handle,
long(self._realpos), data[:chunk]))
  File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 670, in _async_request
    self._send_packet(t, msg)
  File "/usr/local/lib/python2.7/site-packages/paramiko/sftp.py", line 170, in _send_packet
    self._write_all(out)
  File "/usr/local/lib/python2.7/site-packages/paramiko/sftp.py", line 135, in _write_all
    raise EOFError()
 EOFError

Attempt 1 failed. EOFError: 
Writing duplicity-full.20150503T112752Z.vol1.difftar.gpg
ssh: [chan 1] open('/duplicity-full.20150503T112752Z.vol1.difftar.gpg', 'wb')
Backtrace of previous error: Traceback (innermost last):
  File "/usr/local/lib/python2.7/site-packages/duplicity/backend.py", line 365, in inner_retry
    return fn(self, *args)
  File "/usr/local/lib/python2.7/site-packages/duplicity/backend.py", line 515, in put
    self.__do_put(source_path, remote_filename)
  File "/usr/local/lib/python2.7/site-packages/duplicity/backend.py", line 501, in __do_put
    self.backend._put(source_path, remote_filename)
  File "/usr/local/lib/python2.7/site-packages/duplicity/backends/ssh_paramiko_backend.py",
line 288, in _put
    self.sftp.put(source_path.name, remote_filename)
  File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 587, in put
    return self.putfo(fl, remotepath, os.stat(localpath).st_size, callback, confirm)
  File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 541, in putfo
    with self.file(remotepath, 'wb') as fr:
  File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 245, in open
    t, msg = self._request(CMD_OPEN, filename, imode, attrblock)
  File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 648, in _request
    num = self._async_request(type(None), t, *arg)
  File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 670, in _async_request
    self._send_packet(t, msg)
  File "/usr/local/lib/python2.7/site-packages/paramiko/sftp.py", line 170, in _send_packet
    self._write_all(out)
  File "/usr/local/lib/python2.7/site-packages/paramiko/sftp.py", line 135, in _write_all
    raise EOFError()
 EOFError

Attempt 2 failed. EOFError: 
Writing duplicity-full.20150503T112752Z.vol1.difftar.gpg
ssh: [chan 1] open('/duplicity-full.20150503T112752Z.vol1.difftar.gpg', 'wb')
Backtrace of previous error: Traceback (innermost last):
  File "/usr/local/lib/python2.7/site-packages/duplicity/backend.py", line 365, in inner_retry
    return fn(self, *args)
  File "/usr/local/lib/python2.7/site-packages/duplicity/backend.py", line 515, in put
    self.__do_put(source_path, remote_filename)
  File "/usr/local/lib/python2.7/site-packages/duplicity/backend.py", line 501, in __do_put
    self.backend._put(source_path, remote_filename)
  File "/usr/local/lib/python2.7/site-packages/duplicity/backends/ssh_paramiko_backend.py",
line 288, in _put
    self.sftp.put(source_path.name, remote_filename)
  File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 587, in put
    return self.putfo(fl, remotepath, os.stat(localpath).st_size, callback, confirm)
  File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 541, in putfo
    with self.file(remotepath, 'wb') as fr:
  File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 245, in open
    t, msg = self._request(CMD_OPEN, filename, imode, attrblock)
  File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 648, in _request
    num = self._async_request(type(None), t, *arg)
  File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 670, in _async_request
    self._send_packet(t, msg)
  File "/usr/local/lib/python2.7/site-packages/paramiko/sftp.py", line 170, in _send_packet
    self._write_all(out)
  File "/usr/local/lib/python2.7/site-packages/paramiko/sftp.py", line 135, in _write_all
    raise EOFError()
 EOFError

Attempt 3 failed. EOFError: 
Writing duplicity-full.20150503T112752Z.vol1.difftar.gpg
ssh: [chan 1] open('/duplicity-full.20150503T112752Z.vol1.difftar.gpg', 'wb')
Backtrace of previous error: Traceback (innermost last):
  File "/usr/local/lib/python2.7/site-packages/duplicity/backend.py", line 365, in inner_retry
    return fn(self, *args)
  File "/usr/local/lib/python2.7/site-packages/duplicity/backend.py", line 515, in put
    self.__do_put(source_path, remote_filename)
  File "/usr/local/lib/python2.7/site-packages/duplicity/backend.py", line 501, in __do_put
    self.backend._put(source_path, remote_filename)
  File "/usr/local/lib/python2.7/site-packages/duplicity/backends/ssh_paramiko_backend.py",
line 288, in _put
    self.sftp.put(source_path.name, remote_filename)
  File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 587, in put
    return self.putfo(fl, remotepath, os.stat(localpath).st_size, callback, confirm)
  File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 541, in putfo
    with self.file(remotepath, 'wb') as fr:
  File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 245, in open
    t, msg = self._request(CMD_OPEN, filename, imode, attrblock)
  File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 648, in _request
    num = self._async_request(type(None), t, *arg)
  File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 670, in _async_request
    self._send_packet(t, msg)
  File "/usr/local/lib/python2.7/site-packages/paramiko/sftp.py", line 170, in _send_packet
    self._write_all(out)
  File "/usr/local/lib/python2.7/site-packages/paramiko/sftp.py", line 135, in _write_all
    raise EOFError()
 EOFError

Attempt 4 failed. EOFError: 
Writing duplicity-full.20150503T112752Z.vol1.difftar.gpg
ssh: [chan 1] open('/duplicity-full.20150503T112752Z.vol1.difftar.gpg', 'wb')
Backtrace of previous error: Traceback (innermost last):
  File "/usr/local/lib/python2.7/site-packages/duplicity/backend.py", line 365, in inner_retry
    return fn(self, *args)
  File "/usr/local/lib/python2.7/site-packages/duplicity/backend.py", line 515, in put
    self.__do_put(source_path, remote_filename)
  File "/usr/local/lib/python2.7/site-packages/duplicity/backend.py", line 501, in __do_put
    self.backend._put(source_path, remote_filename)
  File "/usr/local/lib/python2.7/site-packages/duplicity/backends/ssh_paramiko_backend.py",
line 288, in _put
    self.sftp.put(source_path.name, remote_filename)
  File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 587, in put
    return self.putfo(fl, remotepath, os.stat(localpath).st_size, callback, confirm)
  File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 541, in putfo
    with self.file(remotepath, 'wb') as fr:
  File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 245, in open
    t, msg = self._request(CMD_OPEN, filename, imode, attrblock)
  File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 648, in _request
    num = self._async_request(type(None), t, *arg)
  File "/usr/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 670, in _async_request
    self._send_packet(t, msg)
  File "/usr/local/lib/python2.7/site-packages/paramiko/sftp.py", line 170, in _send_packet
    self._write_all(out)
  File "/usr/local/lib/python2.7/site-packages/paramiko/sftp.py", line 135, in _write_all
    raise EOFError()
 EOFError

Giving up after 5 attempts. EOFError: 
Releasing lockfile <LinkLockFile:
'/var/db/duplicity/be3234cf4cbafbdf41099030093b6bbe/XXXXXXg.650626177224228893873130' -- '/var/db/duplicity/be3234cf4cbafbdf41099030093b6bbe/lockfile'>
Removing still remembered temporary file /tmp/duplicity-ny_w0E-tempdir/mktemp-cOWPt7-2
Removing still remembered temporary file /tmp/duplicity-ny_w0E-tempdir/mkstemp-hiD03y-1

_______________________________________________
Duplicity-talk mailing list
Duplicity-talk <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Ken Bass | 30 Apr 23:51 2015

librsync failure

I updated my Centos 7 server yesterday which updated the system librsync 
and librsync-devel packages to librsync-1.0.0-1.el7.x86_64 and 
librsync-devel-1.0.0-1.el7.x86_64.
I noticed since that time duplicity no longer runs.

Traceback (most recent call last):
   File "/usr/local/bin/duplicity", line 41, in <module>
     from duplicity import collections
   File "/usr/local/lib64/python2.7/site-packages/duplicity/collections.py", line 30, in <module>
     from duplicity import path
   File "/usr/local/lib64/python2.7/site-packages/duplicity/path.py", line 36, in <module>
     from duplicity import librsync
   File "/usr/local/lib64/python2.7/site-packages/duplicity/librsync.py", line 29, in <module>
     import _librsync
ImportError: librsync.so.1: cannot open shared object file: No such file or directory

I am running 0.6.24, but just to be sure, I downloaded the 0.6.25 tarball. And when I try to compile it:

building 'duplicity._librsync' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/duplicity
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall 
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong 
--param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic 
-D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall 
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong 
--param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic 
-D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c 
duplicity/_librsyncmodule.c -o 
build/temp.linux-x86_64-2.7/duplicity/_librsyncmodule.o
duplicity/_librsyncmodule.c: In function ‘_librsync_new_sigmaker’:
duplicity/_librsyncmodule.c:71:38: error: ‘RS_DEFAULT_STRONG_LEN’ 
undeclared (first use in this function)
                               (size_t)RS_DEFAULT_STRONG_LEN);
                                       ^
duplicity/_librsyncmodule.c:71:38: note: each undeclared identifier is 
reported only once for each function it appears in
duplicity/_librsyncmodule.c:71:30: error: too few arguments to function 
‘rs_sig_begin’
                               (size_t)RS_DEFAULT_STRONG_LEN);
                               ^
In file included from duplicity/_librsyncmodule.c:26:0:
/usr/include/librsync.h:370:11: note: declared here
  rs_job_t *rs_sig_begin(size_t new_block_len,
            ^
error: command 'gcc' failed with exit status 1

I do notice in the Changelog that v0.7.02 (2015/03/10) lists "Fix 
_librsyncmodule.c compilation, bug 1416344, thanks to Kari Hautio."
So this means I am forced to upgrade to 0.7.02? The website said 0.6 
series was being maintained and I got the impression that 0.7 was a 
development version not ready for production. Is 0.7 safe to use? If 
not, is the librsync fix going to be applied to the 0.6 series.

_______________________________________________
Duplicity-talk mailing list
Duplicity-talk <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
E.B. | 24 Apr 06:51 2015
Picon

Incremental backups added onto a segfaulted full backup

Hello,

A full backup process had a segfault many days back. The process left
behind a lock file but after the lockfile was deleted, subsequent incrementals
have succeeded normally. Is data in this backup chain reliable? What is best
course of action? Delete and do a full asap?
Christian Saga | 23 Apr 18:13 2015
Picon

Duplicity and put only backend permissions

Hi there,
I was just thinking about a new setup. Basically allowing duplicity to only write new objects to S3 and not allow deletion of old items.
This would allow me to have the backups secured, in case the server gets hijacked.

Do you know if this would pose problems with duplicity? I feel, it is only writing new objects for full and incremental backups to s3. Only the purge would delete old versions? If I would turn the purge off, then it should work?
Or is there any other action, that would need the possibility to change files in the backend instead of just uploading?

Regards
  Christian
_______________________________________________
Duplicity-talk mailing list
Duplicity-talk <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk
Kenneth Loafman | 23 Apr 02:12 2015

Re: When modifying a file under the backup dir, duplicity still thinks that there are no differences with the backup (stored on GDrive)

The signatures in the sigtar files.  Think of them as uber-large hashes.

On Wed, Apr 22, 2015 at 5:25 PM, Kostas Papadopoulos <kpapad75 <at> travelguide.gr> wrote:
On 4/22/2015 2:51 PM, Kenneth Loafman wrote:
Actually, he has to use --compare-data for verify to look at the local data, otherwise verify just validates that the backup is intact and is not corrupted.


Thanks for the clarification, but then isn't the output of "verify" a bit misleading?
...
Last full backup date: Sun Apr  5 15:55:20 2015

Verify complete: 73 files compared, 0 differences found.
--- Finished state OK at 01:33:47.897 - Runtime 00:01:20.821 ---
Against what are those 73 files compared?

Br, KP


_______________________________________________
Duplicity-talk mailing list
Duplicity-talk <at> nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

Gmane