Дугин Сергей | 28 Feb 04:41 2015
Picon

rsync slow exclude folder

Hi, Rsync.

rsync  version 3.0.9  protocol version 30

rsync -a --exclude="tmp/*" /home/ /backup/home/

It is necessary that the contents of the folder tmp copy, but the folder tmp in
backup was created.

If the folder /home/tmp/ is many millions of files, rsync
think of this folder can be seen through the lsof -p PID

If you do so: rsync -a --exclude="tmp/" /home/ /backup/home/
then backup is done very quickly, but tmp folder in the backup does not
will be created.

How to make so that rsync long thought over such folders, but
This created a backup of them empty?

======================================================================

Нужно  чтобы содержимое папки tmp не копировалось, но
сама папка tmp в
бекапе создавалась.

Если   в  папке  /home/tmp/  находится  много  миллионов файлов, rsync
думает над этой папкой это видно через lsof -p PID

Если сдлать так:  rsync -a --exclude="tmp/" /home/ /backup/home/
то  бекап  делается  очень  быстро,  но при этом папка tmp в
(Continue reading)

thomas veymont | 27 Feb 15:46 2015
Picon

rsync hangs on select() system call

hello,

trying to understand why a rsync client hangs during a transfer. Hopefully someone can advise.

client =  rsync 3.0.6, server = rsync 3.1.1 , transfer is done to a rsync module with such a command:
the client starts to transfer data (a few Mb) then hangs, then manually interrupted (CTRL+C)
a few minutes after. strace output on the client side is:

(...)
4251 17:14:30 select(4, NULL, [3], [3], {60, 0}) = 1 (out [3], left {59, 999998})
14251 17:14:30 write(3, "\374\17\0\7", 4) = 4
14251 17:14:30 select(4, NULL, [3], [3], {60, 0}) = 1 (out [3], left {59, 999998})
14251 17:14:30 write(3, "\337\364(\374\3567\t\361=\373\25 <at> \0'+\00 (...)
14251 17:14:30 select(4, NULL, [3], [3], {60, 0}) = 0 (Timeout)
14251 17:15:30 select(4, NULL, [3], [3], {60, 0}) = 0 (Timeout)
14251 17:16:31 select(4, NULL, [3], [3], {60, 0}) = 0 (Timeout)
14251 17:17:31 select(4, NULL, [3], [3], {60, 0}) = 0 (Timeout)
14251 17:18:31 select(4, NULL, [3], [3], {60, 0}) = 0 (Timeout)
14251 17:19:31 select(4, NULL, [3], [3], {60, 0}) = 0 (Timeout)
14251 17:20:31 select(4, NULL, [3], [3], {60, 0}) = 0 (Timeout)
14251 17:21:31 select(4, NULL, [3], [3], {60, 0}) = 0 (Timeout)
14251 17:22:31 select(4, NULL, [3], [3], {60, 0}) = 0 (Timeout)
14251 17:23:31 select(4, NULL, [3], [3], {60, 0}) = 0 (Timeout)
14251 17:24:31 select(4, NULL, [3], [3], {60, 0}) = 0 (Timeout)
14251 17:25:31 select(4, NULL, [3], [3], {60, 0}) = 0 (Timeout)
14251 17:26:31 select(4, NULL, [3], [3], {60, 0}) = 0 (Timeout)
14251 17:27:31 select(4, NULL, [3], [3], {60, 0}) = ? ERESTARTNOHAND (To be restarted)
14251 17:28:10 --- SIGINT (Interrupt) <at> 0 (0) ---
4251 17:28:10 select(0, NULL, NULL, NULL, {0, 400000}) = 0 (Timeout)
14251 17:28:10 rt_sigaction(SIGUSR1, {SIG_IGN, [], SA_RESTORER, 0x371a432920}, NULL, 8) = 0
14251 17:28:10 rt_sigaction(SIGUSR2, {SIG_IGN, [], SA_RESTORER, 0x371a432920}, NULL, 8) = 0
14251 17:28:10 write(2, "rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(544) [sender=3.0.6]", 89) = 89
14251 17:28:10 write(2, "\n", 1)        = 1
14251 17:28:10 write(1, "_exit_cleanup(code=20, file=rsync.c, line=544): about to call exit(20)\n", 71) = 71
14251 17:28:10 exit_group(20)           = ?

Send-Q ont the client side looks like:

Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name
tcp       41  62844 132.xxx.xxx.xx:41397        152.xx.xx.xx:873           ESTABLISHED 3387/rsync

On the server side, rsync daemon log says :

Feb 25 17:14:25 resif-vm13 rsyncd[2403]: [generator] receiving flist for dir 10
Feb 25 17:14:25 resif-vm13 rsyncd[2403]: recv_file_name(sds_extract/2015/RD/LOR/BHN.D/RD.LOR..BHN.D.2015.055)
Feb 25 17:14:25 resif-vm13 rsyncd[2403]: received 1 names
Feb 25 17:14:25 resif-vm13 rsyncd[2403]: recv_file_list done
Feb 25 17:14:25 resif-vm13 rsyncd[2403]: recv_generator(sds_extract/2015/RD/LOR/BHN.D,19)
Feb 25 17:14:25 resif-vm13 rsyncd[2403]: set modtime of sds_extract/2015/RD/LOR/BHN.D to (1424884220) Wed Feb 25 17:10:20 2015
Feb 25 17:14:25 resif-vm13 rsyncd[2403]: sds_extract/2015/RD/LOR/BHN.D/
Feb 25 17:14:25 resif-vm13 rsyncd[2403]: recv_generator(sds_extract/2015/RD/LOR/BHN.D/RD.LOR..BHN.D.2015.055,20)
Feb 25 17:14:25 resif-vm13 rsyncd[2403]: [generator] receiving flist for dir 11
Feb 25 17:14:25 resif-vm13 rsyncd[2403]: recv_file_name(sds_extract/2015/RD/LOR/BHZ.D/RD.LOR..BHZ.D.2015.055)
Feb 25 17:35:07 resif-vm13 rsyncd[2403]: rsync: connection unexpectedly closed (12005 bytes received so far) [generator]
Feb 25 17:35:07 resif-vm13 rsyncd[2403]: rsync error: error in rsync protocol data stream (code 12) at io.c(226) [generator=3.1.1]
Feb 25 17:35:07 resif-vm13 rsyncd[2403]: [generator] _exit_cleanup(code=12, file=io.c, line=226): about to call exit(12)

please note : clocks on server and client are maybe not exactly the same. I don't have easy console access to client (remote customer's).
Does someone have an idea of why it hangs ? Seems like the client side is hanging on a select() call, which would mean
that one the file selected for transfer is locked for reading (true/false?) ?

any suggestion welcome
Tom
--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Peng Yu | 21 Feb 23:54 2015
Picon

How to figure out what subdirectories are available on an rsync server?

Hi,

http://cran.r-project.org/mirror-howto.html

The above webpage shows the following rsync command to sync the website.

rsync -rtlzv --delete cran.r-project.org::CRAN /dir/on/local/disc

But if I want to sync only a subdirectory of what are available in
ftp://cran.r-project.org/pub/R/, does anybody know how to do it?

Where is "CRAN" from (I don't see "CRAN" in the ftp site)? Is there a
way to check the directory structure of a rsync server? Thanks.

-- 
Regards,
Peng
--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

samba-bugs | 20 Feb 21:45 2015
Picon

[Bug 11111] New: Describe the interaction between --one-file-system and --delete-excluded

https://bugzilla.samba.org/show_bug.cgi?id=11111

            Bug ID: 11111
           Summary: Describe the interaction between --one-file-system and
                     --delete-excluded
           Product: rsync
           Version: 3.1.1
          Hardware: All
                OS: All
            Status: NEW
          Severity: enhancement
          Priority: P5
         Component: core
          Assignee: wayned <at> samba.org
          Reporter: bill+samba <at> bfccomputing.com
        QA Contact: rsync-qa <at> samba.org

Created attachment 10762
  --> https://bugzilla.samba.org/attachment.cgi?id=10762&action=edit
rsync.yo patch

for:

  rsync --one-file-system --delete-excluded --exclude=/bar/ host:/bar/ /foo

where /bar is a mounted filesystem, 

document that ignoring a sender's "other" filesystem does not affect the
operation of --delete-excluded at the matching receiver's patch (/foo/bar).

I ran into this under a pile of scripts where the backups first backed up /foo,
then /foo/bar, then /foo/baz, and the underlying volume's CoW snapshots were
quickly gaining size because /foo was listed in the excludes (first deleted,
then re-populated).

The current documentation isn't wrong - just being explicit would help
eliminate this as a gotcha.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Claus, Andrew | 18 Feb 21:10 2015

Preserve ACLs Changing Group. File Cannot Be Opened on Destination

When using the following to keep duplicate files on multiple computers, I find some files have their ACLs changed and cannot then be opened.

I am using the following command. My goal is to be able to read/write/edit any local or remote file from any machine. Until recently this worked just fine, but now some files cannot be opened once rsync'd.

rsync -avzh --progress --delete --super -A --exclude='~$*'  '/cygdrive/d/claus.a' /cygdrive/x

D is the drive on the remote computer I am currently using, X is its mirror on another remote computer.

When RSYNC'ing I get the following for the files that will not open, after it shows the file transfers:
rsync: set_acl: sys_acl_set_file(claus.a/.rsync commands.txt.1BY2xz, ACL_TYPE_ACCESS): Invalid argument (22)
rsync: set_acl: sys_acl_set_file(claus.a/.test.txt.CA1No2, ACL_TYPE_ACCESS): Invalid argument (22)


On the source machine, the ACLs for files that open and don't open once RSYNC'd are identical. 
# owner: claus.a (or Administrators in some cases, but files with both owners work & fail)
# group: Domain Usersge
user::---
group::---
group:root:rwx
group:Authenticated Users:rwx
group:SYSTEM:rwx
group:Users:rwx
mask:rwx
other:---

However, once RYSNC'd the failing files get "group::rwx". This seems to cause the failure to open.
All files (work or fail) lose the entire "group:root:rwx" ACL, but this seems to have no impact.

Removing the new ACL with setfacl -m g::--- solves my problem, but this is not a long term solution.

Thanks,
Andrew
--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
samba-bugs | 18 Feb 03:04 2015
Picon

[Bug 11101] New: support writing to devices

https://bugzilla.samba.org/show_bug.cgi?id=11101

            Bug ID: 11101
           Summary: support writing to devices
           Product: rsync
           Version: 3.0.6
          Hardware: All
                OS: All
            Status: NEW
          Severity: normal
          Priority: P5
         Component: core
          Assignee: wayned <at> samba.org
          Reporter: jpyeron <at> pdinc.us
        QA Contact: rsync-qa <at> samba.org

see https://bugzilla.redhat.com/show_bug.cgi?id=1193654

when rsyncing to a device, it is important to not delete the device node.

patch on redhat bugzilla.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Tiziano Müller | 17 Feb 15:58 2015
Picon

MinGW support

Hi there

As announced (a lot) earlier [1], we've been working on getting rsync to run on
Windows (using MinGW).

The first results can be found here as 3 different branches:

  https://github.com/stepping-stone/rsync

  https://github.com/stepping-stone/rsync/commits/automake
  https://github.com/stepping-stone/rsync/commits/gnulib
  https://github.com/stepping-stone/rsync/commits/win32

  where win32 -> gnulib -> automake -> master

The "automake" branch is quiet ready and could be pulled up to
a0523a2408f9095f0dac9fe6b204912829f3917e (Use AS_IF instead of plain if/then/fi)
since those are basic fixes/updates to the current build system.
Only the two last commits then introduce automake. I can also send them as
single patches if preferred.

The "gnulib" branch is currently just one big commit which needs to be split
before being ready. Besides, this one needs some testing on exotic platforms
since it changes/updates a lot of compat code.

The "win32" is currently basically a collection of "#ifdef WIN32".

Best regards,
Tiziano

[1] https://lists.samba.org/archive/rsync/2013-September/028701.html

-- 
stepping stone GmbH
Neufeldstrasse 9
CH-3012 Bern

Telefon: +41 31 332 53 63
www.stepping-stone.ch
tiziano.mueller <at>  <at> stepping-stone.ch
--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

QUBE RUBBIK | 11 Feb 15:03 2015
Picon

great feature idea (well, hopefully)

Hello

I was just thinking about a killer feature for rsync, the ability to detect files name changes or move within the source and destination.
At this time rsync has to re-transfer a file if it has been renamed or moved inside a subfolder, with a heavy waste of ressources and bandwidth.

It could be smarter :
with a --smart switch, rsync could take a hash of every file within the source and destination BEFORE TRANSFERING,
then for existing (matching hash) files, it only needs to alter metadata (name, location, chmod etc...) saving plenty of bandwidth

Okay destination has to handle this, I expect the rsync daemon has to handle server side file hashing.

We would have a clever tool to replicate data who only been reorganised with no changes on the files themselves.
No need to resync the whole structure if you added a dir in the path, or someone renamed this particular heavy file

this may save big data on automatic backups, ftp mirrors etc...


What do you think about it?

--smart ?
--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Johan Kröckel | 11 Feb 14:14 2015
Picon

reflink

Will this make it to rsync and in case when?
--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Andrew Hamer-Adams | 3 Feb 03:30 2015
Picon

Preventing an atomic copy

We require an option that forces rsync to truncate the destination file to zero bytes, using the existing inode, before copying (non-atomic).

For various important reasons, we cannot use —inplace as an option due to the way the filesystem we use handles a retrieve from tape.

Using —temp-dir as an option does provide us with the functionality we require, however it has obvious downsides of doubling I/O for no good reason.

Am I missing something obvious? Or should I add this as a feature request?

Cheers
--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
samba-bugs | 31 Jan 15:32 2015
Picon

[Bug 11086] New: rsync --delete has stopped to delete destination files that don't exist at the source

https://bugzilla.samba.org/show_bug.cgi?id=11086

            Bug ID: 11086
           Summary: rsync --delete has stopped to delete destination files
                    that don't exist at the source
           Product: rsync
           Version: 3.1.1
          Hardware: x64
                OS: Linux
            Status: NEW
          Severity: major
          Priority: P5
         Component: core
          Assignee: wayned <at> samba.org
          Reporter: luminarior <at> gmail.com
        QA Contact: rsync-qa <at> samba.org

I have a system Linux root partition wich I backup to another disk
periodically. I use the following script for that action:

#!/bin/bash
NICE=15
PH="/mnt/b"
cd /

nice -n $NICE rm -v /root/.xsession-errors
nice -n $NICE rm -v /root/dead.letter

nice -n $NICE /usr/bin/rsync -avhHxq --progress --stats --force --delete
--delete-before --log-file='/var/log/backup2diskrsync.log'
--exclude='/usr/portage/distfiles/' --exclude='/home/ftp/'
--exclude='/lost+found/' --exclude='/var/tmp/ccache/'
--exclude='/root/.ccache/' --exclude='/var/cache/edb/'
--exclude='/home/myuser/.cache/chromium/'
--exclude='/home/myuser/.opera/cache/' --exclude='/home/myuser/.opera/opcache/'
--exclude='/home/mysearch/search/fsearch/swapfile'
--exclude='/home/myuser/workspace/mysearch/swapfile' / $PH/

The problem is that I don't see any "delete" messages in log file anymore. For
example, I have the following in /usr/src/ directory on backup disk:
# du -sh /mnt/b/usr/src/*
0       /mnt/b/usr/src/linux
944M    /mnt/b/usr/src/linux-3.14.14-gentoo
961M    /mnt/b/usr/src/linux-3.16.5-gentoo
964M    /mnt/b/usr/src/linux-3.17.7-gentoo
972M    /mnt/b/usr/src/linux-3.18.3-gentoo
971M    /mnt/b/usr/src/linux-3.18.5-gentoo
But I don't have this outdated long time ago directories on my root partition:
# du -sh /usr/src/*
0       /usr/src/linux
943M    /usr/src/linux-3.18.5-gentoo

And that's just a one of examples of outdated files and directories. My backup
storage partition will become full soon.

The rsync log (after # /bin/bash backup2diskwithrsync.sh under root):
2015/01/31 17:25:56 [12091] building file list
2015/01/31 17:25:58 [12091] rsync: readlink_stat("/home/myuser/.gvfs") failed:
Permission denied (13)
2015/01/31 17:25:59 [12091] 877346 files to consider
2015/01/31 17:26:01 [12091] >f..t...... etc/openvpn/openvpn-status.log
2015/01/31 17:26:01 [12091] >f..t...... home/incoming/scripts/enp2s0-upd.err
2015/01/31 17:26:01 [12091] >f..t...... home/incoming/scripts/enp2s0-upd.info
2015/01/31 17:26:01 [12091] >f..t...... home/incoming/scripts/ppp0-upd.err
2015/01/31 17:26:01 [12091] >f..t...... home/incoming/scripts/ppp0-upd.info
2015/01/31 17:26:03 [12091] .d..t...... var/log/
2015/01/31 17:26:03 [12091] >f.st...... var/log/backup2diskrsync.log
2015/01/31 17:26:03 [12091] >f.st...... var/log/messages
2015/01/31 17:26:03 [12091] >f.st...... var/log/net-snmpd.log
2015/01/31 17:26:04 [12091] >f.st...... var/log/samba/log.nmbd
2015/01/31 17:26:05 [12091] >f..t...... var/tmp/kdecache-root/icon-cache.kcache
2015/01/31 17:26:05 [12091] >f..t......
var/tmp/kdecache-myuser/plasma_theme_G-Remix-T_v5.3.kcache
2015/01/31 17:26:06 [12091] >f..t......
var/tmp/kdecache-myuser/plasma_theme_internal-system-colors.kcache
2015/01/31 17:26:06 [12091] .d..t......
var/www/localhost/htdocs/rutorrent/share/users/myuserutorrent/settings/erasedata/
2015/01/31 17:26:07 [12091] Number of files: 877,346 (reg: 758,168, dir:
67,562, link: 45,877, dev: 5,678, special: 61)
2015/01/31 17:26:07 [12091] Number of created files: 0
2015/01/31 17:26:07 [12091] Number of deleted files: 0
2015/01/31 17:26:07 [12091] Number of regular files transferred: 12
2015/01/31 17:26:07 [12091] Total file size: 25.31G bytes
2015/01/31 17:26:07 [12091] Total transferred file size: 379.83M bytes
2015/01/31 17:26:07 [12091] Literal data: 379.83M bytes
2015/01/31 17:26:07 [12091] Matched data: 0 bytes
2015/01/31 17:26:07 [12091] File list size: 24.44M
2015/01/31 17:26:07 [12091] File list generation time: 3.099 seconds
2015/01/31 17:26:07 [12091] File list transfer time: 0.000 seconds
2015/01/31 17:26:07 [12091] Total bytes sent: 404.40M
2015/01/31 17:26:07 [12091] Total bytes received: 291
2015/01/31 17:26:07 [12091] sent 404.40M bytes  received 291 bytes  35.17M
bytes/sec
2015/01/31 17:26:07 [12091] total size is 25.31G  speedup is 62.60
2015/01/31 17:26:07 [12091] rsync error: some files/attrs were not transferred
(see previous errors) (code 23) at main.c(1165) [sender=3.1.1]

Error log in the console (some files it doesn't like probably)
rsync: readlink_stat("/home/xor/.gvfs") failed: Permission denied (13)
rsync error: some files/attrs were not transferred (see previous errors) (code
23) at main.c(1165) [sender=3.1.1]

And still there are a lot of garbage on backup partition wich had to be deleted
a looong time ago:
# du -sh /mnt/b/usr/src/*
0       /mnt/b/usr/src/linux
944M    /mnt/b/usr/src/linux-3.14.14-gentoo
961M    /mnt/b/usr/src/linux-3.16.5-gentoo
964M    /mnt/b/usr/src/linux-3.17.7-gentoo
972M    /mnt/b/usr/src/linux-3.18.3-gentoo
971M    /mnt/b/usr/src/linux-3.18.5-gentoo

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Gmane