headers
Torsten Bronger | 21 May 2013 10:20
X-Face
Picon
Picon
Picon
Picon
Favicon
Gravatar

rsync is too often CPU-bound

Hallöchen!

While rsync works perfectly if you have little bandwidth, it is
difficult to use in a fast trustworthy local network.  Firstly, you
have to switch off compression, encryption, and checksum
calculation.  This is easily possible, although it can be hard for
people to collect all options necessary for fast-network tuning.

However, the CPU performance I observed was devastating.
Transfering big files, the NFS server uses <10% CPU, while rsync
daemon uses >80%.  Why is this?

Really critical is this with an NAS with a slow CPU.  In my case, it
limits the transfer rate from 55 MByte/s (NFS) to 12 MByte/s
(rsync).

For me, it means that I use NFS for data transmission, and use rsync
only locally on the client computer with a much faster CPU.  But
this should not be necessary in my opinion.  I think that at least
for big files, CPU and bandwidth performance of rsync and NFS should
be the same.

Tschö,
Torsten.

-- 
Torsten Bronger    Jabber ID: torsten.bronger <at> jabber.rwth-aachen.de
                                  or http://bronger-jmp.appspot.com

--

--
(Continue reading)

Permalink | Reply | 0 comments |
headers
AZ 9901 | 20 May 2013 10:55
Picon

Timeout on deletion

Hello,

I use Rsync client (3.0.9) synchronising to a Rsync daemon (3.0.8) over SSH.

I have sometimes a huge amount of files to delete.
Rsync daemon can take a lot of time to delete them, leading Rsync client into a timeout…

Is there any way to avoid this timeout ?
I would have liked not to increase timeout value on client side (which is already set to 600 seconds), as I
don't really know how long the deletion will take.

Could it be possible for the client and the daemon to communicate with each other from time to time during the
deletion process ?
The client would then know that deletion is still in progress, this would avoid timeout !

Thank you very much,

Best regards,

Ben

--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Permalink | Reply | 0 comments |
headers
garvit sharma | 20 May 2013 06:53
Picon

IRC

Hello All,

                  Is there any irc group for rsync ?. If it does exist please let me know.

Thanks,

--
Regards

Garvit Sharma
Computer Science and Engineering
UG Third year
LNM IIT, Jaipur

No Body is a Scholar by birth, its only hard work and strong determination that makes him master. 

--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Permalink | Reply | 1 comment |
headers
garvit sharma | 20 May 2013 05:19
Picon

contribute to rsync

Hello All,

               Myself Garvit Sharma working in hyderabad central university(HCU) at DCIS. I have started using rsync to sync the data and i found it very interesting. After using rsync a lot i realized to contribute to rsync by adding some extra features into it. Every time for synchronization we need to do it manually by executing command on the command line but instead of doing it manually imagine if it does automatically sync whenever it find changes in the single file.
To do this we need to start a daemon on both the source and destination and these daemon will check the time stamp of last modification of the directory we want and if it finds the new time stamp then it will sync.

The above mentioned idea is my own and i have already started reading and understanding the source of rsync. Please give your comments, your comments matters a lot for me.

Thanks,

--
Regards

Garvit Sharma
Computer Science and Engineering
UG Third year
LNM IIT, Jaipur

No Body is a Scholar by birth, its only hard work and strong determination that makes him master. 

--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Permalink | Reply | 4 comments |
headers
Matthias Schniedermeyer | 20 May 2013 01:20
Picon

Any plans for a 3.1.0 release(-date)?

Hi

If i'm reading git right(git log v3.0.9..) the development branch that 
will eventually result in 3.1.0 is nearly 5 years "in the making".

Personally i'm anticipating the fallocate support, as XFS is great at 
preallocateing (fallocating even a several gigabyte big file only takes 
milliseconds) which reduces fragmentation especially when the copy 
operation takes time.

-- 

Matthias
--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Permalink | Reply | 0 comments |
headers
samba-bugs | 17 May 2013 05:50
Picon
Favicon

[Bug 9894] New: Rsync can silently zero out chunks in a file

https://bugzilla.samba.org/show_bug.cgi?id=9894

           Summary: Rsync can silently zero out chunks in a file
           Product: rsync
           Version: 3.0.9
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: critical
          Priority: P5
         Component: core
        AssignedTo: wayned <at> samba.org
        ReportedBy: an.m <at> outlook.com
         QAContact: rsync-qa <at> samba.org

map_ptr() handles read failures by zeroing out remaining chunk and returning
the partially filled/all zero buffers to callers. It also fills error details
in map_struct->status.

It is caller's responsibility to check the status and handle this condition
appropriately.

It is probably OK to ignore this when we are generating checksums (as it might
lead to a false mismatch, resulting in transfer of a non matching block)

But in context of transfer (send_deflated_token()) when this error is ignored
it leads to silent data corruption for receiver.

Possible solutions are to log errors and disconnect such connection, or retry
till timeout happens. 

No point in giving user a false impression that xfr is successful. Request dev
to look at it with priority.

Cheers,
Ankur

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.
--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Permalink | Reply | 2 comments |
headers
Carl Brewer | 16 May 2013 02:52
Picon

Don't follow bind mounts?


Hello,
The manual says that rsync treats bind mounts on UNIX (Linux) to the 
same filesystem as being on the same filesystem.

I have a server with a pile of bind mounts to the same filesystem for 
some access control/ease of use for FTP users modifying websites.  This 
makes my backups using rsync messy!

Is there any way to stop rsync from following bind mounts to the same 
filesystem?  Short of unmounting them all at backup time and remounting 
afterwards or explicitly excluding each one?  -x -x seemed hopeful, but 
it didn't stop following the mount and I ended up with two copies (or 
more!) of the data on my backup server.   Googling just takes me to the 
man page saying "no", in a roundabout sort of way. I found this :

http://samba.2283325.n4.nabble.com/PATCH-One-more-x-to-not-traverse-cyclic-bind-mounts-td3216379.html

Which never made it into rsync, it seems (didn't work?) and the script 
mnt-excl suggests that it is redundant for rsync >= 2.6.4.  any examples 
of use of :

awk '{print $2}' /proc/mounts | rsync -f 'merge,/- -' host:/dir /dest/

around with some details of its use for backups?

Thank you

Carl

--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Permalink | Reply | 3 comments |
headers
Brian K. White | 15 May 2013 10:13
Favicon
Gravatar

exclude a pattern but only in the top level

Consider the following directory structure

/foo/aaa/*/*
/foo/bbb/*/*
/foo/ccc/*/*

I want to sync all of /foo,
but exclude /foo/aaa
but not exclude any other occurances of "aaa" or "foo/aaa" (be they 
files or dirs) that might occur within the other dirs /foo/bbb/* 
/foo/ccc/* etc

I don't want to exclude /foo/bbb/aaa or /foo/ccc/111/aaa or 
/foo/ccc/111/foo/aaa etc...

Destination is running rsync daemon and has a module named root that 
points to "/" such that normally, for the full /foo with no exclude, 
it's very simple, my rsync command would just be:

rsync -avz /foo ${DEST}::root

That works fine without the exclude.

So, to that I want to add a filter file. (Well, I assume that's what I want)

rsync -avz -f ". filter" /foo ${DEST}::root

If I construct a filter file like this:
- aaa
+ *

or like this
- aaa/
+ *

or like this
- foo/aaa/
+ *

rsync hides the top level /foo/aaa but it also hides any other 
occurrences of the exclude pattern that occur anywhere within the job 
instead of just the top level one.

How can I get it to exclude just the top-level directory "/foo/aaa" ?

-- 
bkw
--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Permalink | Reply | 8 comments |
headers
Lokesh Walase | 15 May 2013 07:50
Picon

Re: Adding prompt before overwrite feature in rsync




On Wed, May 15, 2013 at 11:17 AM, Lokesh Walase <flokeshw24 <at> gmail.com> wrote:



On Sun, May 12, 2013 at 4:46 PM, Lokesh Walase <flokeshw24 <at> gmail.com> wrote:



On Sat, May 11, 2013 at 7:54 PM, Scott Wiersdorf <scott <at> ipartner.net> wrote:
On Sat, May 11, 2013 at 09:56:34AM +0530, Lokesh Walase wrote:
> In "cp" command, we have "-i" option, which prompts before overwriting.
> I read the man page of "rsync" and didnt find anything like that here.
>
> So I m interested in adding this feature to the "rsync" command.
> I have downloaded the source code of "rsync" but I m confused from where to
> start and what to read.

This could turn a large sync into an aborted nightmare; I would
recommend perhaps instead writing a shell script (for example) that
would first do a --dry-run with --itemize-changes, parse that and
present it to the user for confirmation if any file changes are
detected. If the confirmation is given, re-run everything without the
--dry-run, otherwise exit.

Here is a simple bash code that implements the suggestion of Scott.
https://github.com/lokeshw24/rsync_parser

Any suggestions are welcome. 

Thanks Scott for the suggestion, sounds cool :)
Will implement it  !!



Scott
--
Scott Wiersdorf
scott <at> ipartner.net
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html



--
Lokesh Walase



--
Lokesh Walase



--
Lokesh Walase
--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Permalink | Reply | 0 comments |
headers
Carl Brewer | 15 May 2013 02:19
Picon

Using rsync to just -resync permissions and ownership?


G'day,
I'm in the process of fixing a long broken rsync backup on a couple of 
servers.  The old system rsync'd a Linux (CentOS 5) server onto a 
Opensolaris/OpenIndiana server but didn't worry about ownership or 
permissions, across an expensive WAN link.  All the ownerships and 
permissions are different to the original box on the OI box.

I now have a virtual CentOS 5 box that I want to mostly mirror the 
master server onto to have as a warm backup.  I have all the files on 
the OpenIndiana box but with incorrect ownership and permissions.  It's 
around 120GB or so, so I don't want to pump it all down from the master 
server again if I can help it.  I have copied all the files etc off the 
OI box onto the virtual CentOS server, but with incorrect (ie: same as 
on the OI box, not the original CentOS box) ownerships and permissions.

Is there a way using rsync (or something else?) to just change 
ownerships and permissions and not actually copy files no matter what 
the modification time of the files is, on all the files being rsync'ed? 
  I had a poke through the manual and a bit of googling but didn't find 
anything that seemed right.

Thank you for any suggestions

Carl

--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Permalink | Reply | 7 comments |
headers
samba-bugs | 14 May 2013 10:57
Picon
Favicon

[Bug 9882] New: Incorrect exit code when sender over SSH is killed with SIGTERM

https://bugzilla.samba.org/show_bug.cgi?id=9882

           Summary: Incorrect exit code when sender over SSH is killed
                    with SIGTERM
           Product: rsync
           Version: 3.1.0
          Platform: All
        OS/Version: Linux
            Status: NEW
          Severity: normal
          Priority: P5
         Component: core
        AssignedTo: wayned <at> samba.org
        ReportedBy: famzah <at> famzah.net
         QAContact: rsync-qa <at> samba.org

For the tests I've compiled "rsync" from the latest GIT repository sources.
This happens only when SRC is an SSH remote location. Example:

famzah <at> vbox:~$ rsync famzah <at> 127.0.0.1:/boot/initrd.img-3.2.0-36-generic-pae
/tmp/test ; echo "Exit code: $?"
rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(610)
[sender=3.1.0dev]
Exit code: 0

Here is how the process list looks like:

famzah <at> vbox:~$ ps xfww -o pid,user,tty,command
  PID USER     TT       COMMAND
 5076 famzah   ?        sshd: famzah <at> notty  
 5077 famzah   ?         \_ rsync --server --sender -e.Lsf .
/boot/initrd.img-3.2.0-36-generic-pae
 ...
 2996 famzah   pts/1     \_ /bin/bash
 4960 famzah   pts/1     |   \_ rsync
famzah <at> 127.0.0.1:/boot/initrd.img-3.2.0-36-generic-pae /tmp/test
 4961 famzah   pts/1     |       \_ ssh -l famzah 127.0.0.1 rsync --server
--sender -e.Lsf . /boot/initrd.img-3.2.
 5078 famzah   pts/1     |       \_ rsync
famzah <at> 127.0.0.1:/boot/initrd.img-3.2.0-36-generic-pae /tmp/test

I have killed the "rsync" sender process at the _remote_ location with SIGTERM
(PID=5077 in the example above).

Rsync properly indicates that the problem occurred in [sender=3.1.0dev] and
also displays that exit code should be 20. But in the end, the rsync process
terminates with exit code 0 which is incorrect (Success).

While trying this a few times, I noticed that sometimes "rsync" terminates with
a proper exit code if the error occurs in the local "receiver" or "generator"
processes. I get "error in rsync protocol data stream (code 12) at io.c" then
and the final exit code of "rsync" is 12, as expected.

I also tried all other combinations:
  * sync between local SRC and local DST
  * sync between local SRC and remote DST over SSH
  * killing any of the "rsync" processes (receiver, generator, sender, or SSH
tunnel)

The only problem I got with the incorrect final exit code is when I transfer
with remote SRC over SSH and local DST, as demonstrated. Furthermore, the
"sender" needs to be killed with SIGTERM, which seems to be propagated back to
the "receiver" process by the rsync protocol. Killing the "sender" with SIGKILL
causes immediate termination and therefore an rsync protocol error at the side
of the "receiver", which exits properly with a non-zero exit code.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.
--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Permalink | Reply | 0 comments |

Gmane