samba-bugs | 4 May 11:04 2016
Picon

[Bug 11893] New: rsync should check local file access permission before connecting to remote end

https://bugzilla.samba.org/show_bug.cgi?id=11893

            Bug ID: 11893
           Summary: rsync should check local file access permission before
                    connecting to remote end
           Product: rsync
           Version: 3.1.2
          Hardware: All
                OS: All
            Status: NEW
          Severity: minor
          Priority: P5
         Component: core
          Assignee: wayned <at> samba.org
          Reporter: rami.lehti <at> gmail.com
        QA Contact: rsync-qa <at> samba.org

I'm using rsync to transfer large OS image files around and sometimes the local
permissions change so that the sender does not have access to the source file
any more.

Having previously transferred the file there is already a file of matching size
on the target host. Now attempting the transfer again rsync connects to the
remote end and the remote end begins to do its checks before the transfer
proper. Because the image file is large this will take a long time (10
minutes). 

After this rsync will report the "Permission denied" error for the local file. 

It would be more user-friendly to check the local file access permissions
(Continue reading)

Christoph Biedl | 3 May 22:07 2016
Picon
Picon

Yet another filter question

Hello,

Since the very first day I've been using rsync - some 15 years ago -
the filtering rules caused great grieve. Their behaviour is just not
the way I'd expect it be be and as I read the manpage. Usually I end
up with some hand-written recipes, carefully documented,y including all
the gotchas.

This time however I failed and I see no other way than to ask for
advice.

Given the following structure

        project/
            .rsync-filter
            project.git/
                .git/

Now, the following command (rsync 3.1.1)

    rsync -av --del --deleted-excluded -F \
        /path/to/project/ /path/to/backup/project/

should transfer

- sync: project/project.git/.git/
- skip: Everything else in project/project.git/
- sync: Everything else in project/

In other words: Don't transfer the git repo except for .git/ itself.
(Continue reading)

samba-bugs | 28 Apr 17:52 2016
Picon

[Bug 11879] New: escape rrsync restricted folder

https://bugzilla.samba.org/show_bug.cgi?id=11879

            Bug ID: 11879
           Summary: escape rrsync restricted folder
           Product: rsync
           Version: 3.1.2
          Hardware: All
                OS: All
            Status: NEW
          Severity: normal
          Priority: P5
         Component: core
          Assignee: wayned <at> samba.org
          Reporter: fb102email-sambabugzilla <at> yahoo.fr
        QA Contact: rsync-qa <at> samba.org

It is possible to escape rrsync restricted folder by syncing (using rsync -a
...) a symbolic link to the parent folder and then syncing with this symbolic
link.

Concretely, we could do:

ln -s .. parent
rsync -acrvz . login <at> server:

and then we can rsync with login <at> server:parent to read/write files in the
parent folder of the restricted folder.

--

-- 
You are receiving this mail because:
(Continue reading)

samba-bugs | 21 Apr 08:54 2016
Picon

[Bug 11866] New: rsync fails (failed to re-stat) when using double fuzzy + link-dest on renamed files

https://bugzilla.samba.org/show_bug.cgi?id=11866

            Bug ID: 11866
           Summary: rsync fails (failed to re-stat) when using double
                    fuzzy + link-dest on renamed files
           Product: rsync
           Version: 3.1.1
          Hardware: All
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P5
         Component: core
          Assignee: wayned <at> samba.org
          Reporter: mj <at> doze.net
        QA Contact: rsync-qa <at> samba.org

When using rsync with --fuzzy --fuzzy (double fuzzy) and --link-dest, rsync
fails if a file was renamed to a new name on the source that does not exist in
the link-dest.  As an example:

mj <at> backup-server:~/foo$ rm -rf .sync && mkdir .sync && rsync -azSHAXxrsyy
--ignore-existing --fake-super --link-dest=/home/mj/foo/current
--files-from=:/home/mj/files-to-backup root <at> backup-client:/ .sync
rsync: failed to re-stat "/home/mj/foo/.sync/etc/motd.old": No such file or
directory (2)
rsync: failed to re-stat "/home/mj/foo/.sync/var/log/alternatives.log.1": No
such file or directory (2)
rsync: failed to re-stat "/home/mj/foo/.sync/var/log/auth.log.3.gz": No such
file or directory (2)
(Continue reading)

Jethro Tull | 16 Apr 14:29 2016
Picon

addind imap support to rsync

when seeking for syncing two imap maildir folders one has the choice
among many programs such as imapsync, isync, offlineimap ... But all of
these provide individually only a few options.

For example syncing a folder by only appending unexisting files, i.e.
without deleting anything seem to be supported by no one real imap
syncing software stable release. Here "real syncing" means a process
that first collects the data it needs about the files in the destination
folder and proceeds with syncing with the requested options.

For example isync does indeed provide an option for only appending
unexisting files in the destination folder but despite its name it does
not seem to care about the content of the destination folder. The first
time it runs it copies all the emails from the source folder and writes
in a text file what it did to avoid duplicates for the next times it
will run. For syncing two folders that initially contain common stuff
this would eventually result in duplicate emails in the destination
folder.


Moreover each of these programs have their own configuration style that
takes a lot of time to learn. Regarding the many features rsync provides
for syncing files and its ease of use, adding to it the ability to sync
imap maildir folders will probably make of it a definitive choice for
this purpose.

--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Chris Deigan | 14 Apr 07:30 2016
Picon
Gravatar

"Total File Size" Statistic counts each instance of hard linked files

Hi,

This is a question is seeking clarification of intended behaviour. Right
now, rsync reports a statistic of "Total file size". This represents "the
total sum of all file sizes in the transfer" (as described in the man page).

A case I've hit in using this statistic is that it counts each instance of a
file even when it has multiple hard links. We are using --hard-links to
preserve hard links on the destination.

As a result we get a statistic of, for instance, 2TB when the actual sum on
disk (counted with du, using the default behaviour of counting hard linked
files only once) is only around 80GB.

I'm using the statistic for generating backup disk usage numbers that
eventually become billing data, so this has generated a few surprise cases.

There are a few alternatives for my use-case, but I was wondering if
counting hard links multiple times is actually correct behaviour?

My feeling is no, but this consideration isn't apparent in the source or
docs that I've read. Appreciate any comments, particularly from the project
maintainers.

Thanks,
Chris

--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Fabian Cenedese | 12 Apr 09:02 2016
Picon

Re: User controlled i/o block size?

At 01:33 12.04.2016, Greg Freemyer wrote:
>Content-Transfer-Encoding: 7bit
>
>I'm just doing a local copy:
>rsync -avp --progress <source_dir> <dest_dir>

Just as side information: In local copies all files are copied wholly,
the diff algorithm is not in effect. So if a file changes then it still is
copied completely (without --partial, --no-whole-file etc).

Second thing: From what I remember rsync does a lot of stat calls to
get every file's properties. This is more expensive on cygwin/Windows
than on linux directly. Rsync also uses processes/threads which are
easier/faster to create and switched to on linux than on Windows.

A Windows native implementation of rsync could run faster than the
original rsync with cygwin layer. Some time ago somebody announced
a new program using the rsync algorithm. But I never used it so I don't
know about the features or speed.

<http://www.acrosync.com/windows.html>http://www.acrosync.com/windows.html

bye  Fabi

--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Greg Freemyer | 11 Apr 22:08 2016
Picon
Gravatar

User controlled i/o block size?

I hope this isn't a FAQ.

Per the man page I see ways to control the blocksize for hash
comparison reasons, but no way to control it for i/o performance
reasons.

I'm using rsync to copy folder trees full of large files and I'd like
to have control of how much data is read / written at a time.  Maybe
read 10 MB, write 10 MB, etc.

Is there an existing way to do that?

== details ==

When copying a bunch of 1.5 GB files with rsync, I'm only seeing 50%
of the throughput I hope to see.

I haven't looked at the code, or even run strace, but it seems like
the code is doing something like:

while (files)  {
    read 1.5 GB file to ram
    write 1.5 GB file from ram
    fsync()  ensure 1.5 GB file is on disk
} endwhile

I say that because I see several seconds of high-speed reading, then no reads.

When the reads stop, I see writes kick in, then they stop and reads
start up again.

The end result is I'm only using 50% of the available bandwidth.

Not that I'm copying my source folder tree to a newly created folder
tree, so there is not any reading of the destination needed.
My ultimate would be something like:

while (files) {
    while (data_in_file) {
        read user_defined_blocksize to ram from file
        write user_defined_blocksize from ram to file
    }
    fsync()  ensure 1.5 GB file is on disk
} endwhile

Thanks
Greg
--
Greg Freemyer
www.IntelligentAvatar.net

--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Fabian Cenedese | 6 Apr 15:48 2016
Picon

Failed verification message

Hi

We synch between servers, some are NAS with RAIDs, some are
simple Linux servers. Sometimes I get messages like:

WARNING: path/to/file failed verification -- update retained (will try again).

What are the possible reasons for this message? I looked on
the Internet and found old posts about a problem with big files.
I do get this message with big files (GB) but also with small files
(<1MB). Of course if a file is changed while being synched then
a problem is to be expected. But most files are surely not written
at this moment. The error usually only happens once for one
file, the retry or a second synch run okay.

Does this mean that the file content is wrong? Is this a problem
of the connection? What else can be done to improve the reliability
of the backup?

Thanks

bye  Fabi

--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Kenneth Waegeman | 1 Apr 14:57 2016
Picon

weird output rsync version 3.1.1

Hi all,

When running rsync with following parameters, I get some strange output 
for a lot of paths. This does not happen on every path, but I see it often:

  rsync --stats --numeric-ids -lptgoD 
--files-from=/tmp/zkrsync/tmpUaSGa2 -r /data/.snapshots/backup/ 
rsync://osd010.gigalith:5555/zkrs-data
5:
5:
5:
5:
5:
5: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 
26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 
50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 
74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 
98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 
116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 
134 135 136 137 138 139 140 141 142 143 144 145 146
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5: 0
5:
5:
5:
5:
5:
4:
7:
9: 0
11: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 
26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 
50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 
74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 
98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 
116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 
134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 
152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 
170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 
188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 
206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 
224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 
242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 
260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 
278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 
296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 
314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 
332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 
350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 
368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 
386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 
404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 
422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 
440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 
458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 
476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 
494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 
512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 
530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 
548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 
566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 
584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 
602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 
620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637
4:
4: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 
26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 
50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 
74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 
98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 
116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 
134 135 136 137 138 139 140 141 142 143 144 145 146
4: 0
7: 0
9: 0
11: 0
13: 0
15: 0
17: 0
19: 0
21: 0
23: 0
25: 0
27: 0
29: 0
31: 0
33: 0
35: 0
37: 0
39: 0
41: 0
43: 0
45: 0
47: 0
49: 0
51: 0
53: 0
55: 0
57: 0
59: 0
61: 0
63: 0
65: 0
67: 0
69: 0
71: 0
73: 0
75: 0
77: 0
79: 0
81: 0
83: 0
85: 0
87: 0
89: 0
91: 0
93: 0
95: 0
97: 0
99: 0
101: 0
103: 0
105: 0 1 2 3 4 5 6 7 8
5:
5:
5:
5:
5:
5:
5:
5:
5:
5:
5:
5:
107: 0 1 2
109: 0 1 2
111: 0 1 2
113: 0 1 2
115: 0 1 2
117: 0 1 2
119: 0 1 2
121: 0 1 2
123: 0 1 2
125: 0 1 2
127: 0 1 2
129: 0 1 2

Number of files: 2,376 (reg: 2,359, dir: 17)
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 69
Total file size: 37,174,634 bytes
Total transferred file size: 5,438,730 bytes
Literal data: 0 bytes
Matched data: 5,438,730 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 74,092
Total bytes received: 37,174

sent 74,092 bytes  received 37,174 bytes  14,835.47 bytes/sec
total size is 37,174,634  speedup is 334.11

What does this mean?
We are running version 3.1.1

Thank you very much!

Cheers,
Kenneth

--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

tomr | 31 Mar 06:40 2016
Picon

rsync with overlay tree

I maintain a directory structure containing dirs and files that I regularly push to ~50 hosts, which are
divided into 3 groups that have slightly different needs (minor mods in a couple of files).

So ideally I would have 4 directories:
    /path/to/sync/common/   <- common files
    /path/to/sync/group1/   <- group1 specific only
    /path/to/sync/group2/   <- group2 specific only
    /path/to/sync/group3/   <- group3 specific only

Then I'd run an rsync like:
    rsync -av --overlay /path/to/sync/groupN \
     /path/to/sync/common remotehost:

Thinking in terms of a list of files to be transferred, I would like:
- Anything present in common/ added to the file list; then
- Anything present in groupN/ added to the list, clobbering if applicable (regardless of mtime)
- The destination directory to show no sign of the common / overlay structure

I suspect I could populate the list myself and use '--files-from=<LIST>' but I would rather have rsync work
it out if possible.  If all else fails, I will do the merge locally first.

    TEMPDIR=$(mktemp -d)
    cp -r /path/to/sync/common/* $TEMPDIR
    cp -r /path/to/sync/groupN/* $TEMPDIR
    rsync -av $TEMPDIR/* remotehost:
    rm -r $TEMPDIR

Thanks in advance,
tom

--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Gmane