Henri Shustak | 19 Sep 02:39 2014
Picon

Re: rsync with 8bit file names?

>>> When trying to sync my TV folder to a mirror drive, episodes with non-ASCII characters in them cannot be
processed by rsync. Anything I can do about this?
>> 
>> (Q1) What do you mean by a mirror drive? Is this a RAID1 external enclosure or some sort of softRAID? Or is it
just a copy of the TV folder onto another device?
> 
> A copy of the TV folder.

Okay, thanks for confirmation on this point. Have you confirmed that the file system is good on the
destination directory. IE : Running Disk Utility to check the file system?

>> (Q2) Is your Drobo's firmware all up-to date?
> Yes
Great.

>> (Q3) Have you tried via SMB rather than AFP?
> No, but my understanding is that afp in 10.9 and 10.10 *si* smb.
I do not understand your reply? What is "*si*"?

>> (Q2-1) Have you tried this same setup when booted from another OS (e.g. 10.9)?
> No, I moved my machine to the DP with DP7 at the same time that I added this new external store to mirror my TV
folder to.
I would strongly suggest trying this out on a system running 10.9 (not beta software) and seeing if this
resolves the problem.

>> (Q2-2) Have you tried this using some other hardware?
> Limited options. I suppose i could move it to a Mac mini that is still running 10.9 to test, but then again,
it’s about 15 files I can manually deal with out of 7TB which seems like less trouble that moving hardware around.
Sure, but you have asked for assistance from the rsync mailing list. It would be good to have answers to the
following : 
(Continue reading)

Robert Bell | 16 Sep 07:16 2014
Picon
Picon

Recycling and keeping backups - Tower of Hanoi management of backups using rsync

Thanks to Kevin and Paul for responses.

We use a modified Tower of Hanoi scheme (on top of rsync and --link-dest
and recycling) for deciding which backups to keep.

Here is a sample of our holdings for one area:

 	home.20111124.seq.0   set  0
 	home.20130512.seq.512 set 10
 	home.20140203.seq.768 set  9
 	home.20140414.seq.832 set  7
 	home.20140708.seq.896 set  8
 	home.20140815.seq.928 set  6
 	home.20140831.seq.944 set  5
 	home.20140904.seq.948 set  3
 	home.20140908.seq.950 set  2
 	home.20140909.seq.951 set  1
 	home.20140910.seq.952 set  4
 	home.20140911.seq.953 set  1
 	home.20140912.seq.954 set  2
         Found 13 backups as expected up to sequence number 954
         >>>>Marking for recycling home.20140908.seq.950, set number 2

The coverage matches the likelihood of restorations being required - the
coverage tails off exponentially over time.

I found the key to running a Tower of Hanoi scheme is to assign a
sequence number to each backup, from which you can derive a set number.

The dates are for humans - scripts don't have to deal with days of week,
(Continue reading)

Robert Bell | 15 Sep 08:24 2014
Picon
Picon

Re: Backup scripts - recycling old backup directories (Kevin Korb)

Kevin,

Thanks for the reply and interest in this topic.

Comments below.

Regards

Rob.

> I did consider that but rejected it for 2 reasons...
> 
> 1. Backup run time.  We have a 4 hour window to run backups at night.
>  Using recycled directories significantly extended the backup run
> time.  The deletion time is eliminated but frankly, we have the other
> 20 hours of the day to do deletions.  We had to give up using
> - --link-dest when the deletions started to actually take that long even
> though the backups still ran in under 4 hours.

For us, the recycling of old directories significantly shortened the time
to do backups, since the recycled backups have typically 95% of the
files/directories correct (with daily backups and Tower of Hanoi, 
half of our recycled backups are only 5 to 6 days old).

I've just done some tests with a fairly pathological case, all on one
host.

I set up a source tree 's' with 11111 sub-directories and 10000 files,
and then two destinations:
   cp -a s d1
(Continue reading)

Robert Bell | 12 Sep 06:31 2014
Picon
Picon

Backup scripts - recycling old backup directories

Folks,

Kevin Korb wrote:
> Have you considered more advanced methods such as subvolume snapshots
> provided by ZFS and BTRFS?  At work we were forced to abandon rsync
> - --link-dest because of the amount of time it takes to delete old
> backups when the data is primarily many millions of small files
> (shared web hosting company).

We don't have more advanced methods like subvolume snapshots available
to us.

However, we can recycle backup directories.

When we started using rsync with --link-dest back in about 2007, we
deleted old backups, but realised soon after that we could recycle old
backups.

With daily backups, we find about 1.5% of the data and 0.5% of the files
change from one day to the next, so a directory from about 5 days ago
will typically be only 5-10% wrong and can be recycled to be the target
of the latest directory - that's a lot better than recreating the whole
directory tree for a new backup, and then deleting a whole old directory
tree.

We use --delete of course.

Hope this helps someone.

Rob.
(Continue reading)

Henri Shustak | 12 Sep 03:31 2014
Picon

Re: rsync with 8bit file names?

Hello,

Is the source Drobo a NAS or DAS unit? 

Also, I am guessing that the /Volumes/TV is the source for the rsync. Is it possible that the file has
actually moved or been renamed?

If the Drobo (hosted) volume is the source for the rsync command, then perhaps checking the file system
format of that device is also worth while?

Hope this helps. Perhaps someone else on the list has some ideas? 

I would suggest you consider compiling a recent version of rsync for your system. This should be quite
straight forward on 10.10 (beta release). However, I must admit I have yet to try compiling rsync on 10.10 : http://www.lbackup.org/developer/rsync_hfs

Finally, what is the actual command you are running. This information may shed some light on why the command
is failing.

Hope this helps.

---------------------------------------------------------------------
This email is protected by LBackup, an open source backup solution.
http://www.lbackup.org

On 12/09/2014, at 12:40 PM, LuKreme <kremels <at> kreme.com> wrote:

> On 11 Sep 2014, at 16:22 , Henri Shustak <henri.shustak <at> gmail.com> wrote:
>> Guessing the source drive is formatted HFS+ or HFS+J.
> 
> The source drive is a Drobo. The destination is Journaled HFS+.
(Continue reading)

LuKreme | 11 Sep 14:36 2014

rsync with 8bit file names?

When trying to sync my TV folder to a mirror drive, episodes with non-ASCII characters in them cannot be
processed by rsync. Anything I can do about this?

For example, I get this:

building file list ... 
file has vanished: "/Volumes/TV/Raising Hope/Raising Hope - S04E01 - De?\#201ja Vu Man.mkv"

This is rsync under OS X:

$ rsync --version
rsync  version 2.6.9  protocol version 29
Copyright (C) 1996-2006 by Andrew Tridgell, Wayne Davison, and others.
<http://rsync.samba.org/>
Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles,
              inplace, IPv6, 64-bit system inums, 64-bit internal inums

I'd check macports, but since it's not been updated for 10.10 yet I'll need to wait a while.

-- 
Passion is the pill you can swallow forever Taking them one by one One
by One --Agents of Good Roots "Come On"

--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Nirat Shah | 10 Sep 03:57 2014
Picon

--temp-dir option question/issue

Hello, I am using rsync with "--temp-dir=/tmp" option for rsync module.  I noticed that if the "tmp" directory is deleted rsync attempt will fail like -

 rsync: mkstemp "<module_base>/tmp/.xyz_file.Z0m8I6" (in <module>) failed: No such file or directory (2)
 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]

I just wanted to understand how and when the "tmp" directory is created.  Is there a option to create one if it doesn't exist?  The issue I see is that on our system, the "tmp" does seem to be getting deleted once in a while (cause of which I am still trying to figure out).

Thank you very much.
-Nirat.
--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
samba-bugs | 6 Sep 11:56 2014
Picon

[Bug 10799] New: Feature request: detail --dry-run mode when --debug=exit

https://bugzilla.samba.org/show_bug.cgi?id=10799

           Summary: Feature request: detail --dry-run mode when
                    --debug=exit
           Product: rsync
           Version: 3.1.1
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: enhancement
          Priority: P5
         Component: core
        AssignedTo: wayned <at> samba.org
        ReportedBy: davegradwell <at> yahoo.co.uk
         QAContact: rsync-qa <at> samba.org

It would be much appreciated if rsync could mark exits of dry-runs specifically
as such.

Example:
2014/09/06 10:22:41 [3449] [sender] _exit_cleanup(code=0, file=main.c,
line=1165): about to call exit(0) (DRY RUN)

My feeling is that a --dry-run is such a fundamental modifier to the context of
'success' that it merits this special treatment.
I know you see this information when you ask for --stats but I think it is
equally significant to clarify the context of the exit.

With thanks, Dave.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.
--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Chris | 4 Sep 18:30 2014

Backup scripts

I have two scripts that a kind soul on this list wrote for me over 4yrs
ago. I got to looking at them the other day because my old box crashed
and had to build a new one also got a new backup USB drive since I'm
still copying over things from the old one. The first one is for a full
backup:

http://pastebin.com/XF6Zm42A

Works great, does exactly what it's supposed to do. The second is for a
'snapshot' which is where I get a bit confused. I would think that a
'snapshot' would be just the changed files either since the last full
backup or since the last 'snapshot' the night before. It seems though
that it's actually the same as a full backup. I don't profess to be a
script person so I have no idea if it's doing what it should or
something needs to be changed. Below is the 2nd script:

http://pastebin.com/MkBzJnux

Any advice would be appreciated.

Chris

-- 
Chris
31.11°N 97.89°W (Elev. 1092 ft)
11:21:45 up 1 day, 2:52, 1 user, load average: 0.08, 0.21, 0.27
Ubuntu 14.04 LTS, kernel 3.13.0-35-generic

--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Jakob Unterwurzacher | 2 Sep 18:26 2014

[PATCH] rrsync: Add several long options used by BackupPC

rrsync used to throw the error

	/usr/local/bin/rrsync: invalid rsync-command syntax or options

when run under BackupPC 3.2.1, with this patch full and incremental
backups work.
---
 support/rrsync | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/support/rrsync b/support/rrsync
index 6f83f9d..c231ea3 100644
--- a/support/rrsync
+++ b/support/rrsync
 <at>  <at>  -60,6 +60,7  <at>  <at>  our $short_with_num = 'B'; # DO NOT REMOVE ANY
 our %long_opt = (
   'append' => 0,
   'backup-dir' => 2,
+  'block-size' => 1,
   'bwlimit' => 1,
   'checksum-seed' => 1,
   'compare-dest' => 2,
 <at>  <at>  -82,15 +83,19  <at>  <at>  our %long_opt = (
   'force' => 0,
   'from0' => 0,
   'fuzzy' => 0,
+  'group' => 0,
   'groupmap' => 1,
+  'hard-links' => 0,
   'iconv' => 1,
   'ignore-errors' => 0,
   'ignore-existing' => 0,
   'ignore-missing-args' => 0,
+  'ignore-times' => 0,
   'info' => 1,
   'inplace' => 0,
   'link-dest' => 2,
   'list-only' => 0,
+  'links' => 0,
   'log-file' => 3,
   'log-format' => 1,
   'max-delete' => 1,
 <at>  <at>  -103,10 +108,14  <at>  <at>  our %long_opt = (
   'no-relative' => 0,
   'no-specials' => 0,
   'numeric-ids' => 0,
+  'one-file-system' => 0,
   'only-write-batch' => 1,
+  'owner' => 0,
   'partial' => 0,
   'partial-dir' => 2,
   'preallocate' => 0,
+  'perms' => 0,
+  'recursive' => 0,
   'remove-sent-files' => $ro ? -1 : 0,
   'remove-source-files' => $ro ? -1 : 0,
   'safe-links' => 0,
 <at>  <at>  -120,6 +129,7  <at>  <at>  our %long_opt = (
   'super' => 0,
   'temp-dir' => 2,
   'timeout' => 1,
+  'times' => 0,
   'use-qsort' => 0,
   'usermap' => 1,
 );
-- 
1.9.0

--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Dave Gradwell | 30 Aug 12:36 2014
Picon

Cleanest way to find out if it was a --dry-run

Hi, 

As far as I can see, the only way you can tell it's been a dry-run at the receiver-side, is to parse RSYNC_ARGs
in the pre-xfer script.
I think the only way is to crawl through the RSYNC_ARGs to find a value which:
- starts with a single dash, and;
- contains 'n', and;
- comes prior to the "." value, signifying the start of the 'path args' (which seem inherently variable,
ergo scary).

I can't see any other way to determine that it's been a dry run.  On the sender-side, the log contains a handy
(DRY RUN) indicator next to the stats:
2014/08/29 18:54:15 [62154] total size is 327,680,000  speedup is 68,739.25 (DRY RUN)
but I don't think you can output stats at the receiver-side, and I don't think there's any other mention of
the dry run in the receiver-side log.

In my pre-xfer script, I see this:
RSYNC_ARG1=--server
RSYNC_ARG2=-nltrRze.iLsfx
RSYNC_ARG3=--log-format=X
RSYNC_ARG4=--bwlimit=256
RSYNC_ARG5=--delete-during
RSYNC_ARG6=--delete-excluded
RSYNC_ARG7=--force
RSYNC_ARG8=--partial-dir
RSYNC_ARG9=/in-progress/partial
RSYNC_ARG10=--link-dest
RSYNC_ARG11=/latest/data
RSYNC_ARG12=--info=STATS2
RSYNC_ARG13=--debug=EXIT
RSYNC_ARG14=.
RSYNC_ARG15=/in-progress/data/

I'm on Mac OS 10.9, using rsync 3.1.1.

Thanks, Dave.

--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Gmane