McDowell, Blake | 24 Jun 17:47 2016

--partial not working?

Hello,

 

I’m running rsync -avPhi to move large video files to a remote server.  Often we have to stop a transfer midway through to push something else to the server. My hope was that the -P flag would invoke --partial and the transfer would pick-up where it left off. This does not seem to be happening. Rather, the transfer starts over from the beginning, erasing the partially transferred file.


When I redo the transfer my I see this:

>f.st..g.... afc2010039_crhp0026_mv06_concat_Williams_Cecil_J.mov


I understand what all of that means, but I don't know why it doesn't resume the partial transfer.


Do I need to change my rsync command to allow it to resume a transfer of a partially transferred file?


Thanks,

Blake

--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Albert Berger | 23 Jun 22:14 2016
Picon

Operation not supported (95)

Greetings!

During migrating i686 installation of ArchLinux to x86_64, all filesystem was rsynced to a new location.
Maybe this somehow relates to the problem that has appeared after migrating and that did not happen
before: during backuping the system, rsync reports the following error message:

    rsync: set_acl: sys_acl_delete_def_file(var/log/journal/remote): Operation not supported (95)

The full log is as follows:

[root <at> kw al]# /usr/bin/rsync --update -DHAErlptgo --relative --delete-during --verbose
--itemize-changes /var/log/journal /mnt/backup/rootdir/
sending incremental file list
rsync: set_acl: sys_acl_delete_def_file(var/log/journal/remote): Operation not supported (95)
.d..t...... /var/log/
.d..tp..... /var/log/journal/
.d...p..... /var/log/journal/c892f90078ad480d902f236dfd220532/
>f..t...... /var/log/journal/c892f90078ad480d902f236dfd220532/system.journal
>f..t...... /var/log/journal/c892f90078ad480d902f236dfd220532/user-1000.journal
cd+++++++++ /var/log/journal/remote/

sent 16,782,653 bytes  received 169 bytes  6,713,128.80 bytes/sec
total size is 119,537,664  speedup is 7.12
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]

The 'remote' directory is empty on both source and destination. After deleting 'remote' directory on
destination, it's recreated and the same error is reported. The source filesystem is ext4, the
destination is on btrfs and is accessed via samba. Why can this message happen and how can this be corrected?

Thanks.

Albert.

--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Picon
Picon

Re: rsync Digest, Vol 162, Issue 18

Folks,

Am 20.06.2016 um 22:01 schrieb Larry Irwin (gmail):
  > The scripts I use analyze the rsync log after it completes and then 
sftp's a summary to the root of the just completed rsync.
  > If no summary is found or the summary is that it failed, the folder 
rotation for that set is skipped and that folder is re-used on the 
subsequent rsync.
  > The key here is that the folder rotation script runs separately from 
the rsync script(s).

That is what we found to be important some years ago - do most of the 
management outside of the scripts that do rsync.

In particular, our scripts prepare a target backup directory with a name 
of the form:
	slash.20151122.seq.1379.current
   - the name of the area being backed-up is obvious, as is the date.
We use Tower of Hanoi management (see the update at 
https://en.wikipedia.org/wiki/Backup_rotation_scheme#Tower_of_Hanoi), so 
embed a sequence number in the backup directory name, and the suffix 
.current to indicate a current backup target.

we can keep on trying the rsync backups until we get success, and the 
scripts then remove the .current suffix.  This works because the rsync 
includes the --delete option, used since the backup target can be a 
recycled hard-linked directory: also recommended.

  > For each entity I want to rsync, I create a named folder to identify 
it and the rsync'd data is held in sub-folders:
  > daily.[1-7] and monthly.[1-3]
  > When I rsync, I rsync into daily.0 using daily.1 as the link-dest.
  > Then the rotation script checks daily.0/rsync.summary - and if it 
worked, it removes daily.7 and renames the daily folders.
  > On the first of the month, the rotation script removes monthly.3, 
renames the other 2 and makes a complete hard-link copy of daily.1 to 
monthly.1
  > It's been running now for about 4 years and, in my environment, the 
10 copies take about 4 times the space of a single copy.
  > (we do complete copies of linux servers - starting from /)
  > If there's a good spot to post the scripts, I'd be glad to put them up.

I'd recommend Tower of Hanoi, as referenced above, for managing sets of 
backups - far simpler than dealing with days and dates, etc.  For 
example, the above mentions special actions on the first of the month - 
what if that is missed because of some failure - more special case 
scripting? Tower of Hanoi is self-healing....

Regards

Rob.

Dr Robert C. Bell
HPC National Partnerships | Scientific Computing
CSIRO Information Management and Technology
T +61 3 9545 2368 (Docklands and Clayton)
T +61 3 9669 8102 (Docklands only)
Mob +61 428 108 333
Robert.Bell <at> csiro.au<mailto:Robert.Bell <at> csiro.au> | www.csiro.au
https://wiki.csiro.au/display/ASC/Scientific+Computing+Homepage

Also: Bureau of Meteorology Scientific Computing Services

Street: CSIRO SC Lvl 11, 700 Collins Street, Docklands Vic 3008, Australia
Postal: CSIRO SC Lvl 11, GPO Box 1289, Melbourne Vic 3001, Australia
Street: CSIRO IMT, Rivett Room RNW 2.03, Clayton Vic 3168, Australia
Postal: CSIRO IMT, Private Bag 10, Clayton South Vic 3169, Australia
Parcels: CSIRO IMT, Gate 3 Normanby Road, Clayton Vic 3168, Australia

PLEASE NOTE
The information contained in this email may be confidential or
privileged.  Any unauthorised use or disclosure is prohibited.  If
you have received this email in error, please delete it immediately
and notify the sender by return email. Thank you.  To the extent
permitted by law, CSIRO does not represent, warrant and/or
guarantee that the integrity of this communication has been
maintained or that the communication is free of errors, virus,
interception or interference.

Please consider the environment before printing this email.

On 22/06/2016 10:00 pm, rsync-request <at> lists.samba.org wrote:
> Send rsync mailing list submissions to
> 	rsync <at> lists.samba.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> 	https://lists.samba.org/mailman/listinfo/rsync
> or, via email, send a message with subject or body 'help' to
> 	rsync-request <at> lists.samba.org
>
> You can reach the person managing the list at
> 	rsync-owner <at> lists.samba.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of rsync digest..."
>
>
>
> To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>
> ---------------------------------------
>
>
>
> Today's Topics:
>
>    1. Re: rsync script for snapshot backups (Dennis Steinkamp)
>    2. Re: rsync script for snapshot backups (Petros Angelatos)
>
>
>
> _______________________________________
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>
> rsync mailing list
> rsync <at> lists.samba.org
> https://lists.samba.org/mailman/listinfo/rsync
>

--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Dennis Steinkamp | 19 Jun 14:22 2016
Picon

rsync script for snapshot backups

Hey guys,

i tried to create a simple rsync script that should create daily backups from a ZFS storage and put them into a timestamp folder.
After creating the initial full backup, the following backups should only contain "new data" and the rest will be referenced via hardlinks (-link-dest)

This was at least a simple enough scenario to achieve it with my pathetic scripting skills. This is what i came up with:

#!/bin/sh

# rsync copy script for rsync pull from FreeNAS to BackupNAS for Buero dataset

# Set variables
EXPIRED=`date +"%d-%m-%Y" -d "14 days ago"`

# Copy previous timefile to timeold.txt if it exists
if [ -f "/volume1/rsync/Buero/timenow.txt" ]
then
    yes | cp /volume1/rsync/Buero/timenow.txt /volume1/rsync/Buero/timeold.txt
fi
# Create current timefile
    echo `date +"%d-%m-%Y-%H%M"` > /volume1/rsync/Buero/timenow.txt
# rsync command
if [ -f "/volume1/rsync/Buero/timeold.txt" ]
then
    rsync -aqzh \
    --delete --stats --exclude-from=/volume1/rsync/Buero/exclude.txt \
    --log-file=/volume1/Backup_Test/logs/rsync-`date +"%d-%m-%Y-%H%M"`.log \
    --link-dest=/volume1/Backup_Test/`cat /volume1/rsync/Buero/timeold.txt` \
    Test <at> 192.168.2.2::Test/volume1/Backup_Test/`date +"%d-%m-%Y-%H%M"`
else
    rsync -aqzh \
    --delete --stats --exclude-from=/volume1/rsync/Buero/exclude.txt \
    --log-file=/volume1/Backup_Buero/logs/rsync-`date +"%d-%m-%Y-%H%M"`.log \
    Test <at> 192.168.2.2::Test/volume1/Backup_Test/`date +"%d-%m-%Y-%H%M"`
fi

# Delete expired snapshots (2 weeks old)
if [ -d /volume1/Backup_Buero/$EXPIRED-* ]
then
rm -Rf /volume1/Backup_Buero/$EXPIRED-*
fi

Well, it works but there is a huge flaw with his approach and i am not able to solve it on my own unfortunately.
As long as the backups are finishing properly, everything is fine but as soon as one backup job couldn`t be finished for some reason, (like it will be aborted accidently or a power cut occurs)
the whole backup chain is messed up and usually the script creates a new full backup which fills up my backup storage.

What i would like to achieve is, to improve the script so that a backup run that wasn`t finished properly will be resumed, next time the script triggers.
Only if that was successful should the next incremental backup be created so that the files that didn`t changed from the previous backup can be hardlinked properly.

I did a little bit of research and i am not sure if i am on the right track here but apparently this can be done with return codes, but i honestly don`t know how to do this.
Thank you in advance for your help and sorry if this question may seem foolish to most of you people.

Regards

Dennis








--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Albert Berger | 19 Jun 15:21 2016
Picon

Any way to preserve file access times?

Greetings!

Though it's a bit strange for archiving/syncing program not to have an option of
preserving file access times, but even the 'official' patch for enabing this
option (the link to which is on the download page) doesn't work properly.

I wonder is there a way to preserve the access times of files copied
with rsync?

Thanks.

Albert.

--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Albert Berger | 18 Jun 13:22 2016
Picon

/root is wrongly copied during update

For backuping system folders on ArchLinux I use the following command
per folder:

rsync --update -DHAErlptgo --relative --delete-during $dir $backupdir

The problem is that every file in /root folder and its subfolders is
copied each time when this command is run. In all other folders only
new or modified files are copied. The files and the folder itself have
the following attributes:

[root <at> kw var]# getfattr /root/.bash_aliases 
[root <at> kw var]# getfattr /root
[root <at> kw var]# ls -ld /root
drwxr-x--- 29 root root 4096 Jun 18 13:51 /root
[root <at> kw var]# ls -l /root/.bash_aliases 
-rw-r--r-- 1 root root 113 May 17  2015 /root/.bash_aliases

On backup dir:

[root <at> kw var]# getfattr /mnt/backup/home/root/.bash_aliases 
[root <at> kw var]# getfattr /mnt/backup/home/root
[root <at> kw var]# ls -ld /mnt/backup/home/root
drwxr-x---+ 1 root root 0 Jun 18 10:02 /mnt/backup/home/root
[root <at> kw var]# ls -l /mnt/backup/home/root/.bash_aliases 
-rw-r--r--+ 1 root root 113 May 17  2015 /mnt/backup/home/root/.bash_aliases

Could someone advise what can cause such rsync's behaviour?

Thanks.

Albert.

--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

samba-bugs | 16 Jun 08:29 2016
Picon

[Bug 11979] New: rsync -X fails without need

https://bugzilla.samba.org/show_bug.cgi?id=11979

            Bug ID: 11979
           Summary: rsync -X fails without need
           Product: rsync
           Version: 3.1.1
          Hardware: All
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P5
         Component: core
          Assignee: wayned <at> samba.org
          Reporter: hdunkel <at> aixigo.de
        QA Contact: rsync-qa <at> samba.org

Problem: "rsync -aX remote:/somepath ." fails, if the remote site doesn't
support extended attributes. Sample session:

# rsync -aX aix1:/usr/man .
rsync: extended attributes are not supported on this server
rsync error: syntax or usage error (code 1) at main.c(1554) [server=3.1.1]
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(226)
[Receiver=3.1.1]

Since this is a read-only access on the remote side a warning should be
sufficient.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

samba-bugs | 15 Jun 22:16 2016
Picon

[Bug 11978] New: mkstemp failed: File name too long (36) when filename is under the limit

https://bugzilla.samba.org/show_bug.cgi?id=11978

            Bug ID: 11978
           Summary: mkstemp failed: File name too long (36) when filename
                    is under the limit
           Product: rsync
           Version: 3.1.1
          Hardware: All
                OS: All
            Status: NEW
          Severity: normal
          Priority: P5
         Component: core
          Assignee: wayned <at> samba.org
          Reporter: sambabug1.nobled <at> spamgourmet.com
        QA Contact: rsync-qa <at> samba.org

Using mkstemp adds on 7 characters to the filename, which creates a problem
when the filename is below, but close to, the limit of the filesystem.

You can see this when rsync'ing from one ecryptfs filesystem to another, which
will fail even though both have the exact same 143-byte filename limit, if a
filename consists of more than 136 bytes.

Is there any reason the .XXXXXX in the argument to mkstemp can't *replace* the
last 7 characters in the filename, instead of being appended to it?

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Harald Dunkel | 15 Jun 11:07 2016
Picon

failure for "rsync -X"

Hi folks,

Problem: "rsync -aX remote:/somepath ." fails, if the remote
site doesn't support extended attributes. Sample session

# rsync -aX aix1:/usr/man .
rsync: extended attributes are not supported on this server
rsync error: syntax or usage error (code 1) at main.c(1554) [server=3.1.1]
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [Receiver=3.1.1]

Since this is a read-only access on the remote side I wonder
why this results in a fatal error instead of a warning?

rsync is 3.1.1 on both ends.

Every helpful comment is highly appreciated
Harri

--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Arnaud Aujon Chevallier | 9 Jun 11:32 2016
Picon

Can rsync assume that the destination directory is empty ?

Hello,

I'm currently using rsync to backup up to 1 TB of small files of 
relatively small files (hundreds of Ko mostly)

My backup strategy is to use a full backup and then backup the diff 
every day using hardlink with the previous backup. This means that each 
time I use rsync, the destination directory is empty.

Using strace, I can see that rsync call a 'lstat' command to try to see 
if the file already exists in my destination directory. Is there an 
option to tell rsync that the destination directory is empty ?

Do you think that avoiding this call can improve rsync performances in 
this specific case ?

I tried reading the source code, but I'm not exactly sure where this 
lstat call happens.

Thanks a lot,

Arnaud Aujon Chevallier

--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Rodolfo Medina | 5 Jun 12:40 2016
Picon

Deletion of destination files

Hi to all rsync users.

rsync's `--delete' option works fine in the following example: I'm sending all
the content of /home/rodolfo from machine1 to /home/rodolfo in machine2:

 $ rsync --dry-run -vrtul --delete --exclude='/.*' . 192.168.0.2:/home/rodolfo

, and --delete works perfectly.  Instead, in this other example:

 $ rsync --dry-run -vrt --delete --modify-window=1 file1 file2 file3
... /mnt/pendrive/rodolfo

I'm copying various files and directories from different places of my home
directory to a directory named `rodolfo' which is on the pendrive.  I want that
all the files that are not listed among `file1 file2 file3 ...' in the above
command were deleted in the destination, but this does not happen.  In rsync
`man' page I can't find a solution.  Please, whoever can, any suggestion about
how to do what I want.

Thanks in advance for any help,

Rodolfo

--

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Gmane