headers
alinugai | 3 Jun 2002 21:59
Picon
Favicon

Beginner's Questions about Unison

My company is trying to find a solution for what amounts to mostly
1-way replication of files from a single server to perhaps multiple
replica machines.  Both server and replica are running Redhat 7.2.

I have a few questions based on the documentation and a few posts on
this list.

1. If we have around 3 million files and 100GB on our source is this
feasible for unison.  Has anyone used it with this level of data. 
Almost all of our data is static once written and needs only be
synchronized once.

2. I noticed that someone had an out-of-memory error with an 800MB
file which someone said could be corrected by setting rsync = false. 
How does the processing differ in this case?

3. Is there still a file limit size for unison?

4. Can use unison to synchronize to between Host A and Hosts B and C
at the same time without concurrency problems?

Thanks
  Alain Inugai
Permalink | Reply | 1 comment |
headers
royyeow | 6 Jun 2002 11:53
Picon
Favicon

unison using up CPU load

Hi there,

I am running unison 2.7.7 on HP-UX 11.00 to sync up 2 servers (same 
OS).

I am using ssh connection to establish connection. I have script 
running to sync about 750MB. It took around 7 minutes to complete 
that.

But the problem here is, everytime unison starts, it uses up a lot of 
CPU load, on both client and server, as can be seen on top output 
here:

CPU TTY     PID USERNAME PRI NI   SIZE    RES STATE    TIME %WCPU  
%CPU COMMAND
 1   ?    12861 root     241 20 12244K 11576K run      5:31 98.22 
98.05 unison

Any reason why this is happening and how I can solve this?

Please help, thanks!

regards,
Roy
Permalink | Reply | 1 comment |
headers
Mark H. Wilkinson | 6 Jun 2002 10:08

getting at the help

I just found out how hard it is to access the help information within
the Unison binary if you don't actually want to synchronise anything. I
was just wanting to check the semantics of one of the options and
'unison -help' was a bit too brief. I had to mess around creating a
couple of directories in /tmp just to get the user interface to start up
cleanly.

In the process I inadvertently added two 'root = ' lines to the default
profile (by just selecting the default options in the dialog boxes) and
finished up in a state where this would happen:

[mhw <at> sputnik mhw]$ unison /tmp/a /tmp/b
Wrong number of roots: 2 expected, but 4 provided (, , /tmp/b, /tmp/a)
(Maybe you gave roots both on the command line and in the profile?)
[mhw <at> sputnik mhw]$ 

which took a few minutes thought to unravel.

How difficult would it be for the unison gui to carry on running if
there are fatal errors in the profile that has been selected? What about
bringing up the main window as well as the profile selection window when
you just type 'unison'? There should be some way to get the help on the
screen without having to work at it.

-Mark.
Permalink | Reply | 1 comment |
headers
Zhe Yang | 7 Jun 2002 05:47

Re: Beginner's Questions about Unison


> 1. If we have around 3 million files and 100GB on our source is this
> feasible for unison.  Has anyone used it with this level of data.
> Almost all of our data is static once written and needs only be
> synchronized once.

I don't know if anyone has used Unison with such an amount of files.
I guess Unison could work for you, but the first time it's run, it could
take an intolerable amount of time to build the initial archive (the data
structure that Unison uses to track the state of each file by
fingerprinting it), as the content of all the 100GB files need to be read.
Memory could be a problem, too.  But you can use the -path option to make
Unison break up this initial scanning task.

> 2. I noticed that someone had an out-of-memory error with an 800MB
> file which someone said could be corrected by setting rsync = false.
> How does the processing differ in this case?

When rsync = true (the default setting), unison uses the RSYNC algorithm
to transfer big files, and this algorithm creates sizable tables in the
main memory.  Setting rsync to be false forces Unison to perform simple
file transfer---which is slow, but doesn't take much memory.

> 3. Is there still a file limit size for unison?

Yes, and I think it's half gigabytes.  It's a limitation of the current
OCaml compiler.

> 4. Can use unison to synchronize to between Host A and Hosts B and C
> at the same time without concurrency problems?
(Continue reading)

Permalink | Reply | 0 comments |
headers
Zhe Yang | 11 Jun 2002 00:08

Re: getting at the help


> How difficult would it be for the unison gui to carry on running if
> there are fatal errors in the profile that has been selected? What about
> bringing up the main window as well as the profile selection window when
> you just type 'unison'? There should be some way to get the help on the
> screen without having to work at it.

Good points.  But we need to rethink about the logic of the Unison/GUI
states.  Probably it's fairly easy to make the proposed change.
Otherwise we would rather wait till other more critical problems have been
solved.

Zhe
Permalink | Reply | 0 comments |
headers
Zhe Yang | 7 Jun 2002 06:01

Re: unison using up CPU load

> Hi there,
>
> I am running unison 2.7.7 on HP-UX 11.00 to sync up 2 servers (same
> OS).
>
> I am using ssh connection to establish connection. I have script
> running to sync about 750MB. It took around 7 minutes to complete
> that.
>
> But the problem here is, everytime unison starts, it uses up a lot of
> CPU load, on both client and server, as can be seen on top output
> here:
>
> CPU TTY     PID USERNAME PRI NI   SIZE    RES STATE    TIME %WCPU
> %CPU COMMAND
> 1   ?    12861 root     241 20 12244K 11576K run      5:31 98.22
> 98.05 unison
>
> Any reason why this is happening and how I can solve this?

Do you have a lot of small files?  Or, do you explicitly set "fastcheck"
to false?

Unison operates in two stages.  In the first stage, it scans the whole
directory and computes the updates.  In the second stage, it actually
carry out the file transfer.  Scanning through a directory full of
small files and build the data structure that reflect the changes could
consume quite a lot of memory.  This is more the case if fastcheck is off,
when the contents of all the file need to be read to recompute there
fingerprints.
(Continue reading)

Permalink | Reply | 0 comments |
headers
Amy McGovern | 11 Jun 2002 16:36
Picon
Favicon

installing unison on Mac OS X

Hi,

I have been using unison to synchronize several linux boxes for awhile
now and I am very happy with it.  I now have a Mac OS X 10.1.5 machine
and would like to get unison working on this machine so that I can
continue to synchronize with my linux machines.  I am having trouble
getting it to compile and would like to request some help.  I did a
web search and noticed that unison is available for OS X through fink
but I can't get fink to actually list it as an available package.  In
compiling it, I downloaded the objective caml package as listed in the
unison README file.  However, when I type 'make' to compile it, I get
the following errors:

amy> make
zsh: command not found: bash
Building for Unix
NATIVE = true
THREADS = false
STATIC = false
OSTYPE = darwin
OSARCH = linux
ocamlopt: ubase/safelist.mli ---> ubase/safelist.cmi
ocamlopt -I lwt -I ubase -c
/Users/amy/Desktop/unison-2.9.1/ubase/safelist.mli
dyld: ocamlrun can't open library: /usr/local/lib/libdl.dylib  (No
such file or directory, errno = 2)
make: *** [ubase/safelist.cmi] Error 60

Ideas/suggestions?  If there is a unison binary around for OS X, I'm
happy with that :-)  Thanks!!
(Continue reading)

Permalink | Reply | 3 comments |
headers
Zhe Yang | 17 Jun 2002 20:53

Re: installing unison on Mac OS X


Hi,

(Sorry, this mail was buried for a while.)

Are you sure that the Ocaml 3.04 compiler was properly installed?  It
seems that it has trouble finding some dynamically linked library.  Have
you succeed compiling some other Ocaml programs using the native compiler
(ocamlopt)?

Also, you should use "make UI=text" to build unison with the text-based
user interface.  To compile the gtk-based user interface, you need to
install GTK yourselfs from sources.

Zhe

Zhe Yang                                      (O) (+1) (215) 573-5605
University of Pennsylvania                    (H) (+1) (215) 382-4245

On Tue, 11 Jun 2002, Amy McGovern wrote:

> Hi,
>
> I have been using unison to synchronize several linux boxes for awhile
> now and I am very happy with it.  I now have a Mac OS X 10.1.5 machine
> and would like to get unison working on this machine so that I can
> continue to synchronize with my linux machines.  I am having trouble
> getting it to compile and would like to request some help.  I did a
> web search and noticed that unison is available for OS X through fink
> but I can't get fink to actually list it as an available package.  In
(Continue reading)

Permalink | Reply | 2 comments |
headers
Tom Koelman | 13 Jun 2002 08:09
Picon

Windows 2000 socket problem with 2.9.1

Hi,

I am trying to synch to w2000 machines, using 2.9.1. On Machine A I
start

unison -socket 4444

on machine B I type

unison c:\temp\ socket://machineA:4444

it synchronizes all right, but as soon as it is finished, Machine A
complains

Fatal error: Error in waiting on port:
The specified network name is no longer available.
 [read()]

Any suggestions?

Tom
Permalink | Reply | 3 comments |
headers
Zhe Yang | 17 Jun 2002 21:11

Re: Windows 2000 socket problem with 2.9.1


Hi Tom,

This is a known bug and we haven't fixed it yet for the statically linked
Win2K client (built using visual C++).  On the other hand, if you
download the dynamically linked version (built using Cygwin/GNUC), this
problem doesn't exist.  What you need to do is to download
"unison.win32gnuc-*.exe" instead of "unison.win32-*.exe", and install the
following the cygwin and gtk DLLs, as follows:

. (If you haven't done this yet,) to install the cygwin dll, go to
  "www.cygwin.com" and click the link to start.

. Then, to install the gdk, gtk, and glib dlls, download guilib.tar.gz
  from http://www.cis.upenn.edu/~bcpierce/unison/download/resources,
  unpack it and copy the *.dll files into your search path.

Since the dynamically linked version uses the Cygwin API, its behavior is
closer to the Unix version: it understands the symbolic links, and setting
time for read-only files won't fail.  The downside, obviously, is the
somewhat lengthy installation procedure.

Let us know if it works for you.  We are planning to properly document the
cygwin version in the upcoming version.

Cheers,
Zhe

Zhe Yang                                      (O) (+1) (215) 573-5605
University of Pennsylvania                    (H) (+1) (215) 382-4245
(Continue reading)

Permalink | Reply | 1 comment |

Gmane