replication features for remote server locations

Hello' i've a question about cyrus replication system,
i need to syncronize "real time"  3 imap servers  (mailbox and
relative mail flags etc etc) in different geographic locations.
Every server have the same users.
Is it possible using replication system to do it?
Is it possible to consulting mail over one of these servers (not
concurrently by the same user) or the slave server are only
backup (or disaster recovery) servers in replication solution?
Thank's a lot.

--

-- 
Rispetta l'ambiente: se non ti è necessario, non stampare questa mail.
******************************************
Ing. Matteo Cazzador
Email: mcazzador <at> gmail.com
******************************************
----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: replication features for remote server locations

On Wed, Feb 01, 2012 at 05:59:09PM +0100, Matteo Cazzador wrote:
> Hello' i've a question about cyrus replication system,
> i need to syncronize "real time"  3 imap servers  (mailbox and
> relative mail flags etc etc) in different geographic locations.
> Every server have the same users.
> Is it possible using replication system to do it?
> Is it possible to consulting mail over one of these servers (not
> concurrently by the same user) or the slave server are only
> backup (or disaster recovery) servers in replication solution?
> Thank's a lot.

Sorry we didn't respond to your earlier message.

This is possible, but it is unsafe to be fully multi-master - each
user should only be logging in at one of the locations.  And full
3 location multi-master is not well tested.  There are caveats.
We ran like that at FastMail for a while - but there were edge cases
where some of our "fixup" tools could confuse it and cause the
mailbox to fill up with multiple copies of the mailbox, or a delete
to get replicated the wrong way because a mailbox hadn't been created
yet.

In short - you need to know what you're doing and be careful right
now, but it is something we plan to support more robustly.

Bron.
----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: Problem with folder subscriptions and LIST/LSUB

Hello all,

Okay, I now realize this probably is a known issue:

  https://bugzilla.cyrusimap.org/show_bug.cgi?id=3628

I don't ordinarily try to help with these kinds of bugs, because I don't
know enough about imap or the code base to be much more than annoying.
So let me cut to the chase.

1. My wife has our tax information in folders with numbers as names.
2. Those folders cannot currently be subscribed.
3. The improved sorting change did not appear to work for me.
4. If I can't provide her access to these folders, she will not be
   happy.
5. When the Mama ain't happy, ain't nobody happy.

Therefore,

6. I am willing to pay $50 US to get a fix or workaround for this bug.

I realize I could probably dork around with dumping and reloading
databases, copying email, renaming email folders, and reconstructing the
indexes. However, the last time I did that kind of thing (after an
upgrade), I think I fat fingered the steps and screwed up the wife's
seen mail status and priority labels. This has made me a little hesitant
about trying this again without someone helping who is more familiar
with what's involved.

If anyone can provide a work-around patch to close bug 3628, or even a

Re: Problem with folder subscriptions and LIST/LSUB

Quick workaround (assuming that you have root access to the server):

1) using your mail client, create a new folder named newfolder.

2) log in to your server and from a root shell, su to your cyrus user.

3). Navigate the filesystem and cp all the mail files from the directory with the funky name that Cyrus won't
list to newfolder.

4) reconstruct newfolder.

Hth,

Dave

On Feb 1, 2012, at 8:32 PM, "Anthony L. Awtrey" <tony <at> awtrey.com> wrote:

> Hello all,
> 
> Okay, I now realize this probably is a known issue:
> 
>  https://bugzilla.cyrusimap.org/show_bug.cgi?id=3628
> 
> I don't ordinarily try to help with these kinds of bugs, because I don't
> know enough about imap or the code base to be much more than annoying.
> So let me cut to the chase.
> 
> 1. My wife has our tax information in folders with numbers as names.
> 2. Those folders cannot currently be subscribed.
> 3. The improved sorting change did not appear to work for me.

Re: Problem with folder subscriptions and LIST/LSUB

On 02/01/2012 08:47 PM, Dave McMurtrie wrote:
> Quick workaround (assuming that you have root access to the server):
> 
> 1) using your mail client, create a new folder named newfolder.
> 
> 2) log in to your server and from a root shell, su to your cyrus user.
> 
> 3). Navigate the filesystem and cp all the mail files from the directory with the funky name that Cyrus
won't list to newfolder.
> 
> 4) reconstruct newfolder.
> 
> Hth,
> 
> Dave
> 

Thanks Dave, I'll give it a shot.

T
--

-- 
Anthony L. Awtrey

http://awtrey.com/
http://orlandotenor.com/
----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Problem recover replica

----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

RE: Problem recover replica

----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: Problem recover replica

Hi,

Quoting Manel Gimeno Zaragozá <magiza83 <at> hotmail.com>:

>
> Hello,
>
> I'm trying to implement a master-replica cyrus environment and I'm  
> getting some issue that I'm not able to resolve. The scenario is the  
> following:
>
> Everything is working fine between Master & Replica until Replica  
> goes down (maintenance or whatever). Once is down, in master there  
> are still connections and modification. If I bring up Replica the  
> changes made during the "blackout" are no fully transmitted to it.  
> Some of the are transmitted, but not all.
>
> Which is the best way to resincronize the missing parts to Replica?
> I was wordering to rsync imap folders (/var/spool/imap &  
> /var/lib/imap), and It works fine in my test environment, but in  
> produccion it could be GB to sincronize and I think this is not  
> optimum.
> Other option was "sync_client -l -u <all_user>" but I'm guessing  
> that in production I will have the same problem due to the amount of  
> data to sincronize.
>

If you have configured sync_log: 1 cyrus will keek an actionlog for
rolling replication. While "sync_client -r" is not running, or unable
to connect to the sync_server this file keeps track of what has to be
synced. Once sync_client connects to the server it will catch up.

--------------------------------------------------------------------------------

M.Menge                                Tel.: (49) 7071/29-70316
Universität Tübingen                   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung          mail:  
michael.menge <at> zdv.uni-tuebingen.de
Wächterstraße 76
72074 Tübingen

----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Re: Problem recover replica

On Thu, Feb 02, 2012 at 01:11:24PM +0100, Manel Gimeno Zaragozá wrote:

What version?

> Everything is working fine between Master & Replica until Replica goes down (maintenance or whatever).
Once is down, in master there are still connections and modification. If I bring up Replica the changes
made during the "blackout" are no fully transmitted to it. Some of the are transmitted, but not all.
> 
> Which is the best way to resincronize the missing parts to Replica? 

sync_client -r -f $file for each file in the conf/sync/ directory.

> I was wordering to rsync imap folders (/var/spool/imap & /var/lib/imap), and It works fine in my test
environment, but in produccion it could be GB to sincronize and I think this is not optimum.

No, that's awful.  Don't do that.

> Other option was "sync_client -l -u <all_user>" but I'm guessing that in production I will have the same
problem due to the amount of data to sincronize.

It's not too bad.  We run it occasionally if we suspect things have
got confused for other reasons.

> On the other hand, I have also doubts about how "sync_client -r" and "sync_server" are initiated. In my
test machines I should run manually them each time I start cyrus-imapd, I though that it could be automatic
one I start the service cyrus-imapd, but it is not.

I have attached the script that we run.  It has lots of hooks into
our own systems of course.

I would LIKE to make something more generic that's part of Cyrus
itself, rather than having external tooling.  Of course, our external
tooling is a bit special-cased as well.  But at least having a basic
"keep trying to replicate all the records" would be good.

Bron.

----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

RE: Problem recover replica

Quoting Manel Gimeno Zaragozá <magiza83 <at> hotmail.com>:

>
> Hello,
>
> I've noticed that the "missing changes" are due to some of the  
> actions done during the "blackout" are processed un  
> "/var/lib/imap/sync" log and named modified to log-PID.
> It seems that rsync_client have not noticed that Replica is down and  
> it process some of them. Once it notice that replica is down  
> continue login in "log" file. When Replica comes back, sync_client  
> process the changes from "log" without problems, but we have miss  
> some changes.
>
the sync_client in rolling replication locks for the log file and moves
it to log-PID. Then processes the actions in log-PID, deletes this file
and looks for a new log file.

If sync_client can't connect the sync_server it will wait some time
and trying again. If there is still the sync_client process with the
PID, it should catch up. If the sync_client died you can process the
log-PID file with sync_client -r -f log-PID .

--------------------------------------------------------------------------------

M.Menge                                Tel.: (49) 7071/29-70316
Universität Tübingen                   Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung          mail:  
michael.menge <at> zdv.uni-tuebingen.de
Wächterstraße 76
72074 Tübingen

----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/