Wesley Craig | 1 Jul 2010 03:16
Picon

Re: Recomendations for a Migration of a Cyrus mailStore with 70K users.

On 30 Jun 2010, at 13:59, Nestor A Diaz wrote:
> another question : what about if a mail is comming from the lmtp  
> socket
> to the refering mailbox and the mailbox is currently in the renaming
> process ?

 From lmtpengine.c:

     case IMAP_MAILBOX_MOVED:
         prot_printf(pout, "451 4.2.1 Mailbox Moved\r\n");
         break;

i.e., a temp fail error.  It's retried.

> what about if the user have a shared folder and other users
> are accessing his mailbox with write privilegies ??

As a general statement, imapd is sensitive to the MOVING condition.   
The specific circumstance would dictate precisely what would happen,  
but there is no known case that would cause writes to be lost.  The  
user agent would either be informed or the server would retry on the  
user agents's behalf.

It might look weird to the user agent, tho, and the user agent might  
pass that weirdness along to the user.

:wes

----
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
(Continue reading)

John Madden | 1 Jul 2010 21:25
Favicon

mupdate cpu, thread timeouts

Lately, I've been seeing a lot of this from imap frontends, repeating 
over and over:

Jul  1 15:15:52 imap mupdate[18206]: unready for connections
Jul  1 15:15:52 imap mupdate[18206]: synchronizing mailbox list with 
master mupdate server
Jul  1 15:16:51 imap mupdate[18204]: Thread timed out waiting for 
listener_lock
Jul  1 15:16:51 imap mupdate[18204]: Worker thread finished, for a total 
of 4 (3 spare)
Jul  1 15:16:51 imap mupdate[18204]: Thread timed out waiting for 
listener_lock
Jul  1 15:16:51 imap mupdate[18204]: Worker thread finished, for a total 
of 3 (3 spare)
Jul  1 15:16:51 imap mupdate[18204]: Thread timed out waiting for 
listener_lock
Jul  1 15:16:51 imap mupdate[18204]: Worker thread finished, for a total 
of 2 (2 spare)
Jul  1 15:16:51 imap mupdate[18206]: mailbox list synchronization complete
Jul  1 15:16:52 imap mupdate[18206]: New worker thread started, for a 
total of 1
Jul  1 15:16:52 imap mupdate[18206]: New worker thread started, for a 
total of 2
Jul  1 15:16:52 imap mupdate[18206]: New worker thread started, for a 
total of 3
Jul  1 15:16:52 imap mupdate[18206]: New worker thread started, for a 
total of 4
Jul  1 15:16:52 imap mupdate[18206]: New worker thread started, for a 
total of 5
Jul  1 15:16:54 imap mupdate[18203]: unready for connections
(Continue reading)

Wesley Craig | 2 Jul 2010 03:04
Picon

Re: mupdate cpu, thread timeouts

On 01 Jul 2010, at 15:25, John Madden wrote:
> Lately, I've been seeing a lot of this from imap frontends, repeating
> over and over:

These:

> Jul  1 15:16:51 imap mupdate[18204]: Thread timed out waiting for
> listener_lock
> Jul  1 15:16:51 imap mupdate[18204]: Worker thread finished, for a  
> total
> of 4 (3 spare)
> Jul  1 15:16:51 imap mupdate[18204]: Thread timed out waiting for
> listener_lock
> Jul  1 15:16:51 imap mupdate[18204]: Worker thread finished, for a  
> total
> of 3 (3 spare)
> Jul  1 15:16:51 imap mupdate[18204]: Thread timed out waiting for
> listener_lock
> Jul  1 15:16:51 imap mupdate[18204]: Worker thread finished, for a  
> total
> of 2 (2 spare)

> Jul  1 15:16:52 imap mupdate[18206]: New worker thread started, for a
> total of 1
> Jul  1 15:16:52 imap mupdate[18206]: New worker thread started, for a
> total of 2
> Jul  1 15:16:52 imap mupdate[18206]: New worker thread started, for a
> total of 3
> Jul  1 15:16:52 imap mupdate[18206]: New worker thread started, for a
> total of 4
(Continue reading)

Picon
Favicon

Re: postfix lmtp instances

On Thu, 17 Jun 2010, Rudy Gevaert wrote:
> In the past I made an lmtp transport in my postfix master.cf for each 
> backend we had.  As we are going to a lot of backends > 15 instead of 7 
> it will be a little burden to keep the master.cf in sync.
> 
> It would be a lot easier to just use lmtp:hostname.backend.com instead 
> of dedicated lmtp transport.

Or run a frontend in the postfix box just for lmtp, maybe?

--

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh
----
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Picon
Favicon

Re: very long cyr_expire at startup and no mail delivery

On Fri, 25 Jun 2010, Marcus wrote:
> > Yeah - that.  Why on earth is cyr_expire in the startup?  EVENTS is the
> > right place ( actually, we run it out of cron, but that's beside the point )

Because it was like that in the example CMU docs in the start of the
millenium...  We certainly didn't change it much since 2.1 days, so that
config file was added in 2002 I think, maybe even earlier.

BTW, while looking for the source of the example cyrus.conf from 10 years
ago (couldn't find it), I noticed the manpages still haven't documented
cyrus master service options "babysit" and "maxforkrate".

> I thought delprune at startup is standard and therefore I was wondering
> that it takes so long and more evil the service is down while cleaning
> up deliver.db. The debian/lenny cyrus.conf looks like this for the START
> and EVENTS section. So disable delprune in the START section will solve
> that problem? May be a good idea the post this to the debian Wishlist
> for the cyrus package.

Please file the wishlist bug to remind us to update that conf file for the
2.3 packages.

--

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh
----
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
(Continue reading)

Picon
Favicon

Re: migrating Cyrus IMAPd from i386 to amd64 system

On Fri, 25 Jun 2010, Jukka Salmi wrote:
> Since I can afford some hours of mail system downtime I'd prefer to copy
> over a compressed archive containing all the mailboxes, extract it on
> the new system and recursively run reconstruct(8) for all mailboxes;
> then I'd probably copy over the contents of $configdirectory/user/ and
> start Cyrus IMAPd.  Does this make sense?  And what data will I be
> missing? ;-)
> 
> Some basic questions:
> 
> - Is Cyrus on amd64 able to read skiplist files created by Cyrus on
>   i386?  (If not, I'd probably have to use cvt_cyrusdb(8) to dump and
>   restore the relevant files.)

Yes, it is.  Heck, it managed to read (and upgrade in-place, I suppose)
skiplist files from Cyrus 2.1 just fine...

> - What about the cyrus.{cache,header,index,squat} files?  Can I expect
>   Cyrus on amd64 to be able to read them just fine?  (This seems not to
>   be that important since those files are easily recreated IIUC.)

Yes, if they're skiplist.  I wouldn't trust that to work if it is berkeley
DB.

Be careful with the mailbox list, though.  Better dump it to the portable
text file format, update any missing ACL rights, and restore it in the new
box.

--

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
(Continue reading)

John Madden | 2 Jul 2010 15:29
Favicon

Re: mupdate cpu, thread timeouts

>> Jul  1 15:16:51 imap mupdate[18204]: Thread timed out waiting for
>> listener_lock
>> Jul  1 15:16:51 imap mupdate[18204]: Worker thread finished, for a
>> total
>> of 2 (2 spare)
>> Jul  1 15:16:52 imap mupdate[18206]: New worker thread started, for a
>> total of 3
>> Jul  1 15:16:52 imap mupdate[18206]: New worker thread started, for a
>> total of 4
>> Jul  1 15:16:52 imap mupdate[18206]: New worker thread started, for a
>> total of 5
>
> are not interesting.  I might say spurious.  They reflect that the
> mupdate on this machine starts 5 worker threads, and needs none.  These:

Oh, I only threw in the "new worker thread" messages for completeness. 
I'm concerned about the listener_lock timeouts.  mupdate is consuming an 
entire cpu at that these points in time so I assume it's a cpu usage 
thing.  (Also odd is that despite being multi-threaded, it never lands 
on the machine's second cpu.)

>> Jul  1 15:16:54 imap mupdate[18203]: unready for connections
>> Jul  1 15:16:54 imap mupdate[18203]: synchronizing mailbox list with
>> master mupdate server
>
> are the interesting messages.  It says to me that the connection to
> the mupdate master is being lost.  However, there ought to be an
> error message to that effect, which I don't see.  What's happening on
> the mupdate master?

(Continue reading)

D G Teed | 2 Jul 2010 19:11
Picon

Authentication problems since Redhat 5.5 updates

We had a nice trouble free cyrus running until it was updated
with updates from Redhat today.

I've tested with testsaslauthd (no service name given) and it works OK,
so I'd think things are fine on the pam, AD and ldap end.

In the cyrus server's maillog I'm seeing messages like this
from attempts to connect from the remote webmail:

Jul  2 13:54:22 navi imap[4073]: badlogin: webmail.example.com [XXX.YYY.ZZZ.111] CRAM-MD5 [SASL(-13): user not found: no secret in database]

Logins from some other IMAP, like my thunderbird, using a secure IMAP port, work OK.

cyradm can connect, but scripts we have, using IMAP::Admin have stopped working.

# cyrsetquota dteed 100
IMAP::Admin [ initialize ]: try NO Login failed: authentication failure

This is cyrus 2.3.7 from Redhat, identifying as:

name       : Cyrus IMAPD
version    : v2.3.7-Invoca-RPM-2.3.7-7.el5_4.3 2006/07/10 13:46:20
vendor     : Project Cyrus
support-url: http://asg.web.cmu.edu/cyrus
os         : Linux
os-version : 2.6.18-194.8.1.el5
environment: Built w/Cyrus SASL 2.1.22
             Running w/Cyrus SASL 2.1.22
             Built w/Sleepycat Software: Berkeley DB 4.3.29: (February 19, 2009)
             Running w/Sleepycat Software: Berkeley DB 4.3.29: (February 19, 2009)
             Built w/OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
             Running w/OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
             CMU Sieve 2.3
             TCP Wrappers
             NET-SNMP
             mmap = shared
             lock = fcntl
             nonblock = fcntl
             idle = idled

These packages were updated by Redhat related to sasl:

Jul 02 10:36:41 Updated: cyrus-sasl-lib-2.1.22-5.el5_4.3.i386
Jul 02 10:37:11 Updated: cyrus-sasl-plain-2.1.22-5.el5_4.3.i386
Jul 02 10:37:44 Installed: cyrus-sasl-md5-2.1.22-5.el5_4.3.i386
Jul 02 10:38:01 Updated: cyrus-sasl-2.1.22-5.el5_4.3.i386

I tried removing cyrus-sasl-md5 and restarting saslauthd but it did not help.

There has to be something silly getting in the way but what?

--Donald

----
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
D G Teed | 2 Jul 2010 19:43
Picon

Re: Benachrichtung zum Übermittlungsstatus (Fehlgeschlagen)



2010/7/2 D G Teed <donald.teed <at> gmail.com>


Subject: Authentication problems since Redhat 5.5 updates

We had a nice trouble free cyrus running until it was updated
with updates from Redhat today.

I've tested with testsaslauthd (no service name given) and it works OK,
so I'd think things are fine on the pam, AD and ldap end.

In the cyrus server's maillog I'm seeing messages like this
from attempts to connect from the remote webmail:

Jul  2 13:54:22 navi imap[4073]: badlogin: webmail.example.com [XXX.YYY.ZZZ.111] CRAM-MD5 [SASL(-13): user not found: no secret in database]

Logins from some other IMAP, like my thunderbird, using a secure IMAP port, work OK.

cyradm can connect, but scripts we have, using IMAP::Admin have stopped working.

# cyrsetquota dteed 100
IMAP::Admin [ initialize ]: try NO Login failed: authentication failure

This is cyrus 2.3.7 from Redhat, identifying as:

name       : Cyrus IMAPD
version    : v2.3.7-Invoca-RPM-2.3.7-7.el5_4.3 2006/07/10 13:46:20
vendor     : Project Cyrus
support-url: http://asg.web.cmu.edu/cyrus
os         : Linux
os-version : 2.6.18-194.8.1.el5
environment: Built w/Cyrus SASL 2.1.22
             Running w/Cyrus SASL 2.1.22
             Built w/Sleepycat Software: Berkeley DB 4.3.29: (February 19, 2009)
             Running w/Sleepycat Software: Berkeley DB 4.3.29: (February 19, 2009)
             Built w/OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
             Running w/OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
             CMU Sieve 2.3
             TCP Wrappers
             NET-SNMP
             mmap = shared
             lock = fcntl
             nonblock = fcntl
             idle = idled

These packages were updated by Redhat related to sasl:

Jul 02 10:36:41 Updated: cyrus-sasl-lib-2.1.22-5.el5_4.3.i386
Jul 02 10:37:11 Updated: cyrus-sasl-plain-2.1.22-5.el5_4.3.i386
Jul 02 10:37:44 Installed: cyrus-sasl-md5-2.1.22-5.el5_4.3.i386
Jul 02 10:38:01 Updated: cyrus-sasl-2.1.22-5.el5_4.3.i386

I tried removing cyrus-sasl-md5 and restarting saslauthd but it did not help.

There has to be something silly getting in the way but what?

--Donald


I have things working again.  I had disabled Unix authentication in pam
temporarily to try troubleshooting with my account.  That had the side effect
of disabling the cyrus user from authentication.  So that explains the
scripts using IMAP::Admin breaking.

I also removed the package cyrus-sasl-md5 and I believe this has
an impact on the issue I was facing with "CRAM-MD5".

Any tips on how to co-exist with that package are welcomed.

--Donald
 

----
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Jukka Salmi | 2 Jul 2010 20:21
Picon

Re: migrating Cyrus IMAPd from i386 to amd64 system

Jukka Salmi --> info-cyrus (2010-06-25 02:30:49 +0200):
> Hi
> 
> I'm currently running Cyrus IMAPd 2.2.13p1 on NetBSD/i386 without any
> problems.  During the next few days I need to move the users' mailboxes
> to a NetBSD/amd64 system which will run Cyrus IMAPd 2.3.16.  Some hints
> about this would be appreciated.
> 
> The cleanest way would probably be to get Cyrus IMAPd running on the new
> system and then to copy the mailboxes via IMAP (using imapsync(1) or
> similar); but since the upstream on the old system is very slow I'd like
> to avoid this approach if possible.
> 
> Since I can afford some hours of mail system downtime I'd prefer to copy
> over a compressed archive containing all the mailboxes, extract it on
> the new system and recursively run reconstruct(8) for all mailboxes;
> then I'd probably copy over the contents of $configdirectory/user/ and
> start Cyrus IMAPd.  Does this make sense?  And what data will I be
> missing? ;-)
> 
> Some basic questions:
> 
> - Is Cyrus on amd64 able to read skiplist files created by Cyrus on
>   i386?  (If not, I'd probably have to use cvt_cyrusdb(8) to dump and
>   restore the relevant files.)
> 
> - What about the cyrus.{cache,header,index,squat} files?  Can I expect
>   Cyrus on amd64 to be able to read them just fine?  (This seems not to
>   be that important since those files are easily recreated IIUC.)
> 
> 
> TIA, Jukka

Thanks for all the hints I got on this.

Just copying $configdirectory, $partition-default and $sievedir from the
old server to the new one and starting Cyrus there did -- as expected --
not work because of Berkeley DB:

  ctl_cyrusdb[23411]: DBERROR db4: Build signature doesn't match environment

I didn't even try to fix the problem by dumping and restoreing the DB,
but instead removed the copied directories again and did the following:

  - added users to SASLdb (saslpasswd2 -c ...)
  - started cyrus (init script created directories and called mkimap)
  - created users' INBOXes (cyradm)
  - stopped cyrus
  - copied $configdirectory/user, $partition-default/user and $sievedir
    from old server
  - started cyrus
  - for user in $(ls $partition-default/user); do
        reconstruct -f user.$user
    done
  - manually adjusted server and mailbox annotations, user quota and
    ACLs (cyradm)

This resulted in a working Cyrus IMAPd 2.3.16 on the new amd64 system.

However, after a while I started to see the following errors in the
logs:

  ctl_cyrusdb[2085]: DBERROR: error listing log files: No such file or directory
  ctl_cyrusdb[2085]: DBERROR: archive /var/imap/db: cyrusdb error

This seems to be triggered by the `checkpoint' entry from the EVENTS
section of cyrus.conf:

  EVENTS {
    # this is required
    checkpoint    cmd="ctl_cyrusdb -c" period=30

However, running `ctl_cyrusdb -c' manually (as user cyrus...) worked
fine

  ctl_cyrusdb[3296]: checkpointing cyrus databases
  ctl_cyrusdb[3296]: done checkpointing cyrus databases

and since then, I haven't seen the problem occur anymore.  Hmm...

Regards, Jukka

--

-- 
This email fills a much-needed gap in the archives.
----
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html


Gmane