Charles Marcus | 1 Apr 01:09 2007

Re: SQL mail storage - yes yes yes!

On 3/31/2007 Stephen Lee wrote:
> WRT size of the db, what about keeping just the message headers and 
> indices in the db and the body as a file? This is akin to some
> content management systems where the file info is in a db and the
> content resides as a file. Don't know what the logistics and
> performance issues are though. Clearly SQL mail storage is not for
> everyone.

One of the biggest advantages I can think of is single instance storage.

Break the message into its constituent parts (headers, body, 
attachments), and now you can forget about the idiots that send a 15MB 
attachment to everyone in the company - resulting in 523 copies of that 
attachment in the mail system - with SQL storage (done right) - you only 
have one copy - 15MB instead of 523 x 15MB.

Timo? Can you elaborate on how thi sis implemented? Is the message 
broken up into parts? How hard would it be to implement single instance 
storage if an SQL db is used for the storage?

Thanks for all your efforts!

Charles

Jens Knoell | 1 Apr 02:04 2007
Face

Re: SQL mail storage - yes yes yes!


Charles Marcus wrote:
> On 3/31/2007 Stephen Lee wrote:
>> WRT size of the db, what about keeping just the message headers and 
>> indices in the db and the body as a file? This is akin to some
>> content management systems where the file info is in a db and the
>> content resides as a file. Don't know what the logistics and
>> performance issues are though. Clearly SQL mail storage is not for
>> everyone.
>
> One of the biggest advantages I can think of is single instance storage.
>
> Break the message into its constituent parts (headers, body, 
> attachments), and now you can forget about the idiots that send a 15MB 
> attachment to everyone in the company - resulting in 523 copies of 
> that attachment in the mail system - with SQL storage (done right) - 
> you only have one copy - 15MB instead of 523 x 15MB.
>
> Timo? Can you elaborate on how thi sis implemented? Is the message 
> broken up into parts? How hard would it be to implement single 
> instance storage if an SQL db is used for the storage?
>
> Thanks for all your efforts!
>
> Charles
That can be done without SQL as well as long as you're on a *nix system: 
with hard links. One of the reasons that's not usually done is that most 
MTAs split up the message in one copy per recipient before handing them 
to the LDA. At least I'm relatively sure that the splitting happens at 
MTA level.
(Continue reading)

Jeff Rice | 1 Apr 04:16 2007

1.0rc29: Corrupted index cache file


Hi,
I have recently migrated from Courier to Dovecot, and I noticed a 
problem in one of my spam scripts.  Here's the setup:

Spam that needs to be retrained by the filter is placed in a maildir 
from the IMAP client.  A cron job scans that maildir and if it finds a 
message, feeds it to the spam filter and removes it from the training 
maildir.  Depending on whether or not the message was a spam or ham, it 
is either deleted or delivered to Inbox.

It appears I am getting an error when the script deletes a mail message. 
  I haven't checked if this message only occurs when a client is 
connected -- it's possible.

Log excerpt:
Mar 31 18:34:57 finity dovecot: IMAP(XXXXXXX <at> finity.org): Maildir 
/home/vmail/XXXXXXX <at> finity.org/.Spam.Unsure sync: UID inserted in the 
middle of mailbox (8 > 3, file = msg.FHoB:2,)
Mar 31 18:34:57 finity dovecot: IMAP(XXXXXXX <at> finity.org): Disconnected: 
Mailbox is in inconsistent state, please relogin.
Mar 31 18:35:36 finity dovecot: IMAP(XXXXXXX <at> finity.org): Corrupted 
index cache file 
/home/vmail/XXXXXXX <at> finity.org/.Spam.Unsure/dovecot.index.cache: indexid 
changed

Jeff

Ejay Hire | 1 Apr 04:24 2007

Re: SQL mail storage

hi.

I'm sure this is on hold in your head, but just in case.. Can this be
V2.0 minimum, please.

Thanks,
Ejay

On Sat, 2007-03-31 at 17:34 +0300, Timo Sirainen wrote:
> I wasted some time yesterday and today implementing a SQL storage
> plugin. It seems to be working, but:
> 
>  - Saving new messages is done in a regular INSERT statement, which is
> bad. PostgreSQL has at least this COPY TO command which could be used
> instead.
>  - It breaks in stress testing
>  - It's not that well optimized. Especially it could support caching
> some commonly requested fields (same as dovecot.index.cache file).
>  - Currently works only with PostgreSQL.
> 
> Nothing is committed to CVS yet. I'm not sure if I should even do that.
> I'm not sure if those lib-sql API changes were that great. Maybe there's
> a better way..
> 
> If you're interested in trying it, you need the latest CVS HEAD sources
> and these patches:
> 
> http://dovecot.org/patches/sql-storage-changes.diff
> http://dovecot.org/patches/sql-storage-plugin.diff
> 
(Continue reading)

Timo Sirainen | 1 Apr 07:57 2007
Picon
Picon

Re: 1.0rc29: Corrupted index cache file

On 1.4.2007, at 5.16, Jeff Rice wrote:

> Mar 31 18:34:57 finity dovecot: IMAP(XXXXXXX <at> finity.org): Maildir / 
> home/vmail/XXXXXXX <at> finity.org/.Spam.Unsure sync: UID inserted in  
> the middle of mailbox (8 > 3, file = msg.FHoB:2,)

http://wiki.dovecot.org/MailboxFormat/Maildir -> Procmail problems

Timo Sirainen | 1 Apr 08:03 2007
Picon
Picon

Re: SQL mail storage - yes yes yes!

On 1.4.2007, at 2.09, Charles Marcus wrote:

> One of the biggest advantages I can think of is single instance  
> storage.

I was planning on implementing that for dbox also.

> Timo? Can you elaborate on how thi sis implemented? Is the message  
> broken up into parts? How hard would it be to implement single  
> instance storage if an SQL db is used for the storage?

I guess that could be done also. Currently it stores headers and body  
in separate fields but in one row.

If this is implemented I think most of the code is going to be  
generic, so if it's implemented to one storage it could somewhat  
easily be implemented to all of them. Of course not in any standard  
way to mbox/maildir.
Timo Sirainen | 1 Apr 08:35 2007
Picon
Picon

Re: who owns dovecot files and dirs?

On 29.3.2007, at 22.50, Stewart Dean wrote:

> ...and what permissions should they have.  I am thinking of /var/ 
> run/dovecot and the index directory.
> What ownership, group and permissions should they be?  Are there  
> any other files/dirs created for dovecot alone (not the mail  
> folders and INBOXes); if so, how should they be owned and permed?
> I had thought they were to be owned by dovecot, but it turns out  
> that they should not

Dovecot opens pretty much all the configuration etc. files as root  
before dropping the privileges. So in general they could all be 0600  
owned by root. I don't think you should worry about that though. /var/ 
run/dovecot usually gets deleted at boot and Dovecot recreates them,  
so whatever permission changes you do to it they'll get erased  
anyway. /var/lib/dovecot then is currently created 0700, but it only  
contains ssl-parameters.dat which is public data.
Timo Sirainen | 1 Apr 08:48 2007
Picon
Picon

Re: locking question

On 29.3.2007, at 21.30, Stewart Dean wrote:

> There are three applications that have their mitts on files on my  
> mail server, which is running AIXV5.3 and UWIMAP and mbox format.   
> The mail folders and INBOXES are native to that machine, but also  
> are NFS exported to a login server and a mailing list server.  All  
> three machines are running the lockd daemon.
>
> Everybody wants to lock differently
> 1) procmail (delivering for sendmail), which seems to want to use  
> dotlocking, fcntl and lockf locking; for whatever reasons, the  
> compile time tests seem to disallow flock.
> 2) UWIMAP which according to wiki.dovecot.org/Migration/UW uses:
> mbox_read_locks = flock
> mbox_write_locks = dotlock flock
> 3) I want to run dovecot in the same environment as I switch over,  
> for which the locking strategy is supposed to be (according to  
> http://wiki.dovecot.org/MboxLocking)
> mbox_read_locks = fcntl
> mbox_write_locks = dotlock fcntl
>
> Since that same page in the wiki says. "*It's important that all  
> software that's reading or writing to mboxes use the same locking  
> settings.",* I had recompiled procmail so it only usedotlocking and  
> fcntl (thus removing lockf in addition to the disallowed  
> flock)...but now I bumped into the Migration/UW page and there  
> looks to be a conflict.

It doesn't actually hurt to use more locking mechanisms, as long as  
they all use at least one common and they're not in different order.
(Continue reading)

Timo Sirainen | 1 Apr 08:51 2007
Picon
Picon

Re: [ rc28 ] dict{} seems to be ignored

On 28.3.2007, at 11.38, Emiliano Gabrielli (aka AlberT) wrote:

>> If you still can't get it to work, set mail_debug=yes and see if
>> there's anything interesting in the logs.
>
> already done .. nothing and nothing ... I can see the login SQL  
> queries ..
> I can see the quota fetched from the userdb_quota field ... but  
> nothing about
> the quota update on the mysql storage ... dovecot updatets the  
> maildir quota
> on the filesystem ...

If it updates maildir quota, it sees quota=maildir setting somewhere.  
So since it's not in dovecot.conf, apparently it gets that from  
userdb's user_query?

Timo Sirainen | 1 Apr 08:57 2007
Picon
Picon

Re: imaptest10 and stalled messages

On 30.3.2007, at 21.24, Mike Brudenell wrote:

> 2.  How much real-life user load does running imaptest10 with 50  
> simulated
>     clients equate to?  I assume each simulated user is hammering  
> away at
>     its IMAP connection, so should equate to several (how many?)  
> normal
>     users in real-life operation?

I don't know, but most IMAP connections are just idling, so it could  
be anything from 10-100x.

> 3.  I'm concerned by the N/M number at the end of the imaptest10  
> output
>     lines plummeting whenever one process goes into this stalled  
> state:
>     it almost suggests as if the only thing the other processes can  
> do is
>     logout?  Are other sessions really being blocked, or is it just
>     imaptest10 behaving like this?

It's just imaptest. If it detects one connection stalling, it stops  
making new connections to the server until the stall goes away.

That's however a good way to detect what it is that's stalling. Just  
wait for a while until there are only stalled connections, and you  
can truss the imap connections to see what they're doing. Are they  
waiting for some locks?

(Continue reading)


Gmane