Indexing Performance Question (was tpop3d vs dovecot)
Nate <nm_list <at> visp.net>
2007-02-01 01:39:46 GMT
Since posting the previous thread we've setup a new system (Opteron
2.0ghz, 1gb ram, Fedora 6) for testing. I am hoping somebody very
familiar with the indexing portion of dovecot can shed some light on
this for me.
After much testing, I've come to one primary conclusion that dovecot
is possibly unnecessarily scanning or reading files within the
maildir directories. Take a mailbox for instance (my test one has
71570 emails in the cur/ folder so the process runs long enough to
get data of what it's doing). Follow these steps:
1. First pop3 connection. With no index files whatsoever, it takes
nearly a full 10 minutes at 100% io load to scan the 71570 files
(748M) directory and build the index files from scratch which look like this:
-rw------- 1 postfix postdrop 839K Feb 1 00:43 dovecot.index
-rw------- 1 postfix postdrop 1.9M Feb 1 00:57 dovecot.index.cache
-rw------- 1 postfix postdrop 560K Feb 1 00:43 dovecot.index.log
-rw------- 1 postfix postdrop 3.1M Feb 1 00:43 dovecot-uidlist
While it's building index i see a lot of these fly by in lsof
pop3 2023 postfix 8r REG 8,3 14425 285947
/var/spool/mail/test.visp.net/natetest/cur/1128488047.V902I7d42e.s3.visp.net:2,
No problem here, the disk can only go as fast as the disk can go and
thats a *lot* of mail. Although, one curious thing here, it appears
that dovecot is reading or at least scanning the entire contents of
the files within the cur/ directory to generate an index. I assume
this by watching my disk cache go from 100MB to 600MB. Is it
necessary to do that or only read the top X lines of each file, is
that even possible?
(Continue reading)