headers
Thomas Orgis | 4 Mar 2012 14:15
Favicon

Performance with large-ish Mailbox

Hi,

I just set up NOCC as a simple means to web mail access. So far, it's working, which puts it ahead of the
competition that I tried to get running. I like the approach, and I could be happy. But of course I did not
subscribe to the list just to paise (but still, thanks for providing such a tool;-).

My point: I got a mailbox that contains 7 days of mail; the regular POP3 client deletes stuff after a week. In
the world with SPAM (which I filter client-side), this amounts to about 700 mails currently. With that
amount, NOCC running on the same machine as the POP3 server, NOCC takes _very_ much time for any action with
that account.

I have to wonder: Could it be that it grabs a full copy of the mailbox on each run? My normal POP3 client first
fetches an index (which contains message IDs, I suppose), matches that against a local cache and only
transfers the messages that are new. This is very fast, over the internet. Now, getting a listing of POP3
messages and fetching 25 (or 50, whatever the setting for one page is) of them should be much faster than
what is happening. Also, since NOCC also takes very much time on things that are unrelated to the actual
messages (Preferences), I have the suspicion that the full mailbox is loaded on each run, even if it's not needed.

Are my assumptions true here? In that case, performance would be easy to get within reasonable bounds,
unless the API used for POP3 access only allows all or nothing. I don't have PHP experience, so I rather ask
here before haplessly digging around in the code and reading API docs that may or may not be adequate.

I appreciate any pointers...

Alrighty then,

Thomas

PS: Hacking would perhaps be a bit easier for me if you didn't move from Perl to begin with;-)
(Continue reading)

Permalink | Reply | 19 comments |
headers
Tim Gerundt | 5 Mar 2012 14:34
Picon
Gravatar

Re: Performance with large-ish Mailbox

Hi Thomas!

> I have to wonder: Could it be that it grabs a full copy of the
> mailbox on each run?

I think not. The function inbox() from the file "utils\functions.php" 
read only details from the mails NOCC are shown at the moment (25 mails 
per page at default). But to be true, I never touched all of this code, 
since I am part of the project.

I never tested NOCC which such a big mailbox, but I have no problems 
with 250-300 mails in my mailbox. Maybe the problem is somewhere else?

You can change "define('NOCC_DEBUG_LEVEL', 0);" to 
"define('NOCC_DEBUG_LEVEL', 1);" in common.php to see the the time and 
memory NOCC need to render the page.

> PS: Hacking would perhaps be a bit easier for me if you didn't move
> from Perl to begin with;-)

And I had the problem, if it still where writen in Perl. *fg*

But PHP is a good language for websites, so give it a try! ;-)

Greetings,
Tim

------------------------------------------------------------------------------
Try before you buy = See our experts in action!
The most comprehensive online learning library for Microsoft developers
(Continue reading)

Permalink | Reply | 0 comments |
headers
Tim Gerundt | 6 Mar 2012 12:18
Picon
Gravatar

Re: Performance with large-ish Mailbox

Hi Thomas!

> I have to wonder: Could it be that it grabs a full copy of the
> mailbox on each run?

I tested yesterday a little more and you have maybe right. With my 
mailbox (200-300 mails) I thought I had no problems, but my NOCC was 
configured to use IMAP. After I switch to POP3, every page call needs 
much longer. (Over 15 seconds, instead 1-2 seconds with IMAP).

The problem is now, that the function imap_open() 
<http://php.net/manual/en/function.imap-open.php> need so much time. 
Maybe the function parse the whole mailbox if we use a POP3 connection? 
The same mailbox with a IMAP connection need only 1-2 seconds. Since it 
is a own PHP function, I can't make it faster and we need the function 
on every page call. :-(

Can you test it with a IMAP connection?

Greetings,
Tim

------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
Permalink | Reply | 17 comments |
headers
Thomas Orgis | 6 Mar 2012 13:44
Favicon

Re: Performance with large-ish Mailbox

Am Tue, 06 Mar 2012 12:18:37 +0100
schrieb Tim Gerundt <tim <at> gerundt.de>: 

> I tested yesterday a little more and you have maybe right. 

Thanks for testing that...

> The problem is now, that the function imap_open() 
> <http://php.net/manual/en/function.imap-open.php> need so much time. 
> Maybe the function parse the whole mailbox if we use a POP3 connection?

I have that suspicion... but it doesn't make sense: With IMAP, you scan headers only, there should be no
reason not to do that with POP3.

> The same mailbox with a IMAP connection need only 1-2 seconds. Since it 
> is a own PHP function, I can't make it faster and we need the function 
> on every page call. :-(

Well, it's Open Source, isn't it? So we can at least dig around there. And that explains why also other
operations (editing preferences) take so long: the initial imap_open() is called every time, right?

Do you have contact (are subscribed to a list or something) to PHP developers to ask about this
(mis)behaviour of imap_open()? Really, it should not be much slower to get the index of the POP3 inbox on
the very same server than it is on my client over the intertubes:-/

> Can you test it with a IMAP connection?

Sadly, the whole point of using NOCC is that I don't have to hook up a IMAP server up to my mail setup. If I had
IMAP, I'd have installed Squirrelmail and probably not looked further.
(Continue reading)

Permalink | Reply | 16 comments |
headers
Tim Gerundt | 6 Mar 2012 16:46
Picon
Gravatar

Re: Performance with large-ish Mailbox

Hi Thomas!

> Well, it's Open Source, isn't it? So we can at least dig around
> there. And that explains why also other operations (editing
> preferences) take so long: the initial imap_open() is called every
> time, right?

Yes, imap_open() is called every time at the beginning of "action.php". 
You are true, for editing preferences we don't need it, but all other 
actions need it.

> Do you have contact (are subscribed to a list or something) to PHP
> developers to ask about this (mis)behaviour of imap_open()?

No, I have no contact to PHP developers.

> Sadly, the whole point of using NOCC is that I don't have to hook up
> a IMAP server up to my mail setup. If I had IMAP, I'd have installed
> Squirrelmail and probably not looked further.

In this case I would recommend Roundcube <http://roundcube.net/>. This 
open-source webmailer is so cool! ;) But unfortunately it don't support 
POP3 either.

> PS: Do you rely on extra CC for list postings?

Sorry, I just hit "Reply all".

Greetings,
Tim
(Continue reading)

Permalink | Reply | 15 comments |
headers
Thomas Orgis | 6 Mar 2012 17:36
Favicon

Re: Performance with large-ish Mailbox

Hi again,

Am Tue, 06 Mar 2012 16:46:55 +0100
schrieb Tim Gerundt <tim <at> gerundt.de>: 

> Yes, imap_open() is called every time at the beginning of "action.php". 
> You are true, for editing preferences we don't need it, but all other 
> actions need it.

Fine then. Perhaps I can squeeze in a look around that function to get to the point... a quick search brings up
something related:

http://www.maiamailguard.com/maia/ticket/170

Seems like investigating alternatives from PEAR is one way. Though, listing the contents of the mailbox
should not be that slow, especially over local link network.

http://www.phpbuilder.com/board/showthread.php?t=10124629

Hm, that seems strange.

Also; I just tested again, debug message before and after imap_open:

[Tue Mar 06 17:17:21 2012] [error] [client 141.48.166.115] ThOr: opening server:
nesselzelle.de/pop3/notls:110, referer: https://webmail.nesselzelle.de/index.php
[Tue Mar 06 17:17:21 2012] [error] [client 141.48.166.115] ThOr: DONE opening server:
nesselzelle.de/pop3/notls:110, referer: https://webmail.nesselzelle.de/index.php

That is not slow. I remember doing that before posting, actually... Is there some simple option to get a
profile of PHP code, with CPU time / real time counts (assuming there is a lot of waiting going on)?
(Continue reading)

Permalink | Reply | 14 comments |
headers
Hanno Foest | 6 Mar 2012 18:12
Favicon

Re: Performance with large-ish Mailbox

On Tue, Mar 06, 2012 at 05:36:36PM +0100, Thomas Orgis wrote:

> Eh... are you trying to push users away from NOCC? The Roundcube
> interface might be more easily accepted with users that need flashy
> graphics (and that might include the one main user why I'm setting up
> webmail), but I am  still one of the folks thinking that JavaScript
> basically is evil, an uncontrollable inflow of instructions in a
> complete (including holes in Browser engines) programming language.
> I'd prefer plain HTML + CSS, the latter also not being without issues,
> but still a far more restricted set of directives than JavaScript.

Well put. That's precisely the reason for me to use NOCC.

Hanno

------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
Permalink | Reply | 0 comments |
headers
Tim Gerundt | 6 Mar 2012 19:46
Picon
Gravatar

Re: Performance with large-ish Mailbox

Am 06.03.2012 17:36, schrieb Thomas Orgis:
> Hi again,
>
> Am Tue, 06 Mar 2012 16:46:55 +0100
> schrieb Tim Gerundt <tim <at> gerundt.de>:
>
>> Yes, imap_open() is called every time at the beginning of 
>> "action.php".
>> You are true, for editing preferences we don't need it, but all 
>> other
>> actions need it.
>
> Fine then. Perhaps I can squeeze in a look around that function to
> get to the point... a quick search brings up something related:
>
> http://www.maiamailguard.com/maia/ticket/170
>
> Seems like investigating alternatives from PEAR is one way. Though,
> listing the contents of the mailbox should not be that slow,
> especially over local link network.
>
> http://www.phpbuilder.com/board/showthread.php?t=10124629
>
> Hm, that seems strange.
>
> Also; I just tested again, debug message before and after imap_open:
>
> [Tue Mar 06 17:17:21 2012] [error] [client 141.48.166.115] ThOr:
> opening server: nesselzelle.de/pop3/notls:110, referer:
> https://webmail.nesselzelle.de/index.php
(Continue reading)

Permalink | Reply | 12 comments |
headers
Tim Gerundt | 6 Mar 2012 20:04
Picon
Gravatar

Re: Performance with large-ish Mailbox

Sorry, I hit submit before I was finished! ;)

Are you sure, that "opening server" and "DONE opening server" not both 
belongs to the imap_open() call? For me it looks like, that "DONE 
opening server" starts at 17:31:29 and 17:31:57 "num_msg" can call.

A quick and dirty benchmark looks in PHP like this:

$starttime = microtime(true);
// ...
// ... do PHP stuff you will benchmark ...
// ...
// ...
$endtime = microtime(true);
var_dump($endtime - $starttime);

>> Yeah, a lot of structure fetching, I assume it's about 25 in this
>> case... I do have to wonder: At what point does the lengthy apparent
>> mailbox transfer (every mail as a separate request, as it feels...
>> downloading a stream of messages would show a more consistently high
>> server load) happen?

If you at the inbox, it not fetch the body of the mail, only the header 
infos like subject, size and more.

> Is imap_open() really slow for you?

I echoed the time before and after the imap_open() call and it was more 
the 10 seconds.
(Continue reading)

Permalink | Reply | 11 comments |
headers
Thomas Orgis | 7 Mar 2012 01:36
Favicon

Re: Performance with large-ish Mailbox

Am Tue, 06 Mar 2012 20:04:22 +0100
schrieb Tim Gerundt <tim <at> gerundt.de>: 

> Sorry, I hit submit before I was finished! ;)
> 
> Are you sure, that "opening server" and "DONE opening server" not both 
> belongs to the imap_open() call? For me it looks like, that "DONE 
> opening server" starts at 17:31:29 and 17:31:57 "num_msg" can call.

That's the code:

error_log("ThOr: opening server: ".$this->server);
        $conn =  <at> imap_open('{'.$this->server.'}'.$this->folder, $this->login, $this->passwd, 0);
error_log("ThOr: DONE opening server: ".$this->server);

So that in the log...

[Tue Mar 06 17:31:28 2012] [error] [client x] ThOr: opening server: bla
[Tue Mar 06 17:31:29 2012] [error] [client x] ThOr: DONE opening server: bla

Does mean that the call to imap_open didn't need more than one second, doesn't it? I might be wrong, I'm just
transferring my knowledge from other programming languages.

> A quick and dirty benchmark looks in PHP like this:
> 
> $starttime = microtime(true);
> // ...
> // ... do PHP stuff you will benchmark ...
> // ...
> // ...
(Continue reading)

Permalink | Reply | 10 comments |

Gmane