Lyndon Nerenberg | 8 Feb 19:33 2012
Picon

Metadata (5654) Implementation Limits


[I didn't get much response from the imap-protocol list, so I'm trying here, too.]

I'm curious to know what the implementation limits are on servers that support the Metadata extension.  I
would appreciate it if the server authors out there could take a minute to fill out this survey.

I will send a summary to the list once the responses trickle off. (If you want your results to be anonymous,
say so in the comments.)

Thanks,

--lyndon

Implementation name and version:
Do you support /private?:
Do you support /private/vendor?:
Do you support unsolicited Metadata responses without values?:
Do you support unsolicited Metadata responses with values?:
Maximum number of annotations per mailbox:
Maximum size of an <entry>:
Maximum size of a <value>:
Maximum depth supported in <entries>:
Other comments:

Bron Gondwana | 8 Feb 21:15 2012

Designing a new replacement protocol for IMAP

I've given up on the IMAP5 mailing list going endlessly around in circles,
though I'm definitely going to read through the backlogs and remind myself
again what everyone spoke about, because there was some excellent discussion.

So - I will build something and prove that it's good by showing improved
user experiences.

And I'm happy to keep talking here on these mailing lists too.

I've started a wiki page:

http://imapwiki.org/ReplacementProtocol

and plan to blog about what I'm doing on our own (Opera's) blogging platform,
to practice using it:

http://my.opera.com/brongondwana/blog/

I plan to make on a protocol which can replace IMAP in most of its current
uses.  I want to get (at least) two working server implementations, and
two working clients.  I have at least in-principle support from enough
project maintainers to do that.

My strong philosophical ideals are:

* Simple to build clients - morons welcome. If it's not clear the right way
  to do something to someone just reading a protocol dump, we haven't done a
  good enough job.  We can't expect you to read 30 RFCs and resolve their
  conflicts yourself before starting coding, and we won't insult you if you
  didn't get it right.  Where there is necessary complexity, the server
(Continue reading)

Greg Banks | 8 Feb 21:18 2012

Re: Metadata (5654) Implementation Limits


On 09/02/2012, at 5:33, Lyndon Nerenberg <lyndon <at> orthanc.ca> wrote:

>
> [I didn't get much response from the imap-protocol list, so I'm  
> trying here, too.]
>
> I'm curious to know what the implementation limits are on servers  
> that support the Metadata extension.  I would appreciate it if the  
> server authors out there could take a minute to fill out this survey.
>
> I will send a summary to the list once the responses trickle off.  
> (If you want your results to be anonymous, say so in the comments.)
>
> Thanks,
>
> --lyndon
>
> Implementation name and version:

Cyrus (METADATA support rewritten and ANNOTATE support added to future  
2.5 release).

> Do you support /private?:

Yes

> Do you support /private/vendor?:

Yes, with a subset of that space reserved for the server itself. Users  
(Continue reading)

Thomas Koch | 9 Feb 08:20 2012
Picon

Re: Designing a new replacement protocol for IMAP

Bron Gondwana:
> Snide comments from those who don't agree with the goals are welcome too...
> I just won't listen to you.  These goals are very important to me.
Hi Bron,

I'm not an expert in IMAP, just got here out of curiosity about Kolab[1]'s use 
of IMAP folders as data stores.
I found some older initiatives to replace IMAP with an HTTP based approach.[2] 
What do you think of HTTP for mail access? For my bachelor thesis[3] I 
currently argue that CardDAV/CalDAV could be perfectly replaced by AtomPub 
(RFC 5023).
The main advantage would be that most software developers on this planet have 
some vague understanding of HTTP and that imense infrastructure and software 
already exists.

[1] http://wiki.kolab.org/Why_IMAP_Is_Used_For_Storage
[2] http://www.prescod.net/rest/restmail/
    https://www.ietf.org/mailman/listinfo/httpmail
[3] https://github.com/thkoch2001/bachelor-thesis

Best regards,

Thomas Koch, http://www.koch.ro
Adrien de Croy | 9 Feb 09:05 2012

Re: [imap5] Designing a new replacement protocol for IMAP


Whilst HTTP could certainly be used to put together an email system, I 
don't think it would perform well enough to replace existing protocols 
on desktops.

The main issues I see are:

* updates.  Would need to poll or use long polling to get real-time 
updates (e.g. notification of incoming mail).
* slow.  Gut feel based on 17 years writing http proxies is that it 
would be too slow
* complex.  In order to reduce round-trips, you'd have to do many things 
in a single transaction, which would create complexity issues in the 
server and client
* protocol bloat.  if you think IMAP is bloated, try HTTP.
* fundamentally an off-line style protocol for mail, which has strong 
incentives to be online (esp for in-office use).

I think it would also be very problematic in the wild, since any http 
intermediary would feel entitled to mess with traffic.  E.g. heuristic 
caching, intercepting proxies etc.

It's almost too ubiquitous.  Everyone and their dog thinks they know how 
http works.  It's not strictly-enough defined, and has a LOT of optional 
features which aren't relevant for mail (e.g. content negotiation).  
Interop in http is already problematic, but start getting people writing 
mail servers in php etc, and see what starts to happen.  I think it 
would be a nightmare.

Adrien
(Continue reading)

Bron Gondwana | 9 Feb 12:24 2012

Re: Designing a new replacement protocol for IMAP

On Thu, Feb 09, 2012 at 09:05:53PM +1300, Adrien de Croy wrote:
> 
> Whilst HTTP could certainly be used to put together an email system,
> I don't think it would perform well enough to replace existing
> protocols on desktops.
> 
> The main issues I see are:
> 
> * updates.  Would need to poll or use long polling to get real-time
> updates (e.g. notification of incoming mail).

Yes - long polling or "out of band" notification systems.  We have
an HTTP protocol defined kind of "on top of" IMAP plus our own
extensions here at FastMail/Opera (mail.opera.com if you want to
play with it)

We are currently using EventSource to notify the client that
"there is something new", after which the client queries to
find what's updated.

This is possible to be efficient because we use a single counter
for uidvalidity and a single counter for highestmodseq across the
entire user's folders - so any change to uidvalidity means the
folder list needs refreshing, and any change to highestmodseq
means the message view needs checking.

Combined with efficient "what's changed in this view since my
previous modseq value" queries, it's very bandwidth efficient.

> * slow.  Gut feel based on 17 years writing http proxies is that it
(Continue reading)

Giovanni Panozzo | 9 Feb 12:25 2012
Picon

Re: Designing a new replacement protocol for IMAP


> I found some older initiatives to replace IMAP with an HTTP based approach.[2]
> What do you think of HTTP for mail access? For my bachelor thesis[3] I
> currently argue that CardDAV/CalDAV could be perfectly replaced by AtomPub
> (RFC 5023).
> The main advantage would be that most software developers on this planet have
> some vague understanding of HTTP and that imense infrastructure and software
> already exists.

Hi Bron, hi Thomas.
I'm happy to see this interesting traffic on the ML.

Just a small note: these two post are full of "programmer point of view" 
considerations and philosophical ideals. That's good.

But I'm mostly "system integrator", involved also with end-user 
helpdesk. I'm a programmer also, but only form 2% of my time, so I'm not 
a "professional programmer".
I would like to contribute with other key points useful both to system 
integrators, helpdesk and end user (moron end user) ease to use (I have 
a long list... do you want it ? :))

Here are my first notes about HTTP: I think HTTP is welcome. Many mobile 
phone carriers do offer only http/https connectivity, sometimes filtered 
by a transparent proxy too. In some enterprise LANs, users are forced to 
use an http proxy with proxy authentication to access the Internet. No 
other protocols are allowed. HTTP would allow all these users to access 
their e-mail without requiring impossible access upgrades.
And HTTP completed with digest authentication and email body encryption 
also will also help sysadmins to have more security without incurring 
(Continue reading)

Bron Gondwana | 9 Feb 12:32 2012

Re: Designing a new replacement protocol for IMAP

On Thu, Feb 09, 2012 at 12:25:33PM +0100, Giovanni Panozzo wrote:
> PS: I think that crossposting in 3 ML is too much. Which one will be
> the official ML for this discussion ?

(posting to all three for this...)

Let's take the rest of the traffic to just imap5@...

Bron.
Thomas Koch | 10 Feb 15:44 2012
Picon

Re: Designing a new replacement protocol for IMAP

Hi Adrien,

thank you for your insights. I've some more questions inline.

Adrien de Croy:
> * updates.  Would need to poll or use long polling to get real-time
> updates (e.g. notification of incoming mail).
Could Websockets or http://en.wikipedia.org/wiki/Comet_(programming) be of 
help here?

> * complex.  In order to reduce round-trips, you'd have to do many things
> in a single transaction, which would create complexity issues in the
> server and client
I was thinking restful HTTP which by definition is not complex. The question 
is, whether a useful mail fetch protocol could be designed in the constraints 
of rest.

> I think it would also be very problematic in the wild, since any http
> intermediary would feel entitled to mess with traffic.  E.g. heuristic
> caching, intercepting proxies etc.
There is an HTTP header to instruct any cache to pipe the request through and 
do no caching. But anybody on any line in any protocol could mess with your 
data.

Have to go now, sorry. :-)

Best regards,

Thomas Koch, http://www.koch.ro
(Continue reading)

Timo Sirainen | 13 Feb 01:45 2012
Picon
Picon

ANNOTATE-EXPERIMENT-1


I was thinking about finally implementing METADATA + per-mail annotations to Dovecot so I read through the
RFCs. There are mainly two things I'd do differently in ANNOTATE-EXPERIMENT-2:

1. METADATA has case-insensitive entry names, ANNOTATE has case-sensitive. Probably better to keep them
case-insensitive in both.

2. SORT (ANNOTATION) - is this really worth the trouble? There's really no good way for servers to implement
this in an efficient way. It would pretty much require building and maintaining an index for every single
annotation entry name.

And other things I wondered about:

 - If APPEND has annotations, and annotation limits are reached, the whole APPEND fails with
TOOBIG/TOOMANY error? I guess it has to, although that's a bit annoying.

 - ANNOTATE doesn't specify what happens if STORE can change only some of the entries. Should it rollback the
changes to all the previous entries as well? Including changes to previous messages if multiple messages
were specified? Would have been simpler if both METADATA and ANNOTATE supported changing only one
annotation at a time.

 - TOOBIG/TOOMANY doesn't map very nicely if I want to have global "max. annotation size" and "max.
annotation count". Perhaps a global max. count shouldn't exist, since the number of mails/mailboxes
could be limited instead, but a global size similar to mail quota should exist. I guess when that global
size is reached it just fails with TOOBIG until some annotations are deleted.


Gmane