CONDSTORE plus UID EXPUNGE undefined?

Re: CONDSTORE plus UID EXPUNGE undefined?

Re: CONDSTORE plus UID EXPUNGE undefined?

On Tue, Mar 27, 2012 at 12:13:00AM +0000, Adrien W. de Croy wrote:
> Hi all
> 
> Apologies if this is covered somewhere and I missed it.
> 
> I've been reviewing behaviour of HIGHESTMODSEQ wrt UID EXPUNGE.
> 
> My gut feel is that the intent of CONDSTORE would be that the
> HIGHESTMODSEQ be incremented if a UID EXPUNGE results in any messages
> being removed from a mailbox.

Absolutely.  In fact, if you look at QRESYNC, it not only needs to
increment the HIGHESTMODSEQ, it has to increment the MODSEQ of the
message being expunged so that you can give a VANISHED response.

> If we shouldn't increment HMS on UID EXPUNGE, then the HIGHESTMODSEQ
> becomes unreliable as an indication that anything changed on a
> mailbox, since a UID EXPUNGE may have removed messages. I'd like
> clients to be
> able to rely on HMS as an indication that ANY work is required or not
> to resynch.

You can increment HMS by however much you like, whenever you want, and
still be compatible... it could be that something has changed which the
client just didn't ask for.

Bron ( until you run out of 64 bits anyway )

Re[2]: CONDSTORE plus UID EXPUNGE undefined?

Looks like our issue related to MOVE + CONDSTORE.

Since MOVE doesn't do an untagged FETCH response advertising \Deleted 
on the source, but instead only sends untagged EXPUNGE responses, it's 
not clear about incrementing HMS.

The obvious behaviour is to increment HMS.

This I guess belongs in the MOVE I-D in terms of reference to CONDSTORE.

Are there any plans to progress that draft?  It's quite worthwhile IMO.

Adrien

------ Original Message ------
From: "Bron Gondwana" <brong <at> fastmail.fm>
To: "Adrien W. de Croy" <adrien <at> qbik.com>
Cc: "ietf-imapext <at> imc.org" <ietf-imapext <at> imc.org>
Sent: 27/03/2012 1:30:27 p.m.
Subject: Re: CONDSTORE plus UID EXPUNGE undefined?
>On Tue, Mar 27, 2012 at 12:13:00AM +0000, Adrien W. de Croy wrote:
>
>>
>>Hi all
>>
>>Apologies if this is covered somewhere and I missed it.
>>
>>I've been reviewing behaviour of HIGHESTMODSEQ wrt UID EXPUNGE.
>>
>>My gut feel is that the intent of CONDSTORE would be that the
>>HIGHESTMODSEQ be incremented if a UID EXPUNGE results in any messages
>>being removed from a mailbox.
>>
>
>
>Absolutely.  In fact, if you look at QRESYNC, it not only needs to
>increment the HIGHESTMODSEQ, it has to increment the MODSEQ of the
>message being expunged so that you can give a VANISHED response.
>
>
>>
>>If we shouldn't increment HMS on UID EXPUNGE, then the HIGHESTMODSEQ
>>becomes unreliable as an indication that anything changed on a
>>mailbox, since a UID EXPUNGE may have removed messages. I'd like
>>clients to be
>>able to rely on HMS as an indication that ANY work is required or not
>>to resynch.
>>
>
>
>You can increment HMS by however much you like, whenever you want, and
>still be compatible... it could be that something has changed which the
>client just didn't ask for.
>
>Bron ( until you run out of 64 bits anyway )
>
>
>

Re: Re[2]: CONDSTORE plus UID EXPUNGE undefined?

On Tue, Mar 27, 2012 at 12:44:35AM +0000, Adrien W. de Croy wrote:
> Looks like our issue related to MOVE + CONDSTORE.
> 
> Since MOVE doesn't do an untagged FETCH response advertising \Deleted
> on the source, but instead only sends untagged EXPUNGE responses, it's
> not clear about incrementing HMS.
> 
> The obvious behaviour is to increment HMS.
> 
> This I guess belongs in the MOVE I-D in terms of reference to CONDSTORE.
> 
> Are there any plans to progress that draft?  It's quite worthwhile IMO.

Agree that it's worthwhile.  We're already using it internally for our
web interface talking to the Cyrus backends.

My attitude to HMS is - if anything user-visible changes, the HMS must
be bumped such that a client connection can never see two different
responses with exactly the same HMS.  Idempotency is good.

Bron.

Re: CONDSTORE plus UID EXPUNGE undefined?

On 03/27/2012 02:44 AM, Adrien W. de Croy wrote:
> This I guess belongs in the MOVE I-D in terms of reference to CONDSTORE.
>  
> Are there any plans to progress that draft?  It's quite worthwhile IMO.

There have been many. I uploaded one now, with text from Witold, Alexey
and myself.

http://tools.ietf.org/html/draft-gulbrandsen-imap-move

Arnt

Re: CONDSTORE plus UID EXPUNGE undefined?

On Wed, Mar 28, 2012 at 02:21:54PM +0200, Arnt Gulbrandsen wrote:
> 
> On 03/27/2012 02:44 AM, Adrien W. de Croy wrote:
> > This I guess belongs in the MOVE I-D in terms of reference to CONDSTORE.
> >  
> > Are there any plans to progress that draft?  It's quite worthwhile IMO.
> 
> There have been many. I uploaded one now, with text from Witold, Alexey
> and myself.
> 
> http://tools.ietf.org/html/draft-gulbrandsen-imap-move

   Servers may want to avoid implementing this because atomicity
   requires holding one or more locks on several mailboxes at the same
   time.  (This sounds just wrong. COPY requires the same.)

No, COPY doesn't.  You can release the lock on the first mailbox
after taking all the data required, then obtain the lock on the
second mailbox.  At this point, it doesn't matter if someone else
does something to your source messages.

With MOVE, you need to hold the lock on the first mailbox until
after the APPEND has succeeded, then come back and EXPUNGE them.

Bron.

Re: CONDSTORE plus UID EXPUNGE undefined?

------ Original Message ------
From: "Bron Gondwana" <brong <at> fastmail.fm>
To: "Arnt Gulbrandsen" <arnt <at> gulbrandsen.priv.no>
Cc: "ietf-imapext <at> imc.org" <ietf-imapext <at> imc.org>
Sent: 29/03/2012 1:54:43 a.m.
Subject: Re: CONDSTORE plus UID EXPUNGE undefined?
>On Wed, Mar 28, 2012 at 02:21:54PM +0200, Arnt Gulbrandsen wrote:
>
>>
>>
>>On 03/27/2012 02:44 AM, Adrien W. de Croy wrote:
>>> This I guess belongs in the MOVE I-D in terms of reference to CONDSTORE.
>>>
>>> Are there any plans to progress that draft?  It's quite worthwhile IMO.
>>
>>There have been many. I uploaded one now, with text from Witold, Alexey
>>and myself.
>>
>>http://tools.ietf.org/html/draft-gulbrandsen-imap-move
>>
>

  
great, thanks!
>
>
>  Servers may want to avoid implementing this because atomicity
>  requires holding one or more locks on several mailboxes at the same
>  time.  (This sounds just wrong. COPY requires the same.)
>
>No, COPY doesn't.  You can release the lock on the first mailbox
>after taking all the data required, then obtain the lock on the
>second mailbox.  At this point, it doesn't matter if someone else
>does something to your source messages.
>
>With MOVE, you need to hold the lock on the first mailbox until
>after the APPEND has succeeded, then come back and EXPUNGE them.
>
>

  

it can be optimised for MOVE.  Since you don't need to do a deep file 
copy, but a file system move, as long as the dest folder is in the same 
volume as the source, it's usually a very quick operation - much 
quicker than copy.  Worst case it takes the same time if different 
volumes.

So the 2 locks are held a short time only.

  
>
>Bron.
>
>
>

Re: CONDSTORE plus UID EXPUNGE undefined?

> >With MOVE, you need to hold the lock on the first mailbox until
> >after the APPEND has succeeded, then come back and EXPUNGE them.
>   
> it can be optimised for MOVE.  Since you don't need to do a deep file 
> copy, but a file system move, as long as the dest folder is in the same 
> volume as the source, it's usually a very quick operation - much 
> quicker than copy.  Worst case it takes the same time if different 
> volumes.

Assuming you're using a "1 file = 1 email" model, then most filesystems
that support move, also support hardlinks, so a copy is actually
*cheaper* than a move, because you're just adding an extra link, and
adding links is usually a lot cheaper than unlinking.

In fact unlinks are usually sufficiently poorly optimised in
filesystems, that we explicitly added a "delayed expunge" mode to cyrus,
that allows us to push the unlink actions off into the future (eg
weekends) because they completely dominate the time on large expunges
otherwise.

> So the 2 locks are held a short time only.

It's not that they're short time only, it's that it can cause deadlocks
because for a move to be atomic, you have to have both locks at the same
time, while with copy you can get/drop them sequentially.

Rob

Re: CONDSTORE plus UID EXPUNGE undefined?

Bron is right, at least for some architectures. But his argument is
really the same as I've heard umpteen times for move: Carrying out the
copy/store/expunge sequence without the lock COPY can drop early has too
many failure modes and handling them requires too much clientside code.

Having a lock gives you the advantages of the lock at the cost of the lock.

Arnt