headers
martin f krafft | 14 Aug 2007 16:11

use of UIDs vs. MSGID

OfflineIMAP uses the X-OfflineIMAP-x... header to identify messages,
and the IMAP server's UID notion to link them to the server copy.
Since we're dealing with mail all along, and mail messages have
a unique message-id header, I wonder why that's not being used. It's
likely to be indexed on the server and could thus increase the speed
of searches.
Also, why does OfflineIMAP store headers like

  X-OfflineIMAP-x825313192-7365616d75732e6d61646475636b2e6e6574-494e424f58:
  1186324387-0249694837446-v5.99.0

?

As in, why is the header label not constant? And what do all these
numbers mean?

Cheers,

--

-- 
martin;              (greetings from the heart of the sun.)
  \____ echo mailto: !#^."<*>"|tr "<*> mailto:" net <at> madduck

"man kann die menschen nur von ihren eigenen meinungen überzeugen."
                                                    -- charles tschopp

spamtraps: madduck.bogus <at> madduck.net

-- Attached file included as plaintext by Ecartis --
-- File: digital_signature_gpg.asc
-- Desc: Digital signature (see http://martin-krafft.net/gpg/)
(Continue reading)

Permalink | Reply | 2 comments |
headers
John Goerzen | 14 Aug 2007 16:32
Favicon

Re: use of UIDs vs. MSGID

On Tue August 14 2007 9:11:17 am martin f krafft wrote:
> OfflineIMAP uses the X-OfflineIMAP-x... header to identify messages,
> and the IMAP server's UID notion to link them to the server copy.
> Since we're dealing with mail all along, and mail messages have
> a unique message-id header, I wonder why that's not being used. It's

Because it is not guaranteed that there will be exactly one copy with a given 
message-id header in any given mailbox.  Think about scenarios where a user 
copies a message from folder A to folder B, and then copies the same message 
from folder B to folder A.  Folder A will now have two copies of the 
message, but each one could have its own flags.  Also, some mailboxes 
contain messages without that header (common examples are Drafts folders).  

> likely to be indexed on the server and could thus increase the speed
> of searches.

I actually would think this would be significantly slower than using UIDs.  
All servers that do any sort of indexing should be indexing UIDs anyway.  
Servers such as Courier index *only* UIDs, and would have to scan every file 
in a maildir for a given header.

> Also, why does OfflineIMAP store headers like
>
>  
> X-OfflineIMAP-x825313192-7365616d75732e6d61646475636b2e6e6574-494e424f58:
> 1186324387-0249694837446-v5.99.0
>
> ?
>
> As in, why is the header label not constant? And what do all these
(Continue reading)

Permalink | Reply | 1 comment |
headers
martin f krafft | 14 Aug 2007 16:59

Re: use of UIDs vs. MSGID

also sprach John Goerzen <jgoerzen <at> complete.org> [2007.08.14.1632 +0200]:
> Because it is not guaranteed that there will be exactly one copy
> with a given message-id header in any given mailbox.  Think about
> scenarios where a user copies a message from folder A to folder B,
> and then copies the same message from folder B to folder A.
> Folder A will now have two copies of the message, but each one
> could have its own flags.  Also, some mailboxes contain messages
> without that header (common examples are Drafts folders).  
Doh! Thanks for the explanation.

-- 
martin;              (greetings from the heart of the sun.)
  \____ echo mailto: !#^."<*>"|tr "<*> mailto:" net <at> madduck

why do they sterilise the needle for lethal injections?

spamtraps: madduck.bogus <at> madduck.net

-- Attached file included as plaintext by Ecartis --
-- File: digital_signature_gpg.asc
-- Desc: Digital signature (see http://martin-krafft.net/gpg/)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFGwcNDIgvIgzMMSnURAnAcAJ9RIMu/IKyNsp1XrwLD5/O4y+rr3ACeO1sh
qpbRDlS+iysWs1kNSIRL1+Y=
=VRzg
-----END PGP SIGNATURE-----
(Continue reading)

Permalink | Reply | 0 comments |
headers
martin f krafft | 15 Aug 2007 18:37

autorefresh with preauthtunnel?

I am trying to experiment with autorefresh, but found that
offlineimap seems to ignore it. I am using preauthtunnel to create
an SSH tunnel through which I then run imapd directly. I tried
putting autorefresh into [general], the [Account ] stanza, as well
as the remote [Repository ] stanza, but in all cases, offlineimap
would just exit after a single run. -o is not being passed. Here are
the relevant settings:
autorefresh = 600
holdconnectionopen = yes
keepalive = 60

Any idea what's going on?

--

-- 
martin;              (greetings from the heart of the sun.)
  \____ echo mailto: !#^."<*>"|tr "<*> mailto:" net <at> madduck

half a bee, philosophically, must ipso facto half not be.
but half the bee has got to be, vis-a-vis its entity. you see?
but can a bee be said to be or not to be an entire bee,
when half the bee is not a bee, due to some ancient injury?
                                                       -- monty python

spamtraps: madduck.bogus <at> madduck.net

-- Attached file included as plaintext by Ecartis --
-- File: digital_signature_gpg.asc
-- Desc: Digital signature (see http://martin-krafft.net/gpg/)

-----BEGIN PGP SIGNATURE-----
(Continue reading)

Permalink | Reply | 1 comment |
headers
Florian Friesdorf | 16 Aug 2007 12:20
Picon

Re: autorefresh with preauthtunnel?

On Wed, Aug 15, 2007 at 06:37:03PM +0200, martin f krafft wrote:
> I am trying to experiment with autorefresh, but found that
> offlineimap seems to ignore it. I am using preauthtunnel to create
> an SSH tunnel through which I then run imapd directly. I tried
> putting autorefresh into [general], the [Account ] stanza, as well
> as the remote [Repository ] stanza, but in all cases, offlineimap
> would just exit after a single run. -o is not being passed. Here are
> the relevant settings:
> autorefresh = 600
> holdconnectionopen = yes
> keepalive = 60
autorefresh is in minutes. However that should not be the problem.

-florian

-- Attached file included as plaintext by Ecartis --

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7-ecc0.1.6 (GNU/Linux)

iD8DBQBGxCTlgqFlIkofQ2cRAl0nAKCNpvjhjLOK6bJaOVenXOawZTeuEQCbBIV9
YSBGDf2RtXDHMUSlRkXvgsM=
=NKcH
-----END PGP SIGNATURE-----
Permalink | Reply | 0 comments |
headers
Asheesh Laroia | 16 Aug 2007 20:01

Debugging sync logic

I notice that offlineimap gives me some duplicate emails between a remote 
Dovecot 1.0 and a local Maildir spool.  I want to debug this - are there 
any suggestions?  I get even more duplicates doing IMAP<->IMAP syncing.

Also, where is the core sync logic held?  I found the code a little 
confusingly-organized.  (If there's a DeveloperDocumentation page e.g. on 
the wiki, excuse my failure to find it and just slap me the URL.)

I was thinking of adding some assertions inside the code, or maybe running 
offlineimap in an environment like this:

1. Perform random changes on the Maildir and the IMAP spool
2. Synchronize
3. Verify that it didn't create any duplicate messages.
4. GOTO 1

-- Asheesh.

--
Include me out.
Permalink | Reply | 1 comment |
headers
John Goerzen | 16 Aug 2007 21:10
Favicon

Re: Debugging sync logic

On Thu August 16 2007 1:01:01 pm Asheesh Laroia wrote:
> I notice that offlineimap gives me some duplicate emails between a remote
> Dovecot 1.0 and a local Maildir spool.  I want to debug this - are there
> any suggestions?  I get even more duplicates doing IMAP<->IMAP syncing.

Run with -d imap and follow the UIDs.  See where they come from and where 
they go.

> Also, where is the core sync logic held?  I found the code a little
> confusingly-organized.  (If there's a DeveloperDocumentation page e.g. on
> the wiki, excuse my failure to find it and just slap me the URL.)

There are a few places.  offlineimap/folder/Base.py is probably what you 
want, but also look at offlineimap/accounts.py

>
> I was thinking of adding some assertions inside the code, or maybe running
> offlineimap in an environment like this:
>
> 1. Perform random changes on the Maildir and the IMAP spool
> 2. Synchronize
> 3. Verify that it didn't create any duplicate messages.
> 4. GOTO 1

I would greatly appreciate the debugging help.  I've heard from 2 people now 
that something seems to have changed with the newer version that causes this 
problem, but I have yet to be able to duplicate it, and I can't figure out 
what changed that would impact this, either.

-- John
(Continue reading)

Permalink | Reply | 0 comments |
headers
Jean-Baptiste Quenot | 21 Aug 2007 10:24
Gravatar

Local copy of message corrupt

Hi there,

Thanks for  the great program,  it saves  life!  The tool  is very
powerful, so powerful that I already came across the [1]unexpected
mass-deletion of remote messages.

Today I also notice a strange thing: some new messages on the IMAP
server are  perfectly valid, but  the local copy is  corrupt.  The
message locally contains one byte (usually a number such as '3' or
'9').

I use this command to locate those corrupt messages:

$ find ~/mail -size 1c
./hudson-users/cur/1187604320_0.3272.much,U=1634,FMD5=304255666fb0f3ca92e87c4d771f5f6c:2,
./hudson-users/cur/1187600779_0.3272.much,U=1632,FMD5=304255666fb0f3ca92e87c4d771f5f6c:2,
./caraldi/cur/1187615834_1.3272.much,U=1137,FMD5=8851307309c0b28484b1843637c39232:2,S

This is my setup:

* Ubuntu GUN/Linux
* Python 2.5.1c1
* OfflineImap 4.0.16

Does it ring a bell to you?  Thanks in advance,
--

-- 
     Jean-Baptiste Quenot
aka  John Banana   Qwerty
http://caraldi.com/jbq/
(Continue reading)

Permalink | Reply | 4 comments |
headers
John Goerzen | 21 Aug 2007 15:18
Favicon

Re: Local copy of message corrupt

On Tue August 21 2007 3:24:19 am Jean-Baptiste Quenot wrote:
> Hi there,
>
> Thanks for  the great program,  it saves  life!  The tool  is very
> powerful, so powerful that I already came across the [1]unexpected
> mass-deletion of remote messages.
>
> Today I also notice a strange thing: some new messages on the IMAP
> server are  perfectly valid, but  the local copy is  corrupt.  The
> message locally contains one byte (usually a number such as '3' or
> '9').

I do recall occasionally hearing about that, but I think it was years ago.  
One of the annoying things about Python's imaplib.py, unfortunately.

I don't know what to suggest except running it under -1 -d imap -u TTY.TTYUI 
and send me the parts of the log that deal with those messages.

Have you determined what messages on the server they correspond to, and if 
those messages even exist?  The U= is the UID.

- John
Permalink | Reply | 3 comments |
headers
Jean-Baptiste Quenot | 21 Aug 2007 15:31
Gravatar

Re: Local copy of message corrupt

* John Goerzen:

> On Tue August 21 2007 3:24:19 am Jean-Baptiste Quenot wrote:
>
> > Today I also notice a strange  thing: some new messages on the
> > IMAP  server  are  perfectly  valid, but  the  local  copy  is
> > corrupt.   The message  locally contains  one byte  (usually a
> > number such as '3' or '9').
>
> I  do recall  occasionally hearing  about that,  but I  think it
> was  years  ago.  One  of  the  annoying things  about  Python's
> imaplib.py, unfortunately.

Thanks  for your  prompt  answer.  About  imaplib.py  I read  some
discussion  about it  in  the archives,  what's  the progress  for
reimplementing it?.

> I don't know what to suggest  except running it under -1 -d imap
> -u TTY.TTYUI  and send me  the parts of  the log that  deal with
> those messages.

Thanks for the  tip.  I'll post in private  because the attachment
is huge.

> Have you determined what messages  on the server they correspond
> to, and if those messages even exist?

Yes  the messages  exist.   As  a workaround  for  the problem,  I
configure my mail client to use the remote IMAP server.
(Continue reading)

Permalink | Reply | 0 comments |

Gmane