crash on sync of large maildir

My INBOX maildir hold mail for the last 10 years — no Inbox Zero for me!

;)

I've used offlineimap very successfully for a long time, but recently
something started happening to screw it up.  To try and regain some
sanity (and at the advise of John Goerzen), I got a fresh version from
git but I was still running into a problem — though this one was
different.

Now I have the following problem:

    $ offlineimap -f INBOX
    OfflineIMAP 6.3.3
    Copyright (C) 2002 - 2010 John Goerzen <john <at> complete.org>

    This software comes with ABSOLUTELY NO WARRANTY; see the file
    COPYING for details.  This is free software, and you are welcome
    to distribute it under the conditions laid out in COPYING.
    ***** Processing account nichework.com
    Copying folder structure from IMAP to MappedIMAP
    Establishing connection to tunnel:nice env MAILDIR=/home/mah/maildir/everybody.org /usr/lib/dovecot/imap.
    Establishing connection to mail.nichework.com:993.
    Syncing INBOX: IMAP -> MappedIMAP
    Copy message -2 MappedIMAP[INBOX] -> IMAP[INBOX]
    Copy message -6 MappedIMAP[INBOX] -> IMAP[INBOX]
    Copy message -5 MappedIMAP[INBOX] -> IMAP[INBOX]
    Copy message -4 MappedIMAP[INBOX] -> IMAP[INBOX]
    Thread 'New msg sync from INBOX' terminated with exception:

Re: crash on sync of large maildir

On Sat, 30 Apr 2011 23:14:24 -0400, mah <at> everybody.org wrote:
> My INBOX maildir hold mail for the last 10 years — no Inbox Zero for me!
:)

> I've used offlineimap very successfully for a long time, but recently
> something started happening to screw it up.

Mmh, once the UID map between 2 IMAP servers gets screwed, things are
not going to be nice with offlineimap. The whole IMAP<->IMAP syncing
really feels somewhat tacked on in a rather hackish way. IMAP<->Maildir
is much more robust, personally I would advice to use that.

>     Syncing INBOX: IMAP -> MappedIMAP
>     Copy message -2 MappedIMAP[INBOX] -> IMAP[INBOX]
>     Copy message -6 MappedIMAP[INBOX] -> IMAP[INBOX]
>     Copy message -5 MappedIMAP[INBOX] -> IMAP[INBOX]
>     Copy message -4 MappedIMAP[INBOX] -> IMAP[INBOX]
>     Thread 'New msg sync from INBOX' terminated with exception:
>     Traceback (most recent call last):
>     File "/usr/lib/pymodules/python2.7/offlineimap/threadutil.py", line 159, in run
>     Thread.run(self)
>     File "/usr/lib/python2.7/threading.py", line 505, in run
>     self.__target(*self.__args, **self.__kwargs)
>     File "/usr/lib/pymodules/python2.7/offlineimap/folder/Base.py", line 264, in syncmessagesto_neguid_msg
>     self.deletemessage(uid)
>     File "/usr/lib/pymodules/python2.7/offlineimap/folder/UIDMaps.py", line 238, in deletemessage
>     self._mb.deletemessage(self, self.r2l[uid])
>     File "/usr/lib/pymodules/python2.7/offlineimap/folder/IMAP.py", line 567, in deletemessage
>     self.deletemessages_noconvert([uid])
>     File "/usr/lib/pymodules/python2.7/offlineimap/folder/IMAP.py", line 578, in deletemessages_noconvert

Re: ERROR in syncfolder for (username) folder INBOX : null byte in argument for long()

On Thu, 28 Apr 2011 10:28:12 -0700 (PDT), chris coleman wrote:
> Nicolas, I fully agree.
[SNIP lots of good suggestions]
> Your opinons??

No doubt that would all be good. All it takes is someone to implement it


Seriously, the IMAP<->Maildir case is much more robust, the IMAP<->IMAP
case has been tacked on with very little robustness checks (although in
an ingenious way), and that case deserves much more attention in terms
of robustness checks and usefull error output.

Sebastian

Actually, I had just finished a nice Error output series of patches last
week when a hasty "git reset --hard nicolas/master" at the wrong time
destroyed it .

_______________________________________________
OfflineIMAP-project mailing list
OfflineIMAP-project <at> lists.alioth.debian.org
http://lists.alioth.debian.org/mailman/listinfo/offlineimap-project

OfflineIMAP homepage: http://software.complete.org/offlineimap

Re: ERROR in syncfolder for (username) folder INBOX : null byte in argument for long()

On Fri, 29 Apr 2011 07:10:36 -0700 (PDT), chris coleman <christocoleman <at> yahoo.com> wrote:
> Nicolas, Sebastian:
> 
> Here is an update on crashlog1.  
> 
> I removed the files in a WORKING user's folder 
> (~/.offlineimap/Repository-IMAPLocalUsername/FolderValidity/*)  and re-ran 
> offlineimap 6.3.2.  It re-generated those files like you said it would.
> 
> I removed the files in the CRASHIG user's folder, and re-ran it.  It NEVER got 
> to the point of re-generating those files.
> 
> Here is the log from the crashing user.
> 
> Your suggestions on what to do next, will be appreciated !

Most probably this file is corrupted:

~/.offlineimap/Repository-IMAPLocalUsername/UIDMapping/INBOX.Sent

It should be one entry in the form of "number:number" per line and when
reading this file, it got more than 2 entries. So either it lacks some
newline somewhere or it actually got 2 ":" in one line somehow.

>   File "/root/nicolas33-offlineimap-v632/offlineimap/folder/UIDMaps.py", line 
> 49, in _loadmaps
>     (str1, str2) = line.split(':')
> ValueError: too many values to unpack

Sebastian

Re: Prevent deletion of messages during sync per-folder?

On Thu, 28 Apr 2011 11:54:27 -0600, Will Styler wrote:

> My apologies for a rather user-directed question, but I'm so far unable to find a way that I can ask
offlineIMAP to decline to delete local copies of a message from certain folders, for instance, "Trash". 
My IMAP provider (MobileMe) has recently started clearing messages older than 30 days from the server,
and I'd like the local trash folder to keep a long-term archive for referencing elsewhere.  

Fun. And not trivial, as I see it it's not possible today .

What is possible now:
1) Mark a full repository read-only, thus enabling 1-way syncing. This
   is not what you want or need, and it's for a full repo only.

2) put "Trash" in folderexcludes, it will then not sync "Trash" at all,
   but it will keep those in your local "Trash" that are already there. New
   ones wouldn't be put in there though.

(actually this is all I can come up with for now, the "maxage" option
that I though of first wouldn't be of use here).

So what you want is essentially a DONT_DELETE_FOLDER_LIST =
['Trash','INBOX.Otherfolder'] ?

This has been ocasionally been requested for the full repo, but I have
been very reluctant on it, because as you say, moving between folders
etc would lead to duplicate messages. But for specific folders I can see
how it could be useful in certain cases.

> However, I'm aware that this could be difficult to implement, as OfflineIMAP would then want to sync all
those messages which it has kept and the server hasn't back up to the server during the next sync (unless

Re: Prevent deletion of messages during sync per-folder?

On Thu, 28 Apr 2011 20:34:12 +0200, Nicolas Sebrecht wrote:
> Patches has been submitted to go to this direction. A first
> implementation of 1-way sync at Account level was written but never
> merged due to some unaddressed issues. See
> 
>   Subject: [PATCH v3] Re: True 1-way sync (backup)
>   From: Nicolas Sebrecht <nicolas.s-dev <at> laposte.net>
>   Date: Thu, 31 Mar 2011 18:55:38 +0200
>   Message-ID: <20110331165538.GB14175 <at> vidovic>
> 
> for more information.

Uhh, that had not been merged? True. You got hung up on the wrong
debugtype thing :). Here comes v4, the first patch implements the
"empty" debugtype which is always used when debug is used, and the
second patch is the real patch with your issues addressed. Interbranch
diff between v3 and v4 attached to this mail.

More comments before merging?

Sebastian

_______________________________________________
OfflineIMAP-project mailing list
OfflineIMAP-project <at> lists.alioth.debian.org
http://lists.alioth.debian.org/mailman/listinfo/offlineimap-project

[PATCH 1/2] Introduce an *empty* debug type

This debug type will always be enabled whenever any debugging is enables
and it outputs debug messages that cannot be categorized among any of
imap, maildir (e.g. things that concern the sync logic).

Signed-off-by: Sebastian Spaeth <Sebastian <at> SSpaeth.de>
---
 offlineimap/init.py      |    3 ++-
 offlineimap/ui/UIBase.py |    3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/offlineimap/init.py b/offlineimap/init.py
index 296b84b..866c950 100644
--- a/offlineimap/init.py
+++ b/offlineimap/init.py
 <at>  <at>  -221,7 +221,8  <at>  <at>  class OfflineImap:
                 ui._msg("Debug mode: Forcing to singlethreaded.")
                 options.singlethreaded = True

-            for type in options.debugtype.split(','):
+            debugtypes = options.debugtype.split(',') + ['']
+            for type in debugtypes:
                 type = type.strip()
                 ui.add_debug(type)
                 if type.lower() == 'imap':
diff --git a/offlineimap/ui/UIBase.py b/offlineimap/ui/UIBase.py
index 02593e6..6eb1398 100644
--- a/offlineimap/ui/UIBase.py
+++ b/offlineimap/ui/UIBase.py
 <at>  <at>  -25,7 +25,8  <at>  <at>  from StringIO import StringIO
 from Queue import Empty

[PATCH v4 2/2] True 1-way sync (backup)

This commit enables true 1-way syncing between repositories. This has
often been demanded for backup purposes when you do not want to cause
accidental modifications of your backup that would be propagated to the
other side.

This has been implemented by allowing to configure a Repository as
'readonly' to forbid any modification on it.

'readonly' applies to all the type of repositories.

Signed-off-by: Sebastian Spaeth <Sebastian <at> SSpaeth.de>
---
The change to v3 is the improved commit message, an added space after a
comma and changing the debug type from "imap" to "" as was requested by
Nicolas, the changes are really minor. I can send an interpatch diff
though.

 Changelog.draft.rst     |    5 +++++
 offlineimap.conf        |   25 ++++++++++++++++---------
 offlineimap/accounts.py |   24 +++++++++++++++++-------
 3 files changed, 38 insertions(+), 16 deletions(-)

diff --git a/Changelog.draft.rst b/Changelog.draft.rst
index 0a85888..883b9a6 100644
--- a/Changelog.draft.rst
+++ b/Changelog.draft.rst
 <at>  <at>  -13,6 +13,11  <at>  <at>  others.
 New Features
 ------------

Re: ERROR in syncfolder for (username) folder INBOX : null byte in argument for long()

_______________________________________________
OfflineIMAP-project mailing list
OfflineIMAP-project <at> lists.alioth.debian.org
http://lists.alioth.debian.org/mailman/listinfo/offlineimap-project

OfflineIMAP homepage: http://software.complete.org/offlineimap

Re: [PATCH] Re: Don't strip whitespace in the -f option

On Thu, 28 Apr 2011 18:29:31 +0200, Nicolas Sebrecht <nicolas.s-dev <at> laposte.net> wrote:
> No, this is WRONG. Dan didn't produce that patch, sorry.

The code change itself does come from Dan in this case
(mail id:"87oc3z428t.fsf <at> uwo.ca"), and this is as close as a patch gets,
so I don't really feel as the author though 

But whatever works for you is good.

> The best I can do is 
>   Suggested-by: Dan Christensen <jdc <at> uwo.ca>
>   Signed-off-by: Sebastian Spaeth <Sebastian <at> SSpaeth.de>
> 
> and will change it that way.

Fine with me. 

Sebastian

_______________________________________________
OfflineIMAP-project mailing list
OfflineIMAP-project <at> lists.alioth.debian.org
http://lists.alioth.debian.org/mailman/listinfo/offlineimap-project

OfflineIMAP homepage: http://software.complete.org/offlineimap