headers
Christian G. Warden | 1 Oct 2007 01:26
Gravatar

Re: Assorted patches

On Sun, Sep 30, 2007 at 01:54:56PM -0400, Daniel Jacobowitz wrote:
> Here's the result of a lazy Sunday hacking on offlineimap.  Sorry for
> not breaking this into multiple patches.  They're mostly logically
> independent so just ask if that would make a difference.
> First, a new -q (quick) option.  The quick option means to only update
> folders that seem to have had significant changes.  For Maildir, any
> change to any message UID or flags is significant, because checking
> the flags doesn't add a significant cost.  For IMAP, only a change to
> the total number of messages or a change in the UID of the most recent
> message is significant.  This should catch everything except for
> flags changes.
> 
> The difference in bandwidth is astonishing: a quick sync takes 80K
> instead of 5.3MB, and 28 seconds instead of 90.

Thanks, Daniel.  My initial tests show significant speed improvements,
reducing sync time when there are no changes on either side from 255 to
50 seconds.

Christian
Permalink | Reply | 0 comments |
headers
John Goerzen | 1 Oct 2007 18:23
Favicon

Re: Assorted patches

On Sun September 30 2007 12:54:56 pm Daniel Jacobowitz wrote:
> Here's the result of a lazy Sunday hacking on offlineimap.  Sorry for

Wonderful!  You even updated the manpage.  Thanks.  Applied and pushed.  I 
will be making a new OfflineIMAP release shortly.  I haven't reviewed them 
in great detail, but I have glanced at the code and it looks good.

I had never seen the black-on-black in blinkenlights...  I'm curious if your 
terminal normally has a black or a white background?

-- John
Permalink | Reply | 1 comment |
headers
Daniel Jacobowitz | 1 Oct 2007 18:31

Re: Assorted patches

On Mon, Oct 01, 2007 at 11:23:37AM -0500, John Goerzen wrote:
> Wonderful!  You even updated the manpage.  Thanks.  Applied and pushed.  I 
> will be making a new OfflineIMAP release shortly.  I haven't reviewed them 
> in great detail, but I have glanced at the code and it looks good.
> 
> I had never seen the black-on-black in blinkenlights...  I'm curious if your 
> terminal normally has a black or a white background?

Black.  There's a lot of places where something sets the color to
black-on-black to erase... I believe the relevant one is the dots at
the bottom of the screen.  If the last thing before a warning was
erasing a thread's status dot, then the black lingers.

--

-- 
Daniel Jacobowitz
CodeSourcery
Permalink | Reply | 0 comments |
headers
Stewart Smith | 3 Oct 2007 06:39
Gravatar

Re: Synchronization Issue (Folders not deleted)

On Mon, 2007-08-27 at 15:19 +0900, Bernhard Egger wrote:
> Hi,
> 
> I'm using offlineimap to backup my mails on a courier IMAP server. It works
> nicely, although it took about 4 days with 100% CPU and >400MB of RAM to
> download my Junk folder.
> (insanely large, ~300K messages...I know, but still. The first 100K or so
> messages download reasonably fast, but then it gets slower and slower. At the
> end, only about 1 message per 3 seconds is downloaded. With 400MB of RAM one
> would assume that the necessary data structures are all in memory and thus
> accessed/traversed quickly).
This is largely due to the localstatus updates that offlineimap is
doing.

I have a patch (see the archives, or i can send it to you) that uses a
sqlite database per folder for localstatus that *dramatically* speeds
this up and significantly reduces CPU time (i've been using it myself on
my critical mail for several months now... so it does work).

It also saves a heap of RAM :)

I'd be interested to hear you trying an initial sync with my patch and
see how it goes (should be a lot quicker too)

(keep in mind that i have migration code for the old to new format for
localstatus, but not the other way... so you probably want to try this
with a fresh user somewhere and then blow it away)
--

-- 
Stewart Smith (stewart <at> flamingspork.com)
http://www.flamingspork.com/
(Continue reading)

Permalink | Reply | 2 comments |
headers
Stewart Smith | 3 Oct 2007 06:31
Gravatar

Re: Assorted patches

On Sun, 2007-09-30 at 13:54 -0400, Daniel Jacobowitz wrote:
> Here's the result of a lazy Sunday hacking on offlineimap.  Sorry for
> not breaking this into multiple patches.  They're mostly logically
> independent so just ask if that would make a difference.
> First, a new -q (quick) option.  The quick option means to only update
> folders that seem to have had significant changes.  For Maildir, any
> change to any message UID or flags is significant, because checking
> the flags doesn't add a significant cost.  For IMAP, only a change to
> the total number of messages or a change in the UID of the most recent
> message is significant.  This should catch everything except for
> flags changes.
> 
> The difference in bandwidth is astonishing: a quick sync takes 80K
> instead of 5.3MB, and 28 seconds instead of 90.
BEFORE:
rcvd=4863, sent=25921292, time=511
rcvd=5672, sent=15135719, time=509
rcvd=11413, sent=13180623, time=509

AFTER:
rcvd=3405, sent=26544, time=72
rcvd=4175, sent=27820, time=66
rcvd=3877, sent=781402, time=66

NEAT!

(both runs with my localstatus in sqlite patch)

> Third, some speedups in Maildir checking.  It's still pretty slow
> due to a combination of poor performance in os.listdir (never reads
(Continue reading)

Permalink | Reply | 0 comments |
headers
Bernhard Egger | 3 Oct 2007 13:07
Picon

Re: Synchronization Issue (Folders not deleted)


>> I'm using offlineimap to backup my mails on a courier IMAP server. It works
>> nicely, although it took about 4 days with 100% CPU and >400MB of RAM to
>> download my Junk folder.
>> (insanely large, ~300K messages...I know, but still. The first 100K or so
>> messages download reasonably fast, but then it gets slower and slower. At the
>> end, only about 1 message per 3 seconds is downloaded. With 400MB of RAM one
>> would assume that the necessary data structures are all in memory and thus
>> accessed/traversed quickly).
> This is largely due to the localstatus updates that offlineimap is
> doing.
> 
> I have a patch (see the archives, or i can send it to you) that uses a
> sqlite database per folder for localstatus that *dramatically* speeds
> this up and significantly reduces CPU time (i've been using it myself on
> my critical mail for several months now... so it does work).
I would appreciate it if you could send it to me - thanks a lot.

> I'd be interested to hear you trying an initial sync with my patch and
> see how it goes (should be a lot quicker too)
well, the 300K messages folder does not contain that many messages any more
(unfortunately). I just synched once to make a backup of my mail and then
deleted it. I have about 5K messages in there now, and with my +/- 200 junk
messages per day, it'll be quite big quite soon.

Anyways, I'll try it out and then post the results.

Thanks,
Bernhard
(Continue reading)

Permalink | Reply | 1 comment |
headers
Andy Schofield | 3 Oct 2007 19:07
Picon
Picon

Failing to reproduce file hierarchy locally

I've just discovered offlineimap and it seems to be just what I need.
However I am struggling to set it up as I want it.

My server (running postfix + dovecot) sets up a file hierarchy in my
home directory as follows:

ls -aF ~/Maildir

./
../
.archive/
cur/
.Drafts/
new/
.Queue/
.Sent/
subscriptions
tmp/
.Trash/

So new email goes into the topmost directory under new or cur, and if I
put things into the folder called .archive then they go
into .archive/cur

What I want to do is to have offlineimap create an identical hierarchy
on my laptop. However whatever I do, in terms of renaming etc makes no
difference. I tried the config recipe to remove INBOX but no change.
On the laptop I always get ls -aF Maildir

./
(Continue reading)

Permalink | Reply | 6 comments |
headers
Asheesh Laroia | 3 Oct 2007 22:40

Re: Failing to reproduce file hierarchy locally

On Wed, 3 Oct 2007, Andy Schofield wrote:

> I've just discovered offlineimap and it seems to be just what I need. 
> However I am struggling to set it up as I want it.
>
> My server (running postfix + dovecot) sets up a file hierarchy in my 
> home directory as follows:

Use mynametrans from my Python extensions:

http://svn.asheesh.org/svn/public/code/offlineimap_ext.py

See also my offlineimap rc:

http://svn.asheesh.org/svn/public/conf/offlineimaprc

Feel free to publicize these links!

-- Asheesh.

--
Since we're all here, we must not be all there.
 		-- Bob "Mountain" Beck
Permalink | Reply | 5 comments |
headers
Stewart Smith | 4 Oct 2007 06:06
Gravatar

Re: Synchronization Issue (Folders not deleted)

Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable
On Wed, 2007-10-03 at 20:07 +0900, Bernhard Egger wrote:
> > I have a patch (see the archives, or i can send it to you) that uses a
> > sqlite database per folder for localstatus that *dramatically* speeds
> > this up and significantly reduces CPU time (i've been using it myself o=
n
> > my critical mail for several months now... so it does work).
> I would appreciate it if you could send it to me - thanks a lot.

Attached.

Apply in this order:
offlineimap_sqlite_localstatus.patch
updateflags_commit_every100.patch

The second one is just an improvement for syncing lots of changes to
message flags. In the event of crash, you may have to redo the sync of
up to 100 message flags, but this is fine, it just takes you a bit of
extra time/bandwidth/filesystem ops.

> > I'd be interested to hear you trying an initial sync with my patch and
> > see how it goes (should be a lot quicker too)
> well, the 300K messages folder does not contain that many messages any mo=
re
> (unfortunately). I just synched once to make a backup of my mail and then
> deleted it. I have about 5K messages in there now, and with my +/- 200 ju=
nk
> messages per day, it'll be quite big quite soon.
>=20
(Continue reading)

Permalink | Reply | 0 comments |
headers
Andy Schofield | 4 Oct 2007 17:58
Picon
Picon

Re: Failing to reproduce file hierarchy locally

> 
> Use mynametrans from my Python extensions:
> 
> http://svn.asheesh.org/svn/public/code/offlineimap_ext.py
> 
> See also my offlineimap rc:
> 
> http://svn.asheesh.org/svn/public/conf/offlineimaprc
> 
> Feel free to publicize these links!
> 
> -- Asheesh.

That is a great help - they work perfectly! 

However, this structure now messes up the names of folders in Evolution
(as I should have realized - it generates maildir++). What I would like
to do now is to have the top level directory (~/Maildir) as my inbox and
the rest of the substructure as unhidden folders separated by "/"

So I have tried :

[Repository Local]
type = Maildir
localfolders = ~/Maildir
sep = /

[Repository Remote]
type = IMAP
remotehost = XXX.bham.ac.uk
(Continue reading)

Permalink | Reply | 4 comments |

Gmane