Bron Gondwana | 1 Oct 01:32 2008

Re: Question about bodystructure answer on cyrus 2.2.13 (debian etch)

On Tue, Sep 30, 2008 at 11:10:27PM +0200, Thomas Cataldo wrote:
> NIL)("MESSAGE" "RFC822" ("NAME*" {147}
> ISO-8859-1''%6A%27%61%69%20%6C%61%6E%63%E9%20%75%6E%65%20%67%72%6F%73%73%65%20%69%6D%70%72%65%73%73%69%6F%6E%20%73%75%72%20%4C%50%32%33%30%30%52%56)
> NIL NIL "8BIT" 1544 ("Wed, 14 Feb 2007 10:12:39 +0100" "j'ai

> My problem is in the response to the body structure request : cyrus
> tells me he will send me 147 bytes and my lib can deal with that. The
> problem is, the data sent by cyrus is way longer than 147 bytes... can
> someone point me to what I misunderstood ?

The "147" bytes is this bit here:
ISO-8859-1''%6A%27%61%69%20%6C%61%6E%63%E9%20%75%6E%65%20%67%72%6F%73%73%65%20%69%6D%70%72%65%73%73%69%6F%6E%20%73%75%72%20%4C%50%32%33%30%30%52%56

Which is just one "ATOM".  All the stuff after this is other ATOMs,
which aren't part of the 147 bytes.  You can easily get multiple
ATOMs in a single response encoded in this way.

Bron.
----
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Thomas Cataldo | 1 Oct 02:08 2008
Picon

Re: Question about bodystructure answer on cyrus 2.2.13 (debian etch)

On Wed, Oct 1, 2008 at 1:32 AM, Bron Gondwana <brong <at> fastmail.fm> wrote:
> On Tue, Sep 30, 2008 at 11:10:27PM +0200, Thomas Cataldo wrote:
>> NIL)("MESSAGE" "RFC822" ("NAME*" {147}
>> ISO-8859-1''%6A%27%61%69%20%6C%61%6E%63%E9%20%75%6E%65%20%67%72%6F%73%73%65%20%69%6D%70%72%65%73%73%69%6F%6E%20%73%75%72%20%4C%50%32%33%30%30%52%56)
>> NIL NIL "8BIT" 1544 ("Wed, 14 Feb 2007 10:12:39 +0100" "j'ai
>
>> My problem is in the response to the body structure request : cyrus
>> tells me he will send me 147 bytes and my lib can deal with that. The
>> problem is, the data sent by cyrus is way longer than 147 bytes... can
>> someone point me to what I misunderstood ?
>
> The "147" bytes is this bit here:
> ISO-8859-1''%6A%27%61%69%20%6C%61%6E%63%E9%20%75%6E%65%20%67%72%6F%73%73%65%20%69%6D%70%72%65%73%73%69%6F%6E%20%73%75%72%20%4C%50%32%33%30%30%52%56
>
> Which is just one "ATOM".  All the stuff after this is other ATOMs,
> which aren't part of the 147 bytes.  You can easily get multiple
> ATOMs in a single response encoded in this way.

Thank you, I realized what I was doing wrong.

>
> Bron.
>
----
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Adam D | 1 Oct 07:08 2008
Picon

Re: Issues with user.sub files - subscription files hangs system.

Adam D wrote:
> Adam D wrote:
>   
>> Jens wrote:
>>   
>>     
>>> Adam D schrieb:
>>>   
>>>     
>>>       
>>>> Adam D wrote:
>>>>     
>>>>       
>>>>         
>>>>> I have noticed this error in the log:
>>>>> Sep 12 18:57:44 vts-post cyrus/imap[19629]: IOERROR: locking
>>>>> /var/lib/cyrus/domain/W/domain1/user/A/userbox.sub: Interrupted system call
>>>>>
>>>>> Soon afterwards I notice imapd runs at 99.8 % and I can not kill the
>>>>> process.  It looks like the file is locked and can't be written to?  I
>>>>> saw another file called userbox.sub.NEW but even that files can't be
>>>>> read.  When I attempt the terminal freezes.  I also noticed the system
>>>>> can't be rebooted as well.  I don't know what could be causing this. 
>>>>> Does anyone have any incite? 
>>>>>       
>>>>>         
>>>>>           
>>> [...]
>>>   
>>>     
(Continue reading)

Bron Gondwana | 1 Oct 08:40 2008

Re: Issues with user.sub files - subscription files hangs system.

On Tue, Sep 30, 2008 at 10:08:42PM -0700, Adam D wrote:
> > createonpost: yes
> > autocreateinboxfolders:  "Drafts|Sent|Templates|Trash|Junk Mail"
> > autosubscribeinboxfolders: "Drafts|Sent|Templates|Trash|Junk Mail"

I would suspect these in the first instance, having never seen this
behaviour before on any of our production systems.  You're running
the auto-create-folders patch obviously.

The existance of the .NEW file for a skiplist suggests that something
crashed during a checkpoint.  Are you sure that it's a skiplist database
being used for the .sub file?

Can you email me the .sub file if it's not too confidential.  I'm
interested in seeing if anything's wrong with it.  I really do think
it's probably in the autosubscribe code... I don't run the Ubuntu
or Debian packages directly anyway, because we use 2.3.x for the
replication features and some other goodies.

Regards,

Bron.
----
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Simon Matter | 1 Oct 08:47 2008
Picon

Re: Issues with user.sub files - subscription files hangs system.

> Adam D wrote:
>> Adam D wrote:
>>
>>> Jens wrote:
>>>
>>>
>>>> Adam D schrieb:
>>>>
>>>>
>>>>
>>>>> Adam D wrote:
>>>>>
>>>>>
>>>>>
>>>>>> I have noticed this error in the log:
>>>>>> Sep 12 18:57:44 vts-post cyrus/imap[19629]: IOERROR: locking
>>>>>> /var/lib/cyrus/domain/W/domain1/user/A/userbox.sub: Interrupted
>>>>>> system call
>>>>>>
>>>>>> Soon afterwards I notice imapd runs at 99.8 % and I can not kill the
>>>>>> process.  It looks like the file is locked and can't be written to?
>>>>>> I
>>>>>> saw another file called userbox.sub.NEW but even that files can't be
>>>>>> read.  When I attempt the terminal freezes.  I also noticed the
>>>>>> system
>>>>>> can't be rebooted as well.  I don't know what could be causing this.
>>>>>> Does anyone have any incite?
>>>>>>
>>>>>>
>>>>>>
(Continue reading)

Bron Gondwana | 1 Oct 08:53 2008

Re: Issues with user.sub files - subscription files hangs system.

On Tue, Sep 30, 2008 at 10:08:42PM -0700, Adam D wrote:
> >> I thought about that last night as well and we are having the same issue
> >> with a system using Ubuntu 7.10.  I did noticed though 7.04, 7.10, 8.04
> >> are all using 2.2.13 but different patches. 7.10 uses the 2.2.13-11
> >> while 8.10 uses the 2.2.13-13.  I am thinking of trying 8.10 to see if
> >> it works.. if not.. sadly I am wondering if we will have to move the
> >> system over to a full Debian stable/testing?  I would not be happy to
> >> build a Debian testing system to find out we have the same issue.  Would
> >> this be a Ubuntu related issue with their patches?  Has anyone else have
> >> had the same issue without using Ubuntu?

I'm running Ubuntu 8.04 on this laptop, so I've just pulled in the
sources for cyrus-imapd-2.2.

(hmm, I notice they're not shipping my skiplist patches.  I might
prod them about that...)

I don't see the autocreateinbox stuff anywhere in the code.  Odd.
Does Ubuntu actually apply those patches?

Bron.
----
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Adam D | 1 Oct 11:02 2008
Picon

Re: Issues with user.sub files - subscription files hangs system.

Sorry, been doing some extensive testing and every test still end in the
same results.

Bron Gondwana wrote:
> On Tue, Sep 30, 2008 at 10:08:42PM -0700, Adam D wrote:
>   
>>> createonpost: yes
>>> autocreateinboxfolders:  "Drafts|Sent|Templates|Trash|Junk Mail"
>>> autosubscribeinboxfolders: "Drafts|Sent|Templates|Trash|Junk Mail"
>>>       
>
> I would suspect these in the first instance, having never seen this
> behaviour before on any of our production systems.  You're running
> the auto-create-folders patch obviously.
>   
Ahhh, yes... We did not use the auto-create patch on this system.

> The existance of the .NEW file for a skiplist suggests that something
> crashed during a checkpoint.  Are you sure that it's a skiplist database
> being used for the .sub file?
>   
We had converted the sub file to a skiplist from a flat file and either
format still gave the .sub.NEW file.  The server had crashed on a couple
of power outages in a single day but oddly even a new built Debian
system without any of our configs still gave us the same issue. 
Currently the .sub file is a flat file.  From each of our testing this
happens on a fresh raw install to our production server.
> Can you email me the .sub file if it's not too confidential.  I'm
> interested in seeing if anything's wrong with it.  I really do think
> it's probably in the autosubscribe code... I don't run the Ubuntu
(Continue reading)

Adam D | 1 Oct 11:07 2008
Picon

Re: Issues with user.sub files - subscription files hangs system.

Simon Matter wrote:
>> [snip]
>> Has anyone else had this same issue and have been able to fix it?  I
>> really do not know why when deleting the username.sub.NEW file it is
>> recreated and when making changes to the subscriptions an imapd process
>> is spawned and freezes.
>>     
>
> I didn't follow the thread but, did you reconstruct your mailspool? Things
> like that are usually a filesystem full problem or some kind of corruption
> in the mailspool.
>
> Simon
>   
Yes, we ran reconstruct as cyrus user for all mailboxes.  It did take a
while but does reconstruct, reconstructs the sub and seen files?  Would
the corruption be in the mboxlist_db or in the subscription_db?

Thank you

-Adam
----
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

Adam D | 1 Oct 11:15 2008
Picon

Re: Issues with user.sub files - subscription files hangs system.

Bron Gondwana wrote:
> On Tue, Sep 30, 2008 at 10:08:42PM -0700, Adam D wrote:
>   
>>>> I thought about that last night as well and we are having the same issue
>>>> with a system using Ubuntu 7.10.  I did noticed though 7.04, 7.10, 8.04
>>>> are all using 2.2.13 but different patches. 7.10 uses the 2.2.13-11
>>>> while 8.10 uses the 2.2.13-13.  I am thinking of trying 8.10 to see if
>>>> it works.. if not.. sadly I am wondering if we will have to move the
>>>> system over to a full Debian stable/testing?  I would not be happy to
>>>> build a Debian testing system to find out we have the same issue.  Would
>>>> this be a Ubuntu related issue with their patches?  Has anyone else have
>>>> had the same issue without using Ubuntu?
>>>>         
>
> I'm running Ubuntu 8.04 on this laptop, so I've just pulled in the
> sources for cyrus-imapd-2.2.
>
> (hmm, I notice they're not shipping my skiplist patches.  I might
> prod them about that...)
>
> I don't see the autocreateinbox stuff anywhere in the code.  Odd.
> Does Ubuntu actually apply those patches?
>
> Bron

You are correct.  Ubuntu nor Debian are not shipping with the auto
create patches as far as I know.  An oversight and we took the auto
create out of imap.conf but it did not make any difference.

I had thought maybe any of the mail boxes were corrupted so we created a
(Continue reading)

Bron Gondwana | 1 Oct 15:00 2008

Re: Issues with user.sub files - subscription files hangs system.

On Wed, Oct 01, 2008 at 02:07:23AM -0700, Adam D wrote:
> Simon Matter wrote:
> >> [snip]
> >> Has anyone else had this same issue and have been able to fix it?  I
> >> really do not know why when deleting the username.sub.NEW file it is
> >> recreated and when making changes to the subscriptions an imapd process
> >> is spawned and freezes.
> >>     
> >
> > I didn't follow the thread but, did you reconstruct your mailspool? Things
> > like that are usually a filesystem full problem or some kind of corruption
> > in the mailspool.
> >
> > Simon
> >   
> Yes, we ran reconstruct as cyrus user for all mailboxes.  It did take a
> while but does reconstruct, reconstructs the sub and seen files?  Would
> the corruption be in the mboxlist_db or in the subscription_db?

*sigh*

It looks a whole lot like this old cow rearing its ugly head again:

http://lkml.org/lkml/2008/6/17/9

Here's the interesting bit:

    niov = 0;
    if (offset) {
        WRITEV_ADD_TO_IOVEC(iov, niov, (char *) db->base, offset);
(Continue reading)


Gmane