headers
Geir Voll Nielsen | 1 Jan 2007 10:13

lmtpd


Hi,

I am currently running dbmail-2.0.11 in production (Im am about to upgrade
in the near future...). In the mean time I am having a problem with lmtpd
and defunct processess. After running for a while I am seeing this in the
maillog.

serverchild.c,PerformChildTask: maximum number of connections reached,
stopping now
Dec 31 22:01:29 clstr-node-01 dbmail/lmtpd[7238]: pool.c,child_register:
register child [7238]
Dec 31 22:01:29 clstr-node-01 dbmail/lmtpd[7239]: pool.c,child_register:
register child [7239]
Dec 31 22:01:29 clstr-node-01 dbmail/lmtpd[7240]: pool.c,child_register:
register child [7240]
Dec 31 22:01:29 clstr-node-01 dbmail/lmtpd[7241]: pool.c,child_register:
register child [7241]
Dec 31 22:01:29 clstr-node-01 dbmail/lmtpd[7242]: pool.c,child_register:
register child [7242]
Dec 31 22:01:29 clstr-node-01 dbmail/lmtpd[7243]: pool.c,child_register:
register child [7243]
Dec 31 22:01:29 clstr-node-01 dbmail/lmtpd[7244]: pool.c,child_register:
register child [7244]
Dec 31 22:01:29 clstr-node-01 dbmail/lmtpd[7245]: pool.c,child_register:
register child [7245]
Dec 31 22:01:29 clstr-node-01 dbmail/lmtpd[7246]: pool.c,child_register:
register child [7246]
Dec 31 22:01:29 clstr-node-01 dbmail/lmtpd[7247]: pool.c,child_register:
register child [7247]
(Continue reading)

Permalink | Reply | 0 comments |
headers
Tommi Lätti | 1 Jan 2007 18:30

cannot restore backup - urgent help greatly appreciated...

Hello,

Decided to go ahead and do the upgrade today, using postgresql as the 
database and running 2.0.9.

Now, as I'm a good boy I of course took a full pg_dump from the database 
and started then process. Compiled new binaries and so.

Now I ran the migrate2.0to2.2 script against the database and well, not 
everything goes well. I might have cancelled the migration script 
thinking about a better way to do it but can't really remember now.

So then I proceed to take another dump, this time not full, I excluded 
the schema (pg_dump -a). Now what went really wrong was that I actually 
overwrote the original full dump... gahhhhhh... (didn't realize it yet 
at this point)

So in the long run nothing seemed to start working and I was thinking to 
go back to the 2.0.9 and try another time. I drop the database, and go 
for the restore.

Hmm, doesn't have a schema... how's that...

I check the sql dump and I see some very worrying fields, like "SELECT 
pg_catalog.setval('dbmail_sievescripts_idnr_seq', 1, false);". At this 
point I realized that my 'backup' dump is tad bit newer than I hoped for.

So ok, I try different types of getting the sql file (3.3G) pushed back 
into the DB since I still hope I can use the 2.0.9 daemons against it. I 
hope I'm right here?
(Continue reading)

Permalink | Reply | 8 comments |
headers
Tommi Lätti | 1 Jan 2007 19:07

Re: cannot restore backup - urgent help greatly appreciated...

A little update is in order.

I managed to restore the database back. Had to delete all indexes and 
constraints, then restore the data and after that restore the indexes 
and constraints.

Now I run the 2.0.9 imapd and it connects and all, but sees 0 messages. 
I know the messages are there and should be available since I can see 
and read them thru dbmailadministrator.

Also if I copy a message from different account to the inbox, there's 
still nothing shown on the client window. From DBMA I can see the mails 
are actually there and are readable.

Does the migrate2.0_2.2 script make so that the database is then 
unusable for 2.0 series?

Here's a trace5 when I click on Inbox. I should see 4 read messages. MUA 
is Thunderbird.

-- clip --
Jan  2 03:02:17 vanessa dbmail/imap4d[22073]: 
serverchild.c,PerformChildTask: incoming connection from 
[222.228.208.238 (d238.FtokyoFL32.vectant.ne.jp)]
Jan  2 03:02:17 vanessa kernel: Jan  2 03:02:17 vanessa 
dbmail/imap4d[22073]: serverchild.c,PerformChildTask: incoming 
connection from [222.228.208.238 (d238.FtokyoFL32.vectant.ne.jp)]
Jan  2 03:02:17 vanessa dbmail/imap4d[22073]: 
serverchild.c,PerformChildTask: client info init complete, calling 
client handler
(Continue reading)

Permalink | Reply | 7 comments |
headers
Paul J Stevens | 1 Jan 2007 22:08
Picon
Favicon
Gravatar

Re: cannot restore backup - urgent help greatly appreciated...

Tommi Lätti wrote:

> Does the migrate2.0_2.2 script make so that the database is then
> unusable for 2.0 series?

Check the text->bytea conversion of dbmail_messageblks.messageblk, the
first transaction block at the top of the migration script. That's the
only *real* change that makes imapd-2.0 choke on a the database.

> -- clip --

The log-file actually looks clean, but the client will choke on the
conversion issue.

--

-- 
  ________________________________________________________________
  Paul Stevens                                      paul at nfg.nl
  NET FACILITIES GROUP                     GPG/PGP: 1024D/11F8CD31
  The Netherlands________________________________http://www.nfg.nl
Permalink | Reply | 6 comments |
headers
Tommi Lätti | 2 Jan 2007 05:15

Re: cannot restore backup - urgent help greatly appreciated...

Paul J Stevens wrote:
> Tommi Lätti wrote:
> 
>> Does the migrate2.0_2.2 script make so that the database is then
>> unusable for 2.0 series?
> 
> Check the text->bytea conversion of dbmail_messageblks.messageblk, the
> first transaction block at the top of the migration script. That's the
> only *real* change that makes imapd-2.0 choke on a the database.

I'm not sure if the data it's in bytea already. Schema is from 2.0.9, I 
imported the data to that schema. I could of course try putting 2.2.1 
schema on and then importing the data to that.

The tricky question is that I don't really remember how far the migrate 
script went. But since the bytea conversion is at the top, I could 
safely assume that it's definitely converted at least some amount of 
messageblocks, if not all.

Can I cast them back to text? Or should I try to run the migrate script 
again to get the new tables and then dbmail-util -ay from 2.2.1?

>> -- clip --
> 
> The log-file actually looks clean, but the client will choke on the
> conversion issue.

Allrighty, I could check that with tcpdump maybe but I think I can 
assume that it's now bytea... although the schema says text.
(Continue reading)

Permalink | Reply | 5 comments |
headers
Aaron Stone | 2 Jan 2007 05:54
Gravatar

Re: cannot restore backup - urgent help greatly appreciated...

On Tue, 2007-01-02 at 13:15 +0900, Tommi Lätti wrote:

> Allrighty, I could check that with tcpdump maybe but I think I can 
> assume that it's now bytea... although the schema says text.

Well, if the schema says text, then the bytea conversion has not taken
place. If you have a backup, carefully renamed to avoid the default name
of backing up again... run the conversion script. You might want to
break the script into two parts, first the bytea conversion and then the
schema changes.

Aaron
Permalink | Reply | 4 comments |
headers
Tommi Lätti | 2 Jan 2007 06:19

Re: cannot restore backup - urgent help greatly appreciated...

Aaron Stone wrote:
> On Tue, 2007-01-02 at 13:15 +0900, Tommi Lätti wrote:
> 
>> Allrighty, I could check that with tcpdump maybe but I think I can 
>> assume that it's now bytea... although the schema says text.
> 
> Well, if the schema says text, then the bytea conversion has not taken

Is there a way to tell from the data if the bytea conversion has taken 
place already? When I look at the pgsql documentation, I think the text 
should have quite a lot of escapes there. The data I have seems to have 
\012's and so.

> place. If you have a backup, carefully renamed to avoid the default name

I have now multiple snapshots. Not going to do the same little mistake 
again...

> of backing up again... run the conversion script. You might want to
> break the script into two parts, first the bytea conversion and then the
> schema changes.

I just did the script again, line by line manually. No problems nor any 
errors during the script. Bytea seemed to go thru just well also, but I 
can't tell if there's any difference in the data in the messageblocks 
themselves.

--

-- 
br,
Tommi
(Continue reading)

Permalink | Reply | 3 comments |
headers
Tommi Lätti | 2 Jan 2007 07:01

Re: cannot restore backup - urgent help greatly appreciated...

Okay,

dbmail-util -byv goes thru nicely, just 220 un-cached physmessages which 
don't get repaired even with multiple runs of the util.

The tables also seem to have populated (listing only few here):

dbmail_datefield	37092 rows
dbmail_envelope		73303 rows
dbmail_fromfield	13973 rows
dbmail_headername	20 rows
dbmail_headervalue	35894 rows
dbmail_mailboxes	241 rows
dbmail_messageblks	75325 rows
dbmail_messages		26879 rows
dbmail_physmessage	36212 rows
dbmail_referencesfield	25 rows
dbmail_replytofield	4757 rows

but

after starting the imapd I can see the messages, just without any header 
or body data :/

If I copy a message from different account, it shows just nicely.

--

-- 
br,
Tommi
(Continue reading)

Permalink | Reply | 2 comments |
headers
Tommi Lätti | 2 Jan 2007 07:18

Re: cannot restore backup - urgent help greatly appreciated...

I checked again with dbmailadministrator, here are screenshots from what 
the difference looks like...

The mail I copied from another account:
http://www.blosphere.net/~sty/goodmail.png

The mail that were in at 2.0.9 and now converted:
http://www.blosphere.net/~sty/badmail.png

I think a conversion issue. Maybe I did a double bytea conversion? I 
wonder if it is possible.

Any ideas how to recover from this would be welcome.

--

-- 
br,
Tommi
Permalink | Reply | 1 comment |
headers
Tommi Lätti | 2 Jan 2007 12:36

Re: cannot restore backup - urgent help greatly appreciated...

I think I'm done now, and everything is fine. Victory! :)

So what happened...

It became quite clear that the data-only dump I had, had it's 
messageblocks already converted to bytea. Now what went wrong with 
earlier restore attempts was that I was importing the data over to 2.0.9 
schema where the column type is text.

Now, running the conversion script again doesn't help much, if bytea 
data is imported to a text field it'll just break things and it wont work.

So I tried a different approach, created a new db with 2.2.1 schema, and 
imported the data there. I was quite confident after I browsed thru some 
messageblocks and I could see the all-familiar \n\n's there instead of 
\012's :)

So 2.2.1 schema in, remove indexes and constraints, import data, restore 
constraints and indexes, vacuum (to populate the indexes) and then 
dbmail-util -by. First time I've seen the utility to go thru second time 
without any errors.

Fire up the imapd, try mutt... and I have the headers and the bodies 
showing up. *PHEW*.

16 hours of work, and all seems to be well. Undelivered mails are slowly 
getting in from the backup MX.

Sleepy time :)
(Continue reading)

Permalink | Reply | 0 comments |

Gmane