Joe Pruett | 2 Apr 23:52 2009

mix reconstructor?

according to:

http://www.washington.edu/imap/documentation/mixfmt.txt.html

there is no reconstruction tool for mix yet.  i have a user that deleted a 
bunch of messages via pop and didn't realize for a week.  now i want to 
recover the message files, but don't want to recover the existing index 
files.  i was thinking this would be a good use for a reconstructor.

maybe i should just recover the inbox to a folder and make them use an 
imap interface to recover what they want...

any other thoughts from out there?
_______________________________________________
Imap-uw mailing list
Imap-uw <at> u.washington.edu
http://mailman2.u.washington.edu/mailman/listinfo/imap-uw

Mark Crispin | 3 Apr 00:03 2009

Re: mix reconstructor?

On Thu, 2 Apr 2009, Joe Pruett wrote:
> http://www.washington.edu/imap/documentation/mixfmt.txt.html
> there is no reconstruction tool for mix yet.

Out of date.  Look at mixrbld and mixdfix.

> i have a user that deleted a 
> bunch of messages via pop and didn't realize for a week.  now i want to 
> recover the message files, but don't want to recover the existing index 
> files.  i was thinking this would be a good use for a reconstructor.

Unless there were continously multiple sessions with that mailbox open, 
the data is almost certainly gone from the disk by now.  Whenever a 
session gets exclusive access it "burps" out all the expunged messages.

So, basically, the resource is probably backup tape at this point.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.
_______________________________________________
Imap-uw mailing list
Imap-uw <at> u.washington.edu
http://mailman2.u.washington.edu/mailman/listinfo/imap-uw

Joe Pruett | 3 Apr 00:16 2009

Re: mix reconstructor?

> Unless there were continously multiple sessions with that mailbox open, the 
> data is almost certainly gone from the disk by now.  Whenever a session gets 
> exclusive access it "burps" out all the expunged messages.
>
> So, basically, the resource is probably backup tape at this point.

sorry, by recover, i meant recover from tape.  but i didn't want to just 
recover on top of the inbox.  i'm trying the theory of recovering to a new 
folder and then copying via imap rather than try to do any rebuild magic. 
but i'll look at the mixrbld and mixdfix stuff...
_______________________________________________
Imap-uw mailing list
Imap-uw <at> u.washington.edu
http://mailman2.u.washington.edu/mailman/listinfo/imap-uw

Mark Crispin | 3 Apr 00:36 2009

Re: mix reconstructor?

On Thu, 2 Apr 2009, Joe Pruett wrote:
> sorry, by recover, i meant recover from tape.  but i didn't want to just 
> recover on top of the inbox.  i'm trying the theory of recovering to a new 
> folder and then copying via imap rather than try to do any rebuild magic. but 
> i'll look at the mixrbld and mixdfix stuff...

I think that recovering to a new folder, and then letting the user decide 
what to do with it, is best.  There is some black magic possible, but it's 
best left to sorcerers who can talk with the user and determine if 
invoking demons is safe...  ;-)

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.
_______________________________________________
Imap-uw mailing list
Imap-uw <at> u.washington.edu
http://mailman2.u.washington.edu/mailman/listinfo/imap-uw

Djuhaeri Harapan | 3 Apr 16:17 2009
Picon
Picon

Imap-uw and mbox corruption

Hello all,

I try to use imapd-2007e on a linux machine & have the same problem as 
Chuck described in 
http://mailman2.u.washington.edu/pipermail/imap-uw/2008-January/001825.html

The symptom is also the same, the corruption is always at the start of 
the mailbox and it is unreadable characters like:
^W^C^A^ <at>  <at> kM^Z<ÖoÅ^BåÄox 
sSý^Q^B×\<85>ä©gÂ_Ü/Èe^ZòÕ;A^OÂ<99>:m^L~Â#yXå<8e>^X^ <at> !<9f>9Ü<9a><9a><82>^MöPÅ÷^Kqep2GlmX,waJcpLpcRGGASIGebw=

I have tried  Mark's suggestion, "removing the graceful close code in 
imapd.c"
http://mailman2.u.washington.edu/pipermail/imap-uw/2008-January/001826.html

Unfortunately it doesn't help.

Yes, the spool directory is on nfs mounted file system,
but imap-2004a works fine.

Some one can help ?

Kind regards,
Djuhaeri

_______________________________________________
Imap-uw mailing list
Imap-uw <at> u.washington.edu
http://mailman2.u.washington.edu/mailman/listinfo/imap-uw

(Continue reading)

Guy Dawson | 3 Apr 17:00 2009
Picon

Re: mix reconstructor?

Mark Crispin wrote:
> On Thu, 2 Apr 2009, Joe Pruett wrote:
>> sorry, by recover, i meant recover from tape.  but i didn't want to 
>> just recover on top of the inbox.  i'm trying the theory of recovering 
>> to a new folder and then copying via imap rather than try to do any 
>> rebuild magic. but i'll look at the mixrbld and mixdfix stuff...
> 
> I think that recovering to a new folder, and then letting the user 
> decide what to do with it, is best.  There is some black magic possible, 
> but it's best left to sorcerers who can talk with the user and determine 
> if invoking demons is safe...  ;-)

That's what we do. Restore the folder to <folder>.bck and let the user
extract the messages they need. Quick, simple and understandable by
the user...

Guy
-- --------------------------------------------------------------------
Guy Dawson                 I.T. Systems Manager         Crossflight Ltd
guy <at> crossflight.co.uk         07973  797819                01753 776104

*******************************************************************
This message contains the views and opinions of a Crossflight
Limited employee and at this stage are in no way a direct
representation of Crossflight Limited.

Crossflight Limited is an international express courier, mailing
and logistics service provider. This communication and any
attachments are confidential and may be protected from disclosure.
We endorse no advice or opinion contained in this communication
(Continue reading)

Ken.Koch | 3 Apr 20:39 2009
Picon

flock (fcntl) with QFS Shared FS

I have an environment using UW-Imap 2004g on multiple machines that share common mailboxes via a Sun QFS Shared file system mounting from SAN. These mbox formats work fine so far using .locking techniques.

 

I’m looking at converting to MBX or Mix format (finally!) and am doubtful that the flock implementation will work out of the box. In my testing, there have been some random corruption issues with MBX style boxes when I scatter usage among servers to the same mailbox. QFS supports both flock and fcntl locking (I’m on Solaris 10, so I guess it’s fcntl) but c-client is putting those inode exclusive lock files in /tmp. QFS also controls file access at the FS level so that only one client can write to a file at any given time unless I specifically specify multi-writer ability at mount time, which I’m not doing. QFS is a server-client model; a metadata server controls access and locking while data access is direct. I’m wondering if the separate servers not knowing about the /tmp.lockfiles is the reason for the corruption. It’s mainly only noticeable when larger attachments are being delivered (snarfed).

 

Are there any modifications I can do to share the location of these tmp lock files on one of my shared filesystems to ensure the flock’ing success on MBX/Mix formats between multiple servers? Will that help or hurt things? Am I just out of luck when it comes to exclusive locking and multiple servers? The current environment provides good redundancy and load balancing as it is now. I’d like to convert to newer mailbox formats to get better performance for larger mailboxes.

_______________________________________________
Imap-uw mailing list
Imap-uw <at> u.washington.edu
http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Mark Crispin | 4 Apr 00:12 2009

Re: flock (fcntl) with QFS Shared FS

Ken -

You're stepped on a hornet's nest...

I understand what you want to do.  At Messaging Architects (MA), we're 
busy at work on a new generation product that offers a clustered message 
store.  But in the concept of UW imapd, you're setting yourself up for a 
world of hurt.

[1] Almost all network filesystems are completely unsuitable for use with 
mbx format.

With few exceptions -- one being the Cluster File System (CFS) that we 
developed at MA -- network filesystems do NOT, repeat, NOT!! maintain the 
data/inode synchronization semantics of a real filesystem.

mbx is utterly dependent upon full filesystem synchronization semantics. 
mbx uses random access updates which MUST be in COMPLETE synchronization 
at all times.  Using mbx with any network filesystem that does not have 
those semantics is like playing Russian roulette; you WILL eventually lose 
the game.

mix format is less dependent -- in fact, we're working on a variant of mix 
that uses CAstor! --  but I still wouldn't want to risk a network 
filesystem that has not be specifically certified for mix.

[2] Solaris.

By using Solaris, and hence SVR4, you are stuck with the fcntl form of 
locking.  fcntl locking is broken by design.

To understand the insanity of using fcntl, consider the following 
statement from the BSD man pages on fcntl locking:

      This interface follows the completely stupid semantics of System
      V and IEEE Std 1003.1-1988 (``POSIX.1'') that require that all
      locks associated with a file for a given process are removed when
      ANY file descriptor for that file is closed by that process.
      This semantic means that applications must be aware of any files
      that a subroutine library may access.  For example if an
      application for updating the password file locks the password
      file database while making the update, and then calls
      getpwname(3) to retrieve a record, the lock will be lost because
      getpwname(3) opens, reads, and closes the password database.  The
      database close will release all locks that the process has
      associated with the database, even if the library routine never
      requested a lock on the database.  Another minor semantic problem
      with this interface is that locks are not inherited by a child
      process created using the fork(2) function.  The flock(2)
      interface has much more rational last close semantics and allows
      locks to be inherited by child processes.  Flock(2) is
      recommended for applications that want to ensure the integrity of
      their locks when using library routines or wish to pass locks to
      their children.  Note that flock(2) and fcntl(2) locks may be
      safely used concurrently.

Not only must the application must be aware of any files that a subroutine 
library may access, but it must also be aware of file access interactions 
in multiple subroutine libraries and in multiple instances in the same 
subroutine library.

To put it in a more vivid way: by using Solaris -- even with a local 
filesystem -- you are playing the same game of Russian roulette, with 
every cylinder in the revolver is loaded and the safety set.  You're 
pulling strenuously and repeatedly on the trigger, all the time assuring 
yourself that the safety will hold.

In this case, the safety is the flocksim code that I wrote, including its 
elaborate master/slave communication mechanism.  I banged my head against 
the wall for years in an only partially-successful attempt to keep that 
code effective.

I've seen the safety fail on that revolver many times, and I keep on 
fixing it; but every couple of years something new comes up to make it 
fail.

I do not recommend Solaris or any other form of SVR4 -- including HP-UX 
and AIX -- for use with UW or Panda IMAP.  Linux and BSD (including Mac OS 
X) are suitable operating systems.

[3] IMAP is a NAS.  Network filesystems are also a NAS.

This is a fundamental collision.  It usually does not make sense; and 
you're trying to do this in a multi-vendor environment.

The new server that I've developed at MA layers IMAP not only over CFS, 
but over a stateless store protocol.  This was a HUGE undertaking, and was 
accomplished only by divorcing mix from IMAP (the layering goes imap -> 
store -> mix -> CFS).  What's more, numerous changes were made in CFS 
specifically to accomodate mix; and store was built from the onset to 
accomodate IMAP.

None of these lower-level technologies -- store, mix, CFS -- are intended 
to be consumed by end users.  The consumers are our end-user facing 
products, such as the IMAP server.  So it's less of a "layer NAS on top of 
NAS" as it is a "how we implemented the NAS".

And even with all those advantages, it took us MONTHS to get it working.

You don't have any of those advantages.  You are attempting to layer one 
end-user facing NAS on top of another end-user facing NAS.  QFS, NFS, AFS, 
blurdybloopFS, etc. ad nauseum know nothing about what mix wants to do. 
You're crippled by the fcntl form of locking that requires an elaborate 
(ever wonder why imapd is always forking??) kludge to keep locks from 
being dropped.  And you don't have all the engineers responsible for every 
piece of the system working together in one room.

[4] Last, but not least, you're dealing with SUN.

SUN sells a email product.  They are hostile to open-source email software 
that competes with their product.

There is more that I can say about that subject.

...As I said, you're setting yourself up for a world of hurt...

With that said, and to answer your specific question:

In the mbx format, the same /tmp directory must be used for all agents 
that access the mbx format file.

mix does not depend upon the /tmp directory.

Nonetheless, I don't think that it will help.  You have lots of things 
working to break you; and for the most part, you are going to be on your 
own.  I don't support the UW code at all any more (and there are several 
known bugs in UW), and I only support the Panda code in specific 
instances.

Good luck.  You are going to need it.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.
_______________________________________________
Imap-uw mailing list
Imap-uw <at> u.washington.edu
http://mailman2.u.washington.edu/mailman/listinfo/imap-uw

Mark Crispin | 4 Apr 00:30 2009

Re: Imap-uw and mbox corruption

I am aware of the problem, but only recently that it happened on other 
systems than Solaris.  I am now aware that it happens on FreeBSD and 
Linux.

I am astounded by the fact that the first three bytes of the corruption 
always seem to be <CTRL/W><CTRL/C><CTRL/A>, 0x17 0x03 0x01, even on 
different platforms.

I have no idea what is significant about those three bytes, nor why it 
should suddenly have come up.  The only thing that seems related is 
that it happens in association with the imapd process being killed, and 
the switch to using setjmp()/longjmp() in the signal handlers.

Here is the underlying problem:

glibc "improved" things so that there are numerous new mutexes to cover 
possible multi-threading, even for non-threaded applications such as 
imapd.  There were other complications: e.g., putc() is now far slower.

The impact extends to syslog().  imapd, when it receives a signal to 
terminate, wants to issue a log message announcing this fact.  Thanks to 
the mutex, it no longer can do so in the signal handler...even when it has 
no intention of returning back to the interrupted code!

Matters are futher complicated in traditional UNIX mailbox format; imapd 
would like to update the mailbox before it exits (to avoid the problem of 
lost flags) but once again runs afoul of the mutex.

To work around this, I tried to use a setjmp()/longjmp() in the signal 
handler that would take imapd back to the main command loop and then to 
code to save and exit.  Supposedly, longjmp() is supposed to unwind 
whatever context occurred since the setjmp().

The patch that I suggested in January 2008 removed the step of saving the 
mailbox updates after the longjmp().  What this all means is that it 
should still do the longjmp(), but not write anything further to the file 
and just syslog() and exit.  The reports indicate that this doesn't seem 
to have fixed the problem.

I am working with a FreeBSD site that has experienced the problem to try a 
more aggressive version of the patch that removes the longjmp() entirely.

If it isn't the longjmp(), then I don't know what the hell is going on.

If it is the longjmp(), then I'll develop some other way around the issue 
in Panda IMAP.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.
_______________________________________________
Imap-uw mailing list
Imap-uw <at> u.washington.edu
http://mailman2.u.washington.edu/mailman/listinfo/imap-uw

David Houlder | 6 Apr 03:24 2009
Picon
Picon

Re: Imap-uw and mbox corruption (Mark Crispin)

> From:
> Mark Crispin <markrcrispin <at> panda.com>

> To work around this, I tried to use a setjmp()/longjmp() in the signal 
> handler that would take imapd back to the main command loop and then to 
> code to save and exit.  Supposedly, longjmp() is supposed to unwind 
> whatever context occurred since the setjmp().

I don't think longjmp() is async signal safe.

http://unix.derkeiler.com/Newsgroups/comp.unix.programmer/2003-04/0984.html

> If it isn't the longjmp(), then I don't know what the hell is going on.

I suspect that doing longjmp() in an asynchronously called signal 
handler is asking for trouble.

cheers
David


Gmane