Journaled ext3 file system crashed

I fsck'd it and it seems to be in pretty good shape.  This is 2.4.18 - RH9.
This machine can't handle file system problems.  It has to stay RH9 but I
could back out the journalling.  

The machine is only 3 months old.  I don't think it's a hardware issue.  No
error messages and the symptom (aside from being unusable) was I/O error
seeking <different blocks>.  The fsck unloaded the journal (about 6 entries
for the tomcat user on /home and about a dozen for root on /usr) and a later
preen seemed to take a long time on those partitions.  I don't see that I
lost anything although the databases aren't coming up.  

Thoughts?  similar experiences?

I'd like to know what else I could do to both find the problem source and to
make sure I lose as little as possible when recovering.

I used e2image -r and ran the fsck on it to make sure I knew what it was
going to do.  But I don't know anything really about the problem area.  What
else would be a good thing to try?

Dana Bourgeois

Re: Journaled ext3 file system crashed

On Wed, Oct 01, 2003 at 11:54:10PM -0700, Dana Bourgeois wrote:
> I fsck'd it and it seems to be in pretty good shape.  This is 2.4.18 - RH9.
> This machine can't handle file system problems.  It has to stay RH9 but I
> could back out the journalling.  
> 
> The machine is only 3 months old.  I don't think it's a hardware issue.  No
> error messages and the symptom (aside from being unusable) was I/O error
> seeking <different blocks>.  

Absolutely no messages in your log files?  There really should have
been some, unless the filesystem that went read-only when the
filesystem error occured was where your logs are being kept?

Two things I can think of trying:  First of all, make sure you have the
lasted errata kernel from Red Hat installed.  Secondly, consider doing
network-based syslogging (i.e., configure syslog.conf to send its logs
over the network to a logging server).  I generally consider that to
be a really good idea for production servers.

						- Ted

Re: journal buffer_credits problem

Hi,

On Wed, 2003-09-24 at 09:13, Akshat Aranya wrote:

> We need to be careful to not perform file system operations that are 
> specific to the semantics of one particular file
> system. Is there any way that you suggest that can get us a nested 
> transaction (through a VFS operation or an operation exposed by the file 
> system to VFS) when there's a transaction already associated with the 
> current process?

No.

Trouble is, transactions need to be atomic (of course!) but there's only
finite space in the journal.  We need to know, *before* the transaction
starts, whether or not there is space for the transaction in the current
journal segment, or whether we have to start a new compound transaction
in the journal and potentially do a checkpoint to evict some old space.

That's why we can't grow an existing open transaction (or rather, we
can't _guarantee_ it --- there are functions to try to expand a
transaction so that long-running operations like truncating huge files
can try to stay within a single transaction, but the expand can always
fail.)

Cheers,
 Stephen

Re: Freeing blocks not in datazone

Hi,

On Fri, 2003-09-19 at 19:44, Sharif Islam wrote:

> attempt to access beyond end of device
> 09:00: rw=0, want=1635237736, limit=976791680

> Can I run any other diagnostics to find out more about RAID
> drive? Is e2fsck an option now? It seems I still can read/write to the
> aforementioned RIAD drive without any problem.

Unmounting and doing a forced fsck is always advisable if you see such
errors, yes.

--Stephen

Re: ext3 hack: using the journal to replicate a filesystem

Hi,

On Fri, 2003-09-19 at 14:20, Olivier Kaloudoff wrote:

> 	I'm looking for a way to use ext3 in the following
> setup for experimentations purposes:
> 
> 	I have a machine A, a machine B, one of which is
> read-write, and machine B is read-only.
> 
> 	I would want the machine B to be the exact copy
> of machine A, automagically, like it would happen in an
> SQL replication scheme.
> 
> 	Is it possible to do with ext3, using the hidden
> .journal file as a source for the machine B filesystem ?

Fundamentally, you need a clustered cache-coherency mechanism to achieve
this.  Otherwise, when machine A modifies data, there's no way for
machine B to know what has changed.

In other words, you don't just need to know _what_ has changed, you also
need to know _when_.

In theory, though, that could be piggy-backed on top of the ext3
replication mechanism.

A bigger problem would be that the Linux filesystem layer isn't used to
dealing with the fs changing beneath its feet dynamically.  Even a
readonly copy would be hard to keep totally consistent, as there are all

Re: Question about directory permissions

Hi,

On Sat, 2003-09-20 at 02:19, Dana Bourgeois wrote:

> I'm packaging up a web application and it would actually be kind of cool to
> allow users to place jar files in the libs directory of the webapp but force
> the files to be owned by the webapp administrator.
> 
> Is this possible?  My tests seem to prove that it isn't.

No, I'm afraid not.  Setting the gid automatically is as close as you
will get.

--Stephen

chattr +j ==> Oops

[sent this to ext2-devel but haven't seen any other traffic there, so
cc-ing it here]

[Last week] while configuring a new box I came across a site suggesting:

chattr -R -S +j +A /var/spool/postifx

to improve postfix's speed on ext3, in lieu of mounting a separate fs
there w/ data=journal.

So I tried it.  Both on the new box (2.4.21-7um) and on my laptop
(about three csets shy of the v2.6.0-test6 tag).

Both bombed out in the journal.  

My uml provider forwarded this from the uml console:

Assertion failure in journal_revoke() at revoke.c:329: 
"!(__builtin_constant_p(BH_Revoked) ? 
constant_test_bit((BH_Revoked),(&bh->b_state)) : 
variable_test_bit((BH_Revoked),(&bh->b_state)))"
Kernel panic: kernel BUG at revoke.c:329!

I was able to use dmesg(1) to see the prink() the laptop generated
but was unable to copy it down.  It did reference journal... and
var was remounted ro, so I presume it is a similar issue.

I thought the earlier issues with data != ordered were corrected?

Do known issues remain, or is this something new?

ext3

Re: ext3

Hi,

On Thu, 2003-10-09 at 17:42, abhishek patel wrote:
> Hi all . Can anyone tell me what are the other advantages of using
> ext3 over ext2 other than fast fast recovery on crashes

ext3 was designed to be a minimal change to ext2, offering journaling
for fast recovery but no other new features.  

ext3 therefore has largely the same features as ext2: things like ACLs
in the 2.6 ext2 tree are in ext3 too.

The only other things I can think of that mark ext3 out over ext2 are
that the ext3 default ordered journaling mode provides guarantees
against seeing "stale" data in a file after a crash; and that the
"htree" fast directory indexing code in 2.6's ext3 is not in ext2.

Cheers,
 Stephen

ordered vs full journaling mode

Hi,

I was trying to list problems with ordered journaling mode that
are not present in the full (data=journal) journaling mode. Please
letme know if this is incorrect:

Assume that the following sequence of events take place:

1. an already existing file A is deleted
2. new file B is created
3. blocks of A are allocated to B
4. blocks of B are written
5. inode of A & B are updated

In ordered data journaling mode, if the system crashes after writing
the blocks of B but before updating the metadata of A (that is,
between steps 4 and 5), then A might see spurious data. However this
problem cannot happen if we have a full journaling mode.

Is the above example correct ? If not, could you please give me a
situation where ordered data journaling mode will not give as much
reliability as the full journaling mode ?

thanks,
Vijayan