data corrupting bug in 2.4.20 ext3, data=journal

In 2.4.20-pre5 an optimisation was made to the ext3 fsync function
which can very easily cause file data corruption at unmount time.  This
was first reported by Nick Piggin on November 29th (one day after 2.4.20 was
released, and three months after the bug was merged.  Unfortunate timing)

This only affects filesystems which were mounted with the `data=journal'
option.  Or files which are operating under `chattr -j'.  So most people
are unaffected.  The problem is not present in 2.5 kernels.

The symptoms are that any file data which was written within the thirty
seconds prior to the unmount may not make it to disk.   A workaround is
to run `sync' before unmounting.

The optimisation was intended to avoid writing out and waiting on the
inode's buffers when the subsequent commit would do that anyway. This
optimisation was applied to both data=journal and data=ordered modes.
But it is only valid for data=ordered mode.

In data=journal mode the data is left dirty in memory and the unmount
will silently discard it.

The fix is to only apply the optimisation to inodes which are operating
under data=ordered.

--- linux-akpm/fs/ext3/fsync.c~ext3-fsync-fix	Sat Nov 30 23:37:33 2002
+++ linux-akpm-akpm/fs/ext3/fsync.c	Sat Nov 30 23:39:30 2002
 <at>  <at>  -63,10 +63,12  <at>  <at>  int ext3_sync_file(struct file * file, s
 	 */
 	ret = fsync_inode_buffers(inode);

Re: data corrupting bug in 2.4.20 ext3, data=journal

Andrew Morton wrote:
> 
> ...
> The fix is to only apply the optimisation to inodes which are operating
> under data=ordered.
> 

That "fix" didn't fix it.  Sorry about that.

Please avoid ext3/data=journal until it is sorted out.

Re: data corrupting bug in 2.4.20 ext3, data=journal

Andrew Morton wrote:

>In 2.4.20-pre5 an optimisation was made to the ext3 fsync function
>which can very easily cause file data corruption at unmount time.  This
>was first reported by Nick Piggin on November 29th (one day after 2.4.20 was
>released, and three months after the bug was merged.  Unfortunate timing)
>
In fact it was reported on lkml on 18th July IIRC before 2.4.19 was
released if that is any help to you. 2.4.19 and 2.4.20 are affected
and I haven't tested previous releases. I was going to re-report it
sometime, but Alan brought it to light just the other day.

Nick

another idiot and ext3 - The inode is from a bad block in the inode table

Re: another idiot and ext3 - The inode is from a bad block in the inode table

Re: data corrupting bug in 2.4.20 ext3, data=journal

Nick Piggin wrote:
> 
> Andrew Morton wrote:
> 
> >In 2.4.20-pre5 an optimisation was made to the ext3 fsync function
> >which can very easily cause file data corruption at unmount time.  This
> >was first reported by Nick Piggin on November 29th (one day after 2.4.20 was
> >released, and three months after the bug was merged.  Unfortunate timing)
> >
> In fact it was reported on lkml on 18th July IIRC before 2.4.19 was
> released if that is any help to you. 2.4.19 and 2.4.20 are affected
> and I haven't tested previous releases. I was going to re-report it
> sometime, but Alan brought it to light just the other day.
> 

Are you sure?  I can't make it happen on 2.4.19.  And disabling the new
BH_Freed logic (which went into 2.4.20-pre5) makes it go away.

--- linux-akpm/fs/jbd/commit.c~a	Sun Dec  1 23:10:12 2002
+++ linux-akpm-akpm/fs/jbd/commit.c	Sun Dec  1 23:10:27 2002
 <at>  <at>  -695,7 +695,7  <at>  <at>  skip_commit: /* The journal should be un
 		 * use in a different page. */
 		if (__buffer_state(bh, Freed)) {
 			clear_bit(BH_Freed, &bh->b_state);
-			clear_bit(BH_JBDDirty, &bh->b_state);
+//			clear_bit(BH_JBDDirty, &bh->b_state);
 		}
 			
 		if (buffer_jdirty(bh)) {

_

Re: how often to 'fsck -D' ?

* Stephen C. Tweedie <sct <at> redhat.com>:

> That said, I expect 2.4 htree patches to continue to be maintained.

I'd like to try them -- where are they?

--

-- 
Ralf Hildebrandt (Im Auftrag des Referat V a)   Ralf.Hildebrandt <at> charite.de
Charite Campus Mitte                            Tel.  +49 (0)30-450 570-155
Referat V a - Kommunikationsnetze -             Fax.  +49 (0)30-450 570-916
Why you can't find your system administrators:
(S)he's sitting under the desk, hysterical at what the (l)user just asked. 

Re: data corrupting bug in 2.4.20 ext3, data=journal

Andrew Morton wrote:

>Nick Piggin wrote:
>> 
>> Andrew Morton wrote:
>> 
>> >In 2.4.20-pre5 an optimisation was made to the ext3 fsync function
>> >which can very easily cause file data corruption at unmount time.  This
>> >was first reported by Nick Piggin on November 29th (one day after 2.4.20 was
>> >released, and three months after the bug was merged.  Unfortunate timing)
>> >
>> In fact it was reported on lkml on 18th July IIRC before 2.4.19 was
>> released if that is any help to you. 2.4.19 and 2.4.20 are affected
>> and I haven't tested previous releases. I was going to re-report it
>> sometime, but Alan brought it to light just the other day.
>> 
>
>Are you sure?  I can't make it happen on 2.4.19.  And disabling the new
>BH_Freed logic (which went into 2.4.20-pre5) makes it go away.
>
>
>--- linux-akpm/fs/jbd/commit.c~a	Sun Dec  1 23:10:12 2002
>+++ linux-akpm-akpm/fs/jbd/commit.c	Sun Dec  1 23:10:27 2002
> <at>  <at>  -695,7 +695,7  <at>  <at>  skip_commit: /* The journal should be un
> 		 * use in a different page. */
> 		if (__buffer_state(bh, Freed)) {
> 			clear_bit(BH_Freed, &bh->b_state);
>-			clear_bit(BH_JBDDirty, &bh->b_state);
>+//			clear_bit(BH_JBDDirty, &bh->b_state);
> 		}
> 			
> 		if (buffer_jdirty(bh)) {
>
I reported the bug for 2.4.19-rc1 and 2 but I can't remember if I tested 
2.4.19
when it was released. It has an external journal on a seperate disk. I can't
really do any testing with the machine unfortunately.

Regards,
Nick

Re: how often to 'fsck -D' ?

On Mon, 2002-12-02 at 07:56, Ralf Hildebrandt wrote:

> > That said, I expect 2.4 htree patches to continue to be maintained.
> 
> I'd like to try them -- where are they?

Ted posted 2.4 patches a while back, and I've got a set of ext3 patches
including those plus a few bug-fixes and other stuff, available at

	http://people.redhat.com/sct/patches/ext3-2.4/

In particular,

	http://people.redhat.com/sct/patches/ext3-2.4/dev-20021115/2.4.20-rc1.allpatches.patch

contains all of the non-debug patches in one easy-to-swallow capsule,
although it also includes some slightly experimental
performance-improving code.

Cheers,
 Stephen

Re: another idiot and ext3 - The inode is from a bad block in the inode table

On Mon, 2002-12-02 at 02:27, Norm Hanson wrote:
> some more info...
> the drive is connected to a maxtor ata 100 pci card
> here is the /var/log/ from went things went wrong...
> 
> Nov 26 04:02:25 pro180 kernel: hdf: dma_intr: status=0x51 { DriveReady SeekComplete Error }
> Nov 26 04:02:36 pro180 kernel: hdf: dma_intr: error=0x00 { }
> Nov 26 04:02:36 pro180 kernel: hdf: dma_intr: status=0x51 { DriveReady SeekComplete Error }
> Nov 26 04:02:36 pro180 kernel: hdf: dma_intr: error=0x80 { BadSector }, LBAsect=77019210, sector=3600
> Nov 26 04:02:36 pro180 kernel: end_request: I/O error, dev 21:43 (hdf), sector 3600

Yes, things _have_ gone wrong --- the disk is dying.  Ted posted a
message not long ago about how to start recovering from major disk
death, and I've put up a copy at

	http://people.redhat.com/sct/notes/recovery.txt

to get you started.

Cheers,
 Stephen