Re: JFS crash: stack trace
Dave Kleikamp <shaggy <at> linux.vnet.ibm.com>
2010-03-26 12:52:22 GMT
On Fri, 2010-03-26 at 11:43 +0200, Cheusov Aleksey wrote:
> Mar 24 06:39:05 syn-proc4 kernel: [65682.636659] Call Trace:
> Mar 24 06:39:05 syn-proc4 kernel: [65682.636690] [<f8d17478>]
> diWrite+0x16d/0x4a2 [jfs]
> Mar 24 06:39:05 syn-proc4 kernel: [65682.636721] [<c0121355>]
> Mar 24 06:39:05 syn-proc4 kernel: [65682.636744] [<f8d260a6>]
> txCommit+0x18d/0xecc [jfs]
> Mar 24 06:39:05 syn-proc4 kernel: [65682.636770] [<c016b2c7>]
> Mar 24 06:39:05 syn-proc4 kernel: [65682.636790] [<c016ad5d>]
> Mar 24 06:39:05 syn-proc4 kernel: [65682.636808] [<c01f3ea2>]
> Mar 24 06:39:05 syn-proc4 kernel: [65682.636830] [<f8d25d5c>]
> txBegin+0x25/0x1e2 [jfs]
> Mar 24 06:39:05 syn-proc4 kernel: [65682.636858] [<f8d0d0d9>]
> jfs_commit_inode+0x9e/0xde [jfs]
> Mar 24 06:39:05 syn-proc4 kernel: [65682.636883] [<f8d0d328>]
> jfs_write_inode+0x2d/0x3b [jfs]
> Mar 24 06:39:05 syn-proc4 kernel: [65682.636905] [<c01a1120>]
I have no idea what caused the trap. It happens in stable code that
hasn't been changed in ages. It could be due to a memory corruption bug
somewhere else in the kernel, either somewhere else in jfs or elsewhere.
Otherwise, it's something really subtle that I haven't seen before.
> Ok. Even more problems. I tried to remount this fs in read-only but
> 'mount -o remount,ro /my/mountpoint'
> command hadnged up.
If you hadn't rebooted, that isn't surprising. The trap killed this
thread, probably holding locks, that would lock up further operations on
this file system.
> I rebooted machine, upgrade jfsutils to the latest version 1.1.4 and
> tried to fsck it.
> Here is the output:
> # sudo fsck.jfs -f /dev/sdb3
> fsck.jfs version 1.1.14, 06-Apr-2009
> processing started: 3/25/2010 19.17.30
> The current device is: /dev/sdb3
> Block size in bytes: 4096
> Filesystem size in blocks: 363779876
> **Phase 0 - Replay Journal Log
> **Phase 1 - Check Blocks, Files/Directories, and Directory Entries
> Insufficient dynamic storage available for required workspace (1,6).
> CANNOT CONTINUE
It couldn't allocate enough memory. It seems odd since you said that
you ran version 1.1.11 recently. I don't know what would have changed
that would consume any more memory. Maybe more inodes have been created
since the last attempt? Is it possible to add some more swap space and
> This files system contains huge amount of files, hundreds of thousands
> of them, maybe millions,
> and huge amount of hard links. Can this be a problem for JFS and
Have you tried mounting read-only after the reboot? That's not the best
solution, but at least you may be able to recover the data.
IBM Linux Technology Center
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.