Nilfs users (Log-Structured Snapshotting Filesystem)

David Arendt | 3 May 2009 00:55

nilfs_cpfile_delete_checkpoints: cannot delete block

Hi,

Until now nilfs-2.0.12 has run very stable without data corruption.
However on one partition (600G) I have got the following errors while 
running the cleaner:

nilfs_cpfile_delete_checkpoints: cannot delete block
NILFS: GC failed during preparation: cannot delete checkpoints: err=-2

This is a partition mainly holding large temporary render files (can be 
up to 25gb/file). There are currently 132702 snapshots.

As this partition is not used during the next few days, I will leave it 
with the error so if you would like me to test further things, please 
let me know.

Bye,
David Arendt

Ryusuke Konishi | 3 May 2009 10:08

Re: nilfs_cpfile_delete_checkpoints: cannot delete block

Hi David,
On Sun, 03 May 2009 00:55:43 +0200, David Arendt wrote:
> Hi,
> 
> Until now nilfs-2.0.12 has run very stable without data corruption.
> However on one partition (600G) I have got the following errors while 
> running the cleaner:
> 
> nilfs_cpfile_delete_checkpoints: cannot delete block
> NILFS: GC failed during preparation: cannot delete checkpoints: err=-2
> 
> This is a partition mainly holding large temporary render files (can be 
> up to 25gb/file). There are currently 132702 snapshots.
> 
> As this partition is not used during the next few days, I will leave it 
> with the error so if you would like me to test further things, please 
> let me know.
> 
> Bye,
> David Arendt

I have reviewed the function in question, but could not find any
likely problems.

Could you try the following patch?

It's applicable to v2.0.12.

I have some pending patches later than 2.0.12, but they seem to be
independent with your problem.

(Continue reading)

David Arendt | 3 May 2009 11:26

Re: nilfs_cpfile_delete_checkpoints: cannot delete block

Hi,

I have tried your patch.

The more verbose error message is:

nilfs_cpfile_delete_checkpoints: cannot delete block: cno=1407, range = 
[11, 75990)
NILFS: GC failed during preparation: cannot delete checkpoints: err=-2

Bye,
David Arendt

Ryusuke Konishi wrote:
> Hi David,
> On Sun, 03 May 2009 00:55:43 +0200, David Arendt wrote:
>   
>> Hi,
>>
>> Until now nilfs-2.0.12 has run very stable without data corruption.
>> However on one partition (600G) I have got the following errors while 
>> running the cleaner:
>>
>> nilfs_cpfile_delete_checkpoints: cannot delete block
>> NILFS: GC failed during preparation: cannot delete checkpoints: err=-2
>>
>> This is a partition mainly holding large temporary render files (can be 
>> up to 25gb/file). There are currently 132702 snapshots.
>>
>> As this partition is not used during the next few days, I will leave it

(Continue reading)

Ryusuke Konishi | 3 May 2009 11:44

Re: nilfs_cpfile_delete_checkpoints: cannot delete block

Hi!
On Sun, 03 May 2009 11:26:49 +0200, David Arendt wrote:
> Hi,
> 
> I have tried your patch.
> 
> The more verbose error message is:
> 
> nilfs_cpfile_delete_checkpoints: cannot delete block: cno=1407, range = 
> [11, 75990)
> NILFS: GC failed during preparation: cannot delete checkpoints: err=-2

You didn't see any DAT warnings?

If so, do you think the range of deleting checkpoints
(i.e. 11 ~ 75990 - 1) is proper?

How is the output of lscp?

Ryusuke Konishi

> Bye,
> David Arendt
> 
> Ryusuke Konishi wrote:
> > Hi David,
> > On Sun, 03 May 2009 00:55:43 +0200, David Arendt wrote:
> >   
> >> Hi,
> >>

(Continue reading)

David Arendt | 3 May 2009 12:06

Re: nilfs_cpfile_delete_checkpoints: cannot delete block

Hi,

I didn't see any DAT warnings.

Using lscp I see that the first entry is

1428  2009-03-30 02:13:06   cp    -        259      74436

The last entry is

134128  2009-05-03 00:04:28   cp    i      81813        876

If you want to have the full output of lscp please tell me, then I will 
send it to you without sending to the mailinglist as the file has 10mb.

Bye,
David Arendt

Ryusuke Konishi wrote:
> Hi!
> On Sun, 03 May 2009 11:26:49 +0200, David Arendt wrote:
>   
>> Hi,
>>
>> I have tried your patch.
>>
>> The more verbose error message is:
>>
>> nilfs_cpfile_delete_checkpoints: cannot delete block: cno=1407, range = 
>> [11, 75990)

(Continue reading)

David Arendt | 4 May 2009 06:16

Re: nilfs_cpfile_delete_checkpoints: cannot delete block

Hi,

This night. I had lots of:

nilfs_btree_propagate: key = 67, level == 0

On the parition where cleanerd has failed.

A try to umount it resulted in a hang with the following message:

NILFS warning (device sda10): nilfs_segctor_destroy: dirty file(s) after 
the final construction

Bye,
David Arendt

Ryusuke Konishi wrote:
> Hi David,
> On Sun, 03 May 2009 00:55:43 +0200, David Arendt wrote:
>   
>> Hi,
>>
>> Until now nilfs-2.0.12 has run very stable without data corruption.
>> However on one partition (600G) I have got the following errors while 
>> running the cleaner:
>>
>> nilfs_cpfile_delete_checkpoints: cannot delete block
>> NILFS: GC failed during preparation: cannot delete checkpoints: err=-2
>>
>> This is a partition mainly holding large temporary render files (can be

(Continue reading)

Ryusuke Konishi | 5 May 2009 13:23

Re: nilfs_cpfile_delete_checkpoints: cannot delete block

Hi David,
On Mon, 04 May 2009 06:16:24 +0200, David Arendt wrote:
> Hi,
> 
> This night. I had lots of:
> 
> nilfs_btree_propagate: key = 67, level == 0
> 
> On the parition where cleanerd has failed.

This error is related to the GC failure.

Both logs indicate that btree look-up of the 67th block on the
checkpoint file failed.

I suspect inconsistency between the block on page cache and btree; the
block was removed from the btree but were remaining on the page cache.

Could you try the following bugfix patch?

The patch ensures to clear dirty state of page and buffer after
removal of block, and would prevent the inconsistency.

Thanks in advance,
Ryusuke Konishi
--
diff --git a/fs/btnode.c b/fs/btnode.c
index 5e83c60..11a7305 100644
--- a/fs/btnode.c
+++ b/fs/btnode.c

(Continue reading)

Ryusuke Konishi | 5 May 2009 14:33

Picon

[PATCH 1/2] nilfs2: fix possible recovery failure due to block creation without writer

Some function calls in nilfs_prepare_segment_for_recovery() may fail
because they can create blocks on meta data files without configuring
a writable FS-instance.  Concretely, nilfs_mdt_create_block() routine
of meta data files will fail in that case.

This fixes the problem by temporarily attaching a writable FS-instace
during the function is called.

Signed-off-by: Ryusuke Konishi <konishi.ryusuke@...>
---
 fs/nilfs2/recovery.c |    6 ++++--
 1 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/fs/nilfs2/recovery.c b/fs/nilfs2/recovery.c
index 4fc081e..57afa9d 100644
--- a/fs/nilfs2/recovery.c
+++ b/fs/nilfs2/recovery.c
 <at>  <at>  -407,6 +407,7  <at>  <at>  void nilfs_dispose_segment_list(struct list_head *head)
 }

 static int nilfs_prepare_segment_for_recovery(struct the_nilfs *nilfs,
+					      struct nilfs_sb_info *sbi,
 					      struct nilfs_recovery_info *ri)
 {
 	struct list_head *head = &ri->ri_used_segments;
 <at>  <at>  -421,6 +422,7  <at>  <at>  static int nilfs_prepare_segment_for_recovery(struct the_nilfs *nilfs,
 	segnum[2] = ri->ri_segnum;
 	segnum[3] = ri->ri_nextnum;

+	nilfs_attach_writer(nilfs, sbi);

(Continue reading)

Ryusuke Konishi | 5 May 2009 14:33

Picon

[PATCH 2/2] nilfs2: fix circular locking dependency of writer mutex

This fixes the following circular locking dependency problem:

 =======================================================
 [ INFO: possible circular locking dependency detected ]
 2.6.30-rc3 #5
 -------------------------------------------------------
 segctord/3895 is trying to acquire lock:
  (&nilfs->ns_writer_mutex){+.+...}, at: [<d0d02172>]
   nilfs_mdt_get_block+0x89/0x20f [nilfs2]

 but task is already holding lock:
  (&bmap->b_sem){++++..}, at: [<d0d02d99>]
   nilfs_bmap_propagate+0x14/0x2e [nilfs2]

 which lock already depends on the new lock.

The bugfix is done by replacing call sites of nilfs_get_writer() which
are never called from read-only context with direct dereferencing of
pointer to a writable FS-instance.

Signed-off-by: Ryusuke Konishi <konishi.ryusuke@...>
---
 fs/nilfs2/ioctl.c |    8 +++++---
 fs/nilfs2/mdt.c   |   13 +++++++------
 2 files changed, 12 insertions(+), 9 deletions(-)

diff --git a/fs/nilfs2/ioctl.c b/fs/nilfs2/ioctl.c
index 108d281..be387c6 100644
--- a/fs/nilfs2/ioctl.c
+++ b/fs/nilfs2/ioctl.c

(Continue reading)

Ryusuke Konishi | 5 May 2009 14:33

Picon

[PATCH 0/2] nilfs2 bug-fixes against 2.6.30-rc4

This series fixes two known problems.

The "fix possible recovery failure due to block creation without writer"
    fixes possible recovery failure on mount.

The "fix circular locking dependency of writer mutex"
    fixes a possible circular locking problem detected by lockdep check
    on the latest kernel.

The lockdep has detected more circular locking problems, but they are
not solved yet.  I will send them to linux-next for compile test
to work on other problems prior to remains.

This post is for record and review.  These are not applicable to the
standalone package (i.e. nilfs2-module) and users do not have to apply
these piece by piece; I will include them in the next release of
nilfs2-module after merged into upstream.

Thanks,
Ryusuke Konishi
--
Ryusuke Konishi (2):
      nilfs2: fix possible recovery failure due to block creation without writer
      nilfs2: fix circular locking dependency of writer mutex

 fs/nilfs2/ioctl.c    |    8 +++++---
 fs/nilfs2/mdt.c      |   13 +++++++------
 fs/nilfs2/recovery.c |    6 ++++--
 3 files changed, 16 insertions(+), 11 deletions(-)

(Continue reading)

Group

gmane.comp.file-systems.nilfs.user.

Advertisement

Project Web Page

Nilfs users (Log-Structured Snapshotting Filesystem)

Search Archive

Page

1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9

Articles in period: 81

May 2009

Language

Change language

Options

Current view: Threads only / Showing only 20 lines / Not hiding cited text.
Change to All messages, whole messages, or hide cited text.

Post a message
NNTP Newsgroup
Classic Gmane web interface
XML

XML

RSS Feed
List Information

About Gmane

Recent entries

cleaner optimization and online defragmentation: status update (18 Jun 2013)
(unknown) (17 Jun 2013)
[PATCH 20/24] nilfs2: introduce hexdump of internal structures option (17 Jun 2013)
[PATCH 22/24] nilfs2: introduce option of showing internal errors (17 Jun 2013)
[PATCH 21/24] nilfs2: implement hexdump of internal structures option (17 Jun 2013)
[PATCH 19/24] nilfs2: implement superfluous debugging output option (17 Jun 2013)
[PATCH 18/24] nilfs2: introduce superfluous debugging output option (17 Jun 2013)
[PATCH 16/24] nilfs2: introduce dump stack option (17 Jun 2013)
[PATCH 09/24] nilfs2: implement GC subsystem debugging output (17 Jun 2013)
[PATCH 08/24] nilfs2: introduce GC subsystem debugging output option (17 Jun 2013)
[PATCH 04/24] nilfs2: introduce metadata (MDT) files debugging output (17 Jun 2013)
[PATCH 03/24] nilfs2: implement base subsystem debugging output (17 Jun 2013)
[PATCH 01/24] nilfs2: introduce nilfs2_debug() method (17 Jun 2013)
[PATCH 00/24] nilfs2: introduce debugging subsystem implementation (17 Jun 2013)
NiLFS2 partial segment finfo (4 Jun 2013)
My NiLFS2 port to NetBSD: status update (4 Jun 2013)
[PATCH v6 2/2] nilfs2: use atomic64_t type for inodes_count and blocks (29 May 2013)
[PATCH v6 1/2] nilfs2: implement calculation of free inodes count (29 May 2013)
Broken nilfs2 filesystem (27 May 2013)
[PATCH v5 2/2] nilfs2: use atomic_long_t type for inodes_count and blo (24 May 2013)
[PATCH v5 1/2] nilfs2: implement calculation of free inodes count (24 May 2013)

Archives

30	June 2013
48	May 2013
22	April 2013
52	March 2013
6	February 2013
52	January 2013
47	December 2012
58	November 2012
55	October 2012
29	September 2012
46	August 2012
71	July 2012
26	June 2012
30	May 2012
14	April 2012
37	March 2012
23	February 2012
15	January 2012
35	December 2011
21	November 2011
2	October 2011
1	September 2011
40	August 2011
22	July 2011
74	June 2011
47	May 2011
28	April 2011
80	March 2011
75	February 2011
19	January 2011
28	December 2010
25	November 2010
31	October 2010
55	September 2010
31	August 2010
73	July 2010
58	June 2010
97	May 2010
84	April 2010
48	March 2010
37	February 2010
23	January 2010
36	December 2009
135	November 2009
27	October 2009
62	September 2009
109	August 2009
108	July 2009
150	June 2009
81	May 2009
18	April 2009
75	March 2009
26	February 2009
36	January 2009
52	December 2008
16	November 2008
27	October 2008
14	September 2008
15	August 2008
11	July 2008
7	June 2008
12	May 2008
3	April 2008
22	March 2008
20	February 2008
36	January 2008
7	December 2007
4	November 2007
7	October 2007
3	September 2007

Design

Zawodny | Right Menu | Left Menu

Your Own Design

Paste the URL of your CSS below. Download CSS template