headers
Edgar Fuß | 1 May 2011 14:41
Picon
Favicon

5.1 RAID5 write performance

I've the impression that 5.1's fs write performance on RAID5s is exceptionally poor.

On a 5.1 system with a RAID5 accross three 10k discs, un-taring src.tgz takes 161s with WAPL and 347s with softdep.
On the same system, but with a RAID1 accross the same three discs, it takes only 41s.
On that system, dd'in /dev/zero to a file gives 15MB/s on the RAID5 and 41MB/s on the RAID1 (both on the same
three 10k discs).

On 4.0.1 with softdeps, it takes 22s on a RAID1 on two 15k discs and 29s on a RAID5 accross eight 10k discs.
On these 4.0.1 systems, the same dd gives 20MB/s on the RAID5 and 78MB/s on the RAID1.

So while the device throughput ratio for my 5.1 vs 4.0.1 RAID5s is 3:4, the fs througput is more like 1:5
(log/softdep) or 1:11 (softdep/softdep).
Also, on the 5.1 system on identical discs, file system throughput on RAID5 vs. RAID1 is 1:4, while, for the
RAID1, 50% more data need to be written (it's three discs).

Is there something wrong with my 5.1 system (amd64, Adaptec 19160, three Seagate 10k/36G Cheetah drives)
or can someone confirm this behaviour?
Permalink | Reply | 4 comments |
headers
Manuel Bouyer | 2 May 2011 15:23

Re: NFS server problems (lockup) on netbsd-5

On Tue, Apr 26, 2011 at 06:31:19PM +0200, Manuel Bouyer wrote:
> Hello,
> I tried upgrading a NFS server to today's netbsd-5 branch, without success.
> The previous kernel (netbsd-5 from 7 feb 2010) is rock solid.
> I tried a kernel from before the recent nfs-related pullups, it shows
> the same symptoms.
> 
> After a few hours of operation, the network stack is locked up.
> The server doesn't reply to ping, but the console is still alive and
> I can log in (it doesn't always make it to the shell though).
> ps from ddb shows lots of processes waiting on tstile, the lock seems to
> be the softnet_lock. One of the nfsd thread is waiting on livelock, I suspect
> it's also trying to get the softnet_lock but with a tryenter/kpause.
> 
> unfortunably I don't have a core dump (I couldn't get one).

And unfortunably it's not reproductible with a simple testbed (I've been
trying for 3 days).
I wonder if it could be related to the INRENAME change that has
been pulled up ...

--

-- 
Manuel Bouyer <bouyer <at> antioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la difference
--
Permalink | Reply | 1 comment |
headers
David Holland | 2 May 2011 18:57
Picon

Re: NFS server problems (lockup) on netbsd-5

On Mon, May 02, 2011 at 03:23:48PM +0200, Manuel Bouyer wrote:
 > > unfortunably I don't have a core dump (I couldn't get one).
 > 
 > And unfortunably it's not reproductible with a simple testbed (I've been
 > trying for 3 days).
 > I wonder if it could be related to the INRENAME change that has
 > been pulled up ...

Unlikely - that exists only to help work around a protocol bug in puffs.

--

-- 
David A. Holland
dholland <at> netbsd.org
Permalink | Reply | 0 comments |
headers
Patrick Welche | 2 May 2011 19:16
Picon
Picon
Favicon

Re: NFS server problems (lockup) on netbsd-5

On Tue, Apr 26, 2011 at 06:31:19PM +0200, Manuel Bouyer wrote:
> Hello,
> I tried upgrading a NFS server to today's netbsd-5 branch, without success.
> The previous kernel (netbsd-5 from 7 feb 2010) is rock solid.
> I tried a kernel from before the recent nfs-related pullups, it shows
> the same symptoms.
> 
> After a few hours of operation, the network stack is locked up.
> The server doesn't reply to ping, but the console is still alive and
> I can log in (it doesn't always make it to the shell though).
> ps from ddb shows lots of processes waiting on tstile, the lock seems to
> be the softnet_lock. One of the nfsd thread is waiting on livelock, I suspect
> it's also trying to get the softnet_lock but with a tryenter/kpause.
> 
> unfortunably I don't have a core dump (I couldn't get one).
> 
> Any idea about what could be going on ?

Just that I saw this too:
http://mail-index.netbsd.org/current-users/2011/04/18/msg016440.html

Cheers,

Patrick
Permalink | Reply | 0 comments |
headers
Edgar Fuß | 3 May 2011 18:06
Picon
Favicon

Re: 5.1 RAID5 write performance

mrg asked me to build a kernel with HZ=1024 and test.
It got worse.

Anything else I can test or instrument?
Permalink | Reply | 2 comments |
headers
Matthias Scheler | 3 May 2011 18:20
Picon
Favicon

Re: 5.1 RAID5 write performance

On Tue, May 03, 2011 at 06:06:44PM +0200, Edgar Fuß wrote:
> mrg asked me to build a kernel with HZ=1024 and test.
> It got worse.
> 
> Anything else I can test or instrument?

You could try turn off RAIDframe parity maps (which are a very useful
feature in general) with "raidctl -M no raid<n>", reboot your machine
afterwards and repeat your performance tests.

	Kind regards

--

-- 
Matthias Scheler                                  http://zhadum.org.uk/
Permalink | Reply | 1 comment |
headers
Toby | 3 May 2011 19:27
Picon

Re: 5.1 RAID5 write performance

Edgar Fuß <ef <at> math.uni-bonn.de> writes:

> 
> I've the impression that 5.1's fs write performance on RAID5s is exceptionally 
> poor.
> 
I've been working on a raid 5 raidframe setup recently and I've got good write
performance (80- 100MB/sec) and excellent read performance (>500MB/sec ? - 
don't know what's going on here yet). I just use dd for the test, so I was just
testing sequential access. I use 3 recent Hitachi 2.5" 500GB 7200 rpm sata
disks.

Poor raid performance is usually due to partition misalignment or differences
between fs block size and raid data stripe size or a combination of both.

1. Does the partition start on a multiple of the number of data sectors per
stripe unit?  E.g. if you set in your raid conf file as follows: START layout 64
1 1 5 and assuming a 3 disk setup, that makes your setup to have: 64 sectors x
(3 stripe slices - 1 parity slice) = 128 data sectors per stripe In this case,
your partitions should start on a multiple of 128 sectors and the size should be
a multiple of 128 sectors as well. If your filesystem block size is also 128
sectors, then the filesystem block will straddle two raid data blocks and any
filesystem block io operation would require two io operation on the raid device,
AFAIK.

2. AFAIK, setting the fs block size to be equal to the raid data block size
produces the optimal throughput. In the example above, you'd do newfs -b 64k
/dev/≤raid_partition>. E.g. if you're fs block size is 4k and your raid block
size is 64k, writing a file that occupies 16 sequential fs blocks (64k) would
cause the raid device having to write and calculate parity 16 times to the same
(Continue reading)

Permalink | Reply | 0 comments |
headers
Edgar Fuß | 3 May 2011 20:06
Picon
Favicon

sys/dev/isa/fd.c FDUNIT/FDTYPE

sys/dev/isa/fd.c defines FDUNIT and FDTYPE as DIV/MOD 8.
etc/MAKEDEV uses makedisk_p16 for fd*.

Who's right?
As I'm just adding a ninth (ten-sector) fd_type, I prefer the 16 version.
Permalink | Reply | 35 comments |
headers
Martin Husemann | 3 May 2011 23:31
Picon

Re: sys/dev/isa/fd.c FDUNIT/FDTYPE

On Tue, May 03, 2011 at 08:06:56PM +0200, Edgar Fuß wrote:
> sys/dev/isa/fd.c defines FDUNIT and FDTYPE as DIV/MOD 8.
> etc/MAKEDEV uses makedisk_p16 for fd*.
> 
> Who's right?
> As I'm just adding a ninth (ten-sector) fd_type, I prefer the 16 version.

FDUNIT and FDTYPE both derive from minor(dev).
MAKEDEV creates 16 minors, so you get unit 0 and 1, and 8 types each.

What is FDUNIT good for?

Martin
Permalink | Reply | 34 comments |
headers
Matthias Scheler | 4 May 2011 08:55
Picon
Favicon

Re: NFS server problems (lockup) on netbsd-5

On Tue, Apr 26, 2011 at 06:31:19PM +0200, Manuel Bouyer wrote:
> Hello,
> I tried upgrading a NFS server to today's netbsd-5 branch, without success.
> The previous kernel (netbsd-5 from 7 feb 2010) is rock solid.
> I tried a kernel from before the recent nfs-related pullups, it shows
> the same symptoms.

My "netbsd-5" NFS server using sources from the 8th of March has worked
fine for 56 days.

Based on that time frame my best guess is that one of these pullups
might have caused your problem:

http://releng.netbsd.org/cgi-bin/req-5.cgi?show=1580
http://releng.netbsd.org/cgi-bin/req-5.cgi?show=1581
http://releng.netbsd.org/cgi-bin/req-5.cgi?show=1582
http://releng.netbsd.org/cgi-bin/req-5.cgi?show=1583
http://releng.netbsd.org/cgi-bin/req-5.cgi?show=1600

	Kind regards

--

-- 
Matthias Scheler                                  http://zhadum.org.uk/
Permalink | Reply | 2 comments |

Gmane