slow pipes chewing up system time (and real time)?
Lloyd Parkes <lloyd <at> must-have-coffee.gen.nz>
2012-10-16 20:12:05 GMT
I have finally got around to modifying the Mercurial CVS conversion tool so that it handles the NetBSD CVS
repositories and I would like to run it on NetBSD to provide daily incremental repository updates, but I
can't because the NetBSD performance is so bad.
I did all my development work on OS X because that's the biggest box I own and converting the src repository
requires about 13GB of RAM. When I moved to running the tool on NetBSD, I started with the othersrc
repository because that makes testing somewhat more manageable. My problem is that the conversion is
five times slower on NetBSD than it is on OS X in real time even though CPU usage appears to be comparable.
My Mac is 3.4GHz Intel Core i7 with 4 cores and one extra virtual hyper thread core for each real core. The OS X
10.8.2 is run from an external thunderbolt attached RAID 0 disk pair and the data I'm working with is on the
internal WD Caviar Black formatted with a case sensitive journaled HFS+ file system. The cvs server uses a
RAM disk for it's temporary storage.
My NetBSD box is a 3.3GHz AMD Phenom II with 6 cores. NetBSD 6.0 is run from an AHCI attached SSD (an old SSD) and
the data I'm working with is variously on tmpfs or the SSD. The SSD file systems are formatted with FFSv2
without logging, with noatime, and for the designated destination file system, with async. The use of
tmpfs or async doesn't make an obvious change to performance.
"time -l" on the Mac and NetBSD give me similar user CPU and system CPU numbers for the conversion of othersrc
(half a minute and five seconds respectively), but real time on the Mac is 68s and on NetBSD it is 359s. Even
though the reported system time is only five seconds, when I run top on NetBSD, one of the CPUs just sits
there at 100% system.
I don't think the horrible wall clock time is caused by hardware constraints (disk performance etc.)
because I have tested this on tmpfs with no change and after running ktrace I'm a bit suspicious of the pipe
performance. I can't see a way to find out how long each system call is taking, so I don't know for sure
though. The application talks to the cvs server with a stop and wait implementation of the cvs protocol and
the workload is a lot of small requests. I see support in NetBSD for pipes handling lots of large requests