domU is causing misaligned disk writes

Anyone know why my xen xvda devices would be doing (apparently)
unaligned writes to my SAN causing horrible performance and massive
seeking and lots of reading for page cache backfill? BUT writing to
the device in the dom0 is very fast and causes no extra reads?

I am running the 2.6.18-164.11.1.el5xen xen/kernel which came with
CentOS 5.4

After spending a lot of time banging my head on this I seem to have
finally tracked it down to a difference between domU and dom0.  I
never would have thought it would be this but it is extremely
reproduceable. We're talking a difference of 4-5x in write speed.
Reads are equally fast everywhere.

I am using AoE v72 kernel module (initiator) on a Dell R610's to talk
to vblade-19 (target) on Dell R710's all running CentOS 5.4. I have
striped two 7200 RPM SATA disks and exported the md with AoE (although
I have done these tests with individual disks also). Read performance
is excellent:

# dd of=/dev/null if=/dev/xvdg1 bs=4096 count=3000000
3000000+0 records in
3000000+0 records out
12288000000 bytes (12 GB) copied, 106.749 seconds, 115 MB/s

I dropped the cache with:

echo 1 > /proc/sys/vm/drop_caches

on both target and initiator before starting the test. This is great

articles about aoe ha san

------------------------------------------------------------------------------

_______________________________________________
Aoetools-discuss mailing list
Aoetools-discuss@...
https://lists.sourceforge.net/lists/listinfo/aoetools-discuss

Re: [Xen-devel] domU is causing misaligned disk writes

On Tue, Apr 20, 2010 at 02:19:13PM -0700, Tracy Reed wrote:

> > How do you know this is a mis-aligned sectors issue? Is this what your
> > AOE vendor is telling you ?
> 
> No AoE vendor involved. I am using the free stuff. I think it is a
> misalignment issue because during a purely write test it is doing
> massive amounts of reading according to iostat.

How about actually verifying that by e.g. using wireshark and comparing
the I/O patterns in the fast and slow cases? The differences in the
patterns may give clues where to look further.

> #dd if=/dev/zero of=/dev/etherd/e6.1 oflag=direct bs=4096
> count=3000000
> 1764883+0 records in
> 1764883+0 records out
> 7228960768 bytes (7.2 GB) copied, 402.852 seconds, 17.9 MB/s
> 
> But even on my local directly attached SATA workstation disk when
> doing that same dd on an otherwise idle machine I see performance
> like:
> 
> $ dd if=/dev/zero of=foo.test bs=4096 count=4000000
> C755202+0 records in
> 755202+0 records out
> 3093307392 bytes (3.1 GB) copied, 128.552 s, 24.1 MB/s
> 
> which again suggests that oflag=direct isn't doing quite what I expect.

oflag=direct turns off caching on the host dd is running on, i.e. the
initiator. The target still caches writes of course, unless you tell it
not to by passing the "-d" flag to vblade.

Gabor

------------------------------------------------------------------------------