Keri Alleyne | 28 Aug 02:24 2014

AoE Tools malfunction following aoe6-85 drive compilation

Good day.

I would like to report that we are running into some difficulty 
executing commands from AoE Tools following aoe6-85 driver compilation 
on Ubuntu 14.04.1 LTS Server (i386).

Applied Ubuntu patches.

sudo apt-get install build-essential linux-headers-`uname -r`

Extracted aoe6-85.tar.gz


sudo make install

sudo modprobe aoe

dmesg shows:

[  109.114933] aoe: module verification failed: signature and/or 
required key missing - tainting kernel
[  109.132450] aoe: AoE v85 initialised.
[  109.134865] aoe: e0.0: setting 1024 byte data frames
[  109.135635] aoe: 00e04cd770c5 e0.0 v4014 has 20480000 sectors
[  109.138386]  etherd/e0.0: unknown partition table

sudo aoe-stat
       e0.0        10.485GB       eth0 1024  up

(Continue reading)

Catalin Salgau | 10 Jun 19:55 2014

further vblade bpf tweaking

So I'm going to throw some statistics here, and I'd like somebody to 
correct me if they think I'm wrong.
As I see it, the general use-case for AoE deployment consists of one or 
more mostly dedicated machines serving several initiators on one or more 
interfaces (or vlans). This would mean that most, if not all, of the 
traffic would consist of AoE frames.
Let's assume that we are running several targets on the same interface 
(let's say 5, differing by their major number and having 0 as their minor).
If I take a normal sample of traffic under said conditions and pass it 
trough the filter, we would see:
- check Frame type - 2 ops - 0% of frames removed - 100% of frames pass this
- check Response flag - 3 ops - 2 ops - 50% of frames removed - 50% of 
frames pass this
- check Major - 2/4 ops - 80% of frames removed - 10% of frames pass this
- check Minor - 2/4 ops - 0 % of frames removed - 10% of frames pass this

If I were to reorder the checks and move the major/minor checks first, 
response after that and leave the frame type to the end:
- check Major - 2/4 ops - 80% of frames removed - 20% of frames pass this
- check Minor - 2/4 ops - 0 % of frames removed - 20% of frames pass this
- check Response flag - 3 ops - 2 ops - 50% of frames removed - 10% of 
frames pass this
- check Frame type - 2 ops - 0% of frames removed - 10% of frames pass this

This would lead to a fast-fail path for 80% of vblade traffic in 4 ops, 
versus the 9 ops used currently.

If we were to include a moderate amount of other kinds of traffic into 
the mix, major and minor have the nice effect of sitting on top of the 
Total Length and Identification fields of IP, which vary wildly and have 
(Continue reading)

Killer{R} | 9 Jun 23:18 2014

another proposed protocol optimization...

Hello aoetools-discuss,

  ..coalesced read requests. Its quite an expensive to send 60-bytes
  packet per every MTU-sized read. I did experimental read requests
  coalesting in my forked project and it noticable decreased
  packets-per-second rate and also reduced target's CPU usage.
  I did this by appending bunch of such structures prepended by uchar
  indicated their count just after normal Read command packet:
  struct AtaCoalescedRead
        uchar   tag[4];
        uchar   lba[6];
        uchar   resvd[5];
        uchar   sectors;
  + added feature negotiation command so initiator can negitiate with
  target using of that improvement without losing backward compatibility


Best regards,
 Killer{R}                          mailto:support@...

HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
Find What Matters Most in Your Big Data with HPCC Systems
Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
Leverages Graph Analysis for Fast Processing & Easy Data Exploration
Ed Cashin | 8 Jun 14:02 2014

vblade-22-rc1 is first release candidate for version 22


Please try out the release candidate at github.

Its main changes are:

* update version for v22 release candidate 1
* buffer boundary cleanups
* FreeBSD BPF and MTU fixes from Catalin Salgau
* offset and size options by Christoph Biedl

The plan is to have the candidates at github, where releases are easier 
to make.  The release 22 itself will also be published on the 
sourceforge site.


   Ed Cashin <ed.cashin@...>

Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and their 
applications. Written by three acclaimed leaders in the field, 
this first edition is now available. Download your free book today!
Catalin Salgau | 19 May 21:11 2014

[PATCH] Incorrect shelf/slot handling in BPF

The BPF filter program currently included in vblade requires that the 
major and minor fields in a packet header either
- match server's major and minor addresses or
- be both all ones (0xffff and 0xff respectively)
This is against the AoE specification that requires that the two fields 
be tested separately. (as seen in aoe.c:368)
Proposed patch corrects this.

diff --git a/vblade/bpf.c b/vblade/bpf.c
--- a/vblade/bpf.c
+++ b/vblade/bpf.c
 <at>  <at>  -82,32 +82,27  <at>  <at> 
      struct bpf_program *bpf_program;
      struct bpf_insn insns[] = {
-        /* Load the type into register */
+        /* CHECKTYPE: Load the type into register */
          BPF_STMT(BPF_LD+BPF_H+BPF_ABS, 12),
          /* Does it match AoE Type (0x88a2)? No, goto INVALID */
-        BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, 0x88a2, 0, 12),
+        BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, 0x88a2, 0, 10),
          /* Load the flags into register */
          BPF_STMT(BPF_LD+BPF_B+BPF_ABS, 14),
          /* Check to see if the Resp flag is set */
          BPF_STMT(BPF_ALU+BPF_AND+BPF_K, Resp),
          /* Yes, goto INVALID */
-        BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, 0, 0, 9),
-        /* Load the shelf number into register */
+        BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, 0, 0, 7),
+        /* CHECKSHELF: Load the shelf number into register */
(Continue reading)

Catalin Salgau | 16 May 11:31 2014

[PATCH] Obtain interface MTU on FreeBSD

Hi. I've been sitting on some changes for vblade for some time now and I 
remembered to clean them up and contribute them back, so here's one.

diff --git a/vblade/freebsd.c b/vblade/freebsd.c
--- a/vblade/freebsd.c
+++ b/vblade/freebsd.c
 <at>  <at>  -261,7 +261,24  <at>  <at> 
  getmtu(int fd, char *name)
-	return 1500;
+	struct ifreq xx;
+    int s,n;
+	s = socket(AF_INET, SOCK_RAW, 0);
+    if (s == -1)
+    {
+	  perror("Can't get mtu");
+	  return 1500;
+	}
+	xx.ifr_addr.sa_family = AF_INET;
+	strcpy(xx.ifr_name, name);
+	n = ioctl(s,SIOCGIFMTU, &xx);
+	if (n == -1) {
+	  perror("Can't get mtu");
+	  return 1500;
+	}
+	close(s);
+	return xx.ifr_mtu;
(Continue reading)

Turbo Fredriksson | 11 May 18:18 2014

Exposing shares

I'm in the process of adding AoE support to ZoL
(ZFS On Linux -
and have gotten most of the way there (

I'm the author of the iSCSI (not yet accepted) and SMB
(partly accepted) support in ZoL, so adding AoE seemed
like a good and easy addition.

I have however caught on a couple of snags that complicates
the issue.

	1. How do I find the device that is shared?
	    'aoe-stat' doesn't say, and there's noting in the
	    /sys fs that I can see.

	    I can see the process in the list
		debian:~# /bin/ps faxwww | grep 'vblade 9 0 eth0' | grep -v grep
		12175 pts/1    S      0:00 sh -c /usr/sbin/vblade 9 0 eth0 /dev/zvol/rpool/test < /dev/null 2>&1 | logger -t vbladed
		12177 pts/1    S      0:00  \_ /usr/sbin/vblade 9 0 eth0 /dev/zvol/rpool/test
	    but that doesn't seems like a secure way of

	2. How do I terminate (unshare) a device?
	    It's in the process list above, but that also seems
	    a little ... 'wonkey'.
	    It would have been nice if vblade/vbladed wrote
	    a pid file or something.

	    Also, before killing the process, it would be nice
	    to know if 'anyone' where accessing it.
(Continue reading)

Killer{R} | 10 May 21:19 2014

(probably stupid) support&data integrity questions

Hello aoetools-discuss,

1) Do you have forum or support is only this maillist
2) Data integrity   - do somebody had any data cossuption due to AoE
seems doesn't have anything to detect from-wire errors? I mean at
first corrupted packets themselves (I know that in hardware ethernet
check it.. usually but...) and there'is another possible problem that
seems is not addressed by AoE protocol: possible packets duplication
on ethernet level that can be caused for example by bad network
topology, failing hardware (I've seen semi-dead network switches that
flooded ethernet segment with duplicated packets) and so on? Because
there possible data corruption if write-from-the-past-packet
overwrites some data region. Theoretically this could be easily be
fixed by using some sequence numbers, that could be known by server to
be mononically increasing. Futhermore there'is 'tag' field in ATA
command that seems can be used for such purpose but unfortunatelly
according to documentation and vblade sources it completely ignored by


Best regards,
 Killer{R}                          mailto:support@...

Is your legacy SCM system holding you back? Join Perforce May 7 to find out:
&#149; 3 signs your SCM is hindering your productivity
&#149; Requirements for releasing software faster
&#149; Expert tips and advice for migrating your SCM now
(Continue reading)

Killer{R} | 10 May 20:33 2014

AoE performance on small boxes

Hello aoetools-discuss,

  I tried to use AoE vblade on Dlink DNS 325 (arm box with 256 megs of
  ram) with Win2k3 srv/WinAoE 0.97g installed connected by gigabit
  ethernet with 7k jumbo frames enabled on both sides (bigger doesn't
  work somewhy on my HW). And got quite disappointing performance
  rate: while download (from vblade to WinAoE) was quite good
  (50 megs per second), upload was about 20 megs per second, that
  was much less than SMB transfer.
  I decided to try to make world better and patched vblade code, so it
  uses PACKET_RX_RING instead of plain read() and also added quite
  simple write buffering. First thing improved downalod by ~10% - to
  ~55 megs per second, second - dramatically increased upload -
  to 39 megs per second that is almost twice faster.
  Are aoetools devs planning to do somthing like this in future?
  BTW I'm not sure it can give comparable improvements on other
  systems since tested only on that little linux box


Best regards,
 Killer{R}                          mailto:support@...

Is your legacy SCM system holding you back? Join Perforce May 7 to find out:
&#149; 3 signs your SCM is hindering your productivity
&#149; Requirements for releasing software faster
&#149; Expert tips and advice for migrating your SCM now
Christoph Biedl | 8 May 22:43 2014

[PATCH] Implement an offset option for vbladed


the patch below implements a new option -o for vbladed, denoting the
number of sectors that should be skipped at the beginning of the
exported file. Default is 0 (no offset, export from beginning, current


If the exported files are actually raw block devices on the target,
something that might happen when using logical volumes for exports,
several tools running on the target that assume block devices are
for exclusive local access only might create confusion or even
havoc. Most prominently:

* os-prober of grub (might be disabled, though)
  Might mount any block device found that contain a file system.
* (Linux) logical volume management
  Will report any physical volume if the export is used as such.
* (Linux) blkid 
  Reports any block device (minor impact, mostly information leakage)

The overall problem is applications are not aware some block devices 
might be used from a remote client while there's no locking and that
might be hard to implement

Solutions tried but not considered helpful:

* Filter out exported block devices
  Problems: The tools need the ability to filter; new tools might come
(Continue reading)

Denis Ovsienko | 8 May 14:09 2014

AoE support in tcpdump


the master branch of tcpdump now supports basic AoE decoding. If anybody would like to improve it, the right
place for bug reports and pull requests would be here:

    Denis Ovsienko

Is your legacy SCM system holding you back? Join Perforce May 7 to find out:
&#149; 3 signs your SCM is hindering your productivity
&#149; Requirements for releasing software faster
&#149; Expert tips and advice for migrating your SCM now
Aoetools-discuss mailing list
Aoetools-discuss <at>