Picon

question about file system

Hi,
I am using  Coraid (Model: ST3750330NS)
OS is CentOS:

Distributor ID: CentOS
Description:    CentOS release 6.6 (Final)
Release:        6.6
Codename:       Final


kernel:

Linux 2.6.32-71.el6.x86_64 #1 SMP Fri May 20 03:51:51 BST 2011 x86_64 x86_64 x86_64 GNU/Linux

8 GB RAM, 2 x Intel(R) Pentium(R) D CPU 3.00GHz

I use this server (directory connected to shelf through crossover cable) os mail server.
Mail's spool in on etherd device.
I am having big load problems (> 200!)
I would like to investigate if this can be related to file system issue or network issue

I set mtu 9000 to dedicated ethernet:
0a:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)


eth2      Link encap:Ethernet  HWaddr 00:15:17:CB:57:AB
          inet6 addr: fe80::215:17ff:fecb:57ab/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1

In /var/log/messages I see this:


Call Trace:
 [<ffffffff8120051f>] ? security_inode_permission+0x1f/0x30
 [<ffffffff8117ab4d>] ? __link_path_walk+0xfd/0x1040
 [<ffffffff814c97ae>] __mutex_lock_slowpath+0x13e/0x180
 [<ffffffff811794b1>] ? path_put+0x31/0x40
 [<ffffffff814c964b>] mutex_lock+0x2b/0x50
 [<ffffffff81178b7f>] lock_rename+0x3f/0xe0
 [<ffffffff8117c133>] sys_renameat+0x113/0x260
 [<ffffffff81135837>] ? handle_pte_fault+0xf7/0xad0
 [<ffffffff81171b14>] ? cp_new_stat+0xe4/0x100
 [<ffffffff81180210>] ? filldir+0x0/0xe0
 [<ffffffff810d40a2>] ? audit_syscall_entry+0x272/0x2a0
 [<ffffffff8117c29b>] sys_rename+0x1b/0x20
 [<ffffffff81013172>] system_call_fastpath+0x16/0x1b


and this is the output of "top" command:

top - 17:15:32 up  1:29,  1 user,  load average: 160.25, 164.19, 242.60
Tasks: 597 total,   2 running, 595 sleeping,   0 stopped,   0 zombie
Cpu(s): 10.2%us,  5.2%sy,  0.0%ni,  0.0%id, 84.1%wa,  0.0%hi,  0.5%si,  0.0%st
Mem:   8059612k total,  5515976k used,  2543636k free,  1737132k buffers
Swap: 10289144k total,        0k used, 10289144k free,  2254068k cached


(notice high I/O wait)

# aoe-version
              aoetools: 35
  installed aoe driver: 47
    running aoe driver: 47

Do you have any suggestion?
Thank you very much!

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Aoetools-discuss mailing list
Aoetools-discuss@...
https://lists.sourceforge.net/lists/listinfo/aoetools-discuss
Keri Alleyne | 28 Aug 02:24 2014

AoE Tools malfunction following aoe6-85 drive compilation

Good day.

I would like to report that we are running into some difficulty 
executing commands from AoE Tools following aoe6-85 driver compilation 
on Ubuntu 14.04.1 LTS Server (i386).

Applied Ubuntu patches.

sudo apt-get install build-essential linux-headers-`uname -r`

Extracted aoe6-85.tar.gz

make

sudo make install

sudo modprobe aoe

dmesg shows:

[  109.114933] aoe: module verification failed: signature and/or 
required key missing - tainting kernel
[  109.132450] aoe: AoE v85 initialised.
[  109.134865] aoe: e0.0: setting 1024 byte data frames
[  109.135635] aoe: 00e04cd770c5 e0.0 v4014 has 20480000 sectors
[  109.138386]  etherd/e0.0: unknown partition table

sudo aoe-stat
       e0.0        10.485GB       eth0 1024  up

sudo aoe-discover
aoe-discover: /dev/etherd/discover does not exist or is not writeable.

sudo aoe-flush
aoe-flush: /dev/etherd/flush does not exist or is not writeable.

-----------------------------------------------------

As you see, there are some problems running aoe-discover and aoe-flush.
In addition, the tainting of the kernel is a little strange.

Anyway, when this process is repeated on a Ubuntu 12.04 LTS system, the 
AoE Tools work just fine. So there is something different in how AoE 
compiles on 14.04.1.

Any suggestions?

Thanks.

------------------------------------------------------------------------------
Slashdot TV.  
Video for Nerds.  Stuff that matters.
http://tv.slashdot.org/
Catalin Salgau | 10 Jun 19:55 2014
Picon
Picon

further vblade bpf tweaking

So I'm going to throw some statistics here, and I'd like somebody to 
correct me if they think I'm wrong.
As I see it, the general use-case for AoE deployment consists of one or 
more mostly dedicated machines serving several initiators on one or more 
interfaces (or vlans). This would mean that most, if not all, of the 
traffic would consist of AoE frames.
Let's assume that we are running several targets on the same interface 
(let's say 5, differing by their major number and having 0 as their minor).
If I take a normal sample of traffic under said conditions and pass it 
trough the filter, we would see:
- check Frame type - 2 ops - 0% of frames removed - 100% of frames pass this
- check Response flag - 3 ops - 2 ops - 50% of frames removed - 50% of 
frames pass this
- check Major - 2/4 ops - 80% of frames removed - 10% of frames pass this
- check Minor - 2/4 ops - 0 % of frames removed - 10% of frames pass this

If I were to reorder the checks and move the major/minor checks first, 
response after that and leave the frame type to the end:
- check Major - 2/4 ops - 80% of frames removed - 20% of frames pass this
- check Minor - 2/4 ops - 0 % of frames removed - 20% of frames pass this
- check Response flag - 3 ops - 2 ops - 50% of frames removed - 10% of 
frames pass this
- check Frame type - 2 ops - 0% of frames removed - 10% of frames pass this

This would lead to a fast-fail path for 80% of vblade traffic in 4 ops, 
versus the 9 ops used currently.

If we were to include a moderate amount of other kinds of traffic into 
the mix, major and minor have the nice effect of sitting on top of the 
Total Length and Identification fields of IP, which vary wildly and have 
a small chance of collision.
This is not shared, unfortunately, by IPv6, where it would collide with 
part of Flow Label and part of Payload Length. In normal environments 
these bytes would probably be 0, and so would the response flag. If both 
our major and minor are set to 0, we would have a fail in 9 ops to find 
that out, instead of 2 ops in the current case, for said vblade 
instance, but all other instances would fail in 4 ops.
This trade-off would not be convenient in an environment with more 
non-AoE traffic than AoE traffic, but, as stated, I'd argue that that is 
not the general case.

Implementation-wise I would argue against the need to keep both version, 
as the latency difference is small, but flipping between these by a 
command-line argument would be an option that wouldn't cost anything at 
runtime (something like - favour dedicated or mixed environments with a 
flag)

One more note - since I'm deploying this on FreeBSD (and I believe Linux 
also has something similar) I can make sure I don't see frames sent by 
this machine, so I can drop the removed count for the Response flag to 
0% in certain conditions. I'm about to follow this up with freebsd-net <at>  
and maybe raise a question on linux-net <at> .

I submit this for your consideration, and will provide the needed changes.

Cheers,
  Catalin Salgau

------------------------------------------------------------------------------
HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
Find What Matters Most in Your Big Data with HPCC Systems
Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
Leverages Graph Analysis for Fast Processing & Easy Data Exploration
http://p.sf.net/sfu/hpccsystems
Killer{R} | 9 Jun 23:18 2014

another proposed protocol optimization...

Hello aoetools-discuss,

  ..coalesced read requests. Its quite an expensive to send 60-bytes
  packet per every MTU-sized read. I did experimental read requests
  coalesting in my forked project and it noticable decreased
  packets-per-second rate and also reduced target's CPU usage.
  I did this by appending bunch of such structures prepended by uchar
  indicated their count just after normal Read command packet:
  struct AtaCoalescedRead
  {
        uchar   tag[4];
        uchar   lba[6];
        uchar   resvd[5];
        uchar   sectors;
  };
  + added feature negotiation command so initiator can negitiate with
  target using of that improvement without losing backward compatibility

--

-- 
Best regards,
 Killer{R}                          mailto:support@...

------------------------------------------------------------------------------
HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
Find What Matters Most in Your Big Data with HPCC Systems
Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
Leverages Graph Analysis for Fast Processing & Easy Data Exploration
http://p.sf.net/sfu/hpccsystems
Ed Cashin | 8 Jun 14:02 2014
Picon

vblade-22-rc1 is first release candidate for version 22

Hi.

Please try out the release candidate at github.

   https://github.com/ecashin/vblade/releases/tag/v22-rc1

Its main changes are:

* update version for v22 release candidate 1
* buffer boundary cleanups
* FreeBSD BPF and MTU fixes from Catalin Salgau
* offset and size options by Christoph Biedl

The plan is to have the candidates at github, where releases are easier 
to make.  The release 22 itself will also be published on the 
sourceforge site.

--

-- 
   Ed Cashin <ed.cashin@...>

------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and their 
applications. Written by three acclaimed leaders in the field, 
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/NeoTech
Catalin Salgau | 19 May 21:11 2014
Picon
Picon

[PATCH] Incorrect shelf/slot handling in BPF

The BPF filter program currently included in vblade requires that the 
major and minor fields in a packet header either
- match server's major and minor addresses or
- be both all ones (0xffff and 0xff respectively)
This is against the AoE specification that requires that the two fields 
be tested separately. (as seen in aoe.c:368)
Proposed patch corrects this.

diff --git a/vblade/bpf.c b/vblade/bpf.c
--- a/vblade/bpf.c
+++ b/vblade/bpf.c
 <at>  <at>  -82,32 +82,27  <at>  <at> 
  {
      struct bpf_program *bpf_program;
      struct bpf_insn insns[] = {
-        /* Load the type into register */
+        /* CHECKTYPE: Load the type into register */
          BPF_STMT(BPF_LD+BPF_H+BPF_ABS, 12),
          /* Does it match AoE Type (0x88a2)? No, goto INVALID */
-        BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, 0x88a2, 0, 12),
+        BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, 0x88a2, 0, 10),
          /* Load the flags into register */
          BPF_STMT(BPF_LD+BPF_B+BPF_ABS, 14),
          /* Check to see if the Resp flag is set */
          BPF_STMT(BPF_ALU+BPF_AND+BPF_K, Resp),
          /* Yes, goto INVALID */
-        BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, 0, 0, 9),
-        /* Load the shelf number into register */
+        BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, 0, 0, 7),
+        /* CHECKSHELF: Load the shelf number into register */
          BPF_STMT(BPF_LD+BPF_H+BPF_ABS, 16),
-        /* Does it match shelf number? No, goto CHECKBROADCAST */
-        BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, shelf, 0, 2),
-        /* Load the slot number into register */
+        /* Does it match shelf number? Yes, goto CHECKSLOT */
+        BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, shelf, 1, 0),
+        /* Does it match broadcast? No, goto INVALID */
+        BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, 0xffff, 0, 4),
+        /* CHECKSLOT: Load the slot number into register */
          BPF_STMT(BPF_LD+BPF_B+BPF_ABS, 18),
          /* Does it match shelf number? Yes, goto VALID */
-        BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, slot, 4, 0),
-        /* CHECKBROADCAST: is (shelf, slot) == (0xffff, 0xff)? */
-        /* Load the shelf number into register */
-        BPF_STMT(BPF_LD+BPF_H+BPF_ABS, 16),
-        /* Is it 0xffff? No, goto INVALID */
-        BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, 0xffff, 0, 3),
-        /* Load the slot number into register */
-        BPF_STMT(BPF_LD+BPF_B+BPF_ABS, 18),
-        /* Is it 0xff? No, goto INVALID */
+        BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, slot, 1, 0),
+        /* Does it match broadcast? No, goto INVALID */
          BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, 0xff, 0, 1),
          /* VALID: return -1 (allow the packet to be read) */
          BPF_STMT(BPF_RET+BPF_K, -1),

------------------------------------------------------------------------------
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform available
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs
Catalin Salgau | 16 May 11:31 2014
Picon
Picon

[PATCH] Obtain interface MTU on FreeBSD

Hi. I've been sitting on some changes for vblade for some time now and I 
remembered to clean them up and contribute them back, so here's one.

diff --git a/vblade/freebsd.c b/vblade/freebsd.c
--- a/vblade/freebsd.c
+++ b/vblade/freebsd.c
 <at>  <at>  -261,7 +261,24  <at>  <at> 
  int
  getmtu(int fd, char *name)
  {
-	return 1500;
+	struct ifreq xx;
+    int s,n;
+
+	s = socket(AF_INET, SOCK_RAW, 0);
+    if (s == -1)
+    {
+	  perror("Can't get mtu");
+	  return 1500;
+	}
+	xx.ifr_addr.sa_family = AF_INET;
+	strcpy(xx.ifr_name, name);
+	n = ioctl(s,SIOCGIFMTU, &xx);
+	if (n == -1) {
+	  perror("Can't get mtu");
+	  return 1500;
+	}
+	close(s);
+	return xx.ifr_mtu;
  }

  vlong

------------------------------------------------------------------------------
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform available
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs
Turbo Fredriksson | 11 May 18:18 2014

Exposing shares

I'm in the process of adding AoE support to ZoL
(ZFS On Linux - https://github.com/zfsonlinux/zfs)
and have gotten most of the way there (https://github.com/FransUrbo/zfs/tree/aoe).

I'm the author of the iSCSI (not yet accepted) and SMB
(partly accepted) support in ZoL, so adding AoE seemed
like a good and easy addition.

I have however caught on a couple of snags that complicates
the issue.

	1. How do I find the device that is shared?
	    'aoe-stat' doesn't say, and there's noting in the
	    /sys fs that I can see.

	    I can see the process in the list
		debian:~# /bin/ps faxwww | grep 'vblade 9 0 eth0' | grep -v grep
		12175 pts/1    S      0:00 sh -c /usr/sbin/vblade 9 0 eth0 /dev/zvol/rpool/test < /dev/null 2>&1 | logger -t vbladed
		12177 pts/1    S      0:00  \_ /usr/sbin/vblade 9 0 eth0 /dev/zvol/rpool/test
	    but that doesn't seems like a secure way of
	    knowing.

	2. How do I terminate (unshare) a device?
	    It's in the process list above, but that also seems
	    a little ... 'wonkey'.
	    It would have been nice if vblade/vbladed wrote
	    a pid file or something.

	    Also, before killing the process, it would be nice
	    to know if 'anyone' where accessing it.

	3. Actually, getting ANY information on the server doesn't
	   seem to be possible. All the aoe-* commands returns null
	   and vblade/vbladed doesn't have a 'show' or 'stats' option...

So I'm wondering about some 'status API' or somewhat to retrieve
information about shares and their status. Both the three iSCSI
implementations I added support for in ZoL as well as samba have
a well defined 'status' information.

AoE/vblade have _nothing_ of this kind as far as I can see.
--

-- 
Turbo Fredriksson
turbo@...

------------------------------------------------------------------------------
Is your legacy SCM system holding you back? Join Perforce May 7 to find out:
&#149; 3 signs your SCM is hindering your productivity
&#149; Requirements for releasing software faster
&#149; Expert tips and advice for migrating your SCM now
http://p.sf.net/sfu/perforce
Killer{R} | 10 May 21:19 2014

(probably stupid) support&data integrity questions

Hello aoetools-discuss,

1) Do you have forum or support is only this maillist
2) Data integrity   - do somebody had any data cossuption due to AoE
seems doesn't have anything to detect from-wire errors? I mean at
first corrupted packets themselves (I know that in hardware ethernet
check it.. usually but...) and there'is another possible problem that
seems is not addressed by AoE protocol: possible packets duplication
on ethernet level that can be caused for example by bad network
topology, failing hardware (I've seen semi-dead network switches that
flooded ethernet segment with duplicated packets) and so on? Because
there possible data corruption if write-from-the-past-packet
overwrites some data region. Theoretically this could be easily be
fixed by using some sequence numbers, that could be known by server to
be mononically increasing. Futhermore there'is 'tag' field in ATA
command that seems can be used for such purpose but unfortunatelly
according to documentation and vblade sources it completely ignored by
server:(

--

-- 
Best regards,
 Killer{R}                          mailto:support@...

------------------------------------------------------------------------------
Is your legacy SCM system holding you back? Join Perforce May 7 to find out:
&#149; 3 signs your SCM is hindering your productivity
&#149; Requirements for releasing software faster
&#149; Expert tips and advice for migrating your SCM now
http://p.sf.net/sfu/perforce
Killer{R} | 10 May 20:33 2014

AoE performance on small boxes

Hello aoetools-discuss,

  I tried to use AoE vblade on Dlink DNS 325 (arm box with 256 megs of
  ram) with Win2k3 srv/WinAoE 0.97g installed connected by gigabit
  ethernet with 7k jumbo frames enabled on both sides (bigger doesn't
  work somewhy on my HW). And got quite disappointing performance
  rate: while download (from vblade to WinAoE) was quite good
  (50 megs per second), upload was about 20 megs per second, that
  was much less than SMB transfer.
  I decided to try to make world better and patched vblade code, so it
  uses PACKET_RX_RING instead of plain read() and also added quite
  simple write buffering. First thing improved downalod by ~10% - to
  ~55 megs per second, second - dramatically increased upload -
  to 39 megs per second that is almost twice faster.
  Are aoetools devs planning to do somthing like this in future?
  BTW I'm not sure it can give comparable improvements on other
  systems since tested only on that little linux box

--

-- 
Best regards,
 Killer{R}                          mailto:support@...

------------------------------------------------------------------------------
Is your legacy SCM system holding you back? Join Perforce May 7 to find out:
&#149; 3 signs your SCM is hindering your productivity
&#149; Requirements for releasing software faster
&#149; Expert tips and advice for migrating your SCM now
http://p.sf.net/sfu/perforce
Christoph Biedl | 8 May 22:43 2014
Picon

[PATCH] Implement an offset option for vbladed

Hello,

the patch below implements a new option -o for vbladed, denoting the
number of sectors that should be skipped at the beginning of the
exported file. Default is 0 (no offset, export from beginning, current
behaviour).

Rationale:

If the exported files are actually raw block devices on the target,
something that might happen when using logical volumes for exports,
several tools running on the target that assume block devices are
for exclusive local access only might create confusion or even
havoc. Most prominently:

* os-prober of grub (might be disabled, though)
  Might mount any block device found that contain a file system.
* (Linux) logical volume management
  Will report any physical volume if the export is used as such.
* (Linux) blkid 
  Reports any block device (minor impact, mostly information leakage)

The overall problem is applications are not aware some block devices 
might be used from a remote client while there's no locking and that
might be hard to implement

Solutions tried but not considered helpful:

* Filter out exported block devices
  Problems: The tools need the ability to filter; new tools might come
  into existance, requiring new action. Filter languages might be hard
  to use (Linux LVM is quite a nightmare here).
* Optionally open block devices with O_EXCL
  Had no effect. Still exposes block devices where no vbladed is running
  yet.

So the goal was to hide the actual content from these tools. One
approach was to obscure the data using XOR, however it seemed easier
to sacrifice one or more sectors (8 recommended for 4k sector hard
drives) for a simple workaround.

This is based on vblade-20, tests on Debian Linux included a simple
write of each sector with a unique ID and later re-read in order to
detect any re-ordering bugs, however everything worked as expected.

    Christoph

--- a/aoe.c
+++ b/aoe.c
 <at>  <at>  -464,9 +464,10  <at>  <at>  main(int argc, char **argv)
 	int ch, omode = 0, readonly = 0;

 	bufcnt = Bufcount;
+	offset = 0;
 	setbuf(stdin, NULL);
 	progname = *argv;
-	while ((ch = getopt(argc, argv, "b:dsrm:")) != -1) {
+	while ((ch = getopt(argc, argv, "b:dsrm:o:")) != -1) {
 		switch (ch) {
 		case 'b':
 			bufcnt = atoi(optarg);
 <at>  <at>  -485,6 +486,9  <at>  <at>  main(int argc, char **argv)
 		case 'm':
 			setmask(optarg);
 			break;
+		case 'o':
+			offset = atoi(optarg);
+			break;
 		case '?':
 		default:
 			usage();
 <at>  <at>  -505,6 +509,11  <at>  <at>  main(int argc, char **argv)
 	setserial(argv[3], shelf, slot);
 	size = getsize(bfd);
 	size /= 512;
+	if (size < offset) {
+		fprintf(stderr, "Offset %u too big - remaining size is negative!\n", offset);
+		exit(1);
+	}
+	size -= offset;
 	ifname = argv[2];
 	sfd = dial(ifname, bufcnt);
 	getea(sfd, ifname, mac);
diff --git a/ata.c b/ata.c
index 4854f34..7c53b46 100644
--- a/ata.c
+++ b/ata.c
 <at>  <at>  -155,12 +155,12  <at>  <at>  atacmd(Ataregs *p, uchar *dp, int ndp, int payload) // do the ata cmd
 		return 0;
 	}
 	if (p->cmd == 0x20 || p->cmd == 0x24)
-		n = getsec(bfd, dp, lba, p->sectors);
+		n = getsec(bfd, dp, lba+offset, p->sectors);
 	else {
 		// packet should be big enough to contain the data
 		if (payload < 512 * p->sectors)
 			return -1;
-		n = putsec(bfd, dp, lba, p->sectors);
+		n = putsec(bfd, dp, lba+offset, p->sectors);
 	}
 	n /= 512;
 	if (n != p->sectors) {
diff --git a/dat.h b/dat.h
index 064ab1e..f3a93d8 100644
--- a/dat.h
+++ b/dat.h
 <at>  <at>  -169,6 +169,7  <at>  <at>  uchar	mac[6];
 int	bfd;		// block file descriptor
 int	sfd;		// socket file descriptor
 vlong	size;		// size of vblade
+unsigned offset;
 char	*progname;
 char	serial[Nserial+1];

diff --git a/vblade.8 b/vblade.8
index 2cd2b2c..3d8ff13 100644
--- a/vblade.8
+++ b/vblade.8
 <at>  <at>  -55,6 +55,10  <at>  <at>  The -r flag restricts the export of the device to be read-only.
 The -m flag takes an argument, a comma separated list of MAC addresses
 permitted access to the vblade.  A MAC address can be specified in upper
 or lower case, with or without colons.
+.TP
+\fB-o\fP
+The -o flag takes an argument, the number of sectors that should be
+skipped at the beginning of filename.
 .SH EXAMPLE
 In this example, the root user on a host named
 .I nai
------------------------------------------------------------------------------
Is your legacy SCM system holding you back? Join Perforce May 7 to find out:
&#149; 3 signs your SCM is hindering your productivity
&#149; Requirements for releasing software faster
&#149; Expert tips and advice for migrating your SCM now
http://p.sf.net/sfu/perforce
_______________________________________________
Aoetools-discuss mailing list
Aoetools-discuss@...
https://lists.sourceforge.net/lists/listinfo/aoetools-discuss

Gmane