headers
Peter Breuer | 8 Jan 2009 15:27
Picon

Re: How to automatically add a nbd to a raid1?

Hi .. from a hotel in freezing-cold almeria.

"Also sprach Michael Rendell:"
>   Background info: both machines are P4's at 2.4GHz with 512M memory
> running 2.6.18-92.1.22.el5; mkfs is using 4k blocks;  the second mkfs

OK .. I think there was a MD bug with 512MB ram some time back.
Probably not significant, but you might want to grep the archives for
it.

> that was mentioned in the last email finished after about 68 minutes
> with no errors reported in /proc/nbdinfo.

68 mins!

>   Had a look at the strace output - seems to be using cycles in
> the semaphore code (in sem_wait_timeout()):

Semaphore?  How come it's using real semaphores ..  surely I implemented
my own locking?  I did do something _like_ semaphores in shared memory,
but I wouldn't have thought it was visible as system calls in a trace.

>     recv(9, 
> "%`\225\23\0\0\0\1\267\363\320 <at> \0\0\0\0\0\233P\0\0\0\20\0\0\0\0\0\0\0\0\
>     16"..., 64, 0) = 64
>     clock_gettime(CLOCK_MONOTONIC, {1277506, 711333904}) = 0
>     rt_sigprocmask(SIG_BLOCK, [ALRM], NULL, 8) = 0
>     setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={29, 999356}}, NULL) 
> = 0
>     rt_sigaction(SIGALRM, {0x8058540, [ALRM], SA_RESTART}, {0x8058540, [ALRM],
(Continue reading)

Permalink | Reply | 1 comment |
headers
Peter T. Breuer | 8 Jan 2009 15:27
Picon

Re: How to automatically add a nbd to a raid1?

Hi .. from a hotel in freezing-cold almeria.

"Also sprach Michael Rendell:"
>   Background info: both machines are P4's at 2.4GHz with 512M memory
> running 2.6.18-92.1.22.el5; mkfs is using 4k blocks;  the second mkfs

OK .. I think there was a MD bug with 512MB ram some time back.
Probably not significant, but you might want to grep the archives for
it.

> that was mentioned in the last email finished after about 68 minutes
> with no errors reported in /proc/nbdinfo.

68 mins!

>   Had a look at the strace output - seems to be using cycles in
> the semaphore code (in sem_wait_timeout()):

Semaphore?  How come it's using real semaphores ..  surely I implemented
my own locking?  I did do something _like_ semaphores in shared memory,
but I wouldn't have thought it was visible as system calls in a trace.

>     recv(9, 
> "%`\225\23\0\0\0\1\267\363\320 <at> \0\0\0\0\0\233P\0\0\0\20\0\0\0\0\0\0\0\0\
>     16"..., 64, 0) = 64
>     clock_gettime(CLOCK_MONOTONIC, {1277506, 711333904}) = 0
>     rt_sigprocmask(SIG_BLOCK, [ALRM], NULL, 8) = 0
>     setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={29, 999356}}, NULL) 
> = 0
>     rt_sigaction(SIGALRM, {0x8058540, [ALRM], SA_RESTART}, {0x8058540, [ALRM],
(Continue reading)

Permalink | Reply | 0 comments |
headers
Michael Rendell | 18 Dec 2008 02:45
Picon
Picon

mkfs.ext3 taking long time on raid1 with nda disk

Hi Peter,

  Have noticed that creating a file system on a raid1 device
that uses a enbd device sometimes runs into problems: looks
like it is geting errors writing to the server and taking a
while to recover.  Wondering if something is wrong in the
configuration?

The set up is as follows:

/etc/enbd.conf:
    module 0 show_errs=1
    client mDisklessVars /dev/nda nmhd-bs2 1090 -e -n4 -q -t 0

(pwprog is configured but disabled)

cat /proc/nbdinfo
    Device a:       Open
    [a] State:      verify, rw, enabled, validated, show_errs, last error 0, 
lives 0, bp 0
    [a] Queued:     +0R/0W curr (check 0R/0W) +0R/2W max
    [a] Buffersize: 266240  (sectors=520, blocks=65)
    [a] Blocksize:  4096    (log=12)
    [a] Size:       3387392KB
    [a] Blocks:     846848
    [a] Sockets:    4       (+)     (+)     (*)     (+)
    [a] Requested:  0       (0)     (0)     (0)     (0)     0R/0W   max 0
    [a] Despatched: 0       (0)     (0)     (0)     (0)     0R/0W   md5 0W (0 
eq, 0 ne, 0 dn)
    [a] Errored:    0       (0)     (0)     (0)     (0)     0+0
(Continue reading)

Permalink | Reply | 1 comment |
headers
Michael Rendell | 17 Dec 2008 21:11
Picon
Picon

Re: How to automatically add a nbd to a raid1?

On Monday 15 December 2008 11:54, Peter Breuer wrote:
> "Also sprach Michael Rendell:"
>
> > > That's kind of understandable. But the raid disks themselves contain
> > > all the required info in their superblocks. Each will say what the
> > > other members of the array are. You should be able to interrogate them
> > > individually with mdadm --whatever and confirm my contention!
> >
> > While "mdadm --detail --verbose /dev/mdNN" doesn't list the removed
> > devices,
>
> It should .. morally, I mean. It KNOWS what devices are notionally in
> the array.

It may know, but it's not telling (via the mdadm interface anyway);
seems the --verbose option doesn't do much for the --detail mode.

...
> > the kernel module is loaded.  As this is before the enbd-client is
> > started (and before it gets the connection going), it isn't useful
> > for this purpose.
>
> Yes.  But what you want is the ONLINE and OFFLINE notices that enbd
> sends to udev, not the ADD.
>
> The ADD and REMOVE events are just generated by the kernel disk
> partition subsystem as the driver registers control of its majors and
> minors.  They're what udev rules send to the MAKEDEV and REMOVEDEV
> scripts in the enbd distr, via the provided enbd.rules file.
(Continue reading)

Permalink | Reply | 5 comments |
headers
Peter T. Breuer | 15 Dec 2008 19:50
Picon

update to kernel 2.4.26-11

I've updated enbd 2.4.35a for kernel 2.6.26-11 (and tested it working).

There were a few issues - that kernel is weird. I'm still working on
making enbd 2.4.36 work with it, but now that 2.4.35a works, it
shouldn't be long ...

Peter
Permalink | Reply | 1 comment |
headers
Michael Rendell | 15 Dec 2008 16:00
Picon
Picon

Re: How to automatically add a nbd to a raid1?

On Thursday 11 December 2008 20:13, Peter T. Breuer wrote:
> "Also sprach Michael Rendell:"
>
> > > >   I'd like to set up a raid1 disk that has a local disk and a remote
> > > > (enbd) disk.  Was wondering how to set things up so that, on/after
> > > > reboot, the enbd device gets added to the raid when the connection to
> > > > the enbd-server is made?
> > >
> > > There are scripts to handle that in the distribution package. I'm
> > > travelling today but I'll put up a complete answer later (about
> > > midnight CET).
> >
> > When booting, there is a window of time between when the raid drives are
> > started and when the network/enbd get going.
>
> Yes, but don't worry about it. The raid will come up in degraded mode
> if it comes up first (shrug .. it's your choice), then enbd will come
> up, and add itself to the degraded array via whatever scripts exist.
>
> Then the array will silently bring the enbd device up to date or vice
> versa in the background. That won't cost much because RAID is
> intelligent these days and only updates out-of-date blocks, as recorded
> on the bitmap.
>
> > Am using auto-detect on the
> > raid drives,
>
> Uh, I think that means partition type e0 and md compiled into the
> kernel, which does a special hack at bootup to look for them ...
>
(Continue reading)

Permalink | Reply | 1 comment |
headers
Peter T. Breuer | 14 Dec 2008 12:54
Picon

Re: [enbd:144] Re: re arm stability

"Also sprach ptb:"
> But hang on .. in my hotel last night I managed to find a corner case
> for the mmap code that needs investigating. Maybe it's only against
> localhost that it happens, but mmap hangs for the first read on the
> device if not preceded by any write at all.  All I see is the syscall
> hanging.
> 
> I'll investigate and fix. Confirmed in 2.4.36 only. But I'll bet that
> the same thing applies to 2.4.35a.

I lost the bet. 2.4.35a does not have the issue.

The code is present but turned off in enbd 2.4.35a.  Apparently I knew
about the potential for harm.

For 2.4.36 I have now turned it off also while I ask kernel VM gurus
about what goes.

I'm bringing the enbd 2.4.36 code in line with kernel 2.6.26 anyway.
There are several VM changes in that area in 2.6.36 which may help ..
if I can get past the fact they're removed the "nopage" field from
vma_struct, grr ...

Peter
Permalink | Reply | 0 comments |
headers
Michael Rendell | 10 Dec 2008 20:59
Picon
Picon

How to automatically add a nbd to a raid1?

Hello,

  I'd like to set up a raid1 disk that has a local disk and a remote (enbd)
disk.  Was wondering how to set things up so that, on/after reboot, the enbd
device gets added to the raid when the connection to the enbd-server is made?
The raid may already have started in degraded mode (especially likely
if the network or server is down when the client machine boots).
Running "mdadm --re-add /dev/mdNN /dev/ndA" works, but would like this to be
done automatically when the connection is made: is there a way to run
a program/script when a connection is made (like mdadm --monitor)?
or is there some other way of doing this?

Many thanks!

Michael Rendell
Permalink | Reply | 1 comment |
headers
Volker Stoppe | 4 Nov 2008 00:29
Picon

client (-1) cannot open NBD device /dev/nda or similar: No such file or directory

Hallo list!

I am trying to implement ENBD but I stuck at the moment when I want to connect from the client to the Server.
Compiation of the source works fine, the modules are loading without a problem and I have no problem
starting with the server. I would be -very- glad if someone could help! Nobody in the web seems to have my
problem. I followed the instructions from: http://www.it.uc3m.es/~ptb/nbd/ but there has to be
something missing. It is definitly not sudo (installed).

Environment:
Distri: Debian 4.0
Kernel: 2.6.18-6-686
ENBD: 2.4.35

PS: I know that client and server have the same hostname (clones).

Some output of the Systems:

== Server ==
debian32:~/enbd# lsmod | grep enbd
enbd_ioctl              5920  0
enbd                  583032  1 enbd_ioctl

debian32:~/enbd# ./enbd-server 4001 /dev/sda1
enbd-server[3843]: server: will open resources in mode linear
enbd-server[3843]: looking for blksize of /dev/sda1 with fstat...
enbd-server[3843]: blksize of /dev/sda1 is 4096                  
enbd-server[3843]: set final blksize of whole resource to 4096   
enbd-server[3843]: looking for size of fd 4 with seek SEEK_END...
enbd-server[3843]: set size of fd 4 to 98668544                  
enbd-server[3843]: server: set blksize to 4096.
(Continue reading)

Permalink | Reply | 3 comments |
headers
Peter T. Breuer | 22 Jul 2008 11:33
Picon

Re: Compiling issues with enbd-2.4.35a and kernel2.6.25.9

"Also sprach uwe schmeling:"
> No problem I will do it, btw. is there any advanting in using the 2.4.35a?

It's benefitting from most of the development work done in 2.4.36,
without the architectural changes.

In particular it does zero-copy networking. And some network packet
'shepherding' that helps when the new multireq=1 option is set
(this does request merging in userspace, making up for anything the
kernel can't merge, up to the limit set by merge_requests=N).

I.e. "yes".

Peter
Permalink | Reply | 0 comments |
headers
Peter T. Breuer | 21 Jul 2008 16:20
Picon

Re: Compiling issues with enbd-2.4.35a and kernel2.6.25.9

"Also sprach uwe schmeling:"
  [ of end_that_request_last and its little nephews and nieces ]

> > Can you please check for me where it has gone? Or if it has gone.
> > 
> this is now living in blk-core.c but there is only a __end_that_xxx
> which is not exported anymore

Sigh. That is a big change. I'll need to see what to do about it.

BTW, you can probably make the newer 2.4.35 enbd_request struct equal
in size to the 2.4.30 enbd_request struct by adding a "special" field
in the newer struct at the point where it appears in the older struct.
Then recompile and see if the newer client now works with the older
server.

It would likely die horribly, taking your grandmother and pet rabbit
with. But maybe not.

> > You can't, unfortunately. The on-the-wire sizes and shapes of the
> > headers preceding data has likely changed between 2.4.30 and 2.4.35.
> > The enbd_request and enbd_reply structs are defined in include/enbd.h.
> > 
> > In 2.4.35a request is:
> > 
> > struct enbd_request {
> >   __u32  magic;
> >   __u32   type;
> >   __u32 handle;
> >   __u64   from;
(Continue reading)

Permalink | Reply | 0 comments |

Gmane