Distributed Replicated Block Device -> "network RAID 1"

headers John Anthony | 23 May 23:50

_______________________________________________
drbd-user mailing list
drbd-user@...
http://lists.linbit.com/mailman/listinfo/drbd-user

Hi,

I'm running DRBD 8.3.12, and recently hit what looks to me like a bug that was listed as fixed in 8.3.13 --
getting into a state where both nodes are in SyncSource (it's just stuck like that, going nowhere). 
Luckily this happened on a test resource and not a live one, so it's not a big problem, but I was wondering if
there were any known ways of recovering it without doing anything disruptive to the other resources (e.g.
rebooting or unloading the kernel module).

I've tried 'drbdadm down', but it just hangs -- anyone have any other suggestions?  It doesn't really matter
to me if it wipes the resource or anything, I'd just like to have my test device back in a working state
without disturbing anything else.

Thanks,
Zev Weiss

Dears Kindly help as I Can’t create the drbd device because of this
error message

drbdmeta /dev/drbd0 v08 /dev/sda2 internal create-md' terminated with
exit code 40

global {
                usage-count yes;
}
Common {
                Syncer {
                rate 10M;
}
}
resource r0 {
                protocol C;
                on master {
                                device   /dev/drbd1;
                                disk        /dev/sdb1;
                                address 192.168.1.1;7789;
                                meta-disk internal;
}
                on slave{
                                device   /dev/drbd1;
                                disk        /dev/sdb1;
                                address 192.168.1.2;7789;
                                meta-disk internal;
}

}

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          38      305203+  83  Linux
/dev/sda2              39        2349    18563107+  83  Linux
/dev/sda3            2350        2610     2096482+  82  Linux swap / Solaris

Disk /dev/sdb: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1         652     5237158+  83  Linux

[root <at> master ~]# drbdadm create-md r0
v08 Magic number not found
md_offset 5362843648
al_offset 5362810880
bm_offset 5362647040

Found ext3 filesystem which uses 5237156 kB
current configuration leaves usable 5236960 kB

Device size would be truncated, which
would corrupt data and result in
'access beyond end of device' errors.
You need to either
   * use external meta data (recommended)
   * shrink that filesystem first
   * zero out the device (destroy the filesystem)
Operation refused.

Command 'drbdmeta /dev/drbd1 v08 /dev/sdb1 internal create-md'
terminated with exit code 40
drbdadm aborting

--

-- 
Eng. Mahmoud Alshinhab
NOC System Engineer
RHCT -RedHat Certified Technician-
Fedora Ambassador
Wiki : https://fedoraproject.org/wiki/User:Tuxawy
mahmoud.alshinhab@...
tuxawy@...

Looking for guidance from Linbit, or the list, about DRBD's disks
remaining on two separate physical servers, except that the peer
servers are virtualized?

For instance: Writes of approximately 18,000 blocks per second for the
busy hours, as reported by SAR.  How much a performance hit is there
with the same hardware after VMWare is in the picture?  (I'm sure
there are VMWare config settings to optimize disk reads/writes for an
application like DRBD.)

Thanks.

=====Keith

No.

On 05/21/2012 10:24 AM, 陈楠 wrote:
> Actually, we just have one resource. /dev/vg01/share and /dev/vg02/share
> has the same size.
> Business system A
> 
> resource r0 {
>         meta-disk internal;
>         device /dev/drbd0;
>         disk /dev/vg01/share;  # <- Partition A: always vg01
>         on NodeA {
>                 address 2.2.2.150:7788;
>         }
>         on NodeB {
> 		address 2.2.2.151:7788;
>         }
> }
> 
> Business system B
> 
> resource r0 {
>         meta-disk internal;
>         device /dev/drbd0;
>         disk /dev/vg02/share;  # <- Partition A: always vg01
>         on NodeA {
>                 address 2.2.2.150:7788;
>         }
>         on NodeB {
> 		address 2.2.2.151:7788;
>         }
> }

Both nodes:

resource r0 {
        meta-disk internal;
        device /dev/drbd0;
        on NodeA {
	        disk /dev/vg01/share;
                address 2.2.2.150:7788;
        }
        on NodeB {
	        disk /dev/vg02/share;
		address 2.2.2.151:7788;
        }
}

It's OK for your nodes to be different, but *do include that in your
config*. Save yourself from pain down the road.

Regards,
Felix
_______________________________________________
drbd-user mailing list
drbd-user <at> lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Hello,

I am in the process of setting up DRBD on my servers, the network
bandwidth being the bottleneck.  After having evaluated GlusterFS I
realised, that I need the instant read access offered by DRBD.

Logically I am able to separate partitions that would require access
from both nodes, and partitions where an asynchronous master-slave
sync is sufficient.  But as far as I understand, the benefits from
using Protocol A instead of C are limited, when the network is stable.

My question:
Are there any additional benefits from NOT using two primaries or
additional risks when using it? eg. would there be significant
performance gain by using ext4 instead of GFS2/OCFS2? Anything else I
should take into consideration?

Thanks for any ideas or pointers where to look.

Karel

_______________________________________________
drbd-user mailing list
drbd-user@...
http://lists.linbit.com/mailman/listinfo/drbd-user

I'm trying to understand a symptom for a client who uses drbd to run
sets of virtual machines between three pairs of servers (v1a/v1b,
v2a/v2b, v3a/v3b), and I wanted to understand a bit better how DRBD I/O
is buffered depending on what mode is chosen, and buffer settings.

Firstly, it surprised me that even in replication mode "A", the system
still seemed limited by by the bandwidth between nodes.  I found this
out when the customer's bonded interface had flipped over to its 100Mb
backup connection, and suddenly they had I/O problems.  While I was
investigating this and running tests, I noticed that switching to mode A
didn't help, even when measuring short transfers that I'd expect would
fit into reasonable-sized buffers.  What kind of buffer size can I
expect from an "auto-tuned" DRBD?  It seems important to be able to
cover bursts without leaning on the network, so I'd like to know whether
that's possible with some special tuning.

The other problem is the "PingAck not received" messages that have been
littering the logs of the v3a/v3b servers for the last couple of weeks,
e.g. this has been happening every few hours for one DRBD or another:

May 14 08:21:45 v3b kernel: [661127.869500] block drbd10: PingAck did
not arrive in time.
May 14 08:21:45 v3b kernel: [661127.875553] block drbd10: peer( Primary
-> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate ->
DUnknown )
May 14 08:21:45 v3b kernel: [661127.875562] block drbd10: asender terminated
May 14 08:21:45 v3b kernel: [661127.875564] block drbd10: Terminating
drbd10_asender
May 14 08:21:45 v3b kernel: [661127.875597] block drbd10: short read
expecting header on sock: r=-512
May 14 08:21:45 v3b kernel: [661127.882896] block drbd10: Connection closed
May 14 08:21:45 v3b kernel: [661127.882899] block drbd10: conn(
NetworkFailure -> Unconnected )
May 14 08:21:45 v3b kernel: [661127.882904] block drbd10: receiver
terminated
May 14 08:21:45 v3b kernel: [661127.882908] block drbd10: Restarting
drbd10_receiver
May 14 08:21:45 v3b kernel: [661127.882910] block drbd10: receiver
(re)started
May 14 08:21:45 v3b kernel: [661127.882913] block drbd10: conn(
Unconnected -> WFConnection )
May 14 08:21:46 v3b kernel: [661129.123506] block drbd10: Handshake
successful: Agreed network protocol version 91
May 14 08:21:46 v3b kernel: [661129.123511] block drbd10: conn(
WFConnection -> WFReportParams )
May 14 08:21:46 v3b kernel: [661129.123535] block drbd10: Starting
asender thread (from drbd10_receiver [31418])
May 14 08:21:46 v3b kernel: [661129.123876] block drbd10:
data-integrity-alg: <not-used>
May 14 08:21:46 v3b kernel: [661129.123898] block drbd10:
drbd_sync_handshake:
May 14 08:21:46 v3b kernel: [661129.123900] block drbd10: self
C5DC68A8AFD5BFEC:0000000000000000:7EB45F3A26B3BD72:2EC9659EFC4BC513
bits:0 flags:0
May 14 08:21:46 v3b kernel: [661129.123903] block drbd10: peer
F8BB238D22A7ACFF:C5DC68A8AFD5BFED:7EB45F3A26B3BD72:2EC9659EFC4BC513
bits:0 flags:0
May 14 08:21:46 v3b kernel: [661129.123905] block drbd10:
uuid_compare()=-1 by rule 50
May 14 08:21:46 v3b kernel: [661129.123908] block drbd10: peer( Unknown
-> Primary ) conn( WFReportParams -> WFBitMapT ) pdsk( DUnknown ->
UpToDate )
May 14 08:21:46 v3b kernel: [661129.138101] block drbd10: conn(
WFBitMapT -> WFSyncUUID )
May 14 08:21:46 v3b kernel: [661129.139563] block drbd10: helper
command: /sbin/drbdadm before-resync-target minor-10
May 14 08:21:46 v3b kernel: [661129.140282] block drbd10: helper
command: /sbin/drbdadm before-resync-target minor-10 exit code 0 (0x0)
May 14 08:21:46 v3b kernel: [661129.140286] block drbd10: conn(
WFSyncUUID -> SyncTarget ) disk( UpToDate -> Inconsistent )
May 14 08:21:46 v3b kernel: [661129.140292] block drbd10: Began resync
as SyncTarget (will sync 0 KB [0 bits set]).
May 14 08:21:47 v3b kernel: [661129.693954] block drbd10: Resync done
(total 1 sec; paused 0 sec; 0 K/sec)
May 14 08:21:47 v3b kernel: [661129.693961] block drbd10: conn(
SyncTarget -> Connected ) disk( Inconsistent -> UpToDate )
May 14 08:21:47 v3b kernel: [661129.693969] block drbd10: helper
command: /sbin/drbdadm after-resync-target minor-10
May 14 08:21:47 v3b kernel: [661129.694725] block drbd10: helper
command: /sbin/drbdadm after-resync-target minor-10 exit code 0 (0x0)

I've not been able to correlate these ping drops and reconnections to
any of:

1) interface capacity issues (a few times we might make a 400Mb spike,
but sometimes there's none at all);

2) loss of connectivity or ARP problems on the two servers' dedicated
DRBD interfaces (i.e. I've got an unbroken log of pings between the two
servers);

3) any kernel grumbles about the network interface, bonding, RAID or
anything remotely hardware-related.  Apart from the drbd messages
there's no other chatter from the kernel.

The customer's other two pairs of servers have been running 18 months
and not exhibited this behaviour.

The customer hasn't given me the data to show these blips (which are
anything from 2s-30s) correspond to any real performance problems and I
don't have access to the inside of their VMs to check for myself.  So my
questions are - would you expect these disconnections to cause
variations in I/O bandwidth or responsiveness?

And secondly, what should I be doing about it?  My unsatisfactory
response to the customer's worry is to reconnect all the drbds with a
longer ping-timeout, and in 10 hours it hasn't reoccurred, which is an
unusually long record.  I will be more convinced by the end of the day.

Even if that does solve these messages, I'm curious as to the cause.
We've not hit a network bandwidth ceiling, and so we've definitely not
hit an I/O ceiling (which is 4x146GB 15kRPM discs, RAID10, HP RAID).  I
can accept that some VMs will use more bandwidth than others, and so it
wouldn't be surprising that one VM on the machine was the "cause".

But when the disconnections happen, they appear to be completely random.
 Checking with grep/uniq -c, I see out of the 11 devices on the systems,
it happens pretty randomly (and drbd10 is just a test, getting
absolutely zero I/O).

      5 drbd0:
      5 drbd1:
     11 drbd2:
      8 drbd3:
     11 drbd4:
      4 drbd5:
      6 drbd6:
      7 drbd7:
      5 drbd8:
     14 drbd9:
     12 drbd10:
      7 drbd11:

So even if upping the ping time stops the problem, and even if the
effects of the disconnect/reconnect cycles are harmless - why might DRBD
exhibit these symptoms on one pair of servers, but not two other sets?
Is there some I/O pattern that might cause pings to get lost, even over
a lightly-loaded gigabit link?

Thanks for any insights in advance.

--

-- 
Matthew

Hello,

I use 2 servers with CentOS 6.2 

But on 1 server (kvmstorage1) ifs is not starting after a restart or when i shutdown heartbeat and restart is later.

# this is my resources file? it start not all services i have given in the haresources file?
Do i something wrong?

#kvmstorage1
cat /etc/ha.d/haresources
kvmstorage1.localdomain IPaddr::192.168.123.209/24/eth0 drbddisk::main
Filesystem::/dev/drbd0::/datastore::ext4 nfs nfslock rpcidmapd mysql

ResourceManager[2506]:	2012/05/16_21:34:40 info: Running /etc/ha.d/resource.d/Filesystem
/dev/drbd0 /datastore ext4 start
Filesystem[2933]:	2012/05/16_21:34:40 INFO: Running start for /dev/drbd0 on /datastore
Filesystem[2921]:	2012/05/16_21:34:41 INFO:  Success
ResourceManager[2506]:	2012/05/16_21:34:41 info: Running /etc/init.d/nfslock  start
ResourceManager[2506]:	2012/05/16_21:34:41 info: Running /etc/init.d/rpcidmapd  start
ResourceManager[2506]:	2012/05/16_21:34:42 info: Running /etc/init.d/mysqld  start
May 16 21:34:43 kvmstorage1.localdomain heartbeat: [2489]: info: local HA resource acquisition
completed (standby).
May 16 21:34:43 kvmstorage1.localdomain heartbeat: [1583]: info: Standby resource acquisition done [foreign].
May 16 21:34:43 kvmstorage1.localdomain heartbeat: [1583]: info: Initial resource acquisition
complete (auto_failback)
May 16 21:34:43 kvmstorage1.localdomain heartbeat: [1583]: info: remote resource transition completed.

#kvmstorage2
cat /etc/ha.d/haresources
kvmstorage1.localdomain IPaddr::192.168.123.209/24/eth0 drbddisk::main
Filesystem::/dev/drbd0::/datastore::ext4 nfs nfslock rpcidmapd mysql

Filesystem[16037]:	2012/05/16_21:33:53 INFO:  Resource is stopped
ResourceManager[15787]:	2012/05/16_21:33:53 info: Running /etc/ha.d/resource.d/Filesystem
/dev/drbd0 /datastore ext4 start
Filesystem[16117]:	2012/05/16_21:33:53 INFO: Running start for /dev/drbd0 on /datastore
Filesystem[16109]:	2012/05/16_21:33:53 INFO:  Success
ResourceManager[15787]:	2012/05/16_21:33:54 info: Running /etc/init.d/nfs  start
ResourceManager[15787]:	2012/05/16_21:33:54 info: Running /etc/init.d/nfslock  start
ResourceManager[15787]:	2012/05/16_21:33:54 info: Running /etc/init.d/mysqld  start
mach_down[15761]:	2012/05/16_21:33:56 info: /usr/share/heartbeat/mach_down: nice_failback:
foreign resources acquired
May 16 21:33:56 kvmstorage2.localdomain heartbeat: [1528]: info: mach_down takeover complete.
mach_down[15761]:	2012/05/16_21:33:56 info: mach_down takeover complete for node kvmstorage1.localdomain.

127	May 2012
194	April 2012
216	March 2012
220	February 2012
226	January 2012
150	December 2011
161	November 2011
149	October 2011
193	September 2011
187	August 2011
226	July 2011
186	June 2011
201	May 2011
97	April 2011
158	March 2011
195	February 2011
272	January 2011
98	December 2010
130	November 2010
171	October 2010
223	September 2010
102	August 2010
147	July 2010
107	June 2010
224	May 2010
149	April 2010
290	March 2010
142	February 2010
93	January 2010
180	December 2009
202	November 2009
209	October 2009
155	September 2009
91	August 2009
176	July 2009
201	June 2009
163	May 2009
168	April 2009
214	March 2009
244	February 2009
219	January 2009
310	December 2008
132	November 2008
222	October 2008
246	September 2008
307	August 2008
233	July 2008
192	June 2008
225	May 2008
183	April 2008
326	March 2008
336	February 2008
225	January 2008
264	December 2007
151	November 2007
161	October 2007
186	September 2007
218	August 2007
131	July 2007
153	June 2007
124	May 2007
128	April 2007
124	March 2007
189	February 2007
177	January 2007
132	December 2006
176	November 2006
154	October 2006
204	September 2006
149	August 2006
170	July 2006
169	June 2006
134	May 2006
188	April 2006
160	March 2006
192	February 2006
121	January 2006
115	December 2005
144	November 2005
181	October 2005
221	September 2005
256	August 2005
157	July 2005
136	June 2005
89	May 2005
183	April 2005
144	March 2005
88	February 2005
114	January 2005
77	December 2004
157	November 2004
182	October 2004
279	September 2004
168	August 2004
254	July 2004
257	June 2004
285	May 2004
315	April 2004
198	March 2004
132	February 2004
136	January 2004
169	December 2003
245	November 2003
172	October 2003
126	September 2003
251	August 2003
205	July 2003
124	June 2003
179	May 2003
159	April 2003
296	March 2003
258	February 2003
158	January 2003
95	December 2002
108	November 2002
160	October 2002
48	September 2002
120	August 2002
270	July 2002
71	June 2002
71	May 2002
80	April 2002
68	March 2002
45	February 2002
75	January 2002
65	December 2001
162	November 2001
178	October 2001
164	September 2001
83	August 2001
63	July 2001
34	June 2001
50	May 2001
6	April 2001
29	March 2001
49	February 2001
29	January 2001
100	December 2000
151	November 2000
105	October 2000
133	September 2000
96	August 2000
36	July 2000
100	June 2000
76	May 2000
2	November 1999

Changing the name of a resource

Recovering from erroneous sync state

need discribtion and help

Moving a DRBD cluster from physical machines to VMWare machines

Re: drbd wrong lower device doubt

Reasons not to use allow-two-primaries with DRDB

DRBD block script with Xen XL toolstack

drbd wrong lower device doubt

I need Reset DRBD Service

"PingAck not received" messages

NFS not starting with heartbeat

Mon	Tue	Wed	Thu	Fri	Sat	Sun

	1	2	3	4	5	6
7	8	9	10	11	12	13
14	15	16	17	18	19	20
21	22	23	24	25	26	27
28	29	30	31