Amjad Syed | 9 Sep 09:14 2014
Picon

Physical shutdown of one node causes both node to crash in active/passive configuration of 2 node RHEL cluster

Hi,

I have setup a 2 node cluster using RHEL 6.5 .

The cluster .conf looks like this 

 

<?xml version="1.0"?>
<cluster config_version="7" name="oracleha">
        <cman expected_votes="1" two_node="1"/>
        <fencedevices>
           <fencedevice agent= "fence_ipmilan" ipaddr="10.10.63.93" login="ADMIN" name="inspuripmi"  passwd="XXXXX/>
           <fencedevice agent = "fence_ilo2" ipaddr="10.10.63.92" login="test" name="hpipmi"  passwd="XXXXX"/>
          </fencedevices>
      <fence_daemon post_fail_delay="0" post_join_delay="60"/>
        <clusternodes>
           <clusternode name= "192.168.10.10"  nodeid="1" >
           <fence>
               <method name  = "1">
                 <device lanplus = "" name="inspuripmi"  action ="reboot"/>
                 </method>
            </fence>
           </clusternode>
            <clusternode name = "192.168.10.11" nodeid="2">
                 <fence>
                 <method name = "1">
                  <device lanplus = "" name="hpipmi" action ="reboot"/>
                  </method>
               </fence>
            </clusternode>
         </clusternodes>


        <rm>

          <failoverdomains/>
        <resources/>
        <service autostart="1" exclusive="0" name="IP" recovery="relocate">
                <ip address="10.10.5.23" monitor_link="on" sleeptime="10"/>
        </service>
</rm>

</cluster>


The network is as follows:

1)Heartbeat (Bonding) between node 1 and node 2  using ethernet cables 

The ip addresses are 192.168.10.11 and 192.168.10.10 for node 1 and node 2.

2) IPMI.  This is used for fencing and addresses  are 10.10.63.93 and 10.10.63.92

3) External ethernet connected to 10.10.5.x network.

If i do fence_node <ipaddress>, then fencing works,
However if i physically shutdown active node, the passive node also shutdowns. Even if i do ifdown bond0 (on active node), both node shutdown and have to be physically rebooted.

Any thing i am doing wrong ?




--

-- 
Linux-cluster mailing list
Linux-cluster <at> redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster
Neale Ferguson | 8 Sep 16:44 2014
Picon

Re: F_SETLK fails after recovery

Further to the problem described last week. What I'm seeing is that the node (NODE2) that keeps going when
NODE1 fails has many entries in dlm_tool log_plocks output:

1410147734 lvclusdidiz0360 receive plock 10303 LK WR 0-7fffffffffffffff 1/8112/13d5000 w 0
1410147734 lvclusdidiz0360 receive plock 1305c LK WR 0-7fffffffffffffff 1/8147/1390400 w 0
1410147734 lvclusdidiz0360 receive plock 50081 LK WR 0-7fffffffffffffff 1/8182/7ce04400 w 0
1410147736 lvclusdidiz0360 receive plock 10303 LK WR 0-7fffffffffffffff 1/8112/13d5000 w 0
1410147736 lvclusdidiz0360 receive plock 1305c LK WR 0-7fffffffffffffff 1/8147/1390400 w 0
1410147736 lvclusdidiz0360 receive plock 50081 LK WR 0-7fffffffffffffff 1/8182/7ce04400 w 0
1410147738 lvclusdidiz0360 receive plock 10303 LK WR 0-7fffffffffffffff 1/8112/13d5000 w 0
1410147738 lvclusdidiz0360 receive plock 1305c LK WR 0-7fffffffffffffff 1/8147/1390400 w 0
1410147738 lvclusdidiz0360 receive plock 50081 LK WR 0-7fffffffffffffff 1/8182/7ce04400 w 0
1410147740 lvclusdidiz0360 receive plock 10303 LK WR 0-7fffffffffffffff 1/8112/13d5000 w 0
1410147740 lvclusdidiz0360 receive plock 1305c LK WR 0-7fffffffffffffff 1/8147/1390400 w 0
1410147740 lvclusdidiz0360 receive plock 50081 LK WR 0-7fffffffffffffff 1/8182/7ce04400 w 0
1410147742 lvclusdidiz0360 receive plock 10303 LK WR 0-7fffffffffffffff 1/8112/13d5000 w 0
1410147742 lvclusdidiz0360 receive plock 1305c LK WR 0-7fffffffffffffff 1/8147/1390400 w 0
1410147742 lvclusdidiz0360 receive plock 50081 LK WR 0-7fffffffffffffff 1/8182/7ce04400 w 0

i.e. with no corresponding unlock entry. NODE1 is brought down by init 6 and when it restarts it gets as far as
"Starting cman" before NODE2 fences it (I assume we need a higher post_join_delay). When the node is
fenced I see:

1410147774 clvmd purged 0 plocks for 1
1410147774 lvclusdidiz0360 purged 3 plocks for 1

So it looks like it tried to some clean up but then when NODE1 attempts to join NODE2 examines the lockspace
and reports the following:

1410147820 lvclusdidiz0360 wr sect ro 0 rf 0 len 40 "r78067.0"
1410147820 lvclusdidiz0360 wr sect ro 0 rf 0 len 40 "r78068.0"
1410147820 lvclusdidiz0360 wr sect ro 0 rf 0 len 40 "r78059.0"
1410147820 lvclusdidiz0360 wr sect ro 0 rf 0 len 40 "r88464.0"
1410147820 lvclusdidiz0360 wr sect ro 0 rf 0 len 40 "r88478.0"
1410147820 lvclusdidiz0360 store_plocks first 66307 last 88478 r_count 45 p_count 63 sig 5ab0
1410147820 lvclusdidiz0360 receive_plocks_stored 2:8 flags a sig 5ab0 need_plocks 0

So it believes NODE1 will have 45 plocks to process when it comes back. NODE1 receives that plock
information: 

1410147820 lvclusdidiz0360 set_plock_ckpt_node from 0 to 2
1410147820 lvclusdidiz0360 receive_plocks_stored 2:8 flags a sig 5ab0 need_plocks 1

However, when NODE1 attempts to retrieve plocks it reports:

1410147820 lvclusdidiz0360 retrieve_plocks
1410147820 lvclusdidiz0360 retrieve_plocks first 0 last 0 r_count 0 p_count 0 sig 0

Because of the mismatch between sig 0 and sig 5ab0 plocks get disabled and the F_SETLK operation on the gfs2
target will fail on NODE1.

I'm am try to understand the checkpointing process and from where this information is actually being retrieved.

Neale

--

-- 
Linux-cluster mailing list
Linux-cluster <at> redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster

urgrue | 3 Sep 11:07 2014

Possible to apply changes without restart?

Hi,
Using cman/rgmanager in RHEL6 - is it possible to add a resource to my
service and have it be picked up and started without having to restart
cman/rgmanager? I thought ccs --activate did this, and the rgmanager.log
does output:
Sep 03 10:50:07 rgmanager Stopping changed resources.
Sep 03 10:50:07 rgmanager Restarting changed resources.
Sep 03 10:50:07 rgmanager Starting changed resources.

But the resource I added is not running nor is there any mention of it
in the logs. Is it normal or did I do something wrong?

--

-- 
Linux-cluster mailing list
Linux-cluster <at> redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster

Neale Ferguson | 2 Sep 16:56 2014
Picon

F_SETLK fails after recovery

Hi,
 In our two node system if one node fails, the other node takes over the application and uses the shared gfs2
target successfully. However, after the failed node comes back any attempts to lock files on the gfs2
resource results in -ENOSYS. The following test program exhibits the problem - in normal operation the
lock succeeds but in the fail/recover scenario we get -ENOSYS:

#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>

int 
main(int argc, char **argv)
{
	int fd;
	struct flock fl;

	fd = open("/mnt/test.file",O_RDONLY);
	if (fd != -1) {
		if (fcntl(fd, F_SETFL, O_RDONLY|O_DSYNC) != -1) {
			fl.l_type = F_RDLCK;
			fl.l_whence = SEEK_SET;
			fl.l_start = 0;
			fl.l_len = 0;
			if (fcntl(fd, F_SETLK, &fl) != -1)
				printf("File locked successfully\n");
			else
				perror("fcntl(F_SETLK)");
		} else
			perror("fcntl(F_SETFL)");
		close (fd);
	} else 
		perror("open");
}

I've tracked things down to these messages:

1409631951 lockspace lvclusdidiz0360 plock disabled our sig 816fba01 nodeid 2 sig 2f6b
:
1409634840 lockspace lvclusdidiz0360 plock disabled our sig 0 nodeid 2 sig 2f6b

Which indicates the lockspace attribute disable_plock has been set by way of the other node calling send_plocks_stored
().

Looking at the cpg.c:

static void prepare_plocks(struct lockspace *ls)
{

struct change *cg = list_first_entry(&ls->changes, struct change, list);

struct member *memb;
uint32_t sig;

:
:
:
      if (nodes_added(ls))
            store_plocks(ls, &sig);
      send_plocks_stored(ls, sig);
}

If nodes_added(ls) returns false then an uninitialized "sig" value will be passed to
send_plocks_stored(). Do the "our sig" and "sig" values in the above log messages make sense?

If this is not the case, what is supposed to happen in order re-enable plocks on the recovered node?

Neale

--

-- 
Linux-cluster mailing list
Linux-cluster <at> redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster

manish vaidya | 30 Aug 16:12 2014

Please help me on cluster error

i created four node cluster in kvm enviorment But i faced error when create new pv such as pvcreate /dev/sdb1
got error , lock from node 2 & lock from node3

also strange cluster logs

Jun 10 14:46:24 node1 corosync[3266]: [TOTEM ] Retransmit List: 5e

Jun 10 14:46:24 node1 corosync[3266]: [TOTEM ] Retransmit List: 5e
5f
Jun 10 14:46:24 node1 corosync[3266]: [TOTEM ] Retransmit List: 5f
60
Jun 10 14:46:24 node1 corosync[3266]: [TOTEM ] Retransmit List: 61
Jun 10 14:46:24 node1 corosync[3266]: [TOTEM ] Retransmit List: 63
64
Jun 10 14:46:24 node1 corosync[3266]: [TOTEM ] Retransmit List: 69
6a
Jun 10 14:46:24 node1 corosync[3266]: [TOTEM ] Retransmit List: 78
Jun 10 14:46:24 node1 corosync[3266]: [TOTEM ] Retransmit List: 84
85
Jun 10 14:46:24 node1 corosync[3266]: [TOTEM ] Retransmit List: 9a
9b


Please help me on this issue

Get your own FREE website, FREE domain & FREE mobile app with Company email.  
Know More >
--

-- 
Linux-cluster mailing list
Linux-cluster <at> redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster
Neale Ferguson | 28 Aug 21:11 2014
Picon

Delaying fencing during shutdown

Hi,
 In a two node cluster I shutdown one of the nodes and the other node notices the shutdown but on rare occasions
that node will then fence the node that is shutting down. I assume Is this a situation where setting
post_fail_delay would be useful or setting the totem timeout to something higher than its default.

Neale

--

-- 
Linux-cluster mailing list
Linux-cluster <at> redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster

Ferenc Wagner | 26 Aug 22:42 2014
Picon

locating a starting resource

Hi,

crm_resource --locate finds the hosting node of a running (successfully
started) resource just fine.  Is there a way to similarly find out the
location of a resource *being* started, ie. whose resource agent is
already running the start action, but that action is not finished yet?
-- 
Thanks,
Feri.

--

-- 
Linux-cluster mailing list
Linux-cluster <at> redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster

Vasil Valchev | 26 Aug 08:56 2014
Picon

totem token & post_fail_delay question

Hello,

I have a cluster that sometimes has intermittent network issues on the heartbeat network.
Unfortunately improving the network is not an option, so I am looking for a way to tolerate longer interruptions.

Previously it seemed to me the post_fail_delay option is suitable, but after some research it might not be what I am looking for.

If I am correct, when a member leaves (due to token timeout) the cluster will wait the post_fail_delay before fencing. If the member rejoins before that, it will still be fenced, because it has previous state?
From a recent fencing on this cluster there is a strange message:

Aug 24 06:20:45 node2 openais[29048]: [MAIN ] Not killing node node1cl despite it rejoining the cluster with existing state, it has a lower node ID

What does this mean?

And lastly is increasing the totem token timeout the way to go?


Thanks,
Vasil Valchev
--

-- 
Linux-cluster mailing list
Linux-cluster <at> redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster
Ferenc Wagner | 22 Aug 02:37 2014
Picon

on exiting maintenance mode

Hi,

While my Pacemaker cluster was in maintenance mode, resources were moved
(by hand) between the nodes as I rebooted each node in turn.  In the end
the crm status output became perfectly empty, as the reboot of a given
node removed from the output the resources which were located on the
rebooted node at the time of entering maintenance mode.  I expected full
resource discovery on exiting maintenance mode, but it probably did not
happen, as the cluster started up resources already running on other
nodes, which is generally forbidden.  Given that all resources were
running (though possibly migrated during the maintenance), what would
have been the correct way of bringing the cluster out of maintenance
mode?  This should have required no resource actions at all.  Would
cleanup of all resources have helped?  Or is there a better way?
-- 
Thanks,
Feri.

--

-- 
Linux-cluster mailing list
Linux-cluster <at> redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster

Neale Ferguson | 20 Aug 06:45 2014
Picon

clvmd not terminating

We have a sporadic situation where we are attempting to shutdown/restart both nodes of a two node cluster.
One shutdowns completely but one sometimes hangs with:

[root <at> aude2mq036nabzi ~]# service cman stop
Stopping cluster:
   Leaving fence domain... found dlm lockspace /sys/kernel/dlm/clvmd
fence_tool: cannot leave due to active systems
[FAILED]

When the other node is brought back up it has problems with clvmd:

># pvscan
  connect() failed on local socket: Connection refused
  Internal cluster locking initialisation failed.
  WARNING: Falling back to local file-based locking.
  Volume Groups with the clustered attribute will be inaccessible.

Sometimes it works fine but very occasionally we get the above situation. I've encountered the fence
message before, usually when the fence devices were incorrectly configured but it would always fail
because of this. Before I get too far into investigation mode I wondered if the above symptoms ring any
bells for anyone.

Neale

--

-- 
Linux-cluster mailing list
Linux-cluster <at> redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster

Laszlo Budai | 29 Jul 18:58 2014
Picon

RHEL 6.3 - qdisk lost

Dear All,

I have a two node cluster (rgmanager-3.0.12.1-17.el6.x86_64) with a 
shared storage. The storage contains the quorum disk also.
there are some services, and there are some dependencies set.
We are testing what is happening if the storage is disconnected from one 
node (to see the cluster response for such a failure).
So we start from a good cluster (all is OK) and we disconnect the 
storage from the first node.

What I have observed:
1. the cluster is fencing node 1
2. node 2 is trying to start the services, but even if we have 3 
services (let's say B,C,D) which are depending on a fourth one (say A) 
the cluster is trying to start the services in this order: B,C,D,A. 
Obviously it fails for B,C,D and gives us the following messages:

Jul 29 15:49:54 node1 rgmanager[5135]: service:B is not runnable; 
dependency not met
Jul 29 15:49:54 node2 rgmanager[5135]: Not stopping service:B: service 
is failed
Jul 29 15:49:54 node2 rgmanager[5135]: Unable to stop RG service:B in 
recoverable state

it will leave them in "recoverable" state even if service A will start 
successfully (so the dependency would be met now). Why is this happening?

I would expect the rgmanager to start the services in an order that 
would satisfy the dependency relationships. Or if it is not doing that , 
then at least to react to the service state change event (service A has 
started, so dependencies should be evaluated again).

What can be done about it?

Thank you in advance,
Laszlo

--

-- 
Linux-cluster mailing list
Linux-cluster <at> redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster


Gmane