Eivind Olsen | 24 May 10:45 2016
Picon
Gravatar

How to add an unimportant resource to an important cluster?

I have a cluster, running RHEL 6.7, with Ricci, Luci, rgmanager etc.
This is a 2 node cluster, where services are running on one node. The 
service is an Oracle database, and the cluster controls several 
resources:
* LVM volume (using clvmd)
* file system on a logical volume
* IP address
* Oracle listener
* Oracle RDBMS instance

I have now been asked to put another resource (another Oracle RBMS 
instance) but with the requirement that this new resource shouldn't 
cause the rest of the cluster resources to fail over to the other node. 
Basically, what's been asked is to have another resource which will be 
started by the cluster, but if it fails the health check it will be left 
alone.

Is it possible to somehow mark one of the resources as "Not really 
important, attempt to restart if down but don't migrate the entire 
service with all the resources to the other node"?
My gut feeling tells me the better (correct, only etc.) way is probably 
to set up a separate service for this new less important RDBMS instance, 
giving it its own IP address, LVM volume, filesystem, listener etc.

Regards
Eivind Olsen

--

-- 
Linux-cluster mailing list
Linux-cluster <at> redhat.com
(Continue reading)

Jonathan Davies | 15 Apr 16:55 2016
Gravatar

I/O to gfs2 hanging or not hanging after heartbeat loss

Dear linux-cluster,

I have made some observations about the behaviour of gfs2 and would 
appreciate confirmation of whether this is expected behaviour or 
something has gone wrong.

I have a three-node cluster -- let's call the nodes A, B and C. On each 
of nodes A and B, I have a loop that repeatedly writes an increasing 
integer value to a file in the GFS2-mountpoint. On node C, I have a loop 
that reads from both these files from the GFS2-mountpoint. The reads on 
node C show the latest values written by A and B, and stay up-to-date. 
All good so far.

I then cause node A to drop the corosync heartbeat by executing the 
following on node A:

iptables -I INPUT -p udp --dport 5404 -j DROP
iptables -I INPUT -p udp --dport 5405 -j DROP
iptables -I INPUT -p tcp --dport 21064 -j DROP

After a few seconds, I normally observe that all I/O to the GFS2 
filesystem hangs forever on node A: the latest value read by node C is 
the same as the last successful write by node A. This is exactly the 
behaviour I want -- I want to be sure that node A never completes I/O 
that is not able to be seen by other nodes.

However, on some occasions, I observe that node A continues in the loop 
believing that it is successfully writing to the file but, according to 
node C, the file stops being updated. (Meanwhile, the file written by 
node B continues to be up-to-date as read by C.) This is concerning -- 
(Continue reading)

Stefano Panella | 12 Apr 14:45 2016

Help with corosync and GFS2 on multi network setup

Hi everybody,

we have been using corosync directly to provide clustering for GFS2 on our centos 7.2 pools with only one
network interface and all has been working great so far!

We now have a new set-up with two network interfaces for every host in the cluster:
A -> 1 Gbit (the one we would like corosync to use, 10.220.88.X)
B -> 10 Gbit (used for iscsi connection to storage, 10.220.246.X)

when we run corosync in this mode we get the logs continuously spammed by messages like these:

[12880] cl15-02 corosyncdebug   [TOTEM ] entering GATHER state from 0(consensus timeout).
[12880] cl15-02 corosyncdebug   [TOTEM ] Creating commit token because I am the rep.
[12880] cl15-02 corosyncdebug   [TOTEM ] Saving state aru 10 high seq received 10
[12880] cl15-02 corosyncdebug   [MAIN  ] Storing new sequence id for ring 5750
[12880] cl15-02 corosyncdebug   [TOTEM ] entering COMMIT state.
[12880] cl15-02 corosyncdebug   [TOTEM ] got commit token
[12880] cl15-02 corosyncdebug   [TOTEM ] entering RECOVERY state.
[12880] cl15-02 corosyncdebug   [TOTEM ] TRANS [0] member 10.220.88.41:
[12880] cl15-02 corosyncdebug   [TOTEM ] TRANS [1] member 10.220.88.47:
[12880] cl15-02 corosyncdebug   [TOTEM ] position [0] member 10.220.88.41:
[12880] cl15-02 corosyncdebug   [TOTEM ] previous ring seq 574c rep 10.220.88.41
[12880] cl15-02 corosyncdebug   [TOTEM ] aru 10 high delivered 10 received flag 1
[12880] cl15-02 corosyncdebug   [TOTEM ] position [1] member 10.220.88.47:
[12880] cl15-02 corosyncdebug   [TOTEM ] previous ring seq 574c rep 10.220.88.41
[12880] cl15-02 corosyncdebug   [TOTEM ] aru 10 high delivered 10 received flag 1

[12880] cl15-02 corosyncdebug   [TOTEM ] Did not need to originate any messages in recovery.
[12880] cl15-02 corosyncdebug   [TOTEM ] got commit token
[12880] cl15-02 corosyncdebug   [TOTEM ] Sending initial ORF token
(Continue reading)

Daniel Dehennin | 11 Apr 14:29 2016

GFS2 and LVM stripes

Hello,

My OpenNebula cluster has a 4TB GFS2 logical volume supported by two
physical volumes (2TB each).

The result is that near all I/O go to a single PV.

Now I'm looking at a way to convert linear LV to a stripping one and
only found the possibility to go with a mirror[1].

Do you have any advice on the use of GFS2 over stipped LVM?

Regards.

Footnotes: 
[1]  http://community.hpe.com/t5/System-Administration/Need-to-move-the-data-from-Linear-LV-to-stripped-LV-on-RHEL-5-7/td-p/6134323

-- 
Daniel Dehennin
Récupérer ma clef GPG: gpg --recv-keys 0xCC1E9E5B7A6FE2DF
Fingerprint: 3E69 014E 5C23 50E8 9ED6  2AAD CC1E 9E5B 7A6F E2DF
--

-- 
Linux-cluster mailing list
Linux-cluster <at> redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster
Daniel Dehennin | 8 Apr 11:21 2016

GFS2: debugging I/O issues

Hello,

On our virtualisation infrastructure we have a 4To GFS2 over a SAN.

Since one or two weeks we are facing read I/O issues, 5k or 6k IOPS with
an average block size of 5kB.

I'm looking for the possibilities and didn't find anything yet, so my
question: 

    Is it possible that reaching over 80% use of the GFS2 can produce
    such workload?

Regards.

-- 
Daniel Dehennin
Récupérer ma clef GPG: gpg --recv-keys 0xCC1E9E5B7A6FE2DF
Fingerprint: 3E69 014E 5C23 50E8 9ED6  2AAD CC1E 9E5B 7A6F E2DF
--

-- 
Linux-cluster mailing list
Linux-cluster <at> redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster
Bob Peterson | 29 Mar 16:34 2016
Picon

Re: fsck.gfs2 The root dinode block is destroyed.

----- Original Message -----
> Good Morning,
> 
> We have a large cluster with 50 gfs2 SAN mounts.  The mounts range in
> size from 1TB to 15TB each.  We have some with 6-8TB of data but most
> average around 3TB used right now.  We were doing network testing a
> while back to check our redundancy incase of a switch failure, and the
> tests failed..  multiple times.  We ended having the SAN mounts yanked
> out from under the cluster.  Long story short, we seem to have
> corruption.  I can still bring the volumes up with the cluster but
> when i take everything down and do a fsck I get the following:
> 
> 
> (ran with fsck -n /dev/$device)
> 
> Found a copy of the root directory in a journal at block: 0x501ca.
> Damaged root dinode not fixed.
> The root dinode should be at block 0x2f3b98b7 but it seems to be destroyed.
> Found a copy of the root directory in a journal at block: 0x501d2.
> Damaged root dinode not fixed.
> The root dinode should be at block 0x28a3ac7f but it seems to be destroyed.
> Found a copy of the root directory in a journal at block: 0x501da.
> Damaged root dinode not fixed.
> Unable to locate the root directory.
> Can't find any dinodes that might be the root; using master - 1.
> Found a possible root at: 0x16
> The root dinode block is destroyed.
> At this point I recommend reinitializing it.
> Hopefully everything will later be put into lost+found.
> The root dinode was not reinitialized; aborting.
(Continue reading)

Megan . | 29 Mar 15:52 2016
Picon

fsck.gfs2 The root dinode block is destroyed.

Good Morning,

We have a large cluster with 50 gfs2 SAN mounts.  The mounts range in
size from 1TB to 15TB each.  We have some with 6-8TB of data but most
average around 3TB used right now.  We were doing network testing a
while back to check our redundancy incase of a switch failure, and the
tests failed..  multiple times.  We ended having the SAN mounts yanked
out from under the cluster.  Long story short, we seem to have
corruption.  I can still bring the volumes up with the cluster but
when i take everything down and do a fsck I get the following:

(ran with fsck -n /dev/$device)

Found a copy of the root directory in a journal at block: 0x501ca.
Damaged root dinode not fixed.
The root dinode should be at block 0x2f3b98b7 but it seems to be destroyed.
Found a copy of the root directory in a journal at block: 0x501d2.
Damaged root dinode not fixed.
The root dinode should be at block 0x28a3ac7f but it seems to be destroyed.
Found a copy of the root directory in a journal at block: 0x501da.
Damaged root dinode not fixed.
Unable to locate the root directory.
Can't find any dinodes that might be the root; using master - 1.
Found a possible root at: 0x16
The root dinode block is destroyed.
At this point I recommend reinitializing it.
Hopefully everything will later be put into lost+found.
The root dinode was not reinitialized; aborting.

This particular device had 4698 "seems to be destroyed..  found a
(Continue reading)

Robert Hayden | 18 Mar 14:24 2016
Picon

ACPI like feature on RHEL 7 with Pacemaker?

I was testing fence_ipmilan on RHEL 7 cluster and noticed that running the fence agent with the option to power off the remote node, it appears to cleanly stop the remote node instead of removing power immediately.  I suspect something like ACPI intercepting the power off and trying to stop RHEL 7 nicely.  I read through the documentation and did not see any mentions of turning off acpid in RHEL 7 (may because it does not exist) like in the RHEL 6 documentation.

 

Is the proper way to disable acpi like functionality to use the kernel line acpi=off?  Curious on what others are using with HP iLO.

 

Example of remote node’s /var/log/messages … the RHCS_TESTING lines are my unit testing scripts that insert the command I am running.  You can see where I attempt to power off the node and then how system is starting to cleanly stop the node along with pacemaker.

 

Mar 17 15:38:37 node2 RHCS_TESTING: .

Mar 17 15:38:37 node2 RHCS_TESTING: CMD/MSG: fence_ipmilan -P -a x.x.x.x -l XXXXXX -p XXXXX  -L OPERATOR -A password -o off

Mar 17 15:38:37 node2 RHCS_TESTING: .

Mar 17 15:38:37 node2 systemd-logind: Removed session 993.

Mar 17 15:38:37 node2 systemd: Removed slice user-0.slice.

Mar 17 15:38:37 node2 systemd: Stopping user-0.slice.

Mar 17 15:38:37 node2 systemd-logind: Power key pressed.

Mar 17 15:38:37 node2 systemd-logind: Powering Off...

Mar 17 15:38:37 node2 systemd-logind: System is powering down.

Mar 17 15:38:37 node2 systemd: Stopping Availability of block devices...

Mar 17 15:38:37 node2 systemd: Stopping LVM2 PV scan on device 8:2...

Mar 17 15:38:37 node2 systemd: Stopping Pacemaker High Availability Cluster Manager...

Mar 17 15:38:37 node2 systemd: Deactivating swap /dev/mapper/vg00-swaplv00...

Mar 17 15:38:37 node2 pacemakerd[79586]:  notice: Invoking handler for signal 15: Terminated

Mar 17 15:38:37 node2 pacemakerd[79586]:  notice: Shuting down Pacemaker

Mar 17 15:38:37 node2 pacemakerd[79586]:  notice: Stopping crmd: Sent -15 to process 79592

Mar 17 15:38:37 node2 crmd[79592]:  notice: Invoking handler for signal 15: Terminated

Mar 17 15:38:37 node2 crmd[79592]:  notice: Requesting shutdown, upper limit is 1200000ms

Mar 17 15:38:37 node2 crmd[79592]:  notice: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_SHUTDOWN cause=C_SHUTDOWN origin=crm_shutdown ]

Mar 17 15:38:37 node2 multipathd: mpathb: stop event checker thread (140595795502848

 

 

Thanks

Robert

--

-- 
Linux-cluster mailing list
Linux-cluster <at> redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster
Elsaid Younes | 11 Mar 04:55 2016
Picon

Copying the result continuously.


Hi all,

I  wish to be able to run long simulation through gromacs program, using MPI method. I want to modify the input data after every sub-task.
I think that is the meaning of the following code, which is part my script.
cat <<EOF > copyfile.sh #!/bin/sh cp -p result*.dat $SLURM_SUBMIT_DIR EOF chmod u+x copyfile.sh srun -n $SLURM_NNODES -N $SLURM_NNODES cp copyfile.sh $SNIC_TMP
And I have to srun copyfile.sh in the end of every processor.
srun -n $SLURM_NNODES -N $SLURM_NNODES copyfile.sh
Is there something wrong? I need to know what is the meaning of result*?

Thanks in advance,
/Elsaid
--

-- 
Linux-cluster mailing list
Linux-cluster <at> redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster
Shreekant Jena | 5 Mar 07:46 2016
Picon

CMAN Failed to start on Secondary Node

Dear All,

I have a 2 node cluster but after reboot secondary node is showing offline . And cman failed to start .

Please find below logs on secondary node:-

root <at> EI51SPM1 cluster]# clustat
msg_open: Invalid argument
Member Status: Inquorate

Resource Group Manager not running; no service information available.

Membership information not available
[root <at> EI51SPM1 cluster]# tail -10 /var/log/messages
Feb 24 13:36:23 EI51SPM1 ccsd[25487]: Error while processing connect: Connection refused
Feb 24 13:36:23 EI51SPM1 kernel: CMAN: sending membership request
Feb 24 13:36:27 EI51SPM1 ccsd[25487]: Cluster is not quorate.  Refusing connection.
Feb 24 13:36:27 EI51SPM1 ccsd[25487]: Error while processing connect: Connection refused
Feb 24 13:36:28 EI51SPM1 kernel: CMAN: sending membership request
Feb 24 13:36:32 EI51SPM1 ccsd[25487]: Cluster is not quorate.  Refusing connection.
Feb 24 13:36:32 EI51SPM1 ccsd[25487]: Error while processing connect: Connection refused
Feb 24 13:36:32 EI51SPM1 ccsd[25487]: Cluster is not quorate.  Refusing connection.
Feb 24 13:36:32 EI51SPM1 ccsd[25487]: Error while processing connect: Connection refused
Feb 24 13:36:33 EI51SPM1 kernel: CMAN: sending membership request
[root <at> EI51SPM1 cluster]#
[root <at> EI51SPM1 cluster]# cman_tool status
Protocol version: 5.0.1
Config version: 166
Cluster name: IVRS_DB
Cluster ID: 9982
Cluster Member: No
Membership state: Joining
[root <at> EI51SPM1 cluster]# cman_tool nodes
Node  Votes Exp Sts  Name
[root <at> EI51SPM1 cluster]#
[root <at> EI51SPM1 cluster]#


Thanks & regards 
SHREEKANTA JENA

--

-- 
Linux-cluster mailing list
Linux-cluster <at> redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster
Karl-Philipp Richter | 1 Mar 16:50 2016
Picon

How to undo vgchange --clustered y?

Hi,
I invoked `vgchange --clustered y [name]` and accepted the warning that
the volume group might become inaccessible by mistaking inaccessible
with unavailable for other cluster nodes. Since `clvm` doesn't work on
Ubuntu 15.10 and building from source is
painful<ref>http://askubuntu.com/questions/740615/how-to-get-clvmd-running-on-ubuntu-15-10</ref>
I seem to have no chance to ever access the clustered volume group. Is
there any solution to make the volume group accessible again?

-Kalle

--

-- 
Linux-cluster mailing list
Linux-cluster <at> redhat.com
https://www.redhat.com/mailman/listinfo/linux-cluster

Gmane