[Pacemaker] Cluster with two STONITH devices

[Jorge Lopes]

Hi all.

I'm having difficulties orchestrating two STONITH devices in my cluster. I
have been struggling with this in past days and I need some help, please.

A simplified version of my cluster and its goals is as follows:
- The cluster has two physical servers, each with two nodes (VMWare virtual
machines): overall, there are 4 nodes in this simplified version.
- There are two resource groups: group-cluster-a and group-cluster-b.
- To achieve a good CPU balance in the physical servers, the cluster is
asymmetric, with one group running in one server and the other group
running on the other server.
- If the VM of one host becomes not usable, then its resources are started
in its sister VM deployed in the other physical host.
- If one physical host becomes not usable, then all resources are started
in the other physical host.
- Two STONITH levels are used to fence the problematic nodes.

The resources have the following behavior:
- If the resource monitor detects a problem, then Pacemaker tries to
restart the resource in the same node.
- If it fails, then STONITH takes place (vcenter reboots the VM) and
Pacemaker starts the resource in the sister VM present in the other
physical host.
- If restarting the VM fails, I want to power off the physical server and
Pacemaker will start all resources in the other physical host.


The HA stack is:
Ubuntu 14.04 (the node OS, which is a visualized guest running in VMWare
ESXi 5.5)
Pacemaker 1.1.12
Corosync 2.3.4
CRM 2.1.2

The 4 nodes are:
cluster-a-1
cluster-a-2
cluster-b-1
cluster-b-2

The relevant configuration is:

property symmetric-cluster=false
property stonith-enabled=true
property no-quorum-policy=stop

group group-cluster-a vip-cluster-a docker-web
location loc-group-cluster-a-1 group-cluster-a inf: cluster-a-1
location loc-group-cluster-a-2 group-cluster-a 500: cluster-a-2

group group-cluster-b vip-cluster-b docker-srv
location loc-group-cluster-b-1 group-cluster-b 500: cluster-b-1
location loc-group-cluster-b-2 group-cluster-b inf: cluster-b-2


# stonith vcenter definitions for host 1
# run in any of the host2 VM
primitive stonith-vcenter-host1 stonith:external/vcenter \
params \
VI_SERVER="192.168.40.20" \
VI_CREDSTORE="/etc/vicredentials.xml" \
HOSTLIST="cluster-a-1=cluster-a-1;cluster-a-2=cluster-a-2" \
RESETPOWERON="1" \
priority="2" \
pcmk_host_check="static-list" \
pcmk_host_list="cluster-a-1 cluster-a-2" \
op monitor interval="60s"

location loc1-stonith-vcenter-host1 stonith-vcenter-host1 500: cluster-b-1
location loc2-stonith-vcenter-host1 stonith-vcenter-host1 501: cluster-b-2

# stonith vcenter definitions for host 2
# run in any of the host1 VM
primitive stonith-vcenter-host2 stonith:external/vcenter \
params \
VI_SERVER="192.168.40.21" \
VI_CREDSTORE="/etc/vicredentials.xml" \
HOSTLIST="cluster-b-1=cluster-b-1;cluster-b-2=cluster-b-2" \
RESETPOWERON="1" \
priority="2" \
pcmk_host_check="static-list" \
pcmk_host_list="cluster-b-1 cluster-b-2" \
op monitor interval="60s"

location loc1-stonith-vcenter-host2 stonith-vcenter-host2 500: cluster-a-1
location loc2-stonith-vcenter-host2 stonith-vcenter-host2 501: cluster-a-2


# stonith IPMI definitions for host 1 (DELL with iDRAC 7 enterprise
interface at 192.168.40.15)
# run in any of the host2 VM
primitive stonith-ipmi-host1 stonith:external/ipmi \
params hostname="host1" ipaddr="192.168.40.15" userid="root"
passwd="mypassword" interface="lanplus" \
priority="1" \
pcmk_host_check="static-list" \
pcmk_host_list="cluster-a-1 cluster-a-2" \
op start interval="0" timeout="60s" requires="nothing" \
op monitor interval="3600s" timeout="20s" requires="nothing"

location loc1-stonith-ipmi-host1 stonith-ipmi-host1 500: cluster-b-1
location loc2-stonith-ipmi-host1 stonith-ipmi-host1 501: cluster-b-2


# stonith IPMI definitions for host 2 (DELL with iDRAC 7 enterprise
interface at 192.168.40.16)
# run in any of the host1 VM
primitive stonith-ipmi-host2 stonith:external/ipmi \
params hostname="host2" ipaddr="192.168.40.16" userid="root"
passwd="mypassword" interface="lanplus" \
priority="1" \
pcmk_host_check="static-list" \
pcmk_host_list="cluster-b-1 cluster-b-2" \
op start interval="0" timeout="60s" requires="nothing" \
op monitor interval="3600s" timeout="20s" requires="nothing"

location loc1-stonith-ipmi-host2 stonith-ipmi-host2 500: cluster-a-1
location loc2-stonith-ipmi-host2 stonith-ipmi-host2 501: cluster-a-2


What is working:
- When an error is detected in one resource, the resource restart in the
same node, as expected.
- With the STONITH external/ipmi resource *stopped*, a fail in one node
makes the vcenter rebooting it and the resources starts in the sister node.


What is not so good:
- When vcenter reboots one node, then the resource starts in the other node
as expected but then they return to the original node as soon as it becomes
online. This makes a bit of ping-pong and I think it is a consequence of
how the locations are defined. Any suggestion to avoid this? After the
resource was moved to another node, I would prefer that it stays there,
instead of returning it to the original node. I can think of playing with
the resource affinity scores - is this way it should be done?

What is wrong:
Lets consider this scenario.
I have a set of resources provided by a docker agent. My test consists in
stopping the docker service in the node cluster-a-1, which makes the docker
agent to return OCF_ERR_INSTALLED to Pacemaker (this is a change I made in
the docker agent, when compared to the github repository version). With the
IPMI STONITH resource stopped, this leads to the node cluster-a-1 restart,
which is expected.

But with the IPMI STONITH resource started, I notice an erratic behavior:
- Some times, the resources at the node cluster-a-1 are stopped and no
STONITH happens. Also, the resources are not moved to the node cluster-a-2.
In this situation, if I manually restart the node cluster-a-1 (virtual
machine restart), then the IPMI STONITH takes place and restarts the
corresponding physical server.
- Sometimes, the IPMI STONITH starts before the vCenter STONITH, which is
not expected because the vCenter STONITH has higher priority.

I might have something wrong in my stonith definition, but I can't figure
what.
Any idea how to correct this?

And how can I set external/ipmi to power off the physical host, instead of
rebooting it?

[Jorge Lopes]

(I'm a bit confused because I received an auto-reply form
pacemaker-***@oss.clusterlabs.org saying this list is inactive now but
I just received a digest with my mail. I happens that I have resent the
email to the new list with a bit more information, which was missing in the
first message. So here it is that extra bit, anyway).

I also have noticed this pattern (with both STONITH resources running):
1. With the cluster running without errors, I run "stop docker" in node
cluster-a-1.
2. This leads to the vCenter STONITH to act as expected.
3. After the cluster is running again without errors, I run again "stop
docker" in node cluster-a-1.
4. Now, the vCenter STONITH doesn't run and, instead, it is the IPMI
STONITH that runs. This is unexpected for me, as I was expecting to see the
vCenter STONITH to run again.

Post by Jorge Lopes
Hi all.
I'm having difficulties orchestrating two STONITH devices in my cluster. I
have been struggling with this in past days and I need some help, please.
- The cluster has two physical servers, each with two nodes (VMWare
virtual machines): overall, there are 4 nodes in this simplified version.
- There are two resource groups: group-cluster-a and group-cluster-b.
- To achieve a good CPU balance in the physical servers, the cluster is
asymmetric, with one group running in one server and the other group
running on the other server.
- If the VM of one host becomes not usable, then its resources are started
in its sister VM deployed in the other physical host.
- If one physical host becomes not usable, then all resources are started
in the other physical host.
- Two STONITH levels are used to fence the problematic nodes.
- If the resource monitor detects a problem, then Pacemaker tries to
restart the resource in the same node.
- If it fails, then STONITH takes place (vcenter reboots the VM) and
Pacemaker starts the resource in the sister VM present in the other
physical host.
- If restarting the VM fails, I want to power off the physical server and
Pacemaker will start all resources in the other physical host.
Ubuntu 14.04 (the node OS, which is a visualized guest running in VMWare
ESXi 5.5)
Pacemaker 1.1.12
Corosync 2.3.4
CRM 2.1.2
cluster-a-1
cluster-a-2
cluster-b-1
cluster-b-2
property symmetric-cluster=false
property stonith-enabled=true
property no-quorum-policy=stop
group group-cluster-a vip-cluster-a docker-web
location loc-group-cluster-a-1 group-cluster-a inf: cluster-a-1
location loc-group-cluster-a-2 group-cluster-a 500: cluster-a-2
group group-cluster-b vip-cluster-b docker-srv
location loc-group-cluster-b-1 group-cluster-b 500: cluster-b-1
location loc-group-cluster-b-2 group-cluster-b inf: cluster-b-2
# stonith vcenter definitions for host 1
# run in any of the host2 VM
primitive stonith-vcenter-host1 stonith:external/vcenter \
params \
VI_SERVER="192.168.40.20" \
VI_CREDSTORE="/etc/vicredentials.xml" \
HOSTLIST="cluster-a-1=cluster-a-1;cluster-a-2=cluster-a-2" \
RESETPOWERON="1" \
priority="2" \
pcmk_host_check="static-list" \
pcmk_host_list="cluster-a-1 cluster-a-2" \
op monitor interval="60s"
location loc1-stonith-vcenter-host1 stonith-vcenter-host1 500: cluster-b-1
location loc2-stonith-vcenter-host1 stonith-vcenter-host1 501: cluster-b-2
# stonith vcenter definitions for host 2
# run in any of the host1 VM
primitive stonith-vcenter-host2 stonith:external/vcenter \
params \
VI_SERVER="192.168.40.21" \
VI_CREDSTORE="/etc/vicredentials.xml" \
HOSTLIST="cluster-b-1=cluster-b-1;cluster-b-2=cluster-b-2" \
RESETPOWERON="1" \
priority="2" \
pcmk_host_check="static-list" \
pcmk_host_list="cluster-b-1 cluster-b-2" \
op monitor interval="60s"
location loc1-stonith-vcenter-host2 stonith-vcenter-host2 500: cluster-a-1
location loc2-stonith-vcenter-host2 stonith-vcenter-host2 501: cluster-a-2
# stonith IPMI definitions for host 1 (DELL with iDRAC 7 enterprise
interface at 192.168.40.15)
# run in any of the host2 VM
primitive stonith-ipmi-host1 stonith:external/ipmi \
params hostname="host1" ipaddr="192.168.40.15" userid="root"
passwd="mypassword" interface="lanplus" \
priority="1" \
pcmk_host_check="static-list" \
pcmk_host_list="cluster-a-1 cluster-a-2" \
op start interval="0" timeout="60s" requires="nothing" \
op monitor interval="3600s" timeout="20s" requires="nothing"
location loc1-stonith-ipmi-host1 stonith-ipmi-host1 500: cluster-b-1
location loc2-stonith-ipmi-host1 stonith-ipmi-host1 501: cluster-b-2
# stonith IPMI definitions for host 2 (DELL with iDRAC 7 enterprise
interface at 192.168.40.16)
# run in any of the host1 VM
primitive stonith-ipmi-host2 stonith:external/ipmi \
params hostname="host2" ipaddr="192.168.40.16" userid="root"
passwd="mypassword" interface="lanplus" \
priority="1" \
pcmk_host_check="static-list" \
pcmk_host_list="cluster-b-1 cluster-b-2" \
op start interval="0" timeout="60s" requires="nothing" \
op monitor interval="3600s" timeout="20s" requires="nothing"
location loc1-stonith-ipmi-host2 stonith-ipmi-host2 500: cluster-a-1
location loc2-stonith-ipmi-host2 stonith-ipmi-host2 501: cluster-a-2
- When an error is detected in one resource, the resource restart in the
same node, as expected.
- With the STONITH external/ipmi resource *stopped*, a fail in one node
makes the vcenter rebooting it and the resources starts in the sister node.
- When vcenter reboots one node, then the resource starts in the other
node as expected but then they return to the original node as soon as it
becomes online. This makes a bit of ping-pong and I think it is a
consequence of how the locations are defined. Any suggestion to avoid this?
After the resource was moved to another node, I would prefer that it stays
there, instead of returning it to the original node. I can think of playing
with the resource affinity scores - is this way it should be done?
Lets consider this scenario.
I have a set of resources provided by a docker agent. My test consists in
stopping the docker service in the node cluster-a-1, which makes the docker
agent to return OCF_ERR_INSTALLED to Pacemaker (this is a change I made in
the docker agent, when compared to the github repository version). With the
IPMI STONITH resource stopped, this leads to the node cluster-a-1 restart,
which is expected.
- Some times, the resources at the node cluster-a-1 are stopped and no
STONITH happens. Also, the resources are not moved to the node cluster-a-2.
In this situation, if I manually restart the node cluster-a-1 (virtual
machine restart), then the IPMI STONITH takes place and restarts the
corresponding physical server.
- Sometimes, the IPMI STONITH starts before the vCenter STONITH, which is
not expected because the vCenter STONITH has higher priority.
I might have something wrong in my stonith definition, but I can't figure
what.
Any idea how to correct this?
And how can I set external/ipmi to power off the physical host, instead of
rebooting it?

[Digimer]

A recent decision to help reduce confusion with new users was to merge
all the various mailing lists, IRC channels and whatnot under one
banner. So the "Cluster Labs" list and IRC is becoming the only active one.

Please re-ask your question there and we will be happy to help. In the
meantime, look at http://clusterlabs.org/wiki/STONITH_Levels.

Cheers

Post by Jorge Lopes
(I'm a bit confused because I received an auto-reply form
inactive now but I just received a digest with my mail. I happens that I
have resent the email to the new list with a bit more information, which
was missing in the first message. So here it is that extra bit, anyway).
1. With the cluster running without errors, I run "stop docker" in node
cluster-a-1.
2. This leads to the vCenter STONITH to act as expected.
3. After the cluster is running again without errors, I run again "stop
docker" in node cluster-a-1.
4. Now, the vCenter STONITH doesn't run and, instead, it is the IPMI
STONITH that runs. This is unexpected for me, as I was expecting to see
the vCenter STONITH to run again.
Hi all.
I'm having difficulties orchestrating two STONITH devices in my
cluster. I have been struggling with this in past days and I need
some help, please.
- The cluster has two physical servers, each with two nodes (VMWare
virtual machines): overall, there are 4 nodes in this simplified version.
- There are two resource groups: group-cluster-a and group-cluster-b.
- To achieve a good CPU balance in the physical servers, the cluster
is asymmetric, with one group running in one server and the other
group running on the other server.
- If the VM of one host becomes not usable, then its resources are
started in its sister VM deployed in the other physical host.
- If one physical host becomes not usable, then all resources are
started in the other physical host.
- Two STONITH levels are used to fence the problematic nodes.
- If the resource monitor detects a problem, then Pacemaker tries to
restart the resource in the same node.
- If it fails, then STONITH takes place (vcenter reboots the VM) and
Pacemaker starts the resource in the sister VM present in the other
physical host.
- If restarting the VM fails, I want to power off the physical
server and Pacemaker will start all resources in the other physical
host.
Ubuntu 14.04 (the node OS, which is a visualized guest running in
VMWare ESXi 5.5)
Pacemaker 1.1.12
Corosync 2.3.4
CRM 2.1.2
cluster-a-1
cluster-a-2
cluster-b-1
cluster-b-2
property symmetric-cluster=false
property stonith-enabled=true
property no-quorum-policy=stop
group group-cluster-a vip-cluster-a docker-web
location loc-group-cluster-a-1 group-cluster-a inf: cluster-a-1
location loc-group-cluster-a-2 group-cluster-a 500: cluster-a-2
group group-cluster-b vip-cluster-b docker-srv
location loc-group-cluster-b-1 group-cluster-b 500: cluster-b-1
location loc-group-cluster-b-2 group-cluster-b inf: cluster-b-2
# stonith vcenter definitions for host 1
# run in any of the host2 VM
primitive stonith-vcenter-host1 stonith:external/vcenter \
params \
VI_SERVER="192.168.40.20" \
VI_CREDSTORE="/etc/vicredentials.xml" \
HOSTLIST="cluster-a-1=cluster-a-1;cluster-a-2=cluster-a-2" \
RESETPOWERON="1" \
priority="2" \
pcmk_host_check="static-list" \
pcmk_host_list="cluster-a-1 cluster-a-2" \
op monitor interval="60s"
location loc1-stonith-vcenter-host1 stonith-vcenter-host1 500: cluster-b-1
location loc2-stonith-vcenter-host1 stonith-vcenter-host1 501: cluster-b-2
# stonith vcenter definitions for host 2
# run in any of the host1 VM
primitive stonith-vcenter-host2 stonith:external/vcenter \
params \
VI_SERVER="192.168.40.21" \
VI_CREDSTORE="/etc/vicredentials.xml" \
HOSTLIST="cluster-b-1=cluster-b-1;cluster-b-2=cluster-b-2" \
RESETPOWERON="1" \
priority="2" \
pcmk_host_check="static-list" \
pcmk_host_list="cluster-b-1 cluster-b-2" \
op monitor interval="60s"
location loc1-stonith-vcenter-host2 stonith-vcenter-host2 500: cluster-a-1
location loc2-stonith-vcenter-host2 stonith-vcenter-host2 501: cluster-a-2
# stonith IPMI definitions for host 1 (DELL with iDRAC 7 enterprise
interface at 192.168.40.15)
# run in any of the host2 VM
primitive stonith-ipmi-host1 stonith:external/ipmi \
params hostname="host1" ipaddr="192.168.40.15" userid="root"
passwd="mypassword" interface="lanplus" \
priority="1" \
pcmk_host_check="static-list" \
pcmk_host_list="cluster-a-1 cluster-a-2" \
op start interval="0" timeout="60s" requires="nothing" \
op monitor interval="3600s" timeout="20s" requires="nothing"
location loc1-stonith-ipmi-host1 stonith-ipmi-host1 500: cluster-b-1
location loc2-stonith-ipmi-host1 stonith-ipmi-host1 501: cluster-b-2
# stonith IPMI definitions for host 2 (DELL with iDRAC 7 enterprise
interface at 192.168.40.16)
# run in any of the host1 VM
primitive stonith-ipmi-host2 stonith:external/ipmi \
params hostname="host2" ipaddr="192.168.40.16" userid="root"
passwd="mypassword" interface="lanplus" \
priority="1" \
pcmk_host_check="static-list" \
pcmk_host_list="cluster-b-1 cluster-b-2" \
op start interval="0" timeout="60s" requires="nothing" \
op monitor interval="3600s" timeout="20s" requires="nothing"
location loc1-stonith-ipmi-host2 stonith-ipmi-host2 500: cluster-a-1
location loc2-stonith-ipmi-host2 stonith-ipmi-host2 501: cluster-a-2
- When an error is detected in one resource, the resource restart in
the same node, as expected.
- With the STONITH external/ipmi resource *stopped*, a fail in one
node makes the vcenter rebooting it and the resources starts in the
sister node.
- When vcenter reboots one node, then the resource starts in the
other node as expected but then they return to the original node as
soon as it becomes online. This makes a bit of ping-pong and I think
it is a consequence of how the locations are defined. Any suggestion
to avoid this? After the resource was moved to another node, I would
prefer that it stays there, instead of returning it to the original
node. I can think of playing with the resource affinity scores - is
this way it should be done?
Lets consider this scenario.
I have a set of resources provided by a docker agent. My test
consists in stopping the docker service in the node cluster-a-1,
which makes the docker agent to return OCF_ERR_INSTALLED to
Pacemaker (this is a change I made in the docker agent, when
compared to the github repository version). With the IPMI STONITH
resource stopped, this leads to the node cluster-a-1 restart, which
is expected.
- Some times, the resources at the node cluster-a-1 are stopped and
no STONITH happens. Also, the resources are not moved to the node
cluster-a-2. In this situation, if I manually restart the node
cluster-a-1 (virtual machine restart), then the IPMI STONITH takes
place and restarts the corresponding physical server.
- Sometimes, the IPMI STONITH starts before the vCenter STONITH,
which is not expected because the vCenter STONITH has higher priority.
I might have something wrong in my stonith definition, but I can't
figure what.
Any idea how to correct this?
And how can I set external/ipmi to power off the physical host,
instead of rebooting it?
_______________________________________________
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

--
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?

_______________________________________________
Pacemaker mailing list: ***@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[Jorge Lopes]

Any thoughts on this would be much appreciated :)

Post by Jorge Lopes
(I'm a bit confused because I received an auto-reply form
but I just received a digest with my mail. I happens that I have resent the
email to the new list with a bit more information, which was missing in the
first message. So here it is that extra bit, anyway).
1. With the cluster running without errors, I run "stop docker" in node
cluster-a-1.
2. This leads to the vCenter STONITH to act as expected.
3. After the cluster is running again without errors, I run again "stop
docker" in node cluster-a-1.
4. Now, the vCenter STONITH doesn't run and, instead, it is the IPMI
STONITH that runs. This is unexpected for me, as I was expecting to see the
vCenter STONITH to run again.

Post by Jorge Lopes
Hi all.
I'm having difficulties orchestrating two STONITH devices in my cluster.
I have been struggling with this in past days and I need some help, please.
- The cluster has two physical servers, each with two nodes (VMWare
virtual machines): overall, there are 4 nodes in this simplified version.
- There are two resource groups: group-cluster-a and group-cluster-b.
- To achieve a good CPU balance in the physical servers, the cluster is
asymmetric, with one group running in one server and the other group
running on the other server.
- If the VM of one host becomes not usable, then its resources are
started in its sister VM deployed in the other physical host.
- If one physical host becomes not usable, then all resources are started
in the other physical host.
- Two STONITH levels are used to fence the problematic nodes.
- If the resource monitor detects a problem, then Pacemaker tries to
restart the resource in the same node.
- If it fails, then STONITH takes place (vcenter reboots the VM) and
Pacemaker starts the resource in the sister VM present in the other
physical host.
- If restarting the VM fails, I want to power off the physical server and
Pacemaker will start all resources in the other physical host.
Ubuntu 14.04 (the node OS, which is a visualized guest running in VMWare
ESXi 5.5)
Pacemaker 1.1.12
Corosync 2.3.4
CRM 2.1.2
cluster-a-1
cluster-a-2
cluster-b-1
cluster-b-2
property symmetric-cluster=false
property stonith-enabled=true
property no-quorum-policy=stop
group group-cluster-a vip-cluster-a docker-web
location loc-group-cluster-a-1 group-cluster-a inf: cluster-a-1
location loc-group-cluster-a-2 group-cluster-a 500: cluster-a-2
group group-cluster-b vip-cluster-b docker-srv
location loc-group-cluster-b-1 group-cluster-b 500: cluster-b-1
location loc-group-cluster-b-2 group-cluster-b inf: cluster-b-2
# stonith vcenter definitions for host 1
# run in any of the host2 VM
primitive stonith-vcenter-host1 stonith:external/vcenter \
params \
VI_SERVER="192.168.40.20" \
VI_CREDSTORE="/etc/vicredentials.xml" \
HOSTLIST="cluster-a-1=cluster-a-1;cluster-a-2=cluster-a-2" \
RESETPOWERON="1" \
priority="2" \
pcmk_host_check="static-list" \
pcmk_host_list="cluster-a-1 cluster-a-2" \
op monitor interval="60s"
location loc1-stonith-vcenter-host1 stonith-vcenter-host1 500: cluster-b-1
location loc2-stonith-vcenter-host1 stonith-vcenter-host1 501: cluster-b-2
# stonith vcenter definitions for host 2
# run in any of the host1 VM
primitive stonith-vcenter-host2 stonith:external/vcenter \
params \
VI_SERVER="192.168.40.21" \
VI_CREDSTORE="/etc/vicredentials.xml" \
HOSTLIST="cluster-b-1=cluster-b-1;cluster-b-2=cluster-b-2" \
RESETPOWERON="1" \
priority="2" \
pcmk_host_check="static-list" \
pcmk_host_list="cluster-b-1 cluster-b-2" \
op monitor interval="60s"
location loc1-stonith-vcenter-host2 stonith-vcenter-host2 500: cluster-a-1
location loc2-stonith-vcenter-host2 stonith-vcenter-host2 501: cluster-a-2
# stonith IPMI definitions for host 1 (DELL with iDRAC 7 enterprise
interface at 192.168.40.15)
# run in any of the host2 VM
primitive stonith-ipmi-host1 stonith:external/ipmi \
params hostname="host1" ipaddr="192.168.40.15" userid="root"
passwd="mypassword" interface="lanplus" \
priority="1" \
pcmk_host_check="static-list" \
pcmk_host_list="cluster-a-1 cluster-a-2" \
op start interval="0" timeout="60s" requires="nothing" \
op monitor interval="3600s" timeout="20s" requires="nothing"
location loc1-stonith-ipmi-host1 stonith-ipmi-host1 500: cluster-b-1
location loc2-stonith-ipmi-host1 stonith-ipmi-host1 501: cluster-b-2
# stonith IPMI definitions for host 2 (DELL with iDRAC 7 enterprise
interface at 192.168.40.16)
# run in any of the host1 VM
primitive stonith-ipmi-host2 stonith:external/ipmi \
params hostname="host2" ipaddr="192.168.40.16" userid="root"
passwd="mypassword" interface="lanplus" \
priority="1" \
pcmk_host_check="static-list" \
pcmk_host_list="cluster-b-1 cluster-b-2" \
op start interval="0" timeout="60s" requires="nothing" \
op monitor interval="3600s" timeout="20s" requires="nothing"
location loc1-stonith-ipmi-host2 stonith-ipmi-host2 500: cluster-a-1
location loc2-stonith-ipmi-host2 stonith-ipmi-host2 501: cluster-a-2
- When an error is detected in one resource, the resource restart in the
same node, as expected.
- With the STONITH external/ipmi resource *stopped*, a fail in one node
makes the vcenter rebooting it and the resources starts in the sister node.
- When vcenter reboots one node, then the resource starts in the other
node as expected but then they return to the original node as soon as it
becomes online. This makes a bit of ping-pong and I think it is a
consequence of how the locations are defined. Any suggestion to avoid this?
After the resource was moved to another node, I would prefer that it stays
there, instead of returning it to the original node. I can think of playing
with the resource affinity scores - is this way it should be done?
Lets consider this scenario.
I have a set of resources provided by a docker agent. My test consists in
stopping the docker service in the node cluster-a-1, which makes the docker
agent to return OCF_ERR_INSTALLED to Pacemaker (this is a change I made in
the docker agent, when compared to the github repository version). With the
IPMI STONITH resource stopped, this leads to the node cluster-a-1 restart,
which is expected.
- Some times, the resources at the node cluster-a-1 are stopped and no
STONITH happens. Also, the resources are not moved to the node cluster-a-2.
In this situation, if I manually restart the node cluster-a-1 (virtual
machine restart), then the IPMI STONITH takes place and restarts the
corresponding physical server.
- Sometimes, the IPMI STONITH starts before the vCenter STONITH, which is
not expected because the vCenter STONITH has higher priority.
I might have something wrong in my stonith definition, but I can't figure
what.
Any idea how to correct this?
And how can I set external/ipmi to power off the physical host, instead
of rebooting it?