[Pacemaker] Call cib_apply_diff failed (-205): Update was older than existing configuration

[Kostiantyn Ponomarenko]

Hi guys,

I saw this during applying the configuration using a script with crmsh
commands:

+ crm configure primitive STONITH_node-1 stonith:fence_avid_sbb_hw
+ crm configure primitive STONITH_node-0 stonith:fence_avid_sbb_hw params
delay=10
+ crm configure location dont_run_STONITH_node-1_on_node-1 STONITH_node-1
-inf: node-1
+ crm configure location dont_run_STONITH_node-0_on_node-0 STONITH_node-0
-inf: node-0
Call cib_apply_diff failed (-205): Update was older than existing
configuration
ERROR: could not patch cib (rc=205)
INFO: offending xml diff: <diff format="2">
<version>
<source admin_epoch="0" epoch="26" num_updates="3"/>
<target admin_epoch="0" epoch="27" num_updates="3"/>
</version>
<change operation="modify" path="/cib">
<change-list>
<change-attr name="epoch" operation="set" value="27"/>
</change-list>
<change-result>
<cib crm_feature_set="3.0.9" validate-with="pacemaker-2.0" epoch="27"
num_updates="3" admin_epoch="0" cib-last-written="Thu Feb 5 14:56:09 2015"
have-quorum="1" dc-uuid="1"/>
</change-result>
</change>
<change operation="create" path="/cib/configuration/constraints"
position="1">
<rsc_location id="dont_run_STONITH_node-0_on_node-0"
rsc="STONITH_node-0" score="-INFINITY" node="node-0"/>
</change>
</diff>

After that pacemaker stopped on the node on which the script was run.

Thank you,
Kostya

[Kostiantyn Ponomarenko]

Here is a part of my /var/log/cluster/corosync.log:


Feb 05 16:23:36 [3184] isis-seth943f cib: info:
cib_process_ping: Reporting our current digest to node-1:
71d36e21c5df77a4e8b10b29f2a349e2 for 0.70.16 (0x2474c50 0)

Feb 05 16:23:41 [3185] isis-seth943f stonithd: notice: remote_op_done:
Operation reboot of node-0 by node-1 for ***@node-1.58027587: OK

Feb 05 16:23:41 [3189] isis-seth943f crmd: crit:
tengine_stonith_notify: We were alegedly just fenced by node-1 for
node-1!

Feb 05 16:23:41 [3184] isis-seth943f cib: info: cib_perform_op:
Diff: --- 0.70.16 2

Feb 05 16:23:41 [3184] isis-seth943f cib: info: cib_perform_op:
Diff: +++ 0.70.17 (null)

Feb 05 16:23:41 [3184] isis-seth943f cib: info: cib_perform_op:
+ /cib: @num_updates=17

Feb 05 16:23:41 [3184] isis-seth943f cib: info: cib_perform_op:
+ /cib/status/node_state[@id='1']:
@crm-debug-origin=send_stonith_update, @join=down, @expected=down

Feb 05 16:23:41 [3184] isis-seth943f cib: info:
cib_process_request: Completed cib_modify operation for section
status: OK (rc=0, origin=node-1/crmd/91, version=0.70.17)

Feb 05 16:23:41 [3186] isis-seth943f pacemaker_remoted: info:
cancel_recurring_action: Cancelling operation dmdh_monitor_30000

Feb 05 16:23:41 [3182] isis-seth943f pacemakerd: error: pcmk_child_exit:
Child process crmd (3189) exited: Network is down (100)

Feb 05 16:23:41 [3182] isis-seth943f pacemakerd: warning: pcmk_child_exit:
Pacemaker child process crmd no longer wishes to be respawned. Shutting
ourselves down.

Feb 05 16:23:41 [3186] isis-seth943f pacemaker_remoted: warning:
qb_ipcs_event_sendv: new_event_notification (3186-3189-7): Bad file
descriptor (9)

Feb 05 16:23:41 [3186] isis-seth943f pacemaker_remoted: warning:
send_client_notify: Notification of client
crmd/437207d3-a709-48d0-b544-94e2158ea191 failed

Feb 05 16:23:41 [3186] isis-seth943f pacemaker_remoted: info:
cancel_recurring_action: Cancelling operation sm0dh_monitor_30000

Feb 05 16:23:41 [3186] isis-seth943f pacemaker_remoted: warning:
send_client_notify: Notification of client
crmd/437207d3-a709-48d0-b544-94e2158ea191 failed

Feb 05 16:23:41 [3184] isis-seth943f cib: info: cib_perform_op:
Diff: --- 0.70.17 2

Feb 05 16:23:41 [3184] isis-seth943f cib: info: cib_perform_op:
Diff: +++ 0.70.18 (null)

Feb 05 16:23:41 [3184] isis-seth943f cib: info: cib_perform_op:
-- /cib/status/node_state[@id='1']/lrm[@id='1']

Feb 05 16:23:41 [3184] isis-seth943f cib: info: cib_perform_op:
+ /cib: @num_updates=18

Feb 05 16:23:41 [3184] isis-seth943f cib: info:
cib_process_request: Completed cib_delete operation for section
//node_state[@uname='node-0']/lrm: OK (rc=0, origin=node-1/crmd/92,
version=0.70.18)
Feb 05 16:23:41 [3182] isis-seth943f pacemakerd: notice:
pcmk_shutdown_worker: Shuting down Pacemaker

Thank you,
Kostya

On Thu, Feb 5, 2015 at 6:15 PM, Kostiantyn Ponomarenko <

Post by Kostiantyn Ponomarenko
Hi guys,
I saw this during applying the configuration using a script with crmsh
+ crm configure primitive STONITH_node-1 stonith:fence_avid_sbb_hw
+ crm configure primitive STONITH_node-0 stonith:fence_avid_sbb_hw params
delay=10
+ crm configure location dont_run_STONITH_node-1_on_node-1 STONITH_node-1
-inf: node-1
+ crm configure location dont_run_STONITH_node-0_on_node-0 STONITH_node-0
-inf: node-0
Call cib_apply_diff failed (-205): Update was older than existing
configuration
ERROR: could not patch cib (rc=205)
INFO: offending xml diff: <diff format="2">
<version>
<source admin_epoch="0" epoch="26" num_updates="3"/>
<target admin_epoch="0" epoch="27" num_updates="3"/>
</version>
<change operation="modify" path="/cib">
<change-list>
<change-attr name="epoch" operation="set" value="27"/>
</change-list>
<change-result>
<cib crm_feature_set="3.0.9" validate-with="pacemaker-2.0"
epoch="27" num_updates="3" admin_epoch="0" cib-last-written="Thu Feb 5
14:56:09 2015" have-quorum="1" dc-uuid="1"/>
</change-result>
</change>
<change operation="create" path="/cib/configuration/constraints"
position="1">
<rsc_location id="dont_run_STONITH_node-0_on_node-0"
rsc="STONITH_node-0" score="-INFINITY" node="node-0"/>
</change>
</diff>
After that pacemaker stopped on the node on which the script was run.
Thank you,
Kostya

[Kristoffer Grönlund]

Hi,

Post by Kostiantyn Ponomarenko
Hi guys,
I saw this during applying the configuration using a script with crmsh

The CIB patching performed by crmsh has been a bit too sensitive to
CIB version mismatches which can cause errors like the one you are
seeing. This should be fixed in the latest released version of crmsh
(2.1.2), and I would recommend upgrading to that version if you can.

If this problem still occurs with 2.1.2, please let me know [1].

Thanks!

[1]: http://github.com/crmsh/crmsh/issues

Post by Kostiantyn Ponomarenko
+ crm configure primitive STONITH_node-1 stonith:fence_avid_sbb_hw
+ crm configure primitive STONITH_node-0 stonith:fence_avid_sbb_hw params
delay=10
+ crm configure location dont_run_STONITH_node-1_on_node-1 STONITH_node-1
-inf: node-1
+ crm configure location dont_run_STONITH_node-0_on_node-0 STONITH_node-0
-inf: node-0
Call cib_apply_diff failed (-205): Update was older than existing
configuration
ERROR: could not patch cib (rc=205)
INFO: offending xml diff: <diff format="2">
<version>
<source admin_epoch="0" epoch="26" num_updates="3"/>
<target admin_epoch="0" epoch="27" num_updates="3"/>
</version>
<change operation="modify" path="/cib">
<change-list>
<change-attr name="epoch" operation="set" value="27"/>
</change-list>
<change-result>
<cib crm_feature_set="3.0.9" validate-with="pacemaker-2.0" epoch="27"
num_updates="3" admin_epoch="0" cib-last-written="Thu Feb 5 14:56:09 2015"
have-quorum="1" dc-uuid="1"/>
</change-result>
</change>
<change operation="create" path="/cib/configuration/constraints"
position="1">
<rsc_location id="dont_run_STONITH_node-0_on_node-0"
rsc="STONITH_node-0" score="-INFINITY" node="node-0"/>
</change>
</diff>
After that pacemaker stopped on the node on which the script was run.
Thank you,
Kostya
_______________________________________________
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

--
// Kristoffer Grönlund
// ***@suse.com

_______________________________________________
Pacemaker mailing list: ***@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_fro

[Kostiantyn Ponomarenko]

Hi Kristoffer,

Thank you for the help.
I will let you know if I see the same with the latest version.

Post by Kristoffer Grönlund
Hi,

Post by Kostiantyn Ponomarenko
Hi guys,
I saw this during applying the configuration using a script with crmsh

The CIB patching performed by crmsh has been a bit too sensitive to
CIB version mismatches which can cause errors like the one you are
seeing. This should be fixed in the latest released version of crmsh
(2.1.2), and I would recommend upgrading to that version if you can.
If this problem still occurs with 2.1.2, please let me know [1].
Thanks!
[1]: http://github.com/crmsh/crmsh/issues

Post by Kostiantyn Ponomarenko
+ crm configure primitive STONITH_node-1 stonith:fence_avid_sbb_hw
+ crm configure primitive STONITH_node-0 stonith:fence_avid_sbb_hw params
delay=10
+ crm configure location dont_run_STONITH_node-1_on_node-1 STONITH_node-1
-inf: node-1
+ crm configure location dont_run_STONITH_node-0_on_node-0 STONITH_node-0
-inf: node-0
Call cib_apply_diff failed (-205): Update was older than existing
configuration
ERROR: could not patch cib (rc=205)
INFO: offending xml diff: <diff format="2">
<version>
<source admin_epoch="0" epoch="26" num_updates="3"/>
<target admin_epoch="0" epoch="27" num_updates="3"/>
</version>
<change operation="modify" path="/cib">
<change-list>
<change-attr name="epoch" operation="set" value="27"/>
</change-list>
<change-result>
<cib crm_feature_set="3.0.9" validate-with="pacemaker-2.0"

epoch="27"

Post by Kostiantyn Ponomarenko
num_updates="3" admin_epoch="0" cib-last-written="Thu Feb 5 14:56:09

2015"

Post by Kostiantyn Ponomarenko
have-quorum="1" dc-uuid="1"/>
</change-result>
</change>
<change operation="create" path="/cib/configuration/constraints"
position="1">
<rsc_location id="dont_run_STONITH_node-0_on_node-0"
rsc="STONITH_node-0" score="-INFINITY" node="node-0"/>
</change>
</diff>
After that pacemaker stopped on the node on which the script was run.
Thank you,
Kostya
_______________________________________________
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

--
// Kristoffer Grönlund