Discussion:
[Pacemaker] One more globally-unique clone question
Vladislav Bogdanov
2015-01-16 14:25:47 UTC
Permalink
Hi all,

Trying to reproduce problem with early stop of globally-unique clone
instances during move to another node I found one more "interesting"
problem.

Due to the different order of resources in the CIB and extensive use of
constraints between other resources (odd number of resources
cluster-wide) two CLUSTERIP instances are always allocated to the same
node in the new testing cluster.

What would be the best/preferred way to make them run on different nodes
by default?

I see following options:
* Raise priority of globally-unique clone so its instances are always
allocated first of all.
* Use utilization attributes (with high values for nodes and low values
for cluster resources).
* Anything else?

If I configure virtual IPs one-by-one (without clone), I can add a
colocation constraint with negative score between them. I do not see a
way to scale that setup well though (5-10 IPs).
So, what would be the best option to achieve the same with
globally-unique cloned resource?
May be there should be some internal preference/colocation not to place
them together (like default stickiness=1 for clones)?
Or even allow special negative colocation constraint and the same
resource in both 'what' and 'with'
(colocation col1 -1: clone clone)?

Best,
Vladislav


_______________________________________________
Pacemaker mailing list: ***@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
Andrew Beekhof
2015-01-19 23:47:14 UTC
Permalink
Post by Vladislav Bogdanov
Hi all,
Trying to reproduce problem with early stop of globally-unique clone instances during move to another node I found one more "interesting" problem.
Due to the different order of resources in the CIB and extensive use of constraints between other resources (odd number of resources cluster-wide) two CLUSTERIP instances are always allocated to the same node in the new testing cluster.
Ah, so this is why broker-vips:1 was moving.
Post by Vladislav Bogdanov
What would be the best/preferred way to make them run on different nodes by default?
By default they will.
I'm assuming its the constraints that are preventing this.

Getting them to auto-rebalance is the harder problem
Post by Vladislav Bogdanov
* Raise priority of globally-unique clone so its instances are always allocated first of all.
* Use utilization attributes (with high values for nodes and low values for cluster resources).
* Anything else?
If I configure virtual IPs one-by-one (without clone), I can add a colocation constraint with negative score between them. I do not see a way to scale that setup well though (5-10 IPs).
So, what would be the best option to achieve the same with globally-unique cloned resource?
May be there should be some internal preference/colocation not to place them together (like default stickiness=1 for clones)?
Or even allow special negative colocation constraint and the same resource in both 'what' and 'with'
(colocation col1 -1: clone clone)?
Best,
Vladislav
_______________________________________________
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Pacemaker mailing list: ***@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
Vladislav Bogdanov
2015-01-20 05:13:18 UTC
Permalink
Post by Andrew Beekhof
On 17 Jan 2015, at 1:25 am, Vladislav Bogdanov
Hi all,
Trying to reproduce problem with early stop of globally-unique
clone instances during move to another node I found one more
"interesting" problem.
Due to the different order of resources in the CIB and extensive
use of constraints between other resources (odd number of resources
cluster-wide) two CLUSTERIP instances are always allocated to the
same node in the new testing cluster.
Ah, so this is why broker-vips:1 was moving.
That are two different 2-node clusters with different order of resources.
In the first one broker-vips go after even number of resources, and one
instance wants to return to a "mother-node" after it is brought back
online, thus broker-vips:1 is moving.

In the second one, broker-vips go after odd number of resources
(actually three more resources are allocated to one node due to
constraints) and both boker-vips go to another node.
Post by Andrew Beekhof
What would be the best/preferred way to make them run on different nodes by default?
By default they will. I'm assuming its the constraints that are
preventing this.
I only see that they are allocated similar to any other resources.
Post by Andrew Beekhof
Getting them to auto-rebalance is the harder problem
I see. Should it be possible to solve it without priority or utilization
use?
Post by Andrew Beekhof
* Raise priority of globally-unique clone so its instances are
always allocated first of all.
* Use utilization attributes (with high values for nodes and low values
for cluster resources).
* Anything else?
If I configure virtual IPs one-by-one (without clone), I can add a
colocation constraint with negative score between them. I do not
see a way to scale that setup well though (5-10 IPs). So, what
would be the best option to achieve the same with globally-unique
cloned resource? May be there should be some internal
preference/colocation not to place them together (like default
stickiness=1 for clones)? Or even allow special negative colocation
constraint and the same resource in both 'what' and 'with'
(colocation col1 -1: clone clone)?
Best, Vladislav
_______________________________________________ Pacemaker mailing
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
http://bugs.clusterlabs.org
_______________________________________________ Pacemaker mailing
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
http://bugs.clusterlabs.org
_______________________________________________
Pacemaker mailing list: ***@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
Andrew Beekhof
2015-01-21 00:51:40 UTC
Permalink
Post by Vladislav Bogdanov
Post by Andrew Beekhof
On 17 Jan 2015, at 1:25 am, Vladislav Bogdanov
Hi all,
Trying to reproduce problem with early stop of globally-unique
clone instances during move to another node I found one more
"interesting" problem.
Due to the different order of resources in the CIB and extensive
use of constraints between other resources (odd number of resources
cluster-wide) two CLUSTERIP instances are always allocated to the
same node in the new testing cluster.
Ah, so this is why broker-vips:1 was moving.
That are two different 2-node clusters with different order of resources.
In the first one broker-vips go after even number of resources, and one instance wants to return to a "mother-node" after it is brought back online, thus broker-vips:1 is moving.
In the second one, broker-vips go after odd number of resources (actually three more resources are allocated to one node due to constraints) and both boker-vips go to another node.
Post by Andrew Beekhof
What would be the best/preferred way to make them run on different nodes by default?
By default they will. I'm assuming its the constraints that are
preventing this.
I only see that they are allocated similar to any other resources.
Are they allocated in stages though?
Ie. Was there a point at which the "mother-node" was available but constraints prevented broker-vips:1 running there?
Post by Vladislav Bogdanov
Post by Andrew Beekhof
Getting them to auto-rebalance is the harder problem
I see. Should it be possible to solve it without priority or utilization use?
"it" meaning auto-rebalancing or your original issue?
Post by Vladislav Bogdanov
Post by Andrew Beekhof
* Raise priority of globally-unique clone so its instances are
always allocated first of all.
* Use utilization attributes (with high values for nodes and low values
for cluster resources).
* Anything else?
If I configure virtual IPs one-by-one (without clone), I can add a
colocation constraint with negative score between them. I do not
see a way to scale that setup well though (5-10 IPs). So, what
would be the best option to achieve the same with globally-unique
cloned resource? May be there should be some internal
preference/colocation not to place them together (like default
stickiness=1 for clones)? Or even allow special negative colocation
constraint and the same resource in both 'what' and 'with'
(colocation col1 -1: clone clone)?
Best, Vladislav
_______________________________________________ Pacemaker mailing
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
http://bugs.clusterlabs.org
_______________________________________________ Pacemaker mailing
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
http://bugs.clusterlabs.org
_______________________________________________
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Pacemaker mailing list: ***@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
Vladislav Bogdanov
2015-01-21 06:08:23 UTC
Permalink
Post by Andrew Beekhof
Post by Vladislav Bogdanov
Post by Andrew Beekhof
On 17 Jan 2015, at 1:25 am, Vladislav Bogdanov
Hi all,
Trying to reproduce problem with early stop of globally-unique
clone instances during move to another node I found one more
"interesting" problem.
Due to the different order of resources in the CIB and extensive
use of constraints between other resources (odd number of resources
cluster-wide) two CLUSTERIP instances are always allocated to the
same node in the new testing cluster.
Ah, so this is why broker-vips:1 was moving.
That are two different 2-node clusters with different order of resources.
In the first one broker-vips go after even number of resources, and one instance wants to return to a "mother-node" after it is brought back online, thus broker-vips:1 is moving.
In the second one, broker-vips go after odd number of resources (actually three more resources are allocated to one node due to constraints) and both boker-vips go to another node.
Post by Andrew Beekhof
What would be the best/preferred way to make them run on different nodes by default?
By default they will. I'm assuming its the constraints that are
preventing this.
I only see that they are allocated similar to any other resources.
Are they allocated in stages though?
Ie. Was there a point at which the "mother-node" was available but constraints prevented broker-vips:1 running there?
There are three pe-inputs for the node start.
First one starts fence device for the other node, dlm+clvm+gfs and drbd
on the online-back node.
Second one tries to start/promote/move everything else until it is
interrupted (by the drbd RA?).
Third one finishes that attempt.

And yes, CTDB depends on GFS2 filesystem, so broker-vips:1 can't be
allocated immediately due to constraints. It is allocated in the second
pe-input.

May be it is worth sending crm-report to you in order to not overload
list by long listings and you have complete information?
Post by Andrew Beekhof
Post by Vladislav Bogdanov
Post by Andrew Beekhof
Getting them to auto-rebalance is the harder problem
I see. Should it be possible to solve it without priority or utilization use?
"it" meaning auto-rebalancing or your original issue?
I meant auto-rebalancing.
Post by Andrew Beekhof
Post by Vladislav Bogdanov
Post by Andrew Beekhof
* Raise priority of globally-unique clone so its instances are
always allocated first of all.
* Use utilization attributes (with high values for nodes and low values
for cluster resources).
* Anything else?
If I configure virtual IPs one-by-one (without clone), I can add a
colocation constraint with negative score between them. I do not
see a way to scale that setup well though (5-10 IPs). So, what
would be the best option to achieve the same with globally-unique
cloned resource? May be there should be some internal
preference/colocation not to place them together (like default
stickiness=1 for clones)? Or even allow special negative colocation
constraint and the same resource in both 'what' and 'with'
(colocation col1 -1: clone clone)?
Best, Vladislav
_______________________________________________ Pacemaker mailing
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
http://bugs.clusterlabs.org
_______________________________________________ Pacemaker mailing
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
http://bugs.clusterlabs.org
_______________________________________________
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Pacemaker mailing list: ***@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
Andrew Beekhof
2015-02-23 22:58:38 UTC
Permalink
Post by Vladislav Bogdanov
Post by Andrew Beekhof
Post by Vladislav Bogdanov
Post by Andrew Beekhof
On 17 Jan 2015, at 1:25 am, Vladislav Bogdanov
Hi all,
Trying to reproduce problem with early stop of globally-unique
clone instances during move to another node I found one more
"interesting" problem.
Due to the different order of resources in the CIB and extensive
use of constraints between other resources (odd number of resources
cluster-wide) two CLUSTERIP instances are always allocated to the
same node in the new testing cluster.
Ah, so this is why broker-vips:1 was moving.
That are two different 2-node clusters with different order of resources.
In the first one broker-vips go after even number of resources, and one instance wants to return to a "mother-node" after it is brought back online, thus broker-vips:1 is moving.
In the second one, broker-vips go after odd number of resources (actually three more resources are allocated to one node due to constraints) and both boker-vips go to another node.
Post by Andrew Beekhof
What would be the best/preferred way to make them run on different
nodes by default?
By default they will. I'm assuming its the constraints that are
preventing this.
I only see that they are allocated similar to any other resources.
Are they allocated in stages though?
Ie. Was there a point at which the "mother-node" was available but constraints prevented broker-vips:1 running there?
There are three pe-inputs for the node start.
First one starts fence device for the other node, dlm+clvm+gfs and drbd on the online-back node.
Second one tries to start/promote/move everything else until it is interrupted (by the drbd RA?).
Third one finishes that attempt.
I've lost all context on this and I don't seem to be able to reconstruct it :)
Which part of the above is the problem?
Post by Vladislav Bogdanov
And yes, CTDB depends on GFS2 filesystem, so broker-vips:1 can't be allocated immediately due to constraints. It is allocated in the second pe-input.
May be it is worth sending crm-report to you in order to not overload list by long listings and you have complete information?
Post by Andrew Beekhof
Post by Vladislav Bogdanov
Post by Andrew Beekhof
Getting them to auto-rebalance is the harder problem
I see. Should it be possible to solve it without priority or utilization use?
"it" meaning auto-rebalancing or your original issue?
I meant auto-rebalancing.
It should be something we handle internally.
I've made a note of it.
Post by Vladislav Bogdanov
Post by Andrew Beekhof
Post by Vladislav Bogdanov
Post by Andrew Beekhof
* Raise priority of globally-unique clone so its instances are
always allocated first of all.
* Use utilization attributes (with high values for nodes and low values
for cluster resources).
* Anything else?
If I configure virtual IPs one-by-one (without clone), I can add a
colocation constraint with negative score between them. I do not
see a way to scale that setup well though (5-10 IPs). So, what
would be the best option to achieve the same with globally-unique
cloned resource? May be there should be some internal
preference/colocation not to place them together (like default
stickiness=1 for clones)? Or even allow special negative colocation
constraint and the same resource in both 'what' and 'with'
(colocation col1 -1: clone clone)?
Best, Vladislav
_______________________________________________ Pacemaker mailing
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
http://bugs.clusterlabs.org
_______________________________________________ Pacemaker mailing
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
http://bugs.clusterlabs.org
_______________________________________________
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Pacemaker mailing list: ***@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
Vladislav Bogdanov
2015-02-24 05:35:23 UTC
Permalink
Post by Andrew Beekhof
Post by Vladislav Bogdanov
Post by Andrew Beekhof
Post by Vladislav Bogdanov
Post by Andrew Beekhof
On 17 Jan 2015, at 1:25 am, Vladislav Bogdanov
Hi all,
Trying to reproduce problem with early stop of globally-unique
clone instances during move to another node I found one more
"interesting" problem.
Due to the different order of resources in the CIB and extensive
use of constraints between other resources (odd number of resources
cluster-wide) two CLUSTERIP instances are always allocated to the
same node in the new testing cluster.
Ah, so this is why broker-vips:1 was moving.
That are two different 2-node clusters with different order of resources.
In the first one broker-vips go after even number of resources, and one instance wants to return to a "mother-node" after it is brought back online, thus broker-vips:1 is moving.
In the second one, broker-vips go after odd number of resources (actually three more resources are allocated to one node due to constraints) and both boker-vips go to another node.
Post by Andrew Beekhof
What would be the best/preferred way to make them run on different
nodes by default?
By default they will. I'm assuming its the constraints that are
preventing this.
I only see that they are allocated similar to any other resources.
Are they allocated in stages though?
Ie. Was there a point at which the "mother-node" was available but constraints prevented broker-vips:1 running there?
There are three pe-inputs for the node start.
First one starts fence device for the other node, dlm+clvm+gfs and drbd on the online-back node.
Second one tries to start/promote/move everything else until it is interrupted (by the drbd RA?).
Third one finishes that attempt.
I've lost all context on this and I don't seem to be able to reconstruct it :)
Which part of the above is the problem?
;)

In this thread the point is:
* all resources have the same default priority
* there are several triples of resources which are grouped by
order/colocation constraints. Let's call them "triples".
* There is globally-unique cluster-ip clone with clone-max=2
clone-node-max=2 stickiness=0, which is allocated after all "triples"
(it goes after them in CIB).
* If number of "triples" is odd, then in two-node cluster both
cluster-ip instances are allocated to the same node.
Post by Andrew Beekhof
Post by Vladislav Bogdanov
And yes, CTDB depends on GFS2 filesystem, so broker-vips:1 can't be allocated immediately due to constraints. It is allocated in the second pe-input.
May be it is worth sending crm-report to you in order to not overload list by long listings and you have complete information?
Post by Andrew Beekhof
Post by Vladislav Bogdanov
Post by Andrew Beekhof
Getting them to auto-rebalance is the harder problem
I see. Should it be possible to solve it without priority or utilization use?
"it" meaning auto-rebalancing or your original issue?
I meant auto-rebalancing.
It should be something we handle internally.
I've made a note of it.
Post by Vladislav Bogdanov
Post by Andrew Beekhof
Post by Vladislav Bogdanov
Post by Andrew Beekhof
* Raise priority of globally-unique clone so its instances are
always allocated first of all.
* Use utilization attributes (with high values for nodes and low values
for cluster resources).
* Anything else?
If I configure virtual IPs one-by-one (without clone), I can add a
colocation constraint with negative score between them. I do not
see a way to scale that setup well though (5-10 IPs). So, what
would be the best option to achieve the same with globally-unique
cloned resource? May be there should be some internal
preference/colocation not to place them together (like default
stickiness=1 for clones)? Or even allow special negative colocation
constraint and the same resource in both 'what' and 'with'
(colocation col1 -1: clone clone)?
Best, Vladislav
_______________________________________________ Pacemaker mailing
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
http://bugs.clusterlabs.org
_______________________________________________ Pacemaker mailing
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
http://bugs.clusterlabs.org
_______________________________________________
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Pacemaker mailing list: ***@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
Andrew Beekhof
2015-02-26 02:51:31 UTC
Permalink
Post by Vladislav Bogdanov
Post by Andrew Beekhof
Post by Vladislav Bogdanov
Post by Andrew Beekhof
Post by Vladislav Bogdanov
Post by Andrew Beekhof
On 17 Jan 2015, at 1:25 am, Vladislav Bogdanov
Hi all,
Trying to reproduce problem with early stop of globally-unique
clone instances during move to another node I found one more
"interesting" problem.
Due to the different order of resources in the CIB and extensive
use of constraints between other resources (odd number of resources
cluster-wide) two CLUSTERIP instances are always allocated to the
same node in the new testing cluster.
Ah, so this is why broker-vips:1 was moving.
That are two different 2-node clusters with different order of resources.
In the first one broker-vips go after even number of resources, and one instance wants to return to a "mother-node" after it is brought back online, thus broker-vips:1 is moving.
In the second one, broker-vips go after odd number of resources (actually three more resources are allocated to one node due to constraints) and both boker-vips go to another node.
Post by Andrew Beekhof
What would be the best/preferred way to make them run on different
nodes by default?
By default they will. I'm assuming its the constraints that are
preventing this.
I only see that they are allocated similar to any other resources.
Are they allocated in stages though?
Ie. Was there a point at which the "mother-node" was available but constraints prevented broker-vips:1 running there?
There are three pe-inputs for the node start.
First one starts fence device for the other node, dlm+clvm+gfs and drbd on the online-back node.
Second one tries to start/promote/move everything else until it is interrupted (by the drbd RA?).
Third one finishes that attempt.
I've lost all context on this and I don't seem to be able to reconstruct it :)
Which part of the above is the problem?
;)
* all resources have the same default priority
* there are several triples of resources which are grouped by order/colocation constraints. Let's call them "triples".
* There is globally-unique cluster-ip clone with clone-max=2 clone-node-max=2 stickiness=0, which is allocated after all "triples" (it goes after them in CIB).
* If number of "triples" is odd, then in two-node cluster both cluster-ip instances are allocated to the same node.
Very clear, thankyou :-)
Was there a crm_report associated with this somewhere? Its not showing up in this thread.
_______________________________________________
Pacemaker mailing list: ***@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Loading...