[Pacemaker] Suggestions for managing HA of containers from within a Pacemaker container?

[Steven Dake (stdake)]

Hi,

I am working on Containerizing OpenStack in the Kolla project (http://launchpad.net/kolla). One of the key things we want to do over the next few months is add H/A support to our container tech. David Vossel had suggested using systemctl to monitor the containers themselves by running healthchecking scripts within the containers. That idea is sound.

There is another technology called “super-privileged containers”. Essentially it allows more host access for the container, allowing the treatment of Pacemaker as a container rather than a RPM or DEB file. I’d like corosync to run in a separate container. These containers will communicate using their normal mechanisms in a super-privileged mode. We will implement this in Kolla.

Where I am stuck is how does Pacemaker within a container control other containers in the host os. One way I have considered is using the docker —pid=host flag, allowing pacemaker to communicate directly with the host systemctl process. Where I am stuck is our containers don’t run via systemctl, but instead via shell scripts that are executed by third party deployment software.

An example:
Lets say a rabbitmq container wants to run:

The user would run
kolla-mgr deploy messaging

This would run a small bit of code to launch the docker container set for messaging.

Could pacemaker run something like

Kolla-mgr status messaging

To control the lifecycle of the processes?

Or would we be better off with some systemd integration with kolla-mgr?

Thoughts welcome

Regards,
-steve

[Serge Dubrouski]

Hello Steve,

Are you sure that Pacemaker is the right product for your project? Have you
checked Mesos/Marathon or Kubernates? Those are frameworks being developed
for managing containers.

Post by Steven Dake (stdake)
Hi,
I am working on Containerizing OpenStack in the Kolla project (
http://launchpad.net/kolla). One of the key things we want to do over
the next few months is add H/A support to our container tech. David Vossel
had suggested using systemctl to monitor the containers themselves by
running healthchecking scripts within the containers. That idea is sound.
There is another technology called “super-privileged containers”.
Essentially it allows more host access for the container, allowing the
treatment of Pacemaker as a container rather than a RPM or DEB file. I’d
like corosync to run in a separate container. These containers will
communicate using their normal mechanisms in a super-privileged mode. We
will implement this in Kolla.
Where I am stuck is how does Pacemaker within a container control other
containers in the host os. One way I have considered is using the docker
—pid=host flag, allowing pacemaker to communicate directly with the host
systemctl process. Where I am stuck is our containers don’t run via
systemctl, but instead via shell scripts that are executed by third party
deployment software.
The user would run
kolla-mgr deploy messaging
This would run a small bit of code to launch the docker container set for messaging.
Could pacemaker run something like
Kolla-mgr status messaging
To control the lifecycle of the processes?
Or would we be better off with some systemd integration with kolla-mgr?
Thoughts welcome
Regards,
-steve
_______________________________________________
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[Andrew Beekhof]

Post by Serge Dubrouski
Hello Steve,
Are you sure that Pacemaker is the right product for your project? Have you checked Mesos/Marathon or Kubernates? Those are frameworks being developed for managing containers.

And in a few years they'll work out that they need some HA features and try to retrofit them :-)
In the meantime pacemaker is actually rather good at managing containers already and knows a thing or two about HA and how to bring up a complex stack of services.

The one thing that would be really interesting in this area is using the our policy engine as the kubernates scheduler.
So many ideas and so little time.

Post by Serge Dubrouski
Hi,
I am working on Containerizing OpenStack in the Kolla project (http://launchpad.net/kolla). One of the key things we want to do over the next few months is add H/A support to our container tech. David Vossel had suggested using systemctl to monitor the containers themselves by running healthchecking scripts within the containers. That idea is sound.
There is another technology called “super-privileged containers”. Essentially it allows more host access for the container, allowing the treatment of Pacemaker as a container rather than a RPM or DEB file. I’d like corosync to run in a separate container. These containers will communicate using their normal mechanisms in a super-privileged mode. We will implement this in Kolla.
Where I am stuck is how does Pacemaker within a container control other containers in the host os. One way I have considered is using the docker —pid=host flag, allowing pacemaker to communicate directly with the host systemctl process. Where I am stuck is our containers don’t run via systemctl, but instead via shell scripts that are executed by third party deployment software.
The user would run
kolla-mgr deploy messaging
This would run a small bit of code to launch the docker container set for messaging.
Could pacemaker run something like
Kolla-mgr status messaging
To control the lifecycle of the processes?
Or would we be better off with some systemd integration with kolla-mgr?
Thoughts welcome
Regards,
-steve
_______________________________________________
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

_______________________________________________
Pacemaker mailing list: ***@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bu

[Serge Dubrouski]

Pacemaker as a scheduler in Mesos or Kubernates does sound like a very
interesting idea. Packaging corosync into super privileged containers still
doesn't make too much sense to me. What's the reason in isolating something
and then giving it all permissions on a host machine?

Post by Serge Dubrouski

Post by Serge Dubrouski
Hello Steve,
Are you sure that Pacemaker is the right product for your project? Have

you checked Mesos/Marathon or Kubernates? Those are frameworks being
developed for managing containers.
And in a few years they'll work out that they need some HA features and
try to retrofit them :-)
In the meantime pacemaker is actually rather good at managing containers
already and knows a thing or two about HA and how to bring up a complex
stack of services.
The one thing that would be really interesting in this area is using the
our policy engine as the kubernates scheduler.
So many ideas and so little time.

Post by Serge Dubrouski
Hi,
I am working on Containerizing OpenStack in the Kolla project (

http://launchpad.net/kolla). One of the key things we want to do over
the next few months is add H/A support to our container tech. David Vossel
had suggested using systemctl to monitor the containers themselves by
running healthchecking scripts within the containers. That idea is sound.

Post by Serge Dubrouski
There is another technology called “super-privileged containers”.

Essentially it allows more host access for the container, allowing the
treatment of Pacemaker as a container rather than a RPM or DEB file. I’d
like corosync to run in a separate container. These containers will
communicate using their normal mechanisms in a super-privileged mode. We
will implement this in Kolla.

Post by Serge Dubrouski
Where I am stuck is how does Pacemaker within a container control other

containers in the host os. One way I have considered is using the docker
—pid=host flag, allowing pacemaker to communicate directly with the host
systemctl process. Where I am stuck is our containers don’t run via
systemctl, but instead via shell scripts that are executed by third party
deployment software.

Post by Serge Dubrouski
The user would run
kolla-mgr deploy messaging
This would run a small bit of code to launch the docker container set

for messaging.

Post by Serge Dubrouski
Could pacemaker run something like
Kolla-mgr status messaging
To control the lifecycle of the processes?
Or would we be better off with some systemd integration with kolla-mgr?
Thoughts welcome
Regards,
-steve
_______________________________________________
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

_______________________________________________
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[Andrew Beekhof]

Pacemaker as a scheduler in Mesos or Kubernates does sound like a very interesting idea.

Want to help make it work? :-)

Packaging corosync into super privileged containers still doesn't make too much sense to me. What's the reason in isolating something and then giving it all permissions on a host machine?

Probably because someone realised that they wanted to container-ize the software for creating containers and nesting them was too horrible to contemplate.

Post by Serge Dubrouski
Hello Steve,
Are you sure that Pacemaker is the right product for your project? Have you checked Mesos/Marathon or Kubernates? Those are frameworks being developed for managing containers.

And in a few years they'll work out that they need some HA features and try to retrofit them :-)
In the meantime pacemaker is actually rather good at managing containers already and knows a thing or two about HA and how to bring up a complex stack of services.
The one thing that would be really interesting in this area is using the our policy engine as the kubernates scheduler.
So many ideas and so little time.

Post by Serge Dubrouski
Hi,
I am working on Containerizing OpenStack in the Kolla project (http://launchpad.net/kolla). One of the key things we want to do over the next few months is add H/A support to our container tech. David Vossel had suggested using systemctl to monitor the containers themselves by running healthchecking scripts within the containers. That idea is sound.
There is another technology called “super-privileged containers”. Essentially it allows more host access for the container, allowing the treatment of Pacemaker as a container rather than a RPM or DEB file. I’d like corosync to run in a separate container. These containers will communicate using their normal mechanisms in a super-privileged mode. We will implement this in Kolla.
Where I am stuck is how does Pacemaker within a container control other containers in the host os. One way I have considered is using the docker —pid=host flag, allowing pacemaker to communicate directly with the host systemctl process. Where I am stuck is our containers don’t run via systemctl, but instead via shell scripts that are executed by third party deployment software.
The user would run
kolla-mgr deploy messaging
This would run a small bit of code to launch the docker container set for messaging.
Could pacemaker run something like
Kolla-mgr status messaging
To control the lifecycle of the processes?
Or would we be better off with some systemd integration with kolla-mgr?
Thoughts welcome
Regards,
-steve
_______________________________________________
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

_______________________________________________
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
--
Serge Dubrouski.
_______________________________________________
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

_______________________________________________
Pacemaker mailing list: ***@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cl

[David Vossel]

----- Original Message -----

Post by Serge Dubrouski
Pacemaker as a scheduler in Mesos or Kubernates does sound like a very
interesting idea. Packaging corosync into super privileged containers still
doesn't make too much sense to me. What's the reason in isolating something
and then giving it all permissions on a host machine?

because soon everything will run in containers. Take a look at rhel atomic and
the stuff coreos is doing. The only way pacemaker will exist on those distributions
is if it lives in a super privileged container.

Post by Serge Dubrouski

Post by Serge Dubrouski
Hello Steve,
Are you sure that Pacemaker is the right product for your project? Have you
checked Mesos/Marathon or Kubernates? Those are frameworks being developed
for managing containers.

And in a few years they'll work out that they need some HA features and try
to retrofit them :-)
In the meantime pacemaker is actually rather good at managing containers
already and knows a thing or two about HA and how to bring up a complex
stack of services.
The one thing that would be really interesting in this area is using the our
policy engine as the kubernates scheduler.
So many ideas and so little time.

Post by Serge Dubrouski
Hi,
I am working on Containerizing OpenStack in the Kolla project (
http://launchpad.net/kolla ). One of the key things we want to do over the
next few months is add H/A support to our container tech. David Vossel had
suggested using systemctl to monitor the containers themselves by running
healthchecking scripts within the containers. That idea is sound.
There is another technology called “super-privileged containers”.
Essentially it allows more host access for the container, allowing the
treatment of Pacemaker as a container rather than a RPM or DEB file. I’d
like corosync to run in a separate container. These containers will
communicate using their normal mechanisms in a super-privileged mode. We
will implement this in Kolla.
Where I am stuck is how does Pacemaker within a container control other
containers in the host os. One way I have considered is using the docker
—pid=host flag, allowing pacemaker to communicate directly with the host
systemctl process. Where I am stuck is our containers don’t run via
systemctl, but instead via shell scripts that are executed by third party
deployment software.
The user would run
kolla-mgr deploy messaging
This would run a small bit of code to launch the docker container set for messaging.
Could pacemaker run something like
Kolla-mgr status messaging
To control the lifecycle of the processes?
Or would we be better off with some systemd integration with kolla-mgr?
Thoughts welcome
Regards,
-steve
_______________________________________________
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

_______________________________________________
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
--
Serge Dubrouski.
_______________________________________________
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

_______________________________________________
Pacemaker mailing list: ***@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http

[Digimer]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Post by David Vossel
----- Original Message -----

Post by Serge Dubrouski
Pacemaker as a scheduler in Mesos or Kubernates does sound like a
very interesting idea. Packaging corosync into super privileged
containers still doesn't make too much sense to me. What's the
reason in isolating something and then giving it all permissions
on a host machine?

because soon everything will run in containers. Take a look at rhel
atomic and the stuff coreos is doing. The only way pacemaker will
exist on those distributions is if it lives in a super privileged
container.

If I can respectfully disagree, or at least provide a different opinion;

I'm old enough to have seen many supposed "game changers" come and go.
Maybe containers will be different, but to simply assume so at this
stage and to base the future development of HA software on that
assumption seems ... risky.

I think there is an argument for supporting containers, sure, but not
to base plans on the assumption that they will become the be-all and
end-all.

- --
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBAgAGBQJU7kPSAAoJECChztQA3mh0lR4P/3JicT5jSk976BHqNcTPmWsM
DHpumNORtpVPMaktC6lT6PRYeyFtYU2lqYgtBW9tpFbNai5h3qwhscV2O0mUON1L
T4NGB1ZwlVAaf+ziZwxThGor/27hS29mHHVgaqR8ePhmjwxD4VX+A3f1S9QKu59B
0FuFLnWjYFMZwDV0CXGD5IxXf9aoPI47M/AwXE18Aj9TvuwblKw5z03VXUSp1kLh
jFb8WcLjX+i11vXUnVnNsOHZfvAg8ejaA5NZLj4se0cMHwA0O3geAjGNAwqrACcp
R0LRkxD0RBv1II3cx0Z4ZIXTSn1YO51GCFduyjRqQFb+noje3Bjyjv6KIAE8Ncm2
Nwi3flom6z/7FQtSokVm0X3uStqxARX67cqAokLa/g8aGBhoBFUont6A7q+be+W8
rG1FYwTIn/5jI6MDueoSD8NFGOyjNHrfGIIV3Cb2K/tFWPt7xB1aeQmnGaZBHhrp
8zBKsipVQj3ZHSKdoo0mP+uvjqklPKdD53twWngxoPydnapuUkgwG8G/gOFn1mC7
HyaGPF8dB0vv25EijZcF7uC74HQdMvPDUC+6nA35zJ1EgYpINUlyIgInXhRP9ckj
Sw2aUd3yxchsaHt8BE2uLIWRbiSkOkBdVxvTna19iroLU7ivNlc5VOSKqxk7wga+
Ci4+kSFm0oZnSRz1gSqv
=GoJ5
-----END PGP SIGNATURE-----

_______________________________________________
Pacemaker mailing list: ***@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[Andrew Beekhof]

Post by Digimer
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Post by David Vossel
----- Original Message -----

Post by Serge Dubrouski
Pacemaker as a scheduler in Mesos or Kubernates does sound like a
very interesting idea. Packaging corosync into super privileged
containers still doesn't make too much sense to me. What's the
reason in isolating something and then giving it all permissions
on a host machine?

because soon everything will run in containers. Take a look at rhel
atomic and the stuff coreos is doing. The only way pacemaker will
exist on those distributions is if it lives in a super privileged
container.

If I can respectfully disagree, or at least provide a different opinion;
I'm old enough to have seen many supposed "game changers" come and go.
Maybe containers will be different, but to simply assume so at this
stage and to base the future development of HA software on that
assumption seems ... risky.
I think there is an argument for supporting containers, sure, but not
to base plans on the assumption that they will become the be-all and
end-all.

Key phrase: "on these distros".
David's not suggesting everyone run Pacemaker like this, only people that have bought into distros that drank the container brand cool-aid (probably out of a container!).

Post by Digimer
- --
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQIcBAEBAgAGBQJU7kPSAAoJECChztQA3mh0lR4P/3JicT5jSk976BHqNcTPmWsM
DHpumNORtpVPMaktC6lT6PRYeyFtYU2lqYgtBW9tpFbNai5h3qwhscV2O0mUON1L
T4NGB1ZwlVAaf+ziZwxThGor/27hS29mHHVgaqR8ePhmjwxD4VX+A3f1S9QKu59B
0FuFLnWjYFMZwDV0CXGD5IxXf9aoPI47M/AwXE18Aj9TvuwblKw5z03VXUSp1kLh
jFb8WcLjX+i11vXUnVnNsOHZfvAg8ejaA5NZLj4se0cMHwA0O3geAjGNAwqrACcp
R0LRkxD0RBv1II3cx0Z4ZIXTSn1YO51GCFduyjRqQFb+noje3Bjyjv6KIAE8Ncm2
Nwi3flom6z/7FQtSokVm0X3uStqxARX67cqAokLa/g8aGBhoBFUont6A7q+be+W8
rG1FYwTIn/5jI6MDueoSD8NFGOyjNHrfGIIV3Cb2K/tFWPt7xB1aeQmnGaZBHhrp
8zBKsipVQj3ZHSKdoo0mP+uvjqklPKdD53twWngxoPydnapuUkgwG8G/gOFn1mC7
HyaGPF8dB0vv25EijZcF7uC74HQdMvPDUC+6nA35zJ1EgYpINUlyIgInXhRP9ckj
Sw2aUd3yxchsaHt8BE2uLIWRbiSkOkBdVxvTna19iroLU7ivNlc5VOSKqxk7wga+
Ci4+kSFm0oZnSRz1gSqv
=GoJ5
-----END PGP SIGNATURE-----
_______________________________________________
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

_______________________________________________
Pacemaker mailing list: ***@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[Digimer]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Post by Digimer

Post by David Vossel
----- Original Message -----

Post by Serge Dubrouski
Pacemaker as a scheduler in Mesos or Kubernates does sound
like a very interesting idea. Packaging corosync into super
privileged containers still doesn't make too much sense to
me. What's the reason in isolating something and then
giving it all permissions on a host machine?

because soon everything will run in containers. Take a look
at rhel atomic and the stuff coreos is doing. The only way
pacemaker will exist on those distributions is if it lives in
a super privileged container.

If I can respectfully disagree, or at least provide a different opinion;
I'm old enough to have seen many supposed "game changers" come and
go. Maybe containers will be different, but to simply assume so at
this stage and to base the future development of HA software on
that assumption seems ... risky.
I think there is an argument for supporting containers, sure, but
not to base plans on the assumption that they will become the
be-all and end-all.

Key phrase: "on these distros". David's not suggesting everyone
run Pacemaker like this, only people that have bought into
distros that drank the container brand cool-aid (probably out of
a container!).

Ah, ok then. :)

- --
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBAgAGBQJU7orWAAoJECChztQA3mh0fikQALV21hJnW+TeiG1YebyNHlOq
la2haZcwzqUmh+sLFgYP5xgrDhbx5dDyWSvchD1nLAaDreBMDD1lD5TTgFtxPcY7
s2dyp7sEhqVsXRcQtBRdcgN9Zw1IDk6w4lQVN8ST8yxNcLOfK1/Vwk2q3kY3O/TV
SQEq1uqV0FsCDvKYuWz6ReTgVAmaxkKc7wozzD4h867dlAHtvbsznkfWM81Bcbh2
U/QMd5LBeBBqYzl7cKgu9PogzBHynnZxMmKS98eZMmj5EWWA8QoSB2UPWbJE2JNj
lDEl5ms87jPDcHOqnrYg1DJ69Hudj9nxE3PzS+TxnnPJvYyEghmEA6KKMAGQZzqC
uAI1acxs8zV0OZ8nndwW9JsIWLPDUHiBwSXo1LyGJoHh/ke/Y/w30YeZEd2qhweB
5tfkn5H6zWB3V1i2NDCDzw7Ph95hWcHwFwbLkdekEcj0pViX+aBJlH85pmwkiNuV
A4+lAjRkslRfNFZGcjV7Dj9IQMQ9sXgouNe+4mcDH0uNu+3RVjagK4HCy2IFad85
uPXTEOLNGgq08mqn1An78/AimLcBMfWZitpFGo3z00GzMCZ2b/fHIQ7d8KLJt5pt
xznQZPlhtgkKYE9/PZeghXevRmRaWBGCcX3Xil2S4NAuhHx8NxqvLOuNewBBynPx
o3tDOiXwfq99YpXA4zkO
=GHxp
-----END PGP SIGNATURE-----

_______________________________________________
Pacemaker mailing list: ***@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[Andrew Beekhof]

Post by Steven Dake (stdake)
Hi,
I am working on Containerizing OpenStack in the Kolla project (http://launchpad.net/kolla). One of the key things we want to do over the next few months is add H/A support to our container tech. David Vossel had suggested using systemctl to monitor the containers themselves by running healthchecking scripts within the containers. That idea is sound.

systemctl or pacemaker-remote in stand-alone mode?
I can imagine the latter more so than the former.

Post by Steven Dake (stdake)
There is another technology called “super-privileged containers”. Essentially it allows more host access for the container, allowing the treatment of Pacemaker as a container rather than a RPM or DEB file. I’d like corosync to run in a separate container.

Separate to even pacemaker?
My container-foo is weak... does they allow traditional IPC mechanisms between containers?

Post by Steven Dake (stdake)
These containers will communicate using their normal mechanisms in a super-privileged mode. We will implement this in Kolla.
Where I am stuck is how does Pacemaker within a container control other containers in the host os. One way I have considered is using the docker —pid=host flag, allowing pacemaker to communicate directly with the host systemctl process. Where I am stuck is our containers don’t run via systemctl, but instead via shell scripts that are executed by third party deployment software.
The user would run
kolla-mgr deploy messaging
This would run a small bit of code to launch the docker container set for messaging.
Could pacemaker run something like
Kolla-mgr status messaging
To control the lifecycle of the processes?

If you wrote an RA that did so, then it can do whatever you like.
However the concept seems prone to internal split-brain given the number of entities wanting a piece of the container.

Will the containers always be in the pacemaker config or only added when someone runs 'deploy'?
How will pacemaker know when to start monitoring?
What should pacemaker do if it detects the container as failed?
How can it tell the difference between a failure and 'kolla-mgr kill messaging'?
Could pacemaker be the 3rd party that does the starting?
If so, perhaps look at the Docker agent.
If not, you will need to make sure pacemaker doesn't start health checking the container until the 3rd party has completely started it.


I'm not saying it cant work as described, I just want to understand the proposal better before I go too much further.

Post by Steven Dake (stdake)
Or would we be better off with some systemd integration with kolla-mgr?

systemd integration is rarely the answer :-)

Post by Steven Dake (stdake)
Thoughts welcome
Regards,
-steve
_______________________________________________
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

_______________________________________________
Pacemaker mailing list: ***@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[David Vossel]

----- Original Message -----

Post by Steven Dake (stdake)
Hi,

Hey Steve, Good to see you around :)

Post by Steven Dake (stdake)
I am working on Containerizing OpenStack in the Kolla project (
http://launchpad.net/kolla ). One of the key things we want to do over the
next few months is add H/A support to our container tech. David Vossel had
suggested using systemctl to monitor the containers themselves by running
healthchecking scripts within the containers. That idea is sound.

Knowing what I know about OpenStack HA now, that is a bad choice.

Post by Steven Dake (stdake)
There is another technology called “super-privileged containers”. Essentially
it allows more host access for the container, allowing the treatment of

Yep, this is the way to do it. My plan is to have pacemaker running in a container,
and have pacemaker capable of launching resources within containers.

We already have a Docker resource agent. You can find it here,
https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/docker

Using that agent, pacemaker can launch a docker container, and then monitor
the container by performing health checks within the container. Here's an
example of how I'm using this technique to manage a containerized apache
instance.
https://github.com/davidvossel/phd/blob/master/scenarios/docker-apache-ap.scenario#L96

Post by Steven Dake (stdake)
Pacemaker as a container rather than a RPM or DEB file. I’d like corosync to
run in a separate container. These containers will communicate using their

I actually already got pacemaker+corosync running in a container for testing
purposes. If you're interested you can checkout some of that work here,
https://github.com/davidvossel/phd/tree/master/lib/docker . The phd_docker_utils.sh
file holds most of the interesting parts.

Post by Steven Dake (stdake)
normal mechanisms in a super-privileged mode. We will implement this in
Kolla.
Where I am stuck is how does Pacemaker within a container control other
containers in the host os. One way I have considered is using the docker
—pid=host flag, allowing pacemaker to communicate directly with the host
systemctl process. Where I am stuck is our containers don’t run via
systemctl, but instead via shell scripts that are executed by third party
deployment software.
The user would run
kolla-mgr deploy messaging

yes, and from there kolla-mgr hands the containers off to pacemaker to manage.

kolla is the orchestration, pacemaker is the scheduler for performing those tasks.

Post by Steven Dake (stdake)
This would run a small bit of code to launch the docker container set for messaging.
Could pacemaker run something like
Kolla-mgr status messaging
To control the lifecycle of the processes?
Or would we be better off with some systemd integration with kolla-mgr?
Thoughts welcome
Regards,
-steve
_______________________________________________
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

_______________________________________________
Pacemaker mailing list: ***@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: