[Pacemaker] resource monitor priority - timeout on high load

[Radoslaw Garbacz]

Hi,

I have a situation, when resource monitor operations timeout on system high
load.

One thing to do is to increase timeouts, but there can always be load, at
which the timeout event would happen, and on the other hand too long
timeouts would not be very useful, so I would like to increase the priority
of monitoring processes (GNU Linux), but I do not want to be too broad.

So my question is, which pacemaker process should I target to ensure timely
monitoring operations, or maybe even there is a pacemaker configuration
parameter, which would help in such a case.

I know for sure, that agents finish they monitoring requests on time, and
the timeout is due to high load. I have several agents, so I will provide
just some details to illustrate the case:
- timeout: ~60s
- interval: ~60s
- normal (low load) monitor request time: 0.2s
- high load monitor request time: 8s - when pacemaker reports resource
operation timeout


Thanks,

[emmanuel segura]

If I remeber, there is a way to make real time pacemaker process, but
I don't know if this can help you, you need to tell us if your server
high load is system space or userspace.

2015-10-05 19:53 GMT+02:00 Radoslaw Garbacz

Post by Radoslaw Garbacz
Hi,
I have a situation, when resource monitor operations timeout on system high
load.
One thing to do is to increase timeouts, but there can always be load, at
which the timeout event would happen, and on the other hand too long
timeouts would not be very useful, so I would like to increase the priority
of monitoring processes (GNU Linux), but I do not want to be too broad.
So my question is, which pacemaker process should I target to ensure timely
monitoring operations, or maybe even there is a pacemaker configuration
parameter, which would help in such a case.
I know for sure, that agents finish they monitoring requests on time, and
the timeout is due to high load. I have several agents, so I will provide
- timeout: ~60s
- interval: ~60s
- normal (low load) monitor request time: 0.2s
- high load monitor request time: 8s - when pacemaker reports resource
operation timeout
Thanks,
--
Best Regards,
Radoslaw Garbacz
XtremeData Incorporation
_______________________________________________
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

--
.~.
/V\
// \\
/( )\
^`~'^

_______________________________________________
Pacemaker mailing list: ***@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[Radoslaw Garbacz]

I think it would be helpful, my load is definitely user-space.
My goal is for pacemaker to survive high user load, and not shut down
resources on monitoring timeouts, additionally resources should not be
restarted either, i.e. they have to be left alone, unless - of course -
there is a real resource failure.

Thanks,

Post by emmanuel segura
If I remeber, there is a way to make real time pacemaker process, but
I don't know if this can help you, you need to tell us if your server
high load is system space or userspace.
2015-10-05 19:53 GMT+02:00 Radoslaw Garbacz

Post by Radoslaw Garbacz
Hi,
I have a situation, when resource monitor operations timeout on system

high

Post by Radoslaw Garbacz
load.
One thing to do is to increase timeouts, but there can always be load, at
which the timeout event would happen, and on the other hand too long
timeouts would not be very useful, so I would like to increase the

priority

Post by Radoslaw Garbacz
of monitoring processes (GNU Linux), but I do not want to be too broad.
So my question is, which pacemaker process should I target to ensure

timely

Post by Radoslaw Garbacz
monitoring operations, or maybe even there is a pacemaker configuration
parameter, which would help in such a case.
I know for sure, that agents finish they monitoring requests on time, and
the timeout is due to high load. I have several agents, so I will provide
- timeout: ~60s
- interval: ~60s
- normal (low load) monitor request time: 0.2s
- high load monitor request time: 8s - when pacemaker reports resource
operation timeout
Thanks,
--
Best Regards,
Radoslaw Garbacz
XtremeData Incorporation
_______________________________________________
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

--
.~.
/V\
// \\
/( )\
^`~'^
_______________________________________________
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[Dejan Muhamedagic]

Hi,

Post by Radoslaw Garbacz
Hi,
I have a situation, when resource monitor operations timeout on system high
load.
One thing to do is to increase timeouts, but there can always be load, at
which the timeout event would happen, and on the other hand too long
timeouts would not be very useful,

Why do you think so? Did you see actual failures due to a
timeout, i.e. a process just hanging in limbo?

Post by Radoslaw Garbacz
so I would like to increase the priority
of monitoring processes (GNU Linux), but I do not want to be too broad.

If that could make the server a bit bigger too ;-)

Seriously, IMO you have two options: either add more resources
(as in hardware) or increase the monitor timeouts.

Thanks,

Dejan

Post by Radoslaw Garbacz
So my question is, which pacemaker process should I target to ensure timely
monitoring operations, or maybe even there is a pacemaker configuration
parameter, which would help in such a case.
I know for sure, that agents finish they monitoring requests on time, and
the timeout is due to high load. I have several agents, so I will provide
- timeout: ~60s
- interval: ~60s
- normal (low load) monitor request time: 0.2s
- high load monitor request time: 8s - when pacemaker reports resource
operation timeout
Thanks,
--
Best Regards,
Radoslaw Garbacz
XtremeData Incorporation
_______________________________________________
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

_______________________________________________
Pacemaker mailing list: ***@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[Radoslaw Garbacz]

Indeed, I do not see actual failures due to a timeout.

Thanks Dejan,

Post by Dejan Muhamedagic
Hi,

Post by Radoslaw Garbacz
Hi,
I have a situation, when resource monitor operations timeout on system

high

Post by Radoslaw Garbacz
load.
One thing to do is to increase timeouts, but there can always be load, at
which the timeout event would happen, and on the other hand too long
timeouts would not be very useful,

Why do you think so? Did you see actual failures due to a
timeout, i.e. a process just hanging in limbo?

Post by Radoslaw Garbacz
so I would like to increase the priority
of monitoring processes (GNU Linux), but I do not want to be too broad.

If that could make the server a bit bigger too ;-)
Seriously, IMO you have two options: either add more resources
(as in hardware) or increase the monitor timeouts.
Thanks,
Dejan

Post by Radoslaw Garbacz
So my question is, which pacemaker process should I target to ensure

timely

Post by Radoslaw Garbacz
monitoring operations, or maybe even there is a pacemaker configuration
parameter, which would help in such a case.
I know for sure, that agents finish they monitoring requests on time, and
the timeout is due to high load. I have several agents, so I will provide
- timeout: ~60s
- interval: ~60s
- normal (low load) monitor request time: 0.2s
- high load monitor request time: 8s - when pacemaker reports resource
operation timeout
Thanks,
--
Best Regards,
Radoslaw Garbacz
XtremeData Incorporation
_______________________________________________
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

_______________________________________________
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org