Description of problem:

dlm can hang in an infinite loop waiting for a fencing action to occur that has already happened.

Version-Release number of selected component (if applicable):


How reproducible:

difficult race condition

Steps to Reproduce:
1. disconnect a node from a pacemaker cluster in a way where the corosync membership is lost in an unexpected way (network disconnect work work)
2. both pacemaker and the dlm will detect the membership change and attempt to fence the lost node.
3.  If pacemaker's fencing request is completely processed within a short time interval before the dlm's request is processed, those two requests will get merged. The dlm will be alerted that the fencing operation occurred through the successful completion of the stonith_api_kick_helper function, but the dlm will remain in a infinite loop waiting for the time of the last fencing request to  update to what it expects the time should be.


Actual results:

dlm hangs


Expected results:

dlm sees the fencing action occurred and moves on.

Additional info:

here's the code in question.

>	rv = stonith_api_kick_helper(nodeid, 300, 0);

if stonith_api_kick_helper says we fenced the node, the node is fenced. This function should be blocking until the fencing operation completes. If rv==0, then fencing was successful.

>	if (rv) {
>		fprintf(stderr, "kick_helper error %d nodeid %d\n", rv, nodeid);
>		openlog("dlm_stonith", LOG_CONS | LOG_PID, LOG_DAEMON);
>		syslog(LOG_ERR, "kick_helper error %d nodeid %d\n", rv, nodeid);
>		return rv;
>	}
>
>	while (1) {
>		t = stonith_api_time_helper(nodeid, 0);
>		if (t >= fail_time)

We can't trust this statement will ever be true. There are many conditions where both pacemaker and the dlm will fence a node at seemingly the exact same time. depending on the request order and how fast the fencing devices respond we could have a condition where the last fencing action occurs slightly before fail_time.  This is because the two fencing requests stonith receives (one from dlm, one from pacemaker) are merged so we don't double fence a node because of overlapping fencing requests. This only happens if the two requests are received by stonith overlap or occur within a very short amount of time of one another.

>			return 0;
>		sleep(1);
>	}
>


The wait loop should not be necessary. I believe we can remove the entirely and this race condition will be solved.