Bug 1241511

"Fencing will never occur because the cluster is quorate with all nodes"

If stateful cluster nodes fail, they need to be fenced.

If dlm is in charge of fencing, then stateful cluster merges are a situation where you might need to manually intervene (e.g. if no partition maintained quorum).  When pacemaker does fencing, I don't know what's supposed to happen.

If you reproduce this with dlm by itself (get rid of pacemaker) then I could explain the behavior.  Please either reproduce that way, or reassign to pacemaker.

The cluster doesn't require fencing in this situation.
If the dlm requires it, then it is up to the dlm to initiate it.

The expected dlm behavior here remains the same as it's been in the past (since partition/merge handling was added), and there does not appear to be anything to fix.

In the case of a cluster partition that merges, if one partition maintained quorum, then it will kill merged nodes.  Otherwise, as in this case, user intervention is required to select and kill merged nodes.

Hello Michal,

> How reproducible: very frequent
> 
> Steps to Reproduce:
> 1. have quorate pacemaker cluster
> . check nodes uptime
> . disable network communication between all nodes with iptables and wait for
>     all nodes turning inquorate
> . enable at the same time network communication between nodes 
> . check whether fencing occured and if it has not check dlm status and logs

With 3 nodes cluster, unfortunately I cannot reproduce(fencing quickly happens ) if applying iptables manually:-/ Looking at cluster report you attached, I think you may use some automatic method to make a real transient disconnection all of sudden. If so, could you please share your method/scripts to help reproduce?

The reason why I'm here is this patch (https://github.com/ClusterLabs/pacemaker/pull/839) has a problem which will cause both nodes to be fenced in 2-nodes cluster unnecessarily in the following case:

1. Bring both nodes up in the cluster and all resources started.
2. Fence one node by issuing "pkill -9 corosync"
3. Watch logs and surviving node fences the other node and then ends up self fencing 

It will decrease availability in 2-nodes scenario. IMHO, the patch shouldn't let "controld" RA rely on "dlm_tool ls" to get "wait fencing" because this message means there's a node in cluster needing fencing. This commands on each node tell RA the same message, so every node will die. IOW, we need dlm tell RA if this node needs fencing, then that patch should work better.

Thanks for your time;-)

What I do is that I create /root/iptables.sh on each of the cluster node and then I do run from a node outside of the cluster:

for i in 1 2 3; do ssh node$i /root/iptables.sh & done; wait

This way I was able to manifest the problem described in like less than ten attempts on a three node cluster. 

The imortant thing is '&' instead of ';' in the for cycle which will the commands in parallel. 

Hope this helps.

(In reply to michal novacek from comment #10)
> What I do is that I create /root/iptables.sh on each of the cluster node and
> then I do run from a node outside of the cluster:
> 
> for i in 1 2 3; do ssh node$i /root/iptables.sh & done; wait
> 
> This way I was able to manifest the problem described in like less than ten
> attempts on a three node cluster. 
> 
> The imortant thing is '&' instead of ';' in the for cycle which will the
> commands in parallel. 
> 
> Hope this helps.

Hi Michal,

Thanks a lot for your info! I've reproduced this problem now. In case you may interest:
1. setup ntp (optional);
2. put this scritps on every nodes:
---
#!/bin/sh

PATH=$PATH:/usr/sbin/
has_quorum=
hosts="ocfs2test2,ocfs2test3"   # other 2 nodes

iptables -A INPUT -s $hosts -j DROP
echo "iptables: add rules" > /tmp/cron.log

while true; do
	has_quorum=`corosync-quorumtool | awk '{if($1=="Quorate:") print $2;}'`
	if [ $has_quorum == "No" ] ; then
		echo "Quorum lost now" >> /tmp/cron.log
		break;
	fi
done

iptables -D INPUT -s $hosts -j DROP
echo "iptables: remove rules" >> /tmp/cron.log
---
3. concurrently trigger to run by crontab;

Thanks again.