Bug 443358

+++ This bug was initially created as a clone of Bug #251966 +++

Description of problem:

Customer reports a possible split brain condition caused by a malfunction of
fence. Under some conditions the fence cluster group (the one displayed by
"group_tool -v" command) can go into JOIN_START_WAIT state and stays there
forever. This means that when a fence action is required its silently discarded
and other cluster group are allowed to perform their recovery steps. This can
easily lead to a split brain condition in a two node cluster, where the fence
action is not performed and the two nodes may recover the same GFS journal or
mount the same ext3 fs.

How reproducible:
Always

Steps to Reproduce:
1) Configure post_join_delay="60", this is not required but makes it easier to
reproduce the problem
2) Start both nodes at the same time, but keep the network interface for the
heartbeat channel disconnected
3) When both nodes are waiting at the fencing startup, wait a few seconds then
connect the network interface

(This is simple to reproduce with 2 xen guests. Configure 2 xen guests as a
cluster. Shutdown the network bridge from the host then boot both guests. whilst
fenced is waiting enable up the bridge.)

Actual results:
4) the nodes have fencing stuck in JOIN_START_WAIT using two distinct id, from
now on every fence action will be silently discarded, the other clustered
services will perform their recovery action as the fence action was performed

The output of "group_tool -v" shows the services stuck on JOIN_START_WAIT and
using two distinct group id on each node:

Node 1:
type             level name     id       state node id local_done
fence            0     default  00010001 JOIN_START_WAIT 2 200020001 1
[1 2]
dlm              1     clvmd    00020001 none
[1 2]

Node 2:
type             level name     id       state node id local_done
fence            0     default  00010002 JOIN_START_WAIT 1 100020001 1
[1 2]
dlm              1     clvmd    00020001 none
[1 2]

Expected results:
4) either the cluster should not form, or the two clusters should be merged
successfully.

Additional info:
When the nodes start up, they each form a 1-node openais cluster independent of
the other. fence_tool join is run on each node which creates group state in both
clusters. In the situation this bug describes, the dirty flag will not prevent
the clusters from merging because NODE_FLAGS_BEENDOWN is not set:

if (msg->flags & NODE_FLAGS_DIRTY && node->flags & NODE_FLAGS_BEENDOWN)

The attached patch modifies the dirty flag test so that its possible for a
"clean" node (one without state) to join a dirty node regardless of whether its
BEENDOWN.