Description of problem:
Twice now while running revolver on the taft cluster (4 nodes) for the 4.8 regression run, qdisk was unable to deal with a node being shot.


May 12 09:59:19 taft-02 qdiskd[15296]: <warning> qdisk cycle took more than 1 second to complete (1.060000)
May 12 09:59:35 taft-02 qarshd[15549]: Talking to peer 10.15.89.99:57342
May 12 09:59:35 taft-02 qarshd[15549]: Running cmdline: cman_tool status
May 12 09:59:54 taft-02 qdiskd[15296]: <info> Assuming master role
May 12 09:59:55 taft-02 qdiskd[15296]: <notice> Writing eviction notice for node 1
May 12 09:59:55 taft-02 kernel: CMAN: removing node taft-01 from the cluster : Killed by another node
May 12 09:59:56 taft-02 qdiskd[15296]: <notice> Node 1 evicted
May 12 09:59:57 taft-02 qdiskd[15296]: <crit> Node 1 is undead.
May 12 09:59:57 taft-02 qdiskd[15296]: <alert> Writing eviction notice for node 1
May 12 09:59:58 taft-02 qdiskd[15296]: <crit> Node 1 is undead.
May 12 09:59:58 taft-02 qdiskd[15296]: <alert> Writing eviction notice for node 1
May 12 09:59:59 taft-02 qdiskd[15296]: <crit> Node 1 is undead.
May 12 09:59:59 taft-02 qdiskd[15296]: <alert> Writing eviction notice for node 1
May 12 09:59:59 taft-02 qarshd[15563]: Talking to peer 10.15.89.99:57347
May 12 09:59:59 taft-02 qarshd[15563]: Running cmdline: cat /proc/cluster/nodes
May 12 10:00:00 taft-02 qdiskd[15296]: <crit> Node 1 is undead.
May 12 10:00:00 taft-02 qdiskd[15296]: <alert> Writing eviction notice for node 1
May 12 10:00:01 taft-02 qdiskd[15296]: <crit> Node 1 is undead.
May 12 10:00:01 taft-02 qdiskd[15296]: <alert> Writing eviction notice for node 1
May 12 10:00:01 taft-02 kernel: CMAN: Initiating transition, generation 7
May 12 10:00:01 taft-02 kernel: CMAN: Started transition, generation 7
May 12 10:00:01 taft-02 kernel: CMAN: Initiating transition, generation 8
May 12 10:00:02 taft-02 qdiskd[15296]: <crit> Node 1 is undead.
May 12 10:00:02 taft-02 qdiskd[15296]: <alert> Writing eviction notice for node 1
May 12 10:00:03 taft-02 qdiskd[15296]: <crit> Node 1 is undead.
May 12 10:00:03 taft-02 qdiskd[15296]: <alert> Writing eviction notice for node 1
[...]

The above messages then scroll for eternity...

I'll attach the logs.


Version-Release number of selected component (if applicable):
2.6.9-89.ELsmp
cman-1.0.27-1.el4

How reproducible:
2/2