Bug 511113

+++ This bug was initially created as a clone of Bug #510611 +++

Description of problem:

Qdiskd on RHEL5 does not check if IO has failed for interval*tko seconds, relying only on cman kill to evict a node. If the quorum disk is unavailable for all nodes there is no way of acting, and when the disk comes back (let's say due to mpath flapping or unavailability) one of the nodes will be fenced due to cman kill.

On RHEL5 there is the max_error_cycles tunable that causes the node to disconnect itself from qdisk gracefully after having a number of cycles with errors greater than max_error_cycles, but this is not the same as interval*tko, which can deliver a timeout as expected.

How reproducible:

every time.

Steps to Reproduce:

1- Have a cluster of 2 nodes on rhel5 with quorum disk having one vote per machine and 1 vote for quorum disk.

2- unplug the FC cables for both machines, notice that FC link goes down.

3- wait for qdisk interval*tko to elapse.

4- notice that no action takes place and no machines are fenced or rebooted.

5- plug back the FC cable of one of the machines.

6- The machine whose cables were plugged back will send a 'CMAN kill' to the other as soon as it notices that it has missed disk heartbeats for interval*tko.

Actual results:

When both nodes have no FC anymore, no fencing or reboots occur.

Expected results:

Be able to reboot/fence nodes when IO fails for interval*tko seconds.

Additional info:

*** sanitized information ***
*** this data comes from RHEL4 via clone of BZ ***

Example configuration:

        <quorumd interval="5" label="mylabel" min_score="1" tko="10" votes="1" paranoid="1"> 
                <heuristic interval="2" program="ping -c1 -t1 10.0.0.1" score="1"/> 
        </quorumd> 
 
disabled HBA's on both nodes. After multipathd failed the quorum device, qdiskd detected io failures, but did not cause a reboot or fence:
  
Jun 26 11:02:43 node2 qdiskd[4633]: <warning> Error reading node ID block 16 
Jun 26 11:02:43 node2 kernel: SCSI error : <1 0 0 3> return code = 0x10000 
Jun 26 11:02:43 node2 qdiskd[4633]: <err> Error writing to quorum disk 
Jun 26 11:02:43 node2 kernel: end_request: I/O error, dev sdc, sector 80 
Jun 26 11:02:43 node2 kernel: SCSI error : <1 0 0 3> return code = 0x10000 
Jun 26 11:02:43 node2 kernel: end_request: I/O error, dev sdc, sector 88 
 
After reenabling one of the cards we see: 
 
Jun 26 11:02:44 node2 kernel: qla2400 0000:05:00.0: LOOP UP detected (4 Gbps). 
Jun 26 11:02:48 node2 multipathd: 8:32: tur checker reports path is up 
Jun 26 11:02:48 node2 multipathd: 8:32: reinstated 
Jun 26 11:02:48 node2 multipathd: quorum: queue_if_no_path enabled 
Jun 26 11:02:48 node2 multipathd: quorum: Recovered to normal mode 
Jun 26 11:02:48 node2 multipathd: quorum: remaining active paths: 1 
 
leading to eviction of the other node: 
 
Jun 26 11:04:00 node2 kernel: CMAN: Quorum device /dev/sdc timed out 
Jun 26 11:04:02 node2 qdiskd[4633]: <info> Assuming master role 
Jun 26 11:04:07 node2 qdiskd[4633]: <notice> Writing eviction notice for node 1 
Jun 26 11:04:07 node2 kernel: CMAN: removing node node1.qcluster from the cluster : Killed by another node 
Jun 26 11:04:07 node2 fenced[4713]: node1.qcluster not a cluster member after 0 sec post_fail_delay 
Jun 26 11:04:07 node2 fenced[4713]: fencing node "node1.qcluster" 
Jun 26 11:04:12 node2 qdiskd[4633]: <notice> Node 1 evicted 
Jun 26 11:04:17 node2 fenced[4713]: fence "node1.qcluster" success