Description of problem: pcs status shows all nodes started even when hit "partition WITHOUT quorum" with IO's still resuming Version-Release number of selected component (if applicable): # rpm -qa | grep ganesha nfs-ganesha-debuginfo-2.4.4-8.el7rhgs.x86_64 glusterfs-ganesha-3.8.4-29.el7rhgs.x86_64 nfs-ganesha-2.4.4-8.el7rhgs.x86_64 nfs-ganesha-gluster-2.4.4-8.el7rhgs.x86_64 How reproducible: 2/2 times Steps to Reproduce: 1.Create a 6 node ganesha cluster 2.Create 2 volumes.Enable ganesha on it.Mount the volumes to client 3.Run linux untar on 1 volume and scp the files from volume one to volume 2 which is mounted on another client 4.Reboot 3 nodes out of 6 Nodes --> met "partition WITHOUT quorum" .All the nodes are in stopped state in pcs status.IO's are stopped 5.Make one of the node up. -> now its " partition with quorum".IO's will resume.All the nodes are in started state as 4 out of 6 nodes are up. 6.Now reboot one of the node other then the one which were already rebooted.Now again 3 out of 6 nodes are down -->met "partition WITHOUT quorum".IO's resume and all nodes are in started state for more then 15 mins Actual results: It takes longer time for nodes to go in stopped state when met "partition WITHOUT quorum" Expected results: When 50% nodes are down then all the nodes should go to stopped state with no IO's in process Additional info: When 3 nodes are rebooted out of 6 ================ [root@dhcp42-117 ~]# pcs status Cluster name: ganesha-ha-360 Stack: corosync Current DC: dhcp42-119.lab.eng.blr.redhat.com (version 1.1.16-8.el7-94ff4df) - partition WITHOUT quorum Last updated: Thu Jun 22 13:47:07 2017 Last change: Thu Jun 22 13:39:19 2017 by root via crm_attribute on dhcp42-127.lab.eng.blr.redhat.com 6 nodes configured 36 resources configured Online: [ dhcp42-117.lab.eng.blr.redhat.com dhcp42-119.lab.eng.blr.redhat.com dhcp42-129.lab.eng.blr.redhat.com ] OFFLINE: [ dhcp42-114.lab.eng.blr.redhat.com dhcp42-125.lab.eng.blr.redhat.com dhcp42-127.lab.eng.blr.redhat.com ] Full list of resources: Clone Set: nfs_setup-clone [nfs_setup] Stopped: [ dhcp42-114.lab.eng.blr.redhat.com dhcp42-117.lab.eng.blr.redhat.com dhcp42-119.lab.eng.blr.redhat.com dhcp42-125.lab.eng.blr.redhat.com dhcp42-127.lab.eng.blr.redhat.com dhcp42-129.lab.eng.blr.redhat.com ] Clone Set: nfs-mon-clone [nfs-mon] Stopped: [ dhcp42-114.lab.eng.blr.redhat.com dhcp42-117.lab.eng.blr.redhat.com dhcp42-119.lab.eng.blr.redhat.com dhcp42-125.lab.eng.blr.redhat.com dhcp42-127.lab.eng.blr.redhat.com dhcp42-129.lab.eng.blr.redhat.com ] Clone Set: nfs-grace-clone [nfs-grace] Stopped: [ dhcp42-114.lab.eng.blr.redhat.com dhcp42-117.lab.eng.blr.redhat.com dhcp42-119.lab.eng.blr.redhat.com dhcp42-125.lab.eng.blr.redhat.com dhcp42-127.lab.eng.blr.redhat.com dhcp42-129.lab.eng.blr.redhat.com ] Resource Group: dhcp42-117.lab.eng.blr.redhat.com-group dhcp42-117.lab.eng.blr.redhat.com-nfs_block (ocf::heartbeat:portblock): Stopped dhcp42-117.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Stopped dhcp42-117.lab.eng.blr.redhat.com-nfs_unblock (ocf::heartbeat:portblock): Stopped Resource Group: dhcp42-119.lab.eng.blr.redhat.com-group dhcp42-119.lab.eng.blr.redhat.com-nfs_block (ocf::heartbeat:portblock): Stopped dhcp42-119.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Stopped dhcp42-119.lab.eng.blr.redhat.com-nfs_unblock (ocf::heartbeat:portblock): Stopped Resource Group: dhcp42-125.lab.eng.blr.redhat.com-group dhcp42-125.lab.eng.blr.redhat.com-nfs_block (ocf::heartbeat:portblock): Stopped dhcp42-125.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Stopped dhcp42-125.lab.eng.blr.redhat.com-nfs_unblock (ocf::heartbeat:portblock): Stopped Resource Group: dhcp42-127.lab.eng.blr.redhat.com-group dhcp42-127.lab.eng.blr.redhat.com-nfs_block (ocf::heartbeat:portblock): Stopped dhcp42-127.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Stopped dhcp42-127.lab.eng.blr.redhat.com-nfs_unblock (ocf::heartbeat:portblock): Stopped Resource Group: dhcp42-129.lab.eng.blr.redhat.com-group dhcp42-129.lab.eng.blr.redhat.com-nfs_block (ocf::heartbeat:portblock): Stopped dhcp42-129.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Stopped dhcp42-129.lab.eng.blr.redhat.com-nfs_unblock (ocf::heartbeat:portblock): Stopped Resource Group: dhcp42-114.lab.eng.blr.redhat.com-group dhcp42-114.lab.eng.blr.redhat.com-nfs_block (ocf::heartbeat:portblock): Stopped dhcp42-114.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Stopped dhcp42-114.lab.eng.blr.redhat.com-nfs_unblock (ocf::heartbeat:portblock): Stopped Daemon Status: corosync: active/disabled pacemaker: active/enabled pcsd: active/enabled ======================= When 1 node made up.Now 4 nodes are up out of 6 nodes ====================== [root@dhcp42-117 ~]# pcs status Cluster name: ganesha-ha-360 Stack: corosync Current DC: dhcp42-119.lab.eng.blr.redhat.com (version 1.1.16-8.el7-94ff4df) - partition with quorum Last updated: Thu Jun 22 13:49:26 2017 Last change: Thu Jun 22 13:39:19 2017 by root via crm_attribute on dhcp42-127.lab.eng.blr.redhat.com 6 nodes configured 36 resources configured Online: [ dhcp42-117.lab.eng.blr.redhat.com dhcp42-119.lab.eng.blr.redhat.com dhcp42-125.lab.eng.blr.redhat.com dhcp42-129.lab.eng.blr.redhat.com ] OFFLINE: [ dhcp42-114.lab.eng.blr.redhat.com dhcp42-127.lab.eng.blr.redhat.com ] Full list of resources: Clone Set: nfs_setup-clone [nfs_setup] Started: [ dhcp42-117.lab.eng.blr.redhat.com dhcp42-119.lab.eng.blr.redhat.com dhcp42-125.lab.eng.blr.redhat.com dhcp42-129.lab.eng.blr.redhat.com ] Stopped: [ dhcp42-114.lab.eng.blr.redhat.com dhcp42-127.lab.eng.blr.redhat.com ] Clone Set: nfs-mon-clone [nfs-mon] Started: [ dhcp42-117.lab.eng.blr.redhat.com dhcp42-119.lab.eng.blr.redhat.com dhcp42-125.lab.eng.blr.redhat.com dhcp42-129.lab.eng.blr.redhat.com ] Stopped: [ dhcp42-114.lab.eng.blr.redhat.com dhcp42-127.lab.eng.blr.redhat.com ] Clone Set: nfs-grace-clone [nfs-grace] Started: [ dhcp42-117.lab.eng.blr.redhat.com dhcp42-119.lab.eng.blr.redhat.com dhcp42-125.lab.eng.blr.redhat.com dhcp42-129.lab.eng.blr.redhat.com ] Stopped: [ dhcp42-114.lab.eng.blr.redhat.com dhcp42-127.lab.eng.blr.redhat.com ] Resource Group: dhcp42-117.lab.eng.blr.redhat.com-group dhcp42-117.lab.eng.blr.redhat.com-nfs_block (ocf::heartbeat:portblock): Started dhcp42-117.lab.eng.blr.redhat.com dhcp42-117.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Started dhcp42-117.lab.eng.blr.redhat.com dhcp42-117.lab.eng.blr.redhat.com-nfs_unblock (ocf::heartbeat:portblock): Started dhcp42-117.lab.eng.blr.redhat.com Resource Group: dhcp42-119.lab.eng.blr.redhat.com-group dhcp42-119.lab.eng.blr.redhat.com-nfs_block (ocf::heartbeat:portblock): Started dhcp42-119.lab.eng.blr.redhat.com dhcp42-119.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Started dhcp42-119.lab.eng.blr.redhat.com dhcp42-119.lab.eng.blr.redhat.com-nfs_unblock (ocf::heartbeat:portblock): Started dhcp42-119.lab.eng.blr.redhat.com Resource Group: dhcp42-125.lab.eng.blr.redhat.com-group dhcp42-125.lab.eng.blr.redhat.com-nfs_block (ocf::heartbeat:portblock): Started dhcp42-125.lab.eng.blr.redhat.com dhcp42-125.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Started dhcp42-125.lab.eng.blr.redhat.com dhcp42-125.lab.eng.blr.redhat.com-nfs_unblock (ocf::heartbeat:portblock): Started dhcp42-125.lab.eng.blr.redhat.com Resource Group: dhcp42-127.lab.eng.blr.redhat.com-group dhcp42-127.lab.eng.blr.redhat.com-nfs_block (ocf::heartbeat:portblock): Started dhcp42-125.lab.eng.blr.redhat.com dhcp42-127.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Started dhcp42-125.lab.eng.blr.redhat.com dhcp42-127.lab.eng.blr.redhat.com-nfs_unblock (ocf::heartbeat:portblock): Started dhcp42-125.lab.eng.blr.redhat.com Resource Group: dhcp42-129.lab.eng.blr.redhat.com-group dhcp42-129.lab.eng.blr.redhat.com-nfs_block (ocf::heartbeat:portblock): Started dhcp42-129.lab.eng.blr.redhat.com dhcp42-129.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Started dhcp42-129.lab.eng.blr.redhat.com dhcp42-129.lab.eng.blr.redhat.com-nfs_unblock (ocf::heartbeat:portblock): Started dhcp42-129.lab.eng.blr.redhat.com Resource Group: dhcp42-114.lab.eng.blr.redhat.com-group dhcp42-114.lab.eng.blr.redhat.com-nfs_block (ocf::heartbeat:portblock): Started dhcp42-129.lab.eng.blr.redhat.com dhcp42-114.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Started dhcp42-129.lab.eng.blr.redhat.com dhcp42-114.lab.eng.blr.redhat.com-nfs_unblock (ocf::heartbeat:portblock): Started dhcp42-129.lab.eng.blr.redhat.com Daemon Status: corosync: active/disabled pacemaker: active/enabled pcsd: active/enabled ================================== When again 1 node is rebooted.Now 3 nodes are down out of 6 ============================== [root@dhcp42-117 ~]# pcs status Cluster name: ganesha-ha-360 Stack: corosync Current DC: dhcp42-119.lab.eng.blr.redhat.com (version 1.1.16-8.el7-94ff4df) - partition WITHOUT quorum Last updated: Thu Jun 22 13:55:15 2017 Last change: Thu Jun 22 13:39:19 2017 by root via crm_attribute on dhcp42-127.lab.eng.blr.redhat.com 6 nodes configured 36 resources configured Online: [ dhcp42-117.lab.eng.blr.redhat.com dhcp42-119.lab.eng.blr.redhat.com dhcp42-125.lab.eng.blr.redhat.com ] OFFLINE: [ dhcp42-114.lab.eng.blr.redhat.com dhcp42-127.lab.eng.blr.redhat.com dhcp42-129.lab.eng.blr.redhat.com ] Full list of resources: Clone Set: nfs_setup-clone [nfs_setup] Started: [ dhcp42-117.lab.eng.blr.redhat.com dhcp42-119.lab.eng.blr.redhat.com dhcp42-125.lab.eng.blr.redhat.com ] Stopped: [ dhcp42-114.lab.eng.blr.redhat.com dhcp42-127.lab.eng.blr.redhat.com dhcp42-129.lab.eng.blr.redhat.com ] Clone Set: nfs-mon-clone [nfs-mon] Started: [ dhcp42-117.lab.eng.blr.redhat.com dhcp42-119.lab.eng.blr.redhat.com dhcp42-125.lab.eng.blr.redhat.com ] Stopped: [ dhcp42-114.lab.eng.blr.redhat.com dhcp42-127.lab.eng.blr.redhat.com dhcp42-129.lab.eng.blr.redhat.com ] Clone Set: nfs-grace-clone [nfs-grace] Started: [ dhcp42-117.lab.eng.blr.redhat.com dhcp42-119.lab.eng.blr.redhat.com dhcp42-125.lab.eng.blr.redhat.com ] Stopped: [ dhcp42-114.lab.eng.blr.redhat.com dhcp42-127.lab.eng.blr.redhat.com dhcp42-129.lab.eng.blr.redhat.com ] Resource Group: dhcp42-117.lab.eng.blr.redhat.com-group dhcp42-117.lab.eng.blr.redhat.com-nfs_block (ocf::heartbeat:portblock): Started dhcp42-117.lab.eng.blr.redhat.com dhcp42-117.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Started dhcp42-117.lab.eng.blr.redhat.com dhcp42-117.lab.eng.blr.redhat.com-nfs_unblock (ocf::heartbeat:portblock): Started dhcp42-117.lab.eng.blr.redhat.com Resource Group: dhcp42-119.lab.eng.blr.redhat.com-group dhcp42-119.lab.eng.blr.redhat.com-nfs_block (ocf::heartbeat:portblock): Started dhcp42-119.lab.eng.blr.redhat.com dhcp42-119.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Started dhcp42-119.lab.eng.blr.redhat.com dhcp42-119.lab.eng.blr.redhat.com-nfs_unblock (ocf::heartbeat:portblock): Started dhcp42-119.lab.eng.blr.redhat.com Resource Group: dhcp42-125.lab.eng.blr.redhat.com-group dhcp42-125.lab.eng.blr.redhat.com-nfs_block (ocf::heartbeat:portblock): Started dhcp42-125.lab.eng.blr.redhat.com dhcp42-125.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Started dhcp42-125.lab.eng.blr.redhat.com dhcp42-125.lab.eng.blr.redhat.com-nfs_unblock (ocf::heartbeat:portblock): Started dhcp42-125.lab.eng.blr.redhat.com Resource Group: dhcp42-127.lab.eng.blr.redhat.com-group dhcp42-127.lab.eng.blr.redhat.com-nfs_block (ocf::heartbeat:portblock): Started dhcp42-125.lab.eng.blr.redhat.com dhcp42-127.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Started dhcp42-125.lab.eng.blr.redhat.com dhcp42-127.lab.eng.blr.redhat.com-nfs_unblock (ocf::heartbeat:portblock): Started dhcp42-125.lab.eng.blr.redhat.com Resource Group: dhcp42-129.lab.eng.blr.redhat.com-group dhcp42-129.lab.eng.blr.redhat.com-nfs_block (ocf::heartbeat:portblock): Started dhcp42-125.lab.eng.blr.redhat.com dhcp42-129.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Started dhcp42-125.lab.eng.blr.redhat.com dhcp42-129.lab.eng.blr.redhat.com-nfs_unblock (ocf::heartbeat:portblock): Started dhcp42-125.lab.eng.blr.redhat.com Resource Group: dhcp42-114.lab.eng.blr.redhat.com-group dhcp42-114.lab.eng.blr.redhat.com-nfs_block (ocf::heartbeat:portblock): Started dhcp42-125.lab.eng.blr.redhat.com dhcp42-114.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Started dhcp42-125.lab.eng.blr.redhat.com dhcp42-114.lab.eng.blr.redhat.com-nfs_unblock (ocf::heartbeat:portblock): Started dhcp42-125.lab.eng.blr.redhat.com Daemon Status: corosync: active/disabled pacemaker: active/enabled pcsd: active/enabled ===================================== After more then 15 mins ================================== [root@dhcp42-117 ~]# pcs status Cluster name: ganesha-ha-360 Stack: corosync Current DC: dhcp42-119.lab.eng.blr.redhat.com (version 1.1.16-8.el7-94ff4df) - partition WITHOUT quorum Last updated: Thu Jun 22 14:10:19 2017 Last change: Thu Jun 22 13:39:19 2017 by root via crm_attribute on dhcp42-127.lab.eng.blr.redhat.com 6 nodes configured 36 resources configured Online: [ dhcp42-117.lab.eng.blr.redhat.com dhcp42-119.lab.eng.blr.redhat.com dhcp42-125.lab.eng.blr.redhat.com ] OFFLINE: [ dhcp42-114.lab.eng.blr.redhat.com dhcp42-127.lab.eng.blr.redhat.com dhcp42-129.lab.eng.blr.redhat.com ] Full list of resources: Clone Set: nfs_setup-clone [nfs_setup] Stopped: [ dhcp42-114.lab.eng.blr.redhat.com dhcp42-117.lab.eng.blr.redhat.com dhcp42-119.lab.eng.blr.redhat.com dhcp42-125.lab.eng.blr.redhat.com dhcp42-127.lab.eng.blr.redhat.com dhcp42-129.lab.eng.blr.redhat.com ] Clone Set: nfs-mon-clone [nfs-mon] Stopped: [ dhcp42-114.lab.eng.blr.redhat.com dhcp42-117.lab.eng.blr.redhat.com dhcp42-119.lab.eng.blr.redhat.com dhcp42-125.lab.eng.blr.redhat.com dhcp42-127.lab.eng.blr.redhat.com dhcp42-129.lab.eng.blr.redhat.com ] Clone Set: nfs-grace-clone [nfs-grace] Stopped: [ dhcp42-114.lab.eng.blr.redhat.com dhcp42-117.lab.eng.blr.redhat.com dhcp42-119.lab.eng.blr.redhat.com dhcp42-125.lab.eng.blr.redhat.com dhcp42-127.lab.eng.blr.redhat.com dhcp42-129.lab.eng.blr.redhat.com ] Resource Group: dhcp42-117.lab.eng.blr.redhat.com-group dhcp42-117.lab.eng.blr.redhat.com-nfs_block (ocf::heartbeat:portblock): Stopped dhcp42-117.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Stopped dhcp42-117.lab.eng.blr.redhat.com-nfs_unblock (ocf::heartbeat:portblock): Stopped Resource Group: dhcp42-119.lab.eng.blr.redhat.com-group dhcp42-119.lab.eng.blr.redhat.com-nfs_block (ocf::heartbeat:portblock): Stopped dhcp42-119.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Stopped dhcp42-119.lab.eng.blr.redhat.com-nfs_unblock (ocf::heartbeat:portblock): Stopped Resource Group: dhcp42-125.lab.eng.blr.redhat.com-group dhcp42-125.lab.eng.blr.redhat.com-nfs_block (ocf::heartbeat:portblock): Stopped dhcp42-125.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Stopped dhcp42-125.lab.eng.blr.redhat.com-nfs_unblock (ocf::heartbeat:portblock): Stopped Resource Group: dhcp42-127.lab.eng.blr.redhat.com-group dhcp42-127.lab.eng.blr.redhat.com-nfs_block (ocf::heartbeat:portblock): Stopped dhcp42-127.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Stopped dhcp42-127.lab.eng.blr.redhat.com-nfs_unblock (ocf::heartbeat:portblock): Stopped Resource Group: dhcp42-129.lab.eng.blr.redhat.com-group dhcp42-129.lab.eng.blr.redhat.com-nfs_block (ocf::heartbeat:portblock): Stopped dhcp42-129.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Stopped dhcp42-129.lab.eng.blr.redhat.com-nfs_unblock (ocf::heartbeat:portblock): Stopped Resource Group: dhcp42-114.lab.eng.blr.redhat.com-group dhcp42-114.lab.eng.blr.redhat.com-nfs_block (ocf::heartbeat:portblock): Stopped dhcp42-114.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Stopped dhcp42-114.lab.eng.blr.redhat.com-nfs_unblock (ocf::heartbeat:portblock): Stopped Daemon Status: corosync: active/disabled pacemaker: active/enabled pcsd: active/enabled ====================================
The issue is even observed in Rhel 7.3 and RHGS 3.2 # cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.3 (Maipo) # rpm -qa | grep ganesha nfs-ganesha-2.4.1-11.el7rhgs.x86_64 nfs-ganesha-gluster-2.4.1-11.el7rhgs.x86_64 glusterfs-ganesha-3.8.4-18.4.el7rhgs.x86_64
fix is in rhel7.5