Bug 1464021
| Summary: | [GANESHA] Deleting a node from cluster causing all the existing node to stopped state in pcs | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Manisha Saini <msaini> |
| Component: | nfs-ganesha | Assignee: | Kaleb KEITHLEY <kkeithle> |
| Status: | CLOSED NOTABUG | QA Contact: | Manisha Saini <msaini> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | urgent | ||
| Version: | rhgs-3.3 | CC: | jthottan, kkeithle, msaini, rhs-bugs, skoduri, storage-qa-internal |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-06-23 06:39:29 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
sed: can't read /var/run/gluster/shared_storage/nfs-ganesha/exports/ganesha-ha.conf: No such file or directory Warning: Permanently added 'dhcp42-114.lab.eng.blr.redhat.com' (ECDSA) to the list of known hosts. grep: /var/run/gluster/shared_storage/nfs-ganesha/exports/ganesha-ha.conf: No such file or directory grep: /var/run/gluster/shared_storage/nfs-ganesha/exports/ganesha-ha.conf: No such file or directory cp: cannot stat ‘/var/run/gluster/shared_storage/nfs-ganesha/exports/ganesha.conf’: No such file or directory sed: can't read /etc/ganesha/ganesha.conf: No such file or directory ^^^^^ Above messages points to an issue with accessing shared_storage. @Manisha, Could you please check if the shared_storage mount point is accessible on the node from which you are performing the delete-node operation? Soumya, Yes the shared_storage is mounted and accessiable on all the nodes including the one from which the delete node was triggered [root@dhcp42-127 ~]# /usr/libexec/ganesha/ganesha-ha.sh --delete /var/run/gluster/shared_storage/nfs-ganesha/exports/ The path provided here is incorrect. It should be #/usr/libexec/ganesha/ganesha-ha.sh --delete /var/run/gluster/shared_storage/nfs-ganesha/ Note the '/exports' at the end is not needed. Could you please retry post correcting the input path.
It worked without giving export
/usr/libexec/ganesha/ganesha-ha.sh --delete /var/run/gluster/shared_storage/nfs-ganesha/ dhcp42-127.lab.eng.blr.redhat.com
[root@dhcp42-129 ~]# pcs status
Cluster name: ganesha-ha-360
Stack: corosync
Current DC: dhcp42-119.lab.eng.blr.redhat.com (version 1.1.16-8.el7-94ff4df) - partition with quorum
Last updated: Fri Jun 23 12:08:38 2017
Last change: Fri Jun 23 12:06:27 2017 by root via crm_node on dhcp42-125.lab.eng.blr.redhat.com
3 nodes configured
18 resources configured
Online: [ dhcp42-119.lab.eng.blr.redhat.com dhcp42-125.lab.eng.blr.redhat.com dhcp42-129.lab.eng.blr.redhat.com ]
Full list of resources:
Clone Set: nfs_setup-clone [nfs_setup]
Started: [ dhcp42-119.lab.eng.blr.redhat.com dhcp42-125.lab.eng.blr.redhat.com dhcp42-129.lab.eng.blr.redhat.com ]
Clone Set: nfs-mon-clone [nfs-mon]
Started: [ dhcp42-119.lab.eng.blr.redhat.com dhcp42-125.lab.eng.blr.redhat.com dhcp42-129.lab.eng.blr.redhat.com ]
Clone Set: nfs-grace-clone [nfs-grace]
Started: [ dhcp42-119.lab.eng.blr.redhat.com dhcp42-125.lab.eng.blr.redhat.com dhcp42-129.lab.eng.blr.redhat.com ]
Resource Group: dhcp42-119.lab.eng.blr.redhat.com-group
dhcp42-119.lab.eng.blr.redhat.com-nfs_block (ocf::heartbeat:portblock): Started dhcp42-119.lab.eng.blr.redhat.com
dhcp42-119.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Started dhcp42-119.lab.eng.blr.redhat.com
dhcp42-119.lab.eng.blr.redhat.com-nfs_unblock (ocf::heartbeat:portblock): Started dhcp42-119.lab.eng.blr.redhat.com
Resource Group: dhcp42-125.lab.eng.blr.redhat.com-group
dhcp42-125.lab.eng.blr.redhat.com-nfs_block (ocf::heartbeat:portblock): Started dhcp42-125.lab.eng.blr.redhat.com
dhcp42-125.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Started dhcp42-125.lab.eng.blr.redhat.com
dhcp42-125.lab.eng.blr.redhat.com-nfs_unblock (ocf::heartbeat:portblock): Started dhcp42-125.lab.eng.blr.redhat.com
Resource Group: dhcp42-129.lab.eng.blr.redhat.com-group
dhcp42-129.lab.eng.blr.redhat.com-nfs_block (ocf::heartbeat:portblock): Started dhcp42-129.lab.eng.blr.redhat.com
dhcp42-129.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Started dhcp42-129.lab.eng.blr.redhat.com
dhcp42-129.lab.eng.blr.redhat.com-nfs_unblock (ocf::heartbeat:portblock): Started dhcp42-129.lab.eng.blr.redhat.com
Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled
Closing this bug
|
Description of problem: Deleting a node from cluster causing all the existing node to stopped state in pcs Version-Release number of selected component (if applicable): # rpm -qa | grep ganesha nfs-ganesha-2.4.4-8.el7rhgs.x86_64 nfs-ganesha-debuginfo-2.4.4-8.el7rhgs.x86_64 glusterfs-ganesha-3.8.4-29.el7rhgs.x86_64 nfs-ganesha-gluster-2.4.4-8.el7rhgs.x86_64 How reproducible: 2/2 times Steps to Reproduce: 1.Create a 4 node ganesha cluster. 2.Add 2 more nodes to existing 4 node cluster 3.Delete 1 node out of 6 node Actual results: After deleting a node from cluster remaining all nodes goes to stopped state in pcs Expected results: Deleting the node should not impact the existing node.REmaining node should not go to stopped state Additional info: Delete Node- ======= [root@dhcp42-127 ~]# /usr/libexec/ganesha/ganesha-ha.sh --delete /var/run/gluster/shared_storage/nfs-ganesha/exports/ dhcp42-114.lab.eng.blr.redhat.com grep: /var/run/gluster/shared_storage/nfs-ganesha/exports/ganesha-ha.conf: No such file or directory grep: /var/run/gluster/shared_storage/nfs-ganesha/exports/ganesha-ha.conf: No such file or directory Removing group: dhcp42-114.lab.eng.blr.redhat.com-group (and all resources within group) Stopping all resources in group: dhcp42-114.lab.eng.blr.redhat.com-group... Deleting Resource - dhcp42-114.lab.eng.blr.redhat.com-nfs_block Removing Constraint - order-nfs-grace-clone-dhcp42-114.lab.eng.blr.redhat.com-cluster_ip-1-mandatory Deleting Resource - dhcp42-114.lab.eng.blr.redhat.com-cluster_ip-1 Removing Constraint - location-dhcp42-114.lab.eng.blr.redhat.com-group Removing Constraint - location-dhcp42-114.lab.eng.blr.redhat.com-group-dhcp42-117.lab.eng.blr.redhat.com-1000 Removing Constraint - location-dhcp42-114.lab.eng.blr.redhat.com-group-dhcp42-119.lab.eng.blr.redhat.com-2000 Removing Constraint - location-dhcp42-114.lab.eng.blr.redhat.com-group-dhcp42-125.lab.eng.blr.redhat.com-3000 Removing Constraint - location-dhcp42-114.lab.eng.blr.redhat.com-group-dhcp42-127.lab.eng.blr.redhat.com-4000 Removing Constraint - location-dhcp42-114.lab.eng.blr.redhat.com-group-dhcp42-129.lab.eng.blr.redhat.com-5000 Removing Constraint - location-dhcp42-114.lab.eng.blr.redhat.com-group-dhcp42-114.lab.eng.blr.redhat.com-6000 Deleting Resource (and group) - dhcp42-114.lab.eng.blr.redhat.com-nfs_unblock Removing group: dhcp42-117.lab.eng.blr.redhat.com-group (and all resources within group) Stopping all resources in group: dhcp42-117.lab.eng.blr.redhat.com-group... Deleting Resource - dhcp42-117.lab.eng.blr.redhat.com-nfs_block Removing Constraint - order-nfs-grace-clone-dhcp42-117.lab.eng.blr.redhat.com-cluster_ip-1-mandatory Deleting Resource - dhcp42-117.lab.eng.blr.redhat.com-cluster_ip-1 Removing Constraint - location-dhcp42-117.lab.eng.blr.redhat.com-group Removing Constraint - location-dhcp42-117.lab.eng.blr.redhat.com-group-dhcp42-119.lab.eng.blr.redhat.com-1000 Removing Constraint - location-dhcp42-117.lab.eng.blr.redhat.com-group-dhcp42-125.lab.eng.blr.redhat.com-2000 Removing Constraint - location-dhcp42-117.lab.eng.blr.redhat.com-group-dhcp42-127.lab.eng.blr.redhat.com-3000 Removing Constraint - location-dhcp42-117.lab.eng.blr.redhat.com-group-dhcp42-129.lab.eng.blr.redhat.com-4000 Removing Constraint - location-dhcp42-117.lab.eng.blr.redhat.com-group-dhcp42-114.lab.eng.blr.redhat.com-5000 Removing Constraint - location-dhcp42-117.lab.eng.blr.redhat.com-group-dhcp42-117.lab.eng.blr.redhat.com-6000 Deleting Resource (and group) - dhcp42-117.lab.eng.blr.redhat.com-nfs_unblock Removing group: dhcp42-119.lab.eng.blr.redhat.com-group (and all resources within group) Stopping all resources in group: dhcp42-119.lab.eng.blr.redhat.com-group... Deleting Resource - dhcp42-119.lab.eng.blr.redhat.com-nfs_block Removing Constraint - order-nfs-grace-clone-dhcp42-119.lab.eng.blr.redhat.com-cluster_ip-1-mandatory Deleting Resource - dhcp42-119.lab.eng.blr.redhat.com-cluster_ip-1 Removing Constraint - location-dhcp42-119.lab.eng.blr.redhat.com-group Removing Constraint - location-dhcp42-119.lab.eng.blr.redhat.com-group-dhcp42-125.lab.eng.blr.redhat.com-1000 Removing Constraint - location-dhcp42-119.lab.eng.blr.redhat.com-group-dhcp42-127.lab.eng.blr.redhat.com-2000 Removing Constraint - location-dhcp42-119.lab.eng.blr.redhat.com-group-dhcp42-129.lab.eng.blr.redhat.com-3000 Removing Constraint - location-dhcp42-119.lab.eng.blr.redhat.com-group-dhcp42-114.lab.eng.blr.redhat.com-4000 Removing Constraint - location-dhcp42-119.lab.eng.blr.redhat.com-group-dhcp42-117.lab.eng.blr.redhat.com-5000 Removing Constraint - location-dhcp42-119.lab.eng.blr.redhat.com-group-dhcp42-119.lab.eng.blr.redhat.com-6000 Deleting Resource (and group) - dhcp42-119.lab.eng.blr.redhat.com-nfs_unblock Removing group: dhcp42-125.lab.eng.blr.redhat.com-group (and all resources within group) Stopping all resources in group: dhcp42-125.lab.eng.blr.redhat.com-group... Deleting Resource - dhcp42-125.lab.eng.blr.redhat.com-nfs_block Removing Constraint - order-nfs-grace-clone-dhcp42-125.lab.eng.blr.redhat.com-cluster_ip-1-mandatory Deleting Resource - dhcp42-125.lab.eng.blr.redhat.com-cluster_ip-1 Removing Constraint - location-dhcp42-125.lab.eng.blr.redhat.com-group Removing Constraint - location-dhcp42-125.lab.eng.blr.redhat.com-group-dhcp42-127.lab.eng.blr.redhat.com-1000 Removing Constraint - location-dhcp42-125.lab.eng.blr.redhat.com-group-dhcp42-129.lab.eng.blr.redhat.com-2000 Removing Constraint - location-dhcp42-125.lab.eng.blr.redhat.com-group-dhcp42-114.lab.eng.blr.redhat.com-3000 Removing Constraint - location-dhcp42-125.lab.eng.blr.redhat.com-group-dhcp42-117.lab.eng.blr.redhat.com-4000 Removing Constraint - location-dhcp42-125.lab.eng.blr.redhat.com-group-dhcp42-119.lab.eng.blr.redhat.com-5000 Removing Constraint - location-dhcp42-125.lab.eng.blr.redhat.com-group-dhcp42-125.lab.eng.blr.redhat.com-6000 Deleting Resource (and group) - dhcp42-125.lab.eng.blr.redhat.com-nfs_unblock Removing group: dhcp42-127.lab.eng.blr.redhat.com-group (and all resources within group) Stopping all resources in group: dhcp42-127.lab.eng.blr.redhat.com-group... Deleting Resource - dhcp42-127.lab.eng.blr.redhat.com-nfs_block Removing Constraint - order-nfs-grace-clone-dhcp42-127.lab.eng.blr.redhat.com-cluster_ip-1-mandatory Deleting Resource - dhcp42-127.lab.eng.blr.redhat.com-cluster_ip-1 Removing Constraint - location-dhcp42-127.lab.eng.blr.redhat.com-group Removing Constraint - location-dhcp42-127.lab.eng.blr.redhat.com-group-dhcp42-129.lab.eng.blr.redhat.com-1000 Removing Constraint - location-dhcp42-127.lab.eng.blr.redhat.com-group-dhcp42-114.lab.eng.blr.redhat.com-2000 Removing Constraint - location-dhcp42-127.lab.eng.blr.redhat.com-group-dhcp42-117.lab.eng.blr.redhat.com-3000 Removing Constraint - location-dhcp42-127.lab.eng.blr.redhat.com-group-dhcp42-119.lab.eng.blr.redhat.com-4000 Removing Constraint - location-dhcp42-127.lab.eng.blr.redhat.com-group-dhcp42-125.lab.eng.blr.redhat.com-5000 Removing Constraint - location-dhcp42-127.lab.eng.blr.redhat.com-group-dhcp42-127.lab.eng.blr.redhat.com-6000 Deleting Resource (and group) - dhcp42-127.lab.eng.blr.redhat.com-nfs_unblock Removing group: dhcp42-129.lab.eng.blr.redhat.com-group (and all resources within group) Stopping all resources in group: dhcp42-129.lab.eng.blr.redhat.com-group... Deleting Resource - dhcp42-129.lab.eng.blr.redhat.com-nfs_block Removing Constraint - order-nfs-grace-clone-dhcp42-129.lab.eng.blr.redhat.com-cluster_ip-1-mandatory Deleting Resource - dhcp42-129.lab.eng.blr.redhat.com-cluster_ip-1 Removing Constraint - location-dhcp42-129.lab.eng.blr.redhat.com-group Removing Constraint - location-dhcp42-129.lab.eng.blr.redhat.com-group-dhcp42-114.lab.eng.blr.redhat.com-1000 Removing Constraint - location-dhcp42-129.lab.eng.blr.redhat.com-group-dhcp42-117.lab.eng.blr.redhat.com-2000 Removing Constraint - location-dhcp42-129.lab.eng.blr.redhat.com-group-dhcp42-119.lab.eng.blr.redhat.com-3000 Removing Constraint - location-dhcp42-129.lab.eng.blr.redhat.com-group-dhcp42-125.lab.eng.blr.redhat.com-4000 Removing Constraint - location-dhcp42-129.lab.eng.blr.redhat.com-group-dhcp42-127.lab.eng.blr.redhat.com-5000 Removing Constraint - location-dhcp42-129.lab.eng.blr.redhat.com-group-dhcp42-129.lab.eng.blr.redhat.com-6000 Deleting Resource (and group) - dhcp42-129.lab.eng.blr.redhat.com-nfs_unblock grep: /var/run/gluster/shared_storage/nfs-ganesha/exports/ganesha-ha.conf: No such file or directory Adding nfs-grace-clone dhcp42-117.lab.eng.blr.redhat.com-cluster_ip-1 (kind: Mandatory) (Options: first-action=start then-action=start) grep: /var/run/gluster/shared_storage/nfs-ganesha/exports/ganesha-ha.conf: No such file or directory Adding nfs-grace-clone dhcp42-119.lab.eng.blr.redhat.com-cluster_ip-1 (kind: Mandatory) (Options: first-action=start then-action=start) grep: /var/run/gluster/shared_storage/nfs-ganesha/exports/ganesha-ha.conf: No such file or directory Adding nfs-grace-clone dhcp42-125.lab.eng.blr.redhat.com-cluster_ip-1 (kind: Mandatory) (Options: first-action=start then-action=start) grep: /var/run/gluster/shared_storage/nfs-ganesha/exports/ganesha-ha.conf: No such file or directory Adding nfs-grace-clone dhcp42-127.lab.eng.blr.redhat.com-cluster_ip-1 (kind: Mandatory) (Options: first-action=start then-action=start) grep: /var/run/gluster/shared_storage/nfs-ganesha/exports/ganesha-ha.conf: No such file or directory Adding nfs-grace-clone dhcp42-129.lab.eng.blr.redhat.com-cluster_ip-1 (kind: Mandatory) (Options: first-action=start then-action=start) CIB updated dhcp42-114.lab.eng.blr.redhat.com: Stopping Cluster (pacemaker)... dhcp42-114.lab.eng.blr.redhat.com: Successfully destroyed cluster dhcp42-125.lab.eng.blr.redhat.com: Corosync updated dhcp42-127.lab.eng.blr.redhat.com: Corosync updated dhcp42-129.lab.eng.blr.redhat.com: Corosync updated dhcp42-119.lab.eng.blr.redhat.com: Corosync updated dhcp42-117.lab.eng.blr.redhat.com: Corosync updated sed: can't read /var/run/gluster/shared_storage/nfs-ganesha/exports/ganesha-ha.conf: No such file or directory Warning: Permanently added 'dhcp42-114.lab.eng.blr.redhat.com' (ECDSA) to the list of known hosts. grep: /var/run/gluster/shared_storage/nfs-ganesha/exports/ganesha-ha.conf: No such file or directory grep: /var/run/gluster/shared_storage/nfs-ganesha/exports/ganesha-ha.conf: No such file or directory cp: cannot stat ‘/var/run/gluster/shared_storage/nfs-ganesha/exports/ganesha.conf’: No such file or directory sed: can't read /etc/ganesha/ganesha.conf: No such file or directory =================== pcs status =========== [root@dhcp42-119 ~]# pcs status Cluster name: ganesha-ha-360 Stack: corosync Current DC: dhcp42-119.lab.eng.blr.redhat.com (version 1.1.16-8.el7-94ff4df) - partition with quorum Last updated: Thu Jun 22 15:18:03 2017 Last change: Thu Jun 22 15:00:52 2017 by root via crm_node on dhcp42-127.lab.eng.blr.redhat.com 5 nodes configured 30 resources configured Online: [ dhcp42-117.lab.eng.blr.redhat.com dhcp42-119.lab.eng.blr.redhat.com dhcp42-125.lab.eng.blr.redhat.com dhcp42-127.lab.eng.blr.redhat.com dhcp42-129.lab.eng.blr.redhat.com ] Full list of resources: Clone Set: nfs_setup-clone [nfs_setup] Started: [ dhcp42-117.lab.eng.blr.redhat.com dhcp42-119.lab.eng.blr.redhat.com dhcp42-125.lab.eng.blr.redhat.com dhcp42-127.lab.eng.blr.redhat.com dhcp42-129.lab.eng.blr.redhat.com ] Clone Set: nfs-mon-clone [nfs-mon] Started: [ dhcp42-117.lab.eng.blr.redhat.com dhcp42-119.lab.eng.blr.redhat.com dhcp42-125.lab.eng.blr.redhat.com dhcp42-127.lab.eng.blr.redhat.com dhcp42-129.lab.eng.blr.redhat.com ] Clone Set: nfs-grace-clone [nfs-grace] Started: [ dhcp42-117.lab.eng.blr.redhat.com dhcp42-119.lab.eng.blr.redhat.com dhcp42-125.lab.eng.blr.redhat.com dhcp42-127.lab.eng.blr.redhat.com dhcp42-129.lab.eng.blr.redhat.com ] Resource Group: dhcp42-117.lab.eng.blr.redhat.com-group dhcp42-117.lab.eng.blr.redhat.com-nfs_block (ocf::heartbeat:portblock): Stopped dhcp42-117.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Stopped dhcp42-117.lab.eng.blr.redhat.com-nfs_unblock (ocf::heartbeat:portblock): Stopped Resource Group: dhcp42-119.lab.eng.blr.redhat.com-group dhcp42-119.lab.eng.blr.redhat.com-nfs_block (ocf::heartbeat:portblock): Stopped dhcp42-119.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Stopped dhcp42-119.lab.eng.blr.redhat.com-nfs_unblock (ocf::heartbeat:portblock): Stopped Resource Group: dhcp42-125.lab.eng.blr.redhat.com-group dhcp42-125.lab.eng.blr.redhat.com-nfs_block (ocf::heartbeat:portblock): Stopped dhcp42-125.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Stopped dhcp42-125.lab.eng.blr.redhat.com-nfs_unblock (ocf::heartbeat:portblock): Stopped Resource Group: dhcp42-127.lab.eng.blr.redhat.com-group dhcp42-127.lab.eng.blr.redhat.com-nfs_block (ocf::heartbeat:portblock): Stopped dhcp42-127.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Stopped dhcp42-127.lab.eng.blr.redhat.com-nfs_unblock (ocf::heartbeat:portblock): Stopped Resource Group: dhcp42-129.lab.eng.blr.redhat.com-group dhcp42-129.lab.eng.blr.redhat.com-nfs_block (ocf::heartbeat:portblock): Stopped dhcp42-129.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Stopped dhcp42-129.lab.eng.blr.redhat.com-nfs_unblock (ocf::heartbeat:portblock): Stopped Failed Actions: * dhcp42-119.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-129.lab.eng.blr.redhat.com 'unknown error' (1): call=112, status=complete, exitreason='none', last-rc-change='Thu Jun 22 15:00:45 2017', queued=0ms, exec=59ms * dhcp42-125.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-129.lab.eng.blr.redhat.com 'unknown error' (1): call=115, status=complete, exitreason='none', last-rc-change='Thu Jun 22 15:00:45 2017', queued=0ms, exec=36ms * dhcp42-117.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-129.lab.eng.blr.redhat.com 'unknown error' (1): call=109, status=complete, exitreason='none', last-rc-change='Thu Jun 22 15:00:45 2017', queued=0ms, exec=41ms * dhcp42-127.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-129.lab.eng.blr.redhat.com 'unknown error' (1): call=117, status=complete, exitreason='none', last-rc-change='Thu Jun 22 15:00:46 2017', queued=0ms, exec=42ms * dhcp42-129.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-129.lab.eng.blr.redhat.com 'unknown error' (1): call=107, status=complete, exitreason='none', last-rc-change='Thu Jun 22 15:00:44 2017', queued=0ms, exec=38ms * dhcp42-119.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-127.lab.eng.blr.redhat.com 'unknown error' (1): call=128, status=complete, exitreason='none', last-rc-change='Thu Jun 22 15:00:45 2017', queued=0ms, exec=61ms * dhcp42-125.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-127.lab.eng.blr.redhat.com 'unknown error' (1): call=130, status=complete, exitreason='none', last-rc-change='Thu Jun 22 15:00:46 2017', queued=0ms, exec=53ms * dhcp42-117.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-127.lab.eng.blr.redhat.com 'unknown error' (1): call=125, status=complete, exitreason='none', last-rc-change='Thu Jun 22 15:00:45 2017', queued=0ms, exec=56ms * dhcp42-127.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-127.lab.eng.blr.redhat.com 'unknown error' (1): call=120, status=complete, exitreason='none', last-rc-change='Thu Jun 22 15:00:44 2017', queued=0ms, exec=259ms * dhcp42-129.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-127.lab.eng.blr.redhat.com 'unknown error' (1): call=122, status=complete, exitreason='none', last-rc-change='Thu Jun 22 15:00:45 2017', queued=0ms, exec=44ms * dhcp42-119.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-117.lab.eng.blr.redhat.com 'unknown error' (1): call=230, status=complete, exitreason='none', last-rc-change='Thu Jun 22 15:00:45 2017', queued=0ms, exec=40ms * dhcp42-117.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-117.lab.eng.blr.redhat.com 'unknown error' (1): call=228, status=complete, exitreason='none', last-rc-change='Thu Jun 22 15:00:44 2017', queued=0ms, exec=48ms * dhcp42-125.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-117.lab.eng.blr.redhat.com 'unknown error' (1): call=233, status=complete, exitreason='none', last-rc-change='Thu Jun 22 15:00:45 2017', queued=0ms, exec=83ms * dhcp42-127.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-117.lab.eng.blr.redhat.com 'unknown error' (1): call=236, status=complete, exitreason='none', last-rc-change='Thu Jun 22 15:00:45 2017', queued=0ms, exec=37ms * dhcp42-129.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-117.lab.eng.blr.redhat.com 'unknown error' (1): call=238, status=complete, exitreason='none', last-rc-change='Thu Jun 22 15:00:46 2017', queued=0ms, exec=53ms * dhcp42-119.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-119.lab.eng.blr.redhat.com 'unknown error' (1): call=346, status=complete, exitreason='none', last-rc-change='Thu Jun 22 15:00:44 2017', queued=0ms, exec=42ms * dhcp42-117.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-119.lab.eng.blr.redhat.com 'unknown error' (1): call=356, status=complete, exitreason='none', last-rc-change='Thu Jun 22 15:00:46 2017', queued=1ms, exec=62ms * dhcp42-125.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-119.lab.eng.blr.redhat.com 'unknown error' (1): call=348, status=complete, exitreason='none', last-rc-change='Thu Jun 22 15:00:45 2017', queued=0ms, exec=38ms * dhcp42-127.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-119.lab.eng.blr.redhat.com 'unknown error' (1): call=351, status=complete, exitreason='none', last-rc-change='Thu Jun 22 15:00:45 2017', queued=0ms, exec=44ms * dhcp42-129.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-119.lab.eng.blr.redhat.com 'unknown error' (1): call=354, status=complete, exitreason='none', last-rc-change='Thu Jun 22 15:00:45 2017', queued=0ms, exec=34ms * dhcp42-119.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-125.lab.eng.blr.redhat.com 'unknown error' (1): call=207, status=complete, exitreason='none', last-rc-change='Thu Jun 22 15:00:46 2017', queued=0ms, exec=56ms * dhcp42-125.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-125.lab.eng.blr.redhat.com 'unknown error' (1): call=197, status=complete, exitreason='none', last-rc-change='Thu Jun 22 15:00:44 2017', queued=0ms, exec=41ms * dhcp42-117.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-125.lab.eng.blr.redhat.com 'unknown error' (1): call=205, status=complete, exitreason='none', last-rc-change='Thu Jun 22 15:00:46 2017', queued=0ms, exec=47ms * dhcp42-127.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-125.lab.eng.blr.redhat.com 'unknown error' (1): call=199, status=complete, exitreason='none', last-rc-change='Thu Jun 22 15:00:45 2017', queued=0ms, exec=40ms * dhcp42-129.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-125.lab.eng.blr.redhat.com 'unknown error' (1): call=202, status=complete, exitreason='none', last-rc-change='Thu Jun 22 15:00:45 2017', queued=0ms, exec=43ms Daemon Status: corosync: active/disabled pacemaker: active/disabled pcsd: active/enabled ================== [root@dhcp42-125 ~]# rpm -qa | grep pacemaker pacemaker-libs-1.1.16-8.el7.x86_64 pacemaker-cli-1.1.16-8.el7.x86_64 pacemaker-cluster-libs-1.1.16-8.el7.x86_64 pacemaker-1.1.16-8.el7.x86_64 [root@dhcp42-125 ~]# rpm -qa | grep pcs pcs-0.9.157-1.el7.x86_64 [root@dhcp42-125 ~]# rpm -qa | grep corosync corosync-2.4.0-9.el7.x86_64 corosynclib-2.4.0-9.el7.x86_64 [root@dhcp42-125 ~]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.4 Beta (Maipo) Attaching sosreports shortly