Bug 1464021 - [GANESHA] Deleting a node from cluster causing all the existing node to stopped state in pcs
[GANESHA] Deleting a node from cluster causing all the existing node to stopp...
Status: CLOSED NOTABUG
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: nfs-ganesha (Show other bugs)
3.3
Unspecified Unspecified
urgent Severity unspecified
: ---
: ---
Assigned To: Kaleb KEITHLEY
Manisha Saini
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-22 05:54 EDT by Manisha Saini
Modified: 2017-06-23 02:39 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-06-23 02:39:29 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Manisha Saini 2017-06-22 05:54:16 EDT
Description of problem:
Deleting a node from cluster causing all the existing node to stopped state in pcs 

Version-Release number of selected component (if applicable):
# rpm -qa | grep ganesha
nfs-ganesha-2.4.4-8.el7rhgs.x86_64
nfs-ganesha-debuginfo-2.4.4-8.el7rhgs.x86_64
glusterfs-ganesha-3.8.4-29.el7rhgs.x86_64
nfs-ganesha-gluster-2.4.4-8.el7rhgs.x86_64


How reproducible:
2/2 times

Steps to Reproduce:
1.Create a 4 node ganesha cluster.
2.Add 2 more nodes to existing 4 node cluster
3.Delete 1 node out of 6 node

Actual results:
After deleting a node from cluster remaining all nodes goes to stopped state in pcs

Expected results:
Deleting the node should not impact the existing node.REmaining node should not go to stopped state

Additional info:

Delete Node-
=======
[root@dhcp42-127 ~]# /usr/libexec/ganesha/ganesha-ha.sh --delete /var/run/gluster/shared_storage/nfs-ganesha/exports/ dhcp42-114.lab.eng.blr.redhat.com
grep: /var/run/gluster/shared_storage/nfs-ganesha/exports/ganesha-ha.conf: No such file or directory
grep: /var/run/gluster/shared_storage/nfs-ganesha/exports/ganesha-ha.conf: No such file or directory
Removing group: dhcp42-114.lab.eng.blr.redhat.com-group (and all resources within group)
Stopping all resources in group: dhcp42-114.lab.eng.blr.redhat.com-group...
Deleting Resource - dhcp42-114.lab.eng.blr.redhat.com-nfs_block
Removing Constraint - order-nfs-grace-clone-dhcp42-114.lab.eng.blr.redhat.com-cluster_ip-1-mandatory
Deleting Resource - dhcp42-114.lab.eng.blr.redhat.com-cluster_ip-1
Removing Constraint - location-dhcp42-114.lab.eng.blr.redhat.com-group
Removing Constraint - location-dhcp42-114.lab.eng.blr.redhat.com-group-dhcp42-117.lab.eng.blr.redhat.com-1000
Removing Constraint - location-dhcp42-114.lab.eng.blr.redhat.com-group-dhcp42-119.lab.eng.blr.redhat.com-2000
Removing Constraint - location-dhcp42-114.lab.eng.blr.redhat.com-group-dhcp42-125.lab.eng.blr.redhat.com-3000
Removing Constraint - location-dhcp42-114.lab.eng.blr.redhat.com-group-dhcp42-127.lab.eng.blr.redhat.com-4000
Removing Constraint - location-dhcp42-114.lab.eng.blr.redhat.com-group-dhcp42-129.lab.eng.blr.redhat.com-5000
Removing Constraint - location-dhcp42-114.lab.eng.blr.redhat.com-group-dhcp42-114.lab.eng.blr.redhat.com-6000
Deleting Resource (and group) - dhcp42-114.lab.eng.blr.redhat.com-nfs_unblock
Removing group: dhcp42-117.lab.eng.blr.redhat.com-group (and all resources within group)
Stopping all resources in group: dhcp42-117.lab.eng.blr.redhat.com-group...
Deleting Resource - dhcp42-117.lab.eng.blr.redhat.com-nfs_block
Removing Constraint - order-nfs-grace-clone-dhcp42-117.lab.eng.blr.redhat.com-cluster_ip-1-mandatory
Deleting Resource - dhcp42-117.lab.eng.blr.redhat.com-cluster_ip-1
Removing Constraint - location-dhcp42-117.lab.eng.blr.redhat.com-group
Removing Constraint - location-dhcp42-117.lab.eng.blr.redhat.com-group-dhcp42-119.lab.eng.blr.redhat.com-1000
Removing Constraint - location-dhcp42-117.lab.eng.blr.redhat.com-group-dhcp42-125.lab.eng.blr.redhat.com-2000
Removing Constraint - location-dhcp42-117.lab.eng.blr.redhat.com-group-dhcp42-127.lab.eng.blr.redhat.com-3000
Removing Constraint - location-dhcp42-117.lab.eng.blr.redhat.com-group-dhcp42-129.lab.eng.blr.redhat.com-4000
Removing Constraint - location-dhcp42-117.lab.eng.blr.redhat.com-group-dhcp42-114.lab.eng.blr.redhat.com-5000
Removing Constraint - location-dhcp42-117.lab.eng.blr.redhat.com-group-dhcp42-117.lab.eng.blr.redhat.com-6000
Deleting Resource (and group) - dhcp42-117.lab.eng.blr.redhat.com-nfs_unblock
Removing group: dhcp42-119.lab.eng.blr.redhat.com-group (and all resources within group)
Stopping all resources in group: dhcp42-119.lab.eng.blr.redhat.com-group...
Deleting Resource - dhcp42-119.lab.eng.blr.redhat.com-nfs_block
Removing Constraint - order-nfs-grace-clone-dhcp42-119.lab.eng.blr.redhat.com-cluster_ip-1-mandatory
Deleting Resource - dhcp42-119.lab.eng.blr.redhat.com-cluster_ip-1
Removing Constraint - location-dhcp42-119.lab.eng.blr.redhat.com-group
Removing Constraint - location-dhcp42-119.lab.eng.blr.redhat.com-group-dhcp42-125.lab.eng.blr.redhat.com-1000
Removing Constraint - location-dhcp42-119.lab.eng.blr.redhat.com-group-dhcp42-127.lab.eng.blr.redhat.com-2000
Removing Constraint - location-dhcp42-119.lab.eng.blr.redhat.com-group-dhcp42-129.lab.eng.blr.redhat.com-3000
Removing Constraint - location-dhcp42-119.lab.eng.blr.redhat.com-group-dhcp42-114.lab.eng.blr.redhat.com-4000
Removing Constraint - location-dhcp42-119.lab.eng.blr.redhat.com-group-dhcp42-117.lab.eng.blr.redhat.com-5000
Removing Constraint - location-dhcp42-119.lab.eng.blr.redhat.com-group-dhcp42-119.lab.eng.blr.redhat.com-6000
Deleting Resource (and group) - dhcp42-119.lab.eng.blr.redhat.com-nfs_unblock
Removing group: dhcp42-125.lab.eng.blr.redhat.com-group (and all resources within group)
Stopping all resources in group: dhcp42-125.lab.eng.blr.redhat.com-group...
Deleting Resource - dhcp42-125.lab.eng.blr.redhat.com-nfs_block
Removing Constraint - order-nfs-grace-clone-dhcp42-125.lab.eng.blr.redhat.com-cluster_ip-1-mandatory
Deleting Resource - dhcp42-125.lab.eng.blr.redhat.com-cluster_ip-1
Removing Constraint - location-dhcp42-125.lab.eng.blr.redhat.com-group
Removing Constraint - location-dhcp42-125.lab.eng.blr.redhat.com-group-dhcp42-127.lab.eng.blr.redhat.com-1000
Removing Constraint - location-dhcp42-125.lab.eng.blr.redhat.com-group-dhcp42-129.lab.eng.blr.redhat.com-2000
Removing Constraint - location-dhcp42-125.lab.eng.blr.redhat.com-group-dhcp42-114.lab.eng.blr.redhat.com-3000
Removing Constraint - location-dhcp42-125.lab.eng.blr.redhat.com-group-dhcp42-117.lab.eng.blr.redhat.com-4000
Removing Constraint - location-dhcp42-125.lab.eng.blr.redhat.com-group-dhcp42-119.lab.eng.blr.redhat.com-5000
Removing Constraint - location-dhcp42-125.lab.eng.blr.redhat.com-group-dhcp42-125.lab.eng.blr.redhat.com-6000
Deleting Resource (and group) - dhcp42-125.lab.eng.blr.redhat.com-nfs_unblock
Removing group: dhcp42-127.lab.eng.blr.redhat.com-group (and all resources within group)
Stopping all resources in group: dhcp42-127.lab.eng.blr.redhat.com-group...
Deleting Resource - dhcp42-127.lab.eng.blr.redhat.com-nfs_block
Removing Constraint - order-nfs-grace-clone-dhcp42-127.lab.eng.blr.redhat.com-cluster_ip-1-mandatory
Deleting Resource - dhcp42-127.lab.eng.blr.redhat.com-cluster_ip-1
Removing Constraint - location-dhcp42-127.lab.eng.blr.redhat.com-group
Removing Constraint - location-dhcp42-127.lab.eng.blr.redhat.com-group-dhcp42-129.lab.eng.blr.redhat.com-1000
Removing Constraint - location-dhcp42-127.lab.eng.blr.redhat.com-group-dhcp42-114.lab.eng.blr.redhat.com-2000
Removing Constraint - location-dhcp42-127.lab.eng.blr.redhat.com-group-dhcp42-117.lab.eng.blr.redhat.com-3000
Removing Constraint - location-dhcp42-127.lab.eng.blr.redhat.com-group-dhcp42-119.lab.eng.blr.redhat.com-4000
Removing Constraint - location-dhcp42-127.lab.eng.blr.redhat.com-group-dhcp42-125.lab.eng.blr.redhat.com-5000
Removing Constraint - location-dhcp42-127.lab.eng.blr.redhat.com-group-dhcp42-127.lab.eng.blr.redhat.com-6000
Deleting Resource (and group) - dhcp42-127.lab.eng.blr.redhat.com-nfs_unblock
Removing group: dhcp42-129.lab.eng.blr.redhat.com-group (and all resources within group)
Stopping all resources in group: dhcp42-129.lab.eng.blr.redhat.com-group...
Deleting Resource - dhcp42-129.lab.eng.blr.redhat.com-nfs_block
Removing Constraint - order-nfs-grace-clone-dhcp42-129.lab.eng.blr.redhat.com-cluster_ip-1-mandatory
Deleting Resource - dhcp42-129.lab.eng.blr.redhat.com-cluster_ip-1
Removing Constraint - location-dhcp42-129.lab.eng.blr.redhat.com-group
Removing Constraint - location-dhcp42-129.lab.eng.blr.redhat.com-group-dhcp42-114.lab.eng.blr.redhat.com-1000
Removing Constraint - location-dhcp42-129.lab.eng.blr.redhat.com-group-dhcp42-117.lab.eng.blr.redhat.com-2000
Removing Constraint - location-dhcp42-129.lab.eng.blr.redhat.com-group-dhcp42-119.lab.eng.blr.redhat.com-3000
Removing Constraint - location-dhcp42-129.lab.eng.blr.redhat.com-group-dhcp42-125.lab.eng.blr.redhat.com-4000
Removing Constraint - location-dhcp42-129.lab.eng.blr.redhat.com-group-dhcp42-127.lab.eng.blr.redhat.com-5000
Removing Constraint - location-dhcp42-129.lab.eng.blr.redhat.com-group-dhcp42-129.lab.eng.blr.redhat.com-6000
Deleting Resource (and group) - dhcp42-129.lab.eng.blr.redhat.com-nfs_unblock
grep: /var/run/gluster/shared_storage/nfs-ganesha/exports/ganesha-ha.conf: No such file or directory
Adding nfs-grace-clone dhcp42-117.lab.eng.blr.redhat.com-cluster_ip-1 (kind: Mandatory) (Options: first-action=start then-action=start)
grep: /var/run/gluster/shared_storage/nfs-ganesha/exports/ganesha-ha.conf: No such file or directory
Adding nfs-grace-clone dhcp42-119.lab.eng.blr.redhat.com-cluster_ip-1 (kind: Mandatory) (Options: first-action=start then-action=start)
grep: /var/run/gluster/shared_storage/nfs-ganesha/exports/ganesha-ha.conf: No such file or directory
Adding nfs-grace-clone dhcp42-125.lab.eng.blr.redhat.com-cluster_ip-1 (kind: Mandatory) (Options: first-action=start then-action=start)
grep: /var/run/gluster/shared_storage/nfs-ganesha/exports/ganesha-ha.conf: No such file or directory
Adding nfs-grace-clone dhcp42-127.lab.eng.blr.redhat.com-cluster_ip-1 (kind: Mandatory) (Options: first-action=start then-action=start)
grep: /var/run/gluster/shared_storage/nfs-ganesha/exports/ganesha-ha.conf: No such file or directory
Adding nfs-grace-clone dhcp42-129.lab.eng.blr.redhat.com-cluster_ip-1 (kind: Mandatory) (Options: first-action=start then-action=start)
CIB updated
dhcp42-114.lab.eng.blr.redhat.com: Stopping Cluster (pacemaker)...
dhcp42-114.lab.eng.blr.redhat.com: Successfully destroyed cluster
dhcp42-125.lab.eng.blr.redhat.com: Corosync updated
dhcp42-127.lab.eng.blr.redhat.com: Corosync updated
dhcp42-129.lab.eng.blr.redhat.com: Corosync updated
dhcp42-119.lab.eng.blr.redhat.com: Corosync updated
dhcp42-117.lab.eng.blr.redhat.com: Corosync updated
sed: can't read /var/run/gluster/shared_storage/nfs-ganesha/exports/ganesha-ha.conf: No such file or directory
Warning: Permanently added 'dhcp42-114.lab.eng.blr.redhat.com' (ECDSA) to the list of known hosts.
grep: /var/run/gluster/shared_storage/nfs-ganesha/exports/ganesha-ha.conf: No such file or directory
grep: /var/run/gluster/shared_storage/nfs-ganesha/exports/ganesha-ha.conf: No such file or directory
cp: cannot stat ‘/var/run/gluster/shared_storage/nfs-ganesha/exports/ganesha.conf’: No such file or directory
sed: can't read /etc/ganesha/ganesha.conf: No such file or directory

===================


pcs status

===========


[root@dhcp42-119 ~]# pcs status
Cluster name: ganesha-ha-360
Stack: corosync
Current DC: dhcp42-119.lab.eng.blr.redhat.com (version 1.1.16-8.el7-94ff4df) - partition with quorum
Last updated: Thu Jun 22 15:18:03 2017
Last change: Thu Jun 22 15:00:52 2017 by root via crm_node on dhcp42-127.lab.eng.blr.redhat.com

5 nodes configured
30 resources configured

Online: [ dhcp42-117.lab.eng.blr.redhat.com dhcp42-119.lab.eng.blr.redhat.com dhcp42-125.lab.eng.blr.redhat.com dhcp42-127.lab.eng.blr.redhat.com dhcp42-129.lab.eng.blr.redhat.com ]

Full list of resources:

 Clone Set: nfs_setup-clone [nfs_setup]
     Started: [ dhcp42-117.lab.eng.blr.redhat.com dhcp42-119.lab.eng.blr.redhat.com dhcp42-125.lab.eng.blr.redhat.com dhcp42-127.lab.eng.blr.redhat.com dhcp42-129.lab.eng.blr.redhat.com ]
 Clone Set: nfs-mon-clone [nfs-mon]
     Started: [ dhcp42-117.lab.eng.blr.redhat.com dhcp42-119.lab.eng.blr.redhat.com dhcp42-125.lab.eng.blr.redhat.com dhcp42-127.lab.eng.blr.redhat.com dhcp42-129.lab.eng.blr.redhat.com ]
 Clone Set: nfs-grace-clone [nfs-grace]
     Started: [ dhcp42-117.lab.eng.blr.redhat.com dhcp42-119.lab.eng.blr.redhat.com dhcp42-125.lab.eng.blr.redhat.com dhcp42-127.lab.eng.blr.redhat.com dhcp42-129.lab.eng.blr.redhat.com ]
 Resource Group: dhcp42-117.lab.eng.blr.redhat.com-group
     dhcp42-117.lab.eng.blr.redhat.com-nfs_block	(ocf::heartbeat:portblock):	Stopped
     dhcp42-117.lab.eng.blr.redhat.com-cluster_ip-1	(ocf::heartbeat:IPaddr):	Stopped
     dhcp42-117.lab.eng.blr.redhat.com-nfs_unblock	(ocf::heartbeat:portblock):	Stopped
 Resource Group: dhcp42-119.lab.eng.blr.redhat.com-group
     dhcp42-119.lab.eng.blr.redhat.com-nfs_block	(ocf::heartbeat:portblock):	Stopped
     dhcp42-119.lab.eng.blr.redhat.com-cluster_ip-1	(ocf::heartbeat:IPaddr):	Stopped
     dhcp42-119.lab.eng.blr.redhat.com-nfs_unblock	(ocf::heartbeat:portblock):	Stopped
 Resource Group: dhcp42-125.lab.eng.blr.redhat.com-group
     dhcp42-125.lab.eng.blr.redhat.com-nfs_block	(ocf::heartbeat:portblock):	Stopped
     dhcp42-125.lab.eng.blr.redhat.com-cluster_ip-1	(ocf::heartbeat:IPaddr):	Stopped
     dhcp42-125.lab.eng.blr.redhat.com-nfs_unblock	(ocf::heartbeat:portblock):	Stopped
 Resource Group: dhcp42-127.lab.eng.blr.redhat.com-group
     dhcp42-127.lab.eng.blr.redhat.com-nfs_block	(ocf::heartbeat:portblock):	Stopped
     dhcp42-127.lab.eng.blr.redhat.com-cluster_ip-1	(ocf::heartbeat:IPaddr):	Stopped
     dhcp42-127.lab.eng.blr.redhat.com-nfs_unblock	(ocf::heartbeat:portblock):	Stopped
 Resource Group: dhcp42-129.lab.eng.blr.redhat.com-group
     dhcp42-129.lab.eng.blr.redhat.com-nfs_block	(ocf::heartbeat:portblock):	Stopped
     dhcp42-129.lab.eng.blr.redhat.com-cluster_ip-1	(ocf::heartbeat:IPaddr):	Stopped
     dhcp42-129.lab.eng.blr.redhat.com-nfs_unblock	(ocf::heartbeat:portblock):	Stopped

Failed Actions:
* dhcp42-119.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-129.lab.eng.blr.redhat.com 'unknown error' (1): call=112, status=complete, exitreason='none',
    last-rc-change='Thu Jun 22 15:00:45 2017', queued=0ms, exec=59ms
* dhcp42-125.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-129.lab.eng.blr.redhat.com 'unknown error' (1): call=115, status=complete, exitreason='none',
    last-rc-change='Thu Jun 22 15:00:45 2017', queued=0ms, exec=36ms
* dhcp42-117.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-129.lab.eng.blr.redhat.com 'unknown error' (1): call=109, status=complete, exitreason='none',
    last-rc-change='Thu Jun 22 15:00:45 2017', queued=0ms, exec=41ms
* dhcp42-127.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-129.lab.eng.blr.redhat.com 'unknown error' (1): call=117, status=complete, exitreason='none',
    last-rc-change='Thu Jun 22 15:00:46 2017', queued=0ms, exec=42ms
* dhcp42-129.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-129.lab.eng.blr.redhat.com 'unknown error' (1): call=107, status=complete, exitreason='none',
    last-rc-change='Thu Jun 22 15:00:44 2017', queued=0ms, exec=38ms
* dhcp42-119.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-127.lab.eng.blr.redhat.com 'unknown error' (1): call=128, status=complete, exitreason='none',
    last-rc-change='Thu Jun 22 15:00:45 2017', queued=0ms, exec=61ms
* dhcp42-125.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-127.lab.eng.blr.redhat.com 'unknown error' (1): call=130, status=complete, exitreason='none',
    last-rc-change='Thu Jun 22 15:00:46 2017', queued=0ms, exec=53ms
* dhcp42-117.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-127.lab.eng.blr.redhat.com 'unknown error' (1): call=125, status=complete, exitreason='none',
    last-rc-change='Thu Jun 22 15:00:45 2017', queued=0ms, exec=56ms
* dhcp42-127.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-127.lab.eng.blr.redhat.com 'unknown error' (1): call=120, status=complete, exitreason='none',
    last-rc-change='Thu Jun 22 15:00:44 2017', queued=0ms, exec=259ms
* dhcp42-129.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-127.lab.eng.blr.redhat.com 'unknown error' (1): call=122, status=complete, exitreason='none',
    last-rc-change='Thu Jun 22 15:00:45 2017', queued=0ms, exec=44ms
* dhcp42-119.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-117.lab.eng.blr.redhat.com 'unknown error' (1): call=230, status=complete, exitreason='none',
    last-rc-change='Thu Jun 22 15:00:45 2017', queued=0ms, exec=40ms
* dhcp42-117.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-117.lab.eng.blr.redhat.com 'unknown error' (1): call=228, status=complete, exitreason='none',
    last-rc-change='Thu Jun 22 15:00:44 2017', queued=0ms, exec=48ms
* dhcp42-125.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-117.lab.eng.blr.redhat.com 'unknown error' (1): call=233, status=complete, exitreason='none',
    last-rc-change='Thu Jun 22 15:00:45 2017', queued=0ms, exec=83ms
* dhcp42-127.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-117.lab.eng.blr.redhat.com 'unknown error' (1): call=236, status=complete, exitreason='none',
    last-rc-change='Thu Jun 22 15:00:45 2017', queued=0ms, exec=37ms
* dhcp42-129.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-117.lab.eng.blr.redhat.com 'unknown error' (1): call=238, status=complete, exitreason='none',
    last-rc-change='Thu Jun 22 15:00:46 2017', queued=0ms, exec=53ms
* dhcp42-119.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-119.lab.eng.blr.redhat.com 'unknown error' (1): call=346, status=complete, exitreason='none',
    last-rc-change='Thu Jun 22 15:00:44 2017', queued=0ms, exec=42ms
* dhcp42-117.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-119.lab.eng.blr.redhat.com 'unknown error' (1): call=356, status=complete, exitreason='none',
    last-rc-change='Thu Jun 22 15:00:46 2017', queued=1ms, exec=62ms
* dhcp42-125.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-119.lab.eng.blr.redhat.com 'unknown error' (1): call=348, status=complete, exitreason='none',
    last-rc-change='Thu Jun 22 15:00:45 2017', queued=0ms, exec=38ms
* dhcp42-127.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-119.lab.eng.blr.redhat.com 'unknown error' (1): call=351, status=complete, exitreason='none',
    last-rc-change='Thu Jun 22 15:00:45 2017', queued=0ms, exec=44ms
* dhcp42-129.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-119.lab.eng.blr.redhat.com 'unknown error' (1): call=354, status=complete, exitreason='none',
    last-rc-change='Thu Jun 22 15:00:45 2017', queued=0ms, exec=34ms
* dhcp42-119.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-125.lab.eng.blr.redhat.com 'unknown error' (1): call=207, status=complete, exitreason='none',
    last-rc-change='Thu Jun 22 15:00:46 2017', queued=0ms, exec=56ms
* dhcp42-125.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-125.lab.eng.blr.redhat.com 'unknown error' (1): call=197, status=complete, exitreason='none',
    last-rc-change='Thu Jun 22 15:00:44 2017', queued=0ms, exec=41ms
* dhcp42-117.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-125.lab.eng.blr.redhat.com 'unknown error' (1): call=205, status=complete, exitreason='none',
    last-rc-change='Thu Jun 22 15:00:46 2017', queued=0ms, exec=47ms
* dhcp42-127.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-125.lab.eng.blr.redhat.com 'unknown error' (1): call=199, status=complete, exitreason='none',
    last-rc-change='Thu Jun 22 15:00:45 2017', queued=0ms, exec=40ms
* dhcp42-129.lab.eng.blr.redhat.com-nfs_block_start_0 on dhcp42-125.lab.eng.blr.redhat.com 'unknown error' (1): call=202, status=complete, exitreason='none',
    last-rc-change='Thu Jun 22 15:00:45 2017', queued=0ms, exec=43ms


Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled
==================


[root@dhcp42-125 ~]# rpm -qa | grep pacemaker
pacemaker-libs-1.1.16-8.el7.x86_64
pacemaker-cli-1.1.16-8.el7.x86_64
pacemaker-cluster-libs-1.1.16-8.el7.x86_64
pacemaker-1.1.16-8.el7.x86_64
[root@dhcp42-125 ~]# rpm -qa | grep pcs
pcs-0.9.157-1.el7.x86_64
[root@dhcp42-125 ~]# rpm -qa | grep corosync
corosync-2.4.0-9.el7.x86_64
corosynclib-2.4.0-9.el7.x86_64
[root@dhcp42-125 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.4 Beta (Maipo)


Attaching sosreports shortly
Comment 3 Soumya Koduri 2017-06-22 08:27:53 EDT
sed: can't read /var/run/gluster/shared_storage/nfs-ganesha/exports/ganesha-ha.conf: No such file or directory
Warning: Permanently added 'dhcp42-114.lab.eng.blr.redhat.com' (ECDSA) to the list of known hosts.
grep: /var/run/gluster/shared_storage/nfs-ganesha/exports/ganesha-ha.conf: No such file or directory
grep: /var/run/gluster/shared_storage/nfs-ganesha/exports/ganesha-ha.conf: No such file or directory
cp: cannot stat ‘/var/run/gluster/shared_storage/nfs-ganesha/exports/ganesha.conf’: No such file or directory
sed: can't read /etc/ganesha/ganesha.conf: No such file or directory


^^^^^ Above messages points to an issue with accessing shared_storage. 

@Manisha,

Could you please check if the shared_storage mount point is accessible on the node from which you are performing the delete-node operation?
Comment 4 Manisha Saini 2017-06-22 09:03:15 EDT
Soumya,

Yes the shared_storage is mounted and accessiable on all the nodes including the one from which the delete node was triggered
Comment 5 Soumya Koduri 2017-06-23 02:29:33 EDT
[root@dhcp42-127 ~]# /usr/libexec/ganesha/ganesha-ha.sh --delete /var/run/gluster/shared_storage/nfs-ganesha/exports/ 


The path provided here is incorrect. It should be 

#/usr/libexec/ganesha/ganesha-ha.sh --delete /var/run/gluster/shared_storage/nfs-ganesha/  

Note the '/exports' at the end is not needed. Could you please retry post correcting the input path.
Comment 6 Manisha Saini 2017-06-23 02:39:29 EDT
It worked without giving export

/usr/libexec/ganesha/ganesha-ha.sh --delete /var/run/gluster/shared_storage/nfs-ganesha/ dhcp42-127.lab.eng.blr.redhat.com 

[root@dhcp42-129 ~]# pcs status
Cluster name: ganesha-ha-360
Stack: corosync
Current DC: dhcp42-119.lab.eng.blr.redhat.com (version 1.1.16-8.el7-94ff4df) - partition with quorum
Last updated: Fri Jun 23 12:08:38 2017
Last change: Fri Jun 23 12:06:27 2017 by root via crm_node on dhcp42-125.lab.eng.blr.redhat.com

3 nodes configured
18 resources configured

Online: [ dhcp42-119.lab.eng.blr.redhat.com dhcp42-125.lab.eng.blr.redhat.com dhcp42-129.lab.eng.blr.redhat.com ]

Full list of resources:

 Clone Set: nfs_setup-clone [nfs_setup]
     Started: [ dhcp42-119.lab.eng.blr.redhat.com dhcp42-125.lab.eng.blr.redhat.com dhcp42-129.lab.eng.blr.redhat.com ]
 Clone Set: nfs-mon-clone [nfs-mon]
     Started: [ dhcp42-119.lab.eng.blr.redhat.com dhcp42-125.lab.eng.blr.redhat.com dhcp42-129.lab.eng.blr.redhat.com ]
 Clone Set: nfs-grace-clone [nfs-grace]
     Started: [ dhcp42-119.lab.eng.blr.redhat.com dhcp42-125.lab.eng.blr.redhat.com dhcp42-129.lab.eng.blr.redhat.com ]
 Resource Group: dhcp42-119.lab.eng.blr.redhat.com-group
     dhcp42-119.lab.eng.blr.redhat.com-nfs_block	(ocf::heartbeat:portblock):	Started dhcp42-119.lab.eng.blr.redhat.com
     dhcp42-119.lab.eng.blr.redhat.com-cluster_ip-1	(ocf::heartbeat:IPaddr):	Started dhcp42-119.lab.eng.blr.redhat.com
     dhcp42-119.lab.eng.blr.redhat.com-nfs_unblock	(ocf::heartbeat:portblock):	Started dhcp42-119.lab.eng.blr.redhat.com
 Resource Group: dhcp42-125.lab.eng.blr.redhat.com-group
     dhcp42-125.lab.eng.blr.redhat.com-nfs_block	(ocf::heartbeat:portblock):	Started dhcp42-125.lab.eng.blr.redhat.com
     dhcp42-125.lab.eng.blr.redhat.com-cluster_ip-1	(ocf::heartbeat:IPaddr):	Started dhcp42-125.lab.eng.blr.redhat.com
     dhcp42-125.lab.eng.blr.redhat.com-nfs_unblock	(ocf::heartbeat:portblock):	Started dhcp42-125.lab.eng.blr.redhat.com
 Resource Group: dhcp42-129.lab.eng.blr.redhat.com-group
     dhcp42-129.lab.eng.blr.redhat.com-nfs_block	(ocf::heartbeat:portblock):	Started dhcp42-129.lab.eng.blr.redhat.com
     dhcp42-129.lab.eng.blr.redhat.com-cluster_ip-1	(ocf::heartbeat:IPaddr):	Started dhcp42-129.lab.eng.blr.redhat.com
     dhcp42-129.lab.eng.blr.redhat.com-nfs_unblock	(ocf::heartbeat:portblock):	Started dhcp42-129.lab.eng.blr.redhat.com

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled


Closing this bug

Note You need to log in before you can comment on or make changes to this bug.