Bug 1336798

+++ This bug was initially created as a clone of Bug #1333319 +++

+++ This bug was initially created as a clone of Bug #1330365 +++

Description of problem:

Unexporting a volume sometimes fails with "Dynamic export addition/deletion failed".

Version-Release number of selected component (if applicable):

nfs-ganesha-2.3.1-4

How reproducible:

Intermittent

Steps to Reproduce:

Not sure about the exact steps but sometimes the unexport of volume fails with "dynamic export addition/deletion failed"

Performed below steps when i hit this issue.

1.Created a 4 node ganesha cluster.
2.Created a tiered volume, enabled quota and mounted on 2 clients using vers=4
3.Performed IO's from both the mount points.
4.Umounted the volume from both clients.
5.Unexported the volume and it fails with "dynamic export addition/deletion fails" with below messages in logs

[2016-04-25 16:20:35.129472] E [MSGID: 106472] [glusterd-ganesha.c:145:glusterd_check_ganesha_cmd] 0-management: Handling NFS-Ganesha op failed.
[2016-04-25 16:20:35.129582] E [MSGID: 106123] [glusterd-syncop.c:1411:gd_commit_op_phase] 0-management: Commit of operation 'Volume Set' failed on localhost : Dynamic export addition/deletion failed. Please see log file for details


It unexports the volume from the node where ganesha.enable off, was run but from all other nodes, the volume doesnt get unexported 

[root@dhcp37-180 exports]# pwd
/etc/ganesha/exports
[root@dhcp37-180 exports]# ls
[root@dhcp37-180 exports]#

[root@dhcp37-158 exports]# pwd
/etc/ganesha/exports
[root@dhcp37-158 exports]# ls
export.tiervolume.conf
[root@dhcp37-158 exports]#

[root@dhcp37-127 exports]# pwd
/etc/ganesha/exports
[root@dhcp37-127 exports]# ls
export.tiervolume.conf
[root@dhcp37-127 exports]#

[root@dhcp37-174 exports]# pwd
/etc/ganesha/exports
[root@dhcp37-174 exports]# ls
export.tiervolume.conf
[root@dhcp37-174 exports]#

whereas showmount shows volume exported on all the nodes

[root@dhcp37-180 exports]# showmount -e localhost
Export list for localhost:
/tiervolume (everyone)

[root@dhcp37-158 exports]# showmount -e localhost
Export list for localhost:
/tiervolume (everyone)

[root@dhcp37-127 exports]# showmount -e localhost
Export list for localhost:
/tiervolume (everyone)

[root@dhcp37-174 exports]# showmount -e localhost
Export list for localhost:
/tiervolume (everyone)

peer status is proper in this case:

[root@dhcp37-180 exports]# gluster peer status
Number of Peers: 3

Hostname: dhcp37-158.lab.eng.blr.redhat.com
Uuid: 18fa3cca-c714-4c70-b227-cef260fffa27
State: Peer in Cluster (Connected)

Hostname: dhcp37-127.lab.eng.blr.redhat.com
Uuid: 43649367-7f47-41cf-8d63-97896e3504d4
State: Peer in Cluster (Connected)

Hostname: dhcp37-174.lab.eng.blr.redhat.com
Uuid: 1a5a806a-ab58-462b-b939-0b8158a2d914
State: Peer in Cluster (Connected)

After restarting glusterd on all the nodes, showmount doesn't list the volume as exported on localhost:

[root@dhcp37-180 exports]# showmount -e localhost
Export list for localhost:
[root@dhcp37-180 exports]# ls
[root@dhcp37-180 exports]# 

[root@dhcp37-158 exports]# showmount -e localhost
Export list for localhost:
/tiervolume (everyone)
[root@dhcp37-158 exports]# ls
export.tiervolume.conf
[root@dhcp37-158 exports]#

[root@dhcp37-127 exports]# showmount -e localhost
Export list for localhost:
/tiervolume (everyone)
[root@dhcp37-127 exports]# ls
export.tiervolume.conf
[root@dhcp37-127 exports]#

[root@dhcp37-174 exports]# showmount -e localhost
Export list for localhost:
/tiervolume (everyone)
[root@dhcp37-174 exports]# ls
export.tiervolume.conf
[root@dhcp37-174 exports]#


Actual results:

Unexporting a volume sometimes fails with "Dynamic export addition/deletion failed".

Expected results:

The volume should get unexported without any issues.

Additional info:

No AVC's/SElinux related issue seen in the logs

--- Additional comment from Red Hat Bugzilla Rules Engine on 2016-04-25 23:32:51 EDT ---

This bug is automatically being proposed for the current z-stream release of Red Hat Gluster Storage 3 by setting the release flag 'rhgs‑3.1.z' to '?'. 

If this bug should be proposed for a different release, please manually change the proposed release flag.

--- Additional comment from Kaleb KEITHLEY on 2016-04-28 08:52:14 EDT ---

hard to reproduce. Document as known issue for 3.1.3 while we RCA

--- Additional comment from Shashank Raj on 2016-04-29 03:42:05 EDT ---

This issue seems to be hitting most of the time when we unexport the volume after doing some IO's. However still not sure about of the exact steps:

[root@dhcp37-180 ~]# gluster volume set newvolume ganesha.enable off
volume set: failed: Dynamic export addition/deletion failed. Please see log file for details

the volume gets unexported from the node from where we run the command and on the other nodes it remains as it is.

Since, we are hitting this very frequently now, setting the severity to high

--- Additional comment from Vijay Bellur on 2016-05-05 05:35:37 EDT ---

REVIEW: http://review.gluster.org/14225 (NFS-Ganesha : Parse the Export_Id correctly for unexporting volume) posted (#1) for review on master by jiffin tony Thottan (jthottan)

--- Additional comment from Vijay Bellur on 2016-05-05 08:02:29 EDT ---

REVIEW: http://review.gluster.org/14225 (NFS-Ganesha : Parse the Export_Id correctly for unexporting volume) posted (#2) for review on master by jiffin tony Thottan (jthottan)

--- Additional comment from Vijay Bellur on 2016-05-05 14:13:49 EDT ---

COMMIT: http://review.gluster.org/14225 committed in master by Kaleb KEITHLEY (kkeithle) 
------
commit 5440bad91026f27399df51856176aef85c1fcf4d
Author: Jiffin Tony Thottan <jthottan>
Date:   Thu May 5 14:52:32 2016 +0530

    NFS-Ganesha : Parse the Export_Id correctly for unexporting volume
    
    Currently export id parsed using "cut -d ' ' -f8" which might endup
    in giving wrong value. In case of multiple space chracter, output
    may differ. In this all those instance will replaced by awk call
    
    Change-Id: I60dea8ce116900da3c1fc9badf898e51183a2ca1
    BUG: 1333319
    Signed-off-by: Jiffin Tony Thottan <jthottan>
    Reviewed-on: http://review.gluster.org/14225
    Smoke: Gluster Build System <jenkins.com>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.com>
    Reviewed-by: soumya k <skoduri>
    Reviewed-by: Kaleb KEITHLEY <kkeithle>

--- Additional comment from Vijay Bellur on 2016-05-13 02:50:42 EDT ---

REVIEW: http://review.gluster.org/14325 (ganesha/scripts : Fixing refresh config in ganesha-ha.sh) posted (#2) for review on master by jiffin tony Thottan (jthottan)

--- Additional comment from Vijay Bellur on 2016-05-13 08:01:45 EDT ---

COMMIT: http://review.gluster.org/14325 committed in master by Niels de Vos (ndevos) 
------
commit 8299ad374b2b93f123b29e0d411930733a372a04
Author: Jiffin Tony Thottan <jthottan>
Date:   Fri May 13 11:59:55 2016 +0530

    ganesha/scripts : Fixing refresh config in ganesha-ha.sh
    
    The change http://review.gluster.org/#/c/14225/ cause a regression
    for refresh config funtion in ganesha-ha.sh due to a invalid usage
    of awk arguement.
    
    Change-Id: Id5adfb12f99b29bdb3531773cd34bd67cfff8768
    BUG: 1333319
    Signed-off-by: Jiffin Tony Thottan <jthottan>
    Reviewed-on: http://review.gluster.org/14325
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.com>
    Reviewed-by: Kaleb KEITHLEY <kkeithle>
    Smoke: Gluster Build System <jenkins.com>
    Reviewed-by: Niels de Vos <ndevos>