Bug 1336798

Summary: Unexporting a volume sometimes fails with "Dynamic export addition/deletion failed".
Product: [Community] GlusterFS Reporter: Jiffin <jthottan>
Component: ganesha-nfsAssignee: Jiffin <jthottan>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.8.0CC: bugs, jthottan, kkeithle, ndevos, nlevinki, skoduri, sraj, storage-qa-internal
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: glusterfs-3.8rc2 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1333319 Environment:
Last Closed: 2016-06-16 14:06:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1330365, 1333319    
Bug Blocks: 1333528, 1336331    

Description Jiffin 2016-05-17 13:14:04 UTC
+++ This bug was initially created as a clone of Bug #1333319 +++

+++ This bug was initially created as a clone of Bug #1330365 +++

Description of problem:

Unexporting a volume sometimes fails with "Dynamic export addition/deletion failed".

Version-Release number of selected component (if applicable):

nfs-ganesha-2.3.1-4

How reproducible:

Intermittent

Steps to Reproduce:

Not sure about the exact steps but sometimes the unexport of volume fails with "dynamic export addition/deletion failed"

Performed below steps when i hit this issue.

1.Created a 4 node ganesha cluster.
2.Created a tiered volume, enabled quota and mounted on 2 clients using vers=4
3.Performed IO's from both the mount points.
4.Umounted the volume from both clients.
5.Unexported the volume and it fails with "dynamic export addition/deletion fails" with below messages in logs

[2016-04-25 16:20:35.129472] E [MSGID: 106472] [glusterd-ganesha.c:145:glusterd_check_ganesha_cmd] 0-management: Handling NFS-Ganesha op failed.
[2016-04-25 16:20:35.129582] E [MSGID: 106123] [glusterd-syncop.c:1411:gd_commit_op_phase] 0-management: Commit of operation 'Volume Set' failed on localhost : Dynamic export addition/deletion failed. Please see log file for details


It unexports the volume from the node where ganesha.enable off, was run but from all other nodes, the volume doesnt get unexported 

[root@dhcp37-180 exports]# pwd
/etc/ganesha/exports
[root@dhcp37-180 exports]# ls
[root@dhcp37-180 exports]#

[root@dhcp37-158 exports]# pwd
/etc/ganesha/exports
[root@dhcp37-158 exports]# ls
export.tiervolume.conf
[root@dhcp37-158 exports]#

[root@dhcp37-127 exports]# pwd
/etc/ganesha/exports
[root@dhcp37-127 exports]# ls
export.tiervolume.conf
[root@dhcp37-127 exports]#

[root@dhcp37-174 exports]# pwd
/etc/ganesha/exports
[root@dhcp37-174 exports]# ls
export.tiervolume.conf
[root@dhcp37-174 exports]#

whereas showmount shows volume exported on all the nodes

[root@dhcp37-180 exports]# showmount -e localhost
Export list for localhost:
/tiervolume (everyone)

[root@dhcp37-158 exports]# showmount -e localhost
Export list for localhost:
/tiervolume (everyone)

[root@dhcp37-127 exports]# showmount -e localhost
Export list for localhost:
/tiervolume (everyone)

[root@dhcp37-174 exports]# showmount -e localhost
Export list for localhost:
/tiervolume (everyone)

peer status is proper in this case:

[root@dhcp37-180 exports]# gluster peer status
Number of Peers: 3

Hostname: dhcp37-158.lab.eng.blr.redhat.com
Uuid: 18fa3cca-c714-4c70-b227-cef260fffa27
State: Peer in Cluster (Connected)

Hostname: dhcp37-127.lab.eng.blr.redhat.com
Uuid: 43649367-7f47-41cf-8d63-97896e3504d4
State: Peer in Cluster (Connected)

Hostname: dhcp37-174.lab.eng.blr.redhat.com
Uuid: 1a5a806a-ab58-462b-b939-0b8158a2d914
State: Peer in Cluster (Connected)

After restarting glusterd on all the nodes, showmount doesn't list the volume as exported on localhost:

[root@dhcp37-180 exports]# showmount -e localhost
Export list for localhost:
[root@dhcp37-180 exports]# ls
[root@dhcp37-180 exports]# 

[root@dhcp37-158 exports]# showmount -e localhost
Export list for localhost:
/tiervolume (everyone)
[root@dhcp37-158 exports]# ls
export.tiervolume.conf
[root@dhcp37-158 exports]#

[root@dhcp37-127 exports]# showmount -e localhost
Export list for localhost:
/tiervolume (everyone)
[root@dhcp37-127 exports]# ls
export.tiervolume.conf
[root@dhcp37-127 exports]#

[root@dhcp37-174 exports]# showmount -e localhost
Export list for localhost:
/tiervolume (everyone)
[root@dhcp37-174 exports]# ls
export.tiervolume.conf
[root@dhcp37-174 exports]#


Actual results:

Unexporting a volume sometimes fails with "Dynamic export addition/deletion failed".

Expected results:

The volume should get unexported without any issues.

Additional info:

No AVC's/SElinux related issue seen in the logs

--- Additional comment from Red Hat Bugzilla Rules Engine on 2016-04-25 23:32:51 EDT ---

This bug is automatically being proposed for the current z-stream release of Red Hat Gluster Storage 3 by setting the release flag 'rhgs‑3.1.z' to '?'. 

If this bug should be proposed for a different release, please manually change the proposed release flag.

--- Additional comment from Kaleb KEITHLEY on 2016-04-28 08:52:14 EDT ---

hard to reproduce. Document as known issue for 3.1.3 while we RCA

--- Additional comment from Shashank Raj on 2016-04-29 03:42:05 EDT ---

This issue seems to be hitting most of the time when we unexport the volume after doing some IO's. However still not sure about of the exact steps:

[root@dhcp37-180 ~]# gluster volume set newvolume ganesha.enable off
volume set: failed: Dynamic export addition/deletion failed. Please see log file for details

the volume gets unexported from the node from where we run the command and on the other nodes it remains as it is.

Since, we are hitting this very frequently now, setting the severity to high

--- Additional comment from Vijay Bellur on 2016-05-05 05:35:37 EDT ---

REVIEW: http://review.gluster.org/14225 (NFS-Ganesha : Parse the Export_Id correctly for unexporting volume) posted (#1) for review on master by jiffin tony Thottan (jthottan)

--- Additional comment from Vijay Bellur on 2016-05-05 08:02:29 EDT ---

REVIEW: http://review.gluster.org/14225 (NFS-Ganesha : Parse the Export_Id correctly for unexporting volume) posted (#2) for review on master by jiffin tony Thottan (jthottan)

--- Additional comment from Vijay Bellur on 2016-05-05 14:13:49 EDT ---

COMMIT: http://review.gluster.org/14225 committed in master by Kaleb KEITHLEY (kkeithle) 
------
commit 5440bad91026f27399df51856176aef85c1fcf4d
Author: Jiffin Tony Thottan <jthottan>
Date:   Thu May 5 14:52:32 2016 +0530

    NFS-Ganesha : Parse the Export_Id correctly for unexporting volume
    
    Currently export id parsed using "cut -d ' ' -f8" which might endup
    in giving wrong value. In case of multiple space chracter, output
    may differ. In this all those instance will replaced by awk call
    
    Change-Id: I60dea8ce116900da3c1fc9badf898e51183a2ca1
    BUG: 1333319
    Signed-off-by: Jiffin Tony Thottan <jthottan>
    Reviewed-on: http://review.gluster.org/14225
    Smoke: Gluster Build System <jenkins.com>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.com>
    Reviewed-by: soumya k <skoduri>
    Reviewed-by: Kaleb KEITHLEY <kkeithle>

--- Additional comment from Vijay Bellur on 2016-05-13 02:50:42 EDT ---

REVIEW: http://review.gluster.org/14325 (ganesha/scripts : Fixing refresh config in ganesha-ha.sh) posted (#2) for review on master by jiffin tony Thottan (jthottan)

--- Additional comment from Vijay Bellur on 2016-05-13 08:01:45 EDT ---

COMMIT: http://review.gluster.org/14325 committed in master by Niels de Vos (ndevos) 
------
commit 8299ad374b2b93f123b29e0d411930733a372a04
Author: Jiffin Tony Thottan <jthottan>
Date:   Fri May 13 11:59:55 2016 +0530

    ganesha/scripts : Fixing refresh config in ganesha-ha.sh
    
    The change http://review.gluster.org/#/c/14225/ cause a regression
    for refresh config funtion in ganesha-ha.sh due to a invalid usage
    of awk arguement.
    
    Change-Id: Id5adfb12f99b29bdb3531773cd34bd67cfff8768
    BUG: 1333319
    Signed-off-by: Jiffin Tony Thottan <jthottan>
    Reviewed-on: http://review.gluster.org/14325
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.com>
    Reviewed-by: Kaleb KEITHLEY <kkeithle>
    Smoke: Gluster Build System <jenkins.com>
    Reviewed-by: Niels de Vos <ndevos>

Comment 1 Vijay Bellur 2016-05-17 13:15:43 UTC
REVIEW: http://review.gluster.org/14396 (NFS-Ganesha : Parse the Export_Id correctly for unexporting volume) posted (#1) for review on release-3.8 by jiffin tony Thottan (jthottan)

Comment 2 Vijay Bellur 2016-05-18 13:56:18 UTC
COMMIT: http://review.gluster.org/14396 committed in release-3.8 by Kaleb KEITHLEY (kkeithle) 
------
commit f1ffd9a0bbd4989a36dca2fd707226c240a16ec5
Author: Jiffin Tony Thottan <jthottan>
Date:   Thu May 5 14:52:32 2016 +0530

    NFS-Ganesha : Parse the Export_Id correctly for unexporting volume
    
    Currently export id parsed using "cut -d ' ' -f8" which might endup
    in giving wrong value. In case of multiple space chracter, output
    may differ. In this all those instance will replaced by awk call
    
    Upstream reference :
    >Change-Id: I60dea8ce116900da3c1fc9badf898e51183a2ca1
    >BUG: 1333319
    >Signed-off-by: Jiffin Tony Thottan <jthottan>
    >Reviewed-on: http://review.gluster.org/14225
    >Smoke: Gluster Build System <jenkins.com>
    >NetBSD-regression: NetBSD Build System <jenkins.org>
    >CentOS-regression: Gluster Build System <jenkins.com>
    >Reviewed-by: soumya k <skoduri>
    >Reviewed-by: Kaleb KEITHLEY <kkeithle>
    >(cherry picked from commit 5440bad91026f27399df51856176aef85c1fcf4d)
    
    Change-Id: I60dea8ce116900da3c1fc9badf898e51183a2ca1
    BUG: 1336798
    Signed-off-by: Jiffin Tony Thottan <jthottan>
    Reviewed-on: http://review.gluster.org/14396
    Smoke: Gluster Build System <jenkins.com>
    Reviewed-by: Kaleb KEITHLEY <kkeithle>
    Tested-by: Kaleb KEITHLEY <kkeithle>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.com>

Comment 3 Vijay Bellur 2016-05-19 09:10:39 UTC
REVIEW: http://review.gluster.org/14420 (ganesha/scripts : Fixing refresh config in ganesha-ha.sh) posted (#1) for review on release-3.8 by jiffin tony Thottan (jthottan)

Comment 4 Vijay Bellur 2016-05-19 16:50:18 UTC
COMMIT: http://review.gluster.org/14420 committed in release-3.8 by Kaleb KEITHLEY (kkeithle) 
------
commit 2e2ef4f833abe28f7895ea022654aac74e353702
Author: Jiffin Tony Thottan <jthottan>
Date:   Fri May 13 11:59:55 2016 +0530

    ganesha/scripts : Fixing refresh config in ganesha-ha.sh
    
    The change http://review.gluster.org/#/c/14225/ cause a regression
    for refresh config funtion in ganesha-ha.sh due to a invalid usage
    of awk arguement.
    
    >Change-Id: Id5adfb12f99b29bdb3531773cd34bd67cfff8768
    >BUG: 1333319
    >Signed-off-by: Jiffin Tony Thottan <jthottan>
    >Reviewed-on: http://review.gluster.org/14325
    >NetBSD-regression: NetBSD Build System <jenkins.org>
    >CentOS-regression: Gluster Build System <jenkins.com>
    >Reviewed-by: Kaleb KEITHLEY <kkeithle>
    >Smoke: Gluster Build System <jenkins.com>
    >Reviewed-by: Niels de Vos <ndevos>
    >(cherry picked from commit 8299ad374b2b93f123b29e0d411930733a372a04)
    
    Change-Id: Id5adfb12f99b29bdb3531773cd34bd67cfff8768
    BUG: 1336798
    Signed-off-by: Jiffin Tony Thottan <jthottan>
    Reviewed-on: http://review.gluster.org/14420
    Smoke: Gluster Build System <jenkins.com>
    Reviewed-by: Kaleb KEITHLEY <kkeithle>
    Tested-by: Kaleb KEITHLEY <kkeithle>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.com>

Comment 5 Niels de Vos 2016-06-16 14:06:58 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report.

glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user