Bug 1455801

Summary: [Ganesha READDIR Chunking] : Unable to unexport a volume once exported.
Product: [Retired] nfs-ganesha Reporter: Ambarish <asoman>
Component: NFSAssignee: Frank Filz <ffilz>
Status: CLOSED NOTABUG QA Contact: Ambarish <asoman>
Severity: high Docs Contact:
Priority: unspecified    
Version: 2.4CC: amukherj, bturner, dang, jthottan, kkeithle, mbenjamin, rhinduja, rhs-bugs, skoduri
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-05-31 05:50:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ambarish 2017-05-26 07:44:25 UTC
Description of problem:
-----------------------

Toggle ganesha.enable.

Second attempt to unexport the volume fails.

[root@gqas013 ~]# gluster v set testvol ganesha.enable off
volume set: failed: Dynamic export addition/deletion failed. Please see log file for details
[root@gqas013 ~]# 


**From Logs* :

26/05/2017 03:37:18 : epoch 90500000 : gqas013.sbu.lab.eng.bos.redhat.com : ganesha.nfsd-2208[dbus_heartbeat] dbus_message_entrypoint :DBUS :MAJ :Method (RemoveExport) on (org.ganesha.nfsd.exportmgr) failed: name = (org.freedesktop.DBus.Error.InvalidArgs), message = (lookup_export failed with Export id not found)
26/05/2017 03:38:45 : epoch 90500000 : gqas013.sbu.lab.eng.bos.redhat.com : ganesha.nfsd-2208[dbus_heartbeat] dbus_message_entrypoint :DBUS :MAJ :Method (RemoveExport) on (org.ganesha.nfsd.exportmgr) failed: name = (org.freedesktop.DBus.Error.InvalidArgs), message = (lookup_export failed with Export id not found)
26/05/2017 03:38:56 : epoch 90500000 : gqas013.sbu.lab.eng.bos.redhat.com : ganesha.nfsd-2208[dbus_heartbeat] dbus_message_entrypoint :DBUS :MAJ :Method (RemoveExport) on (org.ganesha.nfsd.exportmgr) failed: name = (org.freedesktop.DBus.Error.InvalidArgs), message = (lookup_export failed with Export id not found)
26/05/2017 03:39:15 : epoch 90500000 : gqas013.sbu.lab.eng.bos.redhat.com : ganesha.nfsd-2208[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume testvol exported at : '/'
26/05/2017 03:39:37 : epoch 90500000 : gqas013.sbu.lab.eng.bos.redhat.com : ganesha.nfsd-2208[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume testvol exported at : '/'
                                                

Version-Release number of selected component (if applicable):
--------------------------------------------------------------

glusterfs-ganesha-3.8.4-25.el7rhgs.x86_64
nfs-ganesha-gluster-2.4.4-4.el7rhgs.READDIRbuild3.x86_64


How reproducible:
-----------------

Pretty frequently.

Steps to Reproduce:
-------------------

In description.

Additional info:
-----------------

Volume Name: testvol
Type: Distributed-Replicate
Volume ID: 1a304bd7-25d3-48ff-936f-a85726426cab
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: gqas013.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick0
Brick2: gqas005.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick1
Brick3: gqas006.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick2
Brick4: gqas008.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick3
Options Reconfigured:
nfs.disable: on
transport.address-family: inet
performance.stat-prefetch: off
server.allow-insecure: on
features.cache-invalidation: on
ganesha.enable: on
cluster.enable-shared-storage: enable
nfs-ganesha: enable
[root@gqas013 ~]#

Comment 3 Ambarish 2017-05-26 10:58:09 UTC
Got a confirmation from Manisha that the issue doesn't occur downstream.

which reminded me that the build I am testing is not _just_ the READDIR patches anymore,It includes fixes for CPU hogging,crashes etc.

Maybe this is a regression caused by any of the other patches?

Comment 4 Daniel Gryniewicz 2017-05-26 12:44:09 UTC
Just to allow me to make apples-to-apples, where is the code that's actually being run here?

Comment 5 Daniel Gryniewicz 2017-05-26 12:45:44 UTC
Okay, now I'm confused.  You said "second attemp" failed, but that export is not there, so I'd expect an error.  You cannot remove an export that's not there.

Comment 6 Ambarish 2017-05-26 12:47:48 UTC
(In reply to Daniel Gryniewicz from comment #5)
> Okay, now I'm confused.  You said "second attemp" failed, but that export is
> not there, so I'd expect an error.  You cannot remove an export that's not
> there.

Ohh Im sorry,that's not what I meant.

I meant export -> unexport -> export --> unexport (This guy failed)

Comment 7 Ambarish 2017-05-26 12:53:59 UTC
(In reply to Daniel Gryniewicz from comment #4)
> Just to allow me to make apples-to-apples, where is the code that's actually
> being run here?

The rpms were downloaded from here :

https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=13210154

Comment 8 Daniel Gryniewicz 2017-05-26 13:20:08 UTC
(In reply to Ambarish from comment #7)
> (In reply to Daniel Gryniewicz from comment #4)
> > Just to allow me to make apples-to-apples, where is the code that's actually
> > being run here?
> 
> The rpms were downloaded from here :
> 
> https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=13210154

I can't find any RPMs there, including SRPMs.  The status is expired, so maybe they were removed?  Is there some way I can see the code that built them?

Comment 9 Ambarish 2017-05-27 07:11:43 UTC
Repro'd it on downstream bits - https://bugzilla.redhat.com/show_bug.cgi?id=1456132

Comment 10 Frank Filz 2017-05-30 17:14:52 UTC
If this is conclusively not related to readdir chunking, can we close this bug, or at least set it to a state reflective that the real issue is not readdir chunking?

Comment 11 Jiffin 2017-05-31 05:50:55 UTC
The issue was reproduced when
gluster v set <volname> ganesha.enable on 
gluster v set <volname> ganesha.enable off
gluster v set <volname> ganesha.enable on
gluster v set <volname> ganesha.enable off
in a kind of all of a sudden, it can happen because if ganesha is handling Add export request at the same time it receive RemoveExport request, that request will fail.
If we rerun the steps with delay
gluster v set <volname> ganesha.enable on
check whether volume is exported via showmount command
gluster v set <volname> ganesha.enable off
check whether volume is unexported via showmount command
gluster v set <volname> ganesha.enable on
check whether volume is exported via showmount command
gluster v set <volname> ganesha.enable off
check whether volume is unexported via showmount command

It was not reproducible with above steps
Hence closing this bug as works for me