Bug 1425753

Summary: NFS-Ganesha process segfault when deleting Gluster FSAL exports
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Soumya Koduri <skoduri>
Component: nfs-ganeshaAssignee: Soumya Koduri <skoduri>
Status: CLOSED ERRATA QA Contact: Manisha Saini <msaini>
Severity: high Docs Contact:
Priority: unspecified    
Version: rhgs-3.2CC: amukherj, asrivast, bmohanra, ffilz, jthottan, kkeithle, msaini, rhinduja, rhs-bugs, skoduri, storage-qa-internal
Target Milestone: ---   
Target Release: RHGS 3.3.0   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: nfs-ganesha-2.4.4-1 Doc Type: Bug Fix
Doc Text:
When multiple paths with the same parent volume are exported via NFS-Ganesha, the handles maintained by the server of the files/directories common to those paths would get merged. Due to this, unexporting one of those shares may result in segmentation fault of the server when accessed via another share mount. With this fix, the refcount of such shared objects is maintained and are released only when all the exports accessing them are unexported. There is hence no issues accessing one share while unexporting another one which shares the same parent volume.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2017-09-21 04:47:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1417147    

Description Soumya Koduri 2017-02-22 10:20:23 UTC 
Description of problem:

As reported in the upstream -https://github.com/nfs-ganesha/nfs-ganesha/issues/130#issuecomment-267903225, when there are multiple export entries with same parent volume and one export path is subset of another, the object handles are merged by md-cache across different export entries. Because of this, unexporting one entry shall result in SEGV when those handles are accessed via another share mount.

Version-Release number of selected component (if applicable):
3.2

How reproducible:
Always

Steps to Reproduce:
1. Create 2+ shares using the same gluster volume
2. Delete all shares in any order (except opposite from creation).
3. Ganesha segaults.


Actual results:
NFS-Ganesha process crashes

Expected results:
There should not be any crash. 

Additional info:
Comment 5 Soumya Koduri 2017-03-15 12:28:50 UTC 
Patch merged upstream - https://review.gerrithub.io/#/c/306836/
Comment 7 Manisha Saini 2017-07-03 13:34:38 UTC 
Tested with build 

# rpm -qa | grep ganesha
nfs-ganesha-debuginfo-2.4.4-10.el7rhgs.x86_64
glusterfs-ganesha-3.8.4-31.el7rhgs.x86_64
nfs-ganesha-2.4.4-10.el7rhgs.x86_64
nfs-ganesha-gluster-2.4.4-10.el7rhgs.x86_64


Following are the observation-

1.Export the sub dir with dbus command ->Ganesha process goes to failed state.
While Export subdir with ganesha service restart on all node ->Successfully exports the sub-dir

#dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.AddExport string:/var/run/gluster/shared_storage/nfs-ganesha/exports string:"EXPORT(Path=/ganeshavol5/mani3)"
Error org.freedesktop.DBus.Error.NoReply: Message did not receive a reply (timeout by message bus)


ganesha.log-

03/07/2017 18:51:18 : epoch 96d00000 : dhcp42-125.lab.eng.blr.redhat.com : ganesha.nfsd-12527[dbus_heartbeat] unregister_fsal :FSAL :CRIT :Unregister FSAL GLUSTER with non-zero refcount=2
03/07/2017 18:51:18 : epoch 96d00000 : dhcp42-125.lab.eng.blr.redhat.com : ganesha.nfsd-12527[dbus_heartbeat] glusterfs_unload :FSAL :CRIT :FSAL Gluster unable to unload.  Dying ...

2.Volume stop/delete does not remove the volume export entry from showmount -e (when the sub dir are exported with dbus command) but mount on client gets failed.

# showmount -e
Export list for dhcp42-125.lab.eng.blr.redhat.com:
/ganeshavol5       (everyone)
/ganeshavol5/mani2 (everyone)
/ganeshavol5/mani3 (everyone)
# gluster v stop ganeshavol5
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: ganeshavol5: success
[root@dhcp42-125 exports]# showmount -e
Export list for dhcp42-125.lab.eng.blr.redhat.com:
/ganeshavol5       (everyone)
/ganeshavol5/mani2 (everyone)
/ganeshavol5/mani3 (everyone)

[root@dhcp42-125 exports]# gluster v status ganeshavol5
Volume ganeshavol5 is not started

# gluster v delete ganeshavol5
Deleting volume will erase all information about the volume. Do you want to continue? (y/n) y
volume delete: ganeshavol5: success
[root@dhcp42-125 exports]# showmount -e
Export list for dhcp42-125.lab.eng.blr.redhat.com:
/ganeshavol5       (everyone)
/ganeshavol5/mani2 (everyone)
/ganeshavol5/mani3 (everyone)



#mount -t nfs -o vers=4 10.70.42.42:/ganeshavol5 /mnt/mani/
mount.nfs: Remote I/O error

@Soumya can you comment on the above observations related to sub dir mount
Comment 8 Manisha Saini 2017-07-24 10:31:33 UTC 
In comment #7 dbus command used was incomplete.

Verified this bug on 

# rpm -qa | grep ganesha
nfs-ganesha-gluster-2.4.4-16.el7rhgs.x86_64
nfs-ganesha-2.4.4-16.el7rhgs.x86_64
glusterfs-ganesha-3.8.4-34.el7rhgs.x86_64


Steps:
1.Create 2+ shares of the same volume

# WARNING : Using Gluster CLI will overwrite manual
# changes made to this file. To avoid it, edit the
# file and run ganesha-ha.sh --refresh-config.
EXPORT{
      Export_Id = 2;
      Path = "/ganeshavol3";
      FSAL {
           name = GLUSTER;
           hostname="localhost";
          volume="ganeshavol3";
           }
      Access_type = RW;
      Disable_ACL = true;
      Squash="No_root_squash";
      Pseudo="/ganeshavol3";
      Protocols = "3", "4" ;
      Transports = "UDP","TCP";
      SecType = "sys";
     }
EXPORT{
      Export_Id = 3;
      Path = "/ganeshavol3/dir1";
      FSAL {
           name = GLUSTER;
           hostname="localhost";
          volume="ganeshavol3";
          volpath="/dir1";
           }
      Access_type = RW;
      Disable_ACL = true;
      Squash="No_root_squash";
      Pseudo="/dir1";
      Protocols = "3", "4" ;
      Transports = "UDP","TCP";
      SecType = "sys";
     }
EXPORT{
      Export_Id = 4;
      Path = "/ganeshavol3/dir2";
      FSAL {
           name = GLUSTER;
           hostname="localhost";
          volume="ganeshavol3";
          volpath="/dir2";
           }
      Access_type = RW;
      Disable_ACL = true;
      Squash="No_root_squash";
      Pseudo="/dir2";
      Protocols = "3", "4" ;
      Transports = "UDP","TCP";
      SecType = "sys";
     }


EXPORT{
      Export_Id = 5;
      Path = "/ganeshavol3/dir3";
      FSAL {
           name = GLUSTER;
           hostname="localhost";
          volume="ganeshavol3";
          volpath="/dir3";
           }
      Access_type = RW;
      Disable_ACL = true;
      Squash="No_root_squash";
      Pseudo="/dir3";
      Protocols = "3", "4" ;
      Transports = "UDP","TCP";
      SecType = "sys";
     }

----------


[root@dhcp42-125 exports]# dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.AddExport string:/var/run/gluster/shared_storage/nfs-ganesha/exports/export.ganeshavol3.conf string:"EXPORT(Path=/ganeshavol3/dir1)"
method return sender=:1.36 -> dest=:1.154 reply_serial=2
   string "1 exports added"

[root@dhcp42-125 exports]# dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.AddExport string:/var/run/gluster/shared_storage/nfs-ganesha/exports/export.ganeshavol3.conf string:"EXPORT(Path=/ganeshavol3/dir2)"
method return sender=:1.36 -> dest=:1.155 reply_serial=2
   string "1 exports added"

[root@dhcp42-125 exports]# dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.AddExport string:/var/run/gluster/shared_storage/nfs-ganesha/exports/export.ganeshavol3.conf string:"EXPORT(Path=/ganeshavol3/dir3)"
method return sender=:1.36 -> dest=:1.158 reply_serial=2
   string "1 exports added"

[root@dhcp42-125 exports]# showmount -e
Export list for dhcp42-125.lab.eng.blr.redhat.com:
/ganeshavol3      (everyone)
/ganeshavol3/dir1 (everyone)
/ganeshavol3/dir2 (everyone)
/ganeshavol3/dir3 (everyone)
----------------

2.Start unexporting volume via dbus

------------

[root@dhcp42-125 exports]# dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:4
method return sender=:1.36 -> dest=:1.163 reply_serial=2

[root@dhcp42-125 exports]# showmount -e
Export list for dhcp42-125.lab.eng.blr.redhat.com:
/ganeshavol3      (everyone)
/ganeshavol3/dir1 (everyone)
/ganeshavol3/dir3 (everyone)

[root@dhcp42-125 exports]# dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:3
method return sender=:1.36 -> dest=:1.166 reply_serial=2

[root@dhcp42-125 exports]# showmount -e
Export list for dhcp42-125.lab.eng.blr.redhat.com:
/ganeshavol3      (everyone)
/ganeshavol3/dir3 (everyone)

[root@dhcp42-125 exports]# dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:5
method return sender=:1.36 -> dest=:1.167 reply_serial=2

[root@dhcp42-125 exports]# showmount -e
Export list for dhcp42-125.lab.eng.blr.redhat.com:
/ganeshavol3 (everyone)

-----------


Moving this bug to verified state.
Comment 14 errata-xmlrpc 2017-09-21 04:47:57 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:2779