Bug 1578089

Summary: [Ganesha] Volume exported via ganesha got failed on one of the node while trying to export volumes in loop
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Manisha Saini <msaini>
Component: rpcAssignee: Mohit Agrawal <moagrawa>
Status: CLOSED ERRATA QA Contact: Manisha Saini <msaini>
Severity: medium Docs Contact:
Priority: medium    
Version: rhgs-3.4CC: jahernan, jijoy, jthottan, mchangir, moagrawa, msaini, pasik, pprakash, puebele, rhs-bugs, sabose, sheggodu, skoduri, storage-qa-internal
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: RHGS 3.5.z Batch Update 4   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-6.0-50 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-04-29 07:20:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1577325    
Attachments:
Description Flags
Volume export via Ganesha Script none

Description Manisha Saini 2018-05-14 19:06:26 UTC 
Description of problem:

Hit this issue while reproducing BZ- https://bugzilla.redhat.com/show_bug.cgi?id=1577325#c3

While exporting volume in loop,export got failed on one of the node (Same node from which the ganesha enable volume set operation was running in loop).

Ganesha and Glusterd service was up and running on this particular node on which export got failed


----------------------------
15/05/2018 00:04:40 : epoch 9c470000 : dhcp37-87.lab.eng.blr.redhat.com : ganesha.nfsd-8858[dbus_heartbeat] glusterfs_get_fs :FSAL :CRIT :Unable to initialize volume. Volume: ecvol54
15/05/2018 00:04:45 : epoch 9c470000 : dhcp37-87.lab.eng.blr.redhat.com : ganesha.nfsd-8858[dbus_heartbeat] mdcache_fsal_create_export :FSAL :MAJ :Failed to call create_export on underlying FSAL GLUSTER
15/05/2018 00:04:45 : epoch 9c470000 : dhcp37-87.lab.eng.blr.redhat.com : ganesha.nfsd-8858[dbus_heartbeat] fsal_cfg_commit :CONFIG :CRIT :Could not create export for (/ecvol54) to (/ecvol54)
15/05/2018 00:04:45 : epoch 9c470000 : dhcp37-87.lab.eng.blr.redhat.com : ganesha.nfsd-8858[dbus_heartbeat] gsh_export_addexport :EXPORT :CRIT :0 export entries in /var/run/gluster/shared_storage/nfs-ganesha/exports/export.ecvol54.conf added because (export create, block validation) errors
15/05/2018 00:04:45 : epoch 9c470000 : dhcp37-87.lab.eng.blr.redhat.com : ganesha.nfsd-8858[dbus_heartbeat] dbus_message_entrypoint :DBUS :MAJ :Method (AddExport) on (org.ganesha.nfsd.exportmgr) failed: name = (org.freedesktop.DBus.Error.InvalidFileContent), message = (0 export entries in /var/run/gluster/shared_storage/nfs-ganesha/exports/export.ecvol54.conf added because (export create, block validation) errors. Details:
Config File (/var/run/gluster/shared_storage/nfs-ganesha/exports/export.ecvol54.conf:7): 1 validation errors in block FSAL
Config File (/var/run/gluster/shared_storage/nfs-ganesha/exports/export.ecvol54.conf:7): Errors processing block (FSAL)
Config File (/var/run/gluster/shared_storage/nfs-ganesha/exports/export.ecvol54.conf:4): 1 validation errors in block EXPORT
Config File (/var/run/gluster/shared_storage/nfs-ganesha/exports/export.ecvol54.conf:4): Errors found in configuration block EXPORT
------------------------------


Version-Release number of selected component (if applicable):

# rpm -qa | grep ganesha
nfs-ganesha-2.5.5-6.el7rhgs.x86_64
nfs-ganesha-gluster-2.5.5-6.el7rhgs.x86_64
glusterfs-ganesha-3.12.2-9.el7rhgs.x86_64


How reproducible:
1/1

Steps to Reproduce:
1.Create 5 node ganesha cluster
2.Create and start 98 volumes
3.Start exporting the volumes in loop via ganesha 


Actual results:
Volume export got failed on one of the node.I terminated the script when failure was observed in order to provide setup in same state for BZ- 1577325.


Expected results:
Volume export should not fail.

Additional info:


Node On which the export got failed--

----------
# showmount -e
Export list for dhcp37-87.lab.eng.blr.redhat.com:
/ecvol1  (everyone)
/ecvol2  (everyone)
/ecvol3  (everyone)
/ecvol4  (everyone)
/ecvol5  (everyone)
/ecvol6  (everyone)
/ecvol7  (everyone)
/ecvol8  (everyone)
/ecvol9  (everyone)
/ecvol10 (everyone)
/ecvol11 (everyone)
/ecvol12 (everyone)
/ecvol13 (everyone)
/ecvol14 (everyone)
/ecvol15 (everyone)
/ecvol16 (everyone)
/ecvol17 (everyone)
/ecvol18 (everyone)
/ecvol19 (everyone)
/ecvol20 (everyone)
/ecvol21 (everyone)
/ecvol22 (everyone)
/ecvol23 (everyone)
/ecvol24 (everyone)
/ecvol25 (everyone)
/ecvol26 (everyone)
/ecvol27 (everyone)
/ecvol28 (everyone)
/ecvol29 (everyone)
/ecvol30 (everyone)
/ecvol31 (everyone)
/ecvol32 (everyone)
/ecvol33 (everyone)
/ecvol34 (everyone)
/ecvol35 (everyone)
/ecvol36 (everyone)
/ecvol37 (everyone)
/ecvol38 (everyone)
/ecvol39 (everyone)
/ecvol40 (everyone)
/ecvol41 (everyone)
/ecvol42 (everyone)
/ecvol43 (everyone)
/ecvol44 (everyone)
/ecvol45 (everyone)
/ecvol46 (everyone)
/ecvol47 (everyone)
/ecvol48 (everyone)
/ecvol49 (everyone)
/ecvol50 (everyone)
/ecvol51 (everyone)
/ecvol52 (everyone)
/ecvol53 (everyone)

-----------------

On of the other node where export got Passed---


------------------

# showmount -e
Export list for dhcp47-44.lab.eng.blr.redhat.com:
/ecvol1  (everyone)
/ecvol2  (everyone)
/ecvol3  (everyone)
/ecvol4  (everyone)
/ecvol5  (everyone)
/ecvol6  (everyone)
/ecvol7  (everyone)
/ecvol8  (everyone)
/ecvol9  (everyone)
/ecvol10 (everyone)
/ecvol11 (everyone)
/ecvol12 (everyone)
/ecvol13 (everyone)
/ecvol14 (everyone)
/ecvol15 (everyone)
/ecvol16 (everyone)
/ecvol17 (everyone)
/ecvol18 (everyone)
/ecvol19 (everyone)
/ecvol20 (everyone)
/ecvol21 (everyone)
/ecvol22 (everyone)
/ecvol23 (everyone)
/ecvol24 (everyone)
/ecvol25 (everyone)
/ecvol26 (everyone)
/ecvol27 (everyone)
/ecvol28 (everyone)
/ecvol29 (everyone)
/ecvol30 (everyone)
/ecvol31 (everyone)
/ecvol32 (everyone)
/ecvol33 (everyone)
/ecvol34 (everyone)
/ecvol35 (everyone)
/ecvol36 (everyone)
/ecvol37 (everyone)
/ecvol38 (everyone)
/ecvol39 (everyone)
/ecvol40 (everyone)
/ecvol41 (everyone)
/ecvol42 (everyone)
/ecvol43 (everyone)
/ecvol44 (everyone)
/ecvol45 (everyone)
/ecvol46 (everyone)
/ecvol47 (everyone)
/ecvol48 (everyone)
/ecvol49 (everyone)
/ecvol50 (everyone)
/ecvol51 (everyone)
/ecvol52 (everyone)
/ecvol53 (everyone)
/ecvol54 (everyone)

-----------------
--------------
Export got failed for ecvol54
--------------
# gluster v info ecvol54
 
Volume Name: ecvol54
Type: Disperse
Volume ID: dc957ade-8332-47f0-ad20-5427221b0f6a
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (4 + 2) = 6
Transport-type: tcp
Bricks:
Brick1: dhcp37-87.lab.eng.blr.redhat.com:/gluster/brick2/54
Brick2: dhcp46-209.lab.eng.blr.redhat.com:/gluster/brick2/54
Brick3: dhcp47-3.lab.eng.blr.redhat.com:/gluster/brick2/54
Brick4: dhcp47-82.lab.eng.blr.redhat.com:/gluster/brick2/54
Brick5: dhcp47-44.lab.eng.blr.redhat.com:/gluster/brick2/54
Brick6: dhcp37-87.lab.eng.blr.redhat.com:/gluster/brick4/b54
Options Reconfigured:
ganesha.enable: on
features.cache-invalidation: on
transport.address-family: inet
nfs.disable: on
nfs-ganesha: enable
cluster.enable-shared-storage: enable


[root@dhcp37-87 ganesha]# service nfs-ganesha status
Redirecting to /bin/systemctl status nfs-ganesha.service
● nfs-ganesha.service - NFS-Ganesha file server
   Loaded: loaded (/usr/lib/systemd/system/nfs-ganesha.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2018-05-14 18:17:36 IST; 6h ago
     Docs: http://github.com/nfs-ganesha/nfs-ganesha/wiki
 Main PID: 8858 (ganesha.nfsd)
    Tasks: 705
   CGroup: /system.slice/nfs-ganesha.service
           └─8858 /usr/bin/ganesha.nfsd -L /var/log/ganesha/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT -E 6555417408369524736

May 14 18:17:34 dhcp37-87.lab.eng.blr.redhat.com systemd[1]: Starting NFS-Ganesha file server...
May 14 18:17:36 dhcp37-87.lab.eng.blr.redhat.com systemd[1]: Started NFS-Ganesha file server.



Script was checking for export after each volume set operation ---

----------------------------------Iteration51-----------------------------
volume set: success
*****************Node1********************
/ecvol51 (everyone)
*****************Node2********************
/ecvol51 (everyone)
*****************Node3********************
/ecvol51 (everyone)
*****************Node4********************
/ecvol51 (everyone)
*****************Node5********************
/ecvol51 (everyone)
----------------------------------Iteration52-----------------------------
volume set: success
*****************Node1********************
/ecvol52 (everyone)
*****************Node2********************
/ecvol52 (everyone)
*****************Node3********************
/ecvol52 (everyone)
*****************Node4********************
/ecvol52 (everyone)
*****************Node5********************
/ecvol52 (everyone)

----------------------------------Iteration53-----------------------------
volume set: success
*****************Node1********************
/ecvol53 (everyone)
*****************Node2********************
/ecvol53 (everyone)
*****************Node3********************
/ecvol53 (everyone)
*****************Node4********************
/ecvol53 (everyone)
*****************Node5********************
/ecvol53 (everyone)
----------------------------------Iteration54-----------------------------
volume set: success
*****************Node1********************
*****************Node2********************
/ecvol54 (everyone)
*****************Node3********************
/ecvol54 (everyone)
*****************Node4********************

----------------
Comment 7 Manisha Saini 2018-05-15 06:31:22 UTC 
Resetting needinfo on Atin which got cleared.
Comment 23 Manisha Saini 2018-05-21 11:05:12 UTC 
Created attachment 1439566 [details]
Volume export via Ganesha Script
Comment 26 Atin Mukherjee 2018-11-10 08:05:42 UTC 
So what's the conclusion on this bug? Did we manage to hit this again in the regression cycle? If not, any particular reason why this bug needs to be kept open? If this is constantly reproducible on VMs have we checked if the resource footprint for the same are according to the standard requirements of RHGS?
Comment 46 errata-xmlrpc 2021-04-29 07:20:37 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (glusterfs bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1462
Comment 47 Red Hat Bugzilla 2023-09-15 00:08:07 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days