Description of problem: Hit this issue while reproducing BZ- https://bugzilla.redhat.com/show_bug.cgi?id=1577325#c3 While exporting volume in loop,export got failed on one of the node (Same node from which the ganesha enable volume set operation was running in loop). Ganesha and Glusterd service was up and running on this particular node on which export got failed ---------------------------- 15/05/2018 00:04:40 : epoch 9c470000 : dhcp37-87.lab.eng.blr.redhat.com : ganesha.nfsd-8858[dbus_heartbeat] glusterfs_get_fs :FSAL :CRIT :Unable to initialize volume. Volume: ecvol54 15/05/2018 00:04:45 : epoch 9c470000 : dhcp37-87.lab.eng.blr.redhat.com : ganesha.nfsd-8858[dbus_heartbeat] mdcache_fsal_create_export :FSAL :MAJ :Failed to call create_export on underlying FSAL GLUSTER 15/05/2018 00:04:45 : epoch 9c470000 : dhcp37-87.lab.eng.blr.redhat.com : ganesha.nfsd-8858[dbus_heartbeat] fsal_cfg_commit :CONFIG :CRIT :Could not create export for (/ecvol54) to (/ecvol54) 15/05/2018 00:04:45 : epoch 9c470000 : dhcp37-87.lab.eng.blr.redhat.com : ganesha.nfsd-8858[dbus_heartbeat] gsh_export_addexport :EXPORT :CRIT :0 export entries in /var/run/gluster/shared_storage/nfs-ganesha/exports/export.ecvol54.conf added because (export create, block validation) errors 15/05/2018 00:04:45 : epoch 9c470000 : dhcp37-87.lab.eng.blr.redhat.com : ganesha.nfsd-8858[dbus_heartbeat] dbus_message_entrypoint :DBUS :MAJ :Method (AddExport) on (org.ganesha.nfsd.exportmgr) failed: name = (org.freedesktop.DBus.Error.InvalidFileContent), message = (0 export entries in /var/run/gluster/shared_storage/nfs-ganesha/exports/export.ecvol54.conf added because (export create, block validation) errors. Details: Config File (/var/run/gluster/shared_storage/nfs-ganesha/exports/export.ecvol54.conf:7): 1 validation errors in block FSAL Config File (/var/run/gluster/shared_storage/nfs-ganesha/exports/export.ecvol54.conf:7): Errors processing block (FSAL) Config File (/var/run/gluster/shared_storage/nfs-ganesha/exports/export.ecvol54.conf:4): 1 validation errors in block EXPORT Config File (/var/run/gluster/shared_storage/nfs-ganesha/exports/export.ecvol54.conf:4): Errors found in configuration block EXPORT ------------------------------ Version-Release number of selected component (if applicable): # rpm -qa | grep ganesha nfs-ganesha-2.5.5-6.el7rhgs.x86_64 nfs-ganesha-gluster-2.5.5-6.el7rhgs.x86_64 glusterfs-ganesha-3.12.2-9.el7rhgs.x86_64 How reproducible: 1/1 Steps to Reproduce: 1.Create 5 node ganesha cluster 2.Create and start 98 volumes 3.Start exporting the volumes in loop via ganesha Actual results: Volume export got failed on one of the node.I terminated the script when failure was observed in order to provide setup in same state for BZ- 1577325. Expected results: Volume export should not fail. Additional info: Node On which the export got failed-- ---------- # showmount -e Export list for dhcp37-87.lab.eng.blr.redhat.com: /ecvol1 (everyone) /ecvol2 (everyone) /ecvol3 (everyone) /ecvol4 (everyone) /ecvol5 (everyone) /ecvol6 (everyone) /ecvol7 (everyone) /ecvol8 (everyone) /ecvol9 (everyone) /ecvol10 (everyone) /ecvol11 (everyone) /ecvol12 (everyone) /ecvol13 (everyone) /ecvol14 (everyone) /ecvol15 (everyone) /ecvol16 (everyone) /ecvol17 (everyone) /ecvol18 (everyone) /ecvol19 (everyone) /ecvol20 (everyone) /ecvol21 (everyone) /ecvol22 (everyone) /ecvol23 (everyone) /ecvol24 (everyone) /ecvol25 (everyone) /ecvol26 (everyone) /ecvol27 (everyone) /ecvol28 (everyone) /ecvol29 (everyone) /ecvol30 (everyone) /ecvol31 (everyone) /ecvol32 (everyone) /ecvol33 (everyone) /ecvol34 (everyone) /ecvol35 (everyone) /ecvol36 (everyone) /ecvol37 (everyone) /ecvol38 (everyone) /ecvol39 (everyone) /ecvol40 (everyone) /ecvol41 (everyone) /ecvol42 (everyone) /ecvol43 (everyone) /ecvol44 (everyone) /ecvol45 (everyone) /ecvol46 (everyone) /ecvol47 (everyone) /ecvol48 (everyone) /ecvol49 (everyone) /ecvol50 (everyone) /ecvol51 (everyone) /ecvol52 (everyone) /ecvol53 (everyone) ----------------- On of the other node where export got Passed--- ------------------ # showmount -e Export list for dhcp47-44.lab.eng.blr.redhat.com: /ecvol1 (everyone) /ecvol2 (everyone) /ecvol3 (everyone) /ecvol4 (everyone) /ecvol5 (everyone) /ecvol6 (everyone) /ecvol7 (everyone) /ecvol8 (everyone) /ecvol9 (everyone) /ecvol10 (everyone) /ecvol11 (everyone) /ecvol12 (everyone) /ecvol13 (everyone) /ecvol14 (everyone) /ecvol15 (everyone) /ecvol16 (everyone) /ecvol17 (everyone) /ecvol18 (everyone) /ecvol19 (everyone) /ecvol20 (everyone) /ecvol21 (everyone) /ecvol22 (everyone) /ecvol23 (everyone) /ecvol24 (everyone) /ecvol25 (everyone) /ecvol26 (everyone) /ecvol27 (everyone) /ecvol28 (everyone) /ecvol29 (everyone) /ecvol30 (everyone) /ecvol31 (everyone) /ecvol32 (everyone) /ecvol33 (everyone) /ecvol34 (everyone) /ecvol35 (everyone) /ecvol36 (everyone) /ecvol37 (everyone) /ecvol38 (everyone) /ecvol39 (everyone) /ecvol40 (everyone) /ecvol41 (everyone) /ecvol42 (everyone) /ecvol43 (everyone) /ecvol44 (everyone) /ecvol45 (everyone) /ecvol46 (everyone) /ecvol47 (everyone) /ecvol48 (everyone) /ecvol49 (everyone) /ecvol50 (everyone) /ecvol51 (everyone) /ecvol52 (everyone) /ecvol53 (everyone) /ecvol54 (everyone) ----------------- -------------- Export got failed for ecvol54 -------------- # gluster v info ecvol54 Volume Name: ecvol54 Type: Disperse Volume ID: dc957ade-8332-47f0-ad20-5427221b0f6a Status: Started Snapshot Count: 0 Number of Bricks: 1 x (4 + 2) = 6 Transport-type: tcp Bricks: Brick1: dhcp37-87.lab.eng.blr.redhat.com:/gluster/brick2/54 Brick2: dhcp46-209.lab.eng.blr.redhat.com:/gluster/brick2/54 Brick3: dhcp47-3.lab.eng.blr.redhat.com:/gluster/brick2/54 Brick4: dhcp47-82.lab.eng.blr.redhat.com:/gluster/brick2/54 Brick5: dhcp47-44.lab.eng.blr.redhat.com:/gluster/brick2/54 Brick6: dhcp37-87.lab.eng.blr.redhat.com:/gluster/brick4/b54 Options Reconfigured: ganesha.enable: on features.cache-invalidation: on transport.address-family: inet nfs.disable: on nfs-ganesha: enable cluster.enable-shared-storage: enable [root@dhcp37-87 ganesha]# service nfs-ganesha status Redirecting to /bin/systemctl status nfs-ganesha.service ● nfs-ganesha.service - NFS-Ganesha file server Loaded: loaded (/usr/lib/systemd/system/nfs-ganesha.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2018-05-14 18:17:36 IST; 6h ago Docs: http://github.com/nfs-ganesha/nfs-ganesha/wiki Main PID: 8858 (ganesha.nfsd) Tasks: 705 CGroup: /system.slice/nfs-ganesha.service └─8858 /usr/bin/ganesha.nfsd -L /var/log/ganesha/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT -E 6555417408369524736 May 14 18:17:34 dhcp37-87.lab.eng.blr.redhat.com systemd[1]: Starting NFS-Ganesha file server... May 14 18:17:36 dhcp37-87.lab.eng.blr.redhat.com systemd[1]: Started NFS-Ganesha file server. Script was checking for export after each volume set operation --- ----------------------------------Iteration51----------------------------- volume set: success *****************Node1******************** /ecvol51 (everyone) *****************Node2******************** /ecvol51 (everyone) *****************Node3******************** /ecvol51 (everyone) *****************Node4******************** /ecvol51 (everyone) *****************Node5******************** /ecvol51 (everyone) ----------------------------------Iteration52----------------------------- volume set: success *****************Node1******************** /ecvol52 (everyone) *****************Node2******************** /ecvol52 (everyone) *****************Node3******************** /ecvol52 (everyone) *****************Node4******************** /ecvol52 (everyone) *****************Node5******************** /ecvol52 (everyone) ----------------------------------Iteration53----------------------------- volume set: success *****************Node1******************** /ecvol53 (everyone) *****************Node2******************** /ecvol53 (everyone) *****************Node3******************** /ecvol53 (everyone) *****************Node4******************** /ecvol53 (everyone) *****************Node5******************** /ecvol53 (everyone) ----------------------------------Iteration54----------------------------- volume set: success *****************Node1******************** *****************Node2******************** /ecvol54 (everyone) *****************Node3******************** /ecvol54 (everyone) *****************Node4******************** ----------------
Resetting needinfo on Atin which got cleared.
Created attachment 1439566 [details] Volume export via Ganesha Script
So what's the conclusion on this bug? Did we manage to hit this again in the regression cycle? If not, any particular reason why this bug needs to be kept open? If this is constantly reproducible on VMs have we checked if the resource footprint for the same are according to the standard requirements of RHGS?
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (glusterfs bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:1462
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days