Description of problem: Linux untars failed with "Operation not permitted"/"Permission denied" when find's were running in parallel from another client (vers=3) "ganesha.nfsd-24864[work-140] glusterfs_setattr2 :FSAL :CRIT :setattrs failed with error Operation not permitted" messages were observed in ganesha.log snippet-- ------------ # tar xvf linux-4.9.5.tar.xz > msaini tar: linux-4.9.5/Documentation/devicetree/bindings/ata/nvidia,tegra124-ahci.txt: Cannot utime: Permission denied tar: linux-4.9.5/Documentation/devicetree/bindings/leds/backlight: Cannot change ownership to uid 0, gid 0: Operation not permitted tar: linux-4.9.5/Documentation/devicetree/bindings/phy/rockchip-usb-phy.txt: Cannot change mode to rw-rw-r--: Operation not permitted tar: linux-4.9.5/Documentation/devicetree/bindings/sound/wm8904.txt: Cannot utime: Permission denied tar: linux-4.9.5/arch/alpha/include/uapi/asm/Kbuild: Cannot open: Operation not permitted tar: linux-4.9.5/arch/arm/boot/dts/imx6q-rex-pro.dts: Cannot change ownership to uid 0, gid 0: Operation not permitted tar: linux-4.9.5/arch/arm/boot/dts/vf-colibri-eval-v3.dtsi: Cannot change ownership to uid 0, gid 0: Operation not permitted tar: linux-4.9.5/arch/arm/mach-mvebu/coherency.h: Cannot change ownership to uid 0, gid 0: Operation not permitted tar: linux-4.9.5/arch/mips/include/asm/sibyte/sb1250_uart.h: Cannot open: Permission denied tar: linux-4.9.5/arch/mips/include/asm/sn/sn0/hubmd.h: Cannot change ownership to uid 0, gid 0: Operation not permitted tar: linux-4.9.5/arch/powerpc/include/asm/reg_booke.h: Cannot open: Operation not permitted tar: linux-4.9.5/drivers/hid/wacom_sys.c: Cannot change mode to rw-rw-r--: Operation not permitted tar: linux-4.9.5/drivers/media/rc/keymaps/rc-hauppauge.c: Cannot open: Operation not permitted tar: linux-4.9.5/drivers/net/ethernet/mellanox/mlxsw/spectrum_buffers.c: Cannot utime: Permission denied tar: linux-4.9.5/include/media/drv-intf/si476x.h: Cannot open: Operation not permitted tar: linux-4.9.5/include/uapi/linux/personality.h: Cannot open: Operation not permitted tar: linux-4.9.5/include/uapi/linux/phantom.h: Cannot open: Permission denied tar: linux-4.9.5/include/uapi/linux/qrtr.h: Cannot open: Permission denied tar: linux-4.9.5/include/uapi/linux/snmp.h: Cannot change mode to rw-rw-r--: Operation not permitted tar: Exiting with failure status due to previous errors --------------- ganesha.log- -------------- 20/05/2018 02:29:57 : epoch 1af30000 : moonshine.lab.eng.blr.redhat.com : ganesha.nfsd-24864[work-204] glusterfs_setattr2 :FSAL :CRIT :setattrs failed with error Permission denied 20/05/2018 02:30:09 : epoch 1af30000 : moonshine.lab.eng.blr.redhat.com : ganesha.nfsd-24864[work-185] glusterfs_setattr2 :FSAL :CRIT :setattrs failed with error Operation not permitted 20/05/2018 02:30:16 : epoch 1af30000 : moonshine.lab.eng.blr.redhat.com : ganesha.nfsd-24864[work-51] glusterfs_setattr2 :FSAL :CRIT :setattrs failed with error Operation not permitted 20/05/2018 02:30:28 : epoch 1af30000 : moonshine.lab.eng.blr.redhat.com : ganesha.nfsd-24864[work-219] glusterfs_setattr2 :FSAL :CRIT :setattrs failed with error Permission denied 20/05/2018 02:31:04 : epoch 1af30000 : moonshine.lab.eng.blr.redhat.com : ganesha.nfsd-24864[work-137] glusterfs_setattr2 :FSAL :CRIT :setattrs failed with error Operation not permitted 20/05/2018 02:31:19 : epoch 1af30000 : moonshine.lab.eng.blr.redhat.com : ganesha.nfsd-24864[work-153] glusterfs_setattr2 :FSAL :CRIT :setattrs failed with error Operation not permitted 20/05/2018 02:31:34 : epoch 1af30000 : moonshine.lab.eng.blr.redhat.com : ganesha.nfsd-24864[work-69] glusterfs_setattr2 :FSAL :CRIT :setattrs failed with error Operation not permitted 20/05/2018 02:31:56 : epoch 1af30000 : moonshine.lab.eng.blr.redhat.com : ganesha.nfsd-24864[work-150] glusterfs_setattr2 :FSAL :CRIT :setattrs failed with error Operation not permitted 20/05/2018 02:33:32 : epoch 1af30000 : moonshine.lab.eng.blr.redhat.com : ganesha.nfsd-24864[work-69] glusterfs_setattr2 :FSAL :CRIT :setattrs failed with error Operation not permitted 20/05/2018 02:34:10 : epoch 1af30000 : moonshine.lab.eng.blr.redhat.com : ganesha.nfsd-24864[work-49] glusterfs_setattr2 :FSAL :CRIT :setattrs failed with error Operation not permitted 20/05/2018 02:36:59 : epoch 1af30000 : moonshine.lab.eng.blr.redhat.com : ganesha.nfsd-24864[work-203] glusterfs_setattr2 :FSAL :CRIT :setattrs failed with error Operation not permitted 20/05/2018 02:38:02 : epoch 1af30000 : moonshine.lab.eng.blr.redhat.com : ganesha.nfsd-24864[work-94] glusterfs_setattr2 :FSAL :CRIT :setattrs failed with error Operation not permitted 20/05/2018 02:38:46 : epoch 1af30000 : moonshine.lab.eng.blr.redhat.com : ganesha.nfsd-24864[work-104] glusterfs_setattr2 :FSAL :CRIT :setattrs failed with error Permission denied 20/05/2018 02:42:32 : epoch 1af30000 : moonshine.lab.eng.blr.redhat.com : ganesha.nfsd-24864[work-140] glusterfs_setattr2 :FSAL :CRIT :setattrs failed with error Operation not permitted 20/05/2018 02:42:53 : epoch 1af30000 : moonshine.lab.eng.blr.redhat.com : ganesha.nfsd-24864[work-115] glusterfs_setattr2 :FSAL :CRIT :setattrs failed with error Operation not permitted 20/05/2018 02:42:54 : epoch 1af30000 : moonshine.lab.eng.blr.redhat.com : ganesha.nfsd-24864[work-226] glusterfs_setattr2 :FSAL :CRIT :setattrs failed with error Operation not permitted ----------------------------- Version-Release number of selected component (if applicable): nfs-ganesha-gluster-2.5.5-7.el7rhgs.x86_64 nfs-ganesha-debuginfo-2.5.5-7.el7rhgs.x86_64 nfs-ganesha-2.5.5-7.el7rhgs.x86_64 glusterfs-ganesha-3.12.2-11.el7rhgs.x86_64 How reproducible: 3/3 Steps to Reproduce: 1.Create 6 node Ganesha cluster 2.Create 6*(4+2) Distributed-Disperse volume 3.Export the volume via ganesha 4.Mount the volume to 2 clients with vers=3 5.Run following workload Client 1- Linux untars Client 2- find's in loop ( while true;do find . -mindepth 1 -type f;done ) Actual results: Linux unatrs failed with permission denied messages Expected results: Linux untar should pass Additional info: The same test passed with v4 protocol
So, the only functions that can return EPERM in glusterfs_setattr2() are glfs_ftruncate() and glfs_h_setattrs(), so this appears to be returned from gfapi.
In this one, we should add something to log which function had the error. Also, I'm not sure we should LogCrit, which untar shouldn't result in any permission errors, I'm not sure that it's impossible for permission errors to be expected in setattr2. It would be interesting to know the details of the files that are generating the errors. It might be interesting to run pjdfstests in a loop under the same scenario and see if any of it's tests fail unexpectedly. I suspect the repeated readdir are causing something to get flushed from mdcache and then a slightly different sequence of operations occurs resulting in libgfapi doing permission checks that without the mdcache evictions, would not be happening.
I wonder if this is related at all to https://bugzilla.redhat.com/show_bug.cgi?id=1580120
Good catch, Jiffin. Fix here: https://review.gerrithub.io/#/c/ffilz/nfs-ganesha/+/412483
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:2610