Description of problem: ======================= On a distrbuted-disperse volume, when performed the steps mentioned in "Steps to reproduce", Linux kernel untar failed with errors immediate after add-brick. Version-Release number of selected component (if applicable): 3.12.2-8.el7rhgs.x86_64 How reproducible: Reported at first hit Steps to Reproduce: =================== 1) Create a Distributed-Disperse volume and start it. 2) Mount it on multiple clients. 3) From clients start below ops, client-1: Copy linux kernel package on mount point and create two dirs 1 and 2 and start linux kernel untar ; tar -xvf linux-4.9.27.tar.gz -C 1 client-2: tar -xvf linux-4.9.27.tar.gz -C 2 Client-3: while true; do ls -lRt;done Client-4: while true; do find;done 4) While step-3 in-progress, add bricks to the volume. Actual results: =============== Linux kernel untar failed with errors immediate after add-brick Expected results: ================= Linux kernel untar should continue without any errors/failures.
Linux untart from both clients failed with below errors. Client-1: ========== linux-4.9.27/Documentation/arm/VFP/ linux-4.9.27/Documentation/arm/VFP/release-notes.txt linux-4.9.27/Documentation/arm/cluster-pm-race-avoidance.txt tar: linux-4.9.27/Documentation/arm/cluster-pm-race-avoidance.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/firmware.txt tar: linux-4.9.27/Documentation/arm/firmware.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/kernel_mode_neon.txt tar: linux-4.9.27/Documentation/arm/kernel_mode_neon.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/kernel_user_helpers.txt tar: linux-4.9.27/Documentation/arm/kernel_user_helpers.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/keystone/ tar: linux-4.9.27/Documentation/arm/keystone: Cannot mkdir: Invalid argument linux-4.9.27/Documentation/arm/keystone/Overview.txt tar: linux-4.9.27/Documentation/arm/keystone/Overview.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/keystone/knav-qmss.txt tar: linux-4.9.27/Documentation/arm/keystone/knav-qmss.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/mem_alignment tar: linux-4.9.27/Documentation/arm/mem_alignment: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/memory.txt tar: linux-4.9.27/Documentation/arm/memory.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/nwfpe/ tar: linux-4.9.27/Documentation/arm/nwfpe: Cannot mkdir: Invalid argument linux-4.9.27/Documentation/arm/nwfpe/NOTES tar: linux-4.9.27/Documentation/arm/nwfpe/NOTES: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/nwfpe/README tar: linux-4.9.27/Documentation/arm/nwfpe/README: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/nwfpe/README.FPE tar: linux-4.9.27/Documentation/arm/nwfpe/README.FPE: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/nwfpe/TODO tar: linux-4.9.27/Documentation/arm/nwfpe/TODO: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/pxa/ tar: linux-4.9.27/Documentation/arm/pxa: Cannot mkdir: Invalid argument linux-4.9.27/Documentation/arm/pxa/mfp.txt tar: linux-4.9.27/Documentation/arm/pxa/mfp.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/sti/ tar: linux-4.9.27/Documentation/arm/sti: Cannot mkdir: Invalid argument linux-4.9.27/Documentation/arm/sti/overview.txt tar: linux-4.9.27/Documentation/arm/sti/overview.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/sti/stih407-overview.txt tar: linux-4.9.27/Documentation/arm/sti/stih407-overview.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/sti/stih415-overview.txt gzip: stdin: Bad file descriptor tar: linux-4.9.27/Documentation/arm/sti/stih415-overview.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/sti/stih416-overview.txt tar: linux-4.9.27/Documentation/arm/sti/stih416-overview.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/sti/stih418-overview.txt tar: linux-4.9.27/Documentation/arm/sti/stih418-overview.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/stm32/ tar: linux-4.9.27/Documentation/arm/stm32: Cannot mkdir: Invalid argument linux-4.9.27/Documentation/arm/stm32/overview.txt tar: linux-4.9.27/Documentation/arm/stm32/overview.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/stm32/stm32f429-overview.txt tar: linux-4.9.27/Documentation/arm/stm32/stm32f429-overview.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/sunxi/ tar: linux-4.9.27/Documentation/arm/sunxi: Cannot mkdir: Invalid argument linux-4.9.27/Documentation/arm/sunxi/README tar: linux-4.9.27/Documentation/arm/sunxi/README: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/sunxi/clocks.txt tar: linux-4.9.27/Documentation/arm/sunxi/clocks.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/swp_emulation tar: linux-4.9.27/Documentation/arm/swp_emulation: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/tcm.txt tar: linux-4.9.27/Documentation/arm/tcm.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/uefi.txt tar: linux-4.9.27/Documentation/arm/uefi.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/vlocks.txt tar: linux-4.9.27/Documentation/arm/vlocks.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm64/ tar: linux-4.9.27/Documentation/arm: Cannot utime: Invalid argument tar: linux-4.9.27/Documentation/arm: Cannot change ownership to uid 0, gid 0: Invalid argument tar: linux-4.9.27/Documentation/arm: Cannot change mode to rwxrwxr-x: Invalid argument tar: linux-4.9.27/Documentation/arm64: Cannot mkdir: Invalid argument linux-4.9.27/Documentation/arm64/acpi_object_usage.txt tar: linux-4.9.27/Documentation/arm64/acpi_object_usage.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm64/arm-acpi.txt tar: Unexpected EOF in archive tar: linux-4.9.27/Documentation: Cannot utime: Invalid argument tar: linux-4.9.27/Documentation: Cannot change ownership to uid 0, gid 0: Invalid argument tar: linux-4.9.27/Documentation: Cannot change mode to rwxrwxr-x: Invalid argument tar: linux-4.9.27: Cannot utime: Invalid argument tar: linux-4.9.27: Cannot change ownership to uid 0, gid 0: Invalid argument tar: linux-4.9.27: Cannot change mode to rwxrwxr-x: Invalid argument tar: Error is not recoverable: exiting now Client-2: ========= linux-4.9.27/Documentation/arm/SA1100/FreeBird tar: linux-4.9.27/Documentation/arm/SA1100/FreeBird: Cannot utime: Stale file handle tar: linux-4.9.27/Documentation/arm/SA1100/FreeBird: Cannot change ownership to uid 0, gid 0: Stale file handle tar: linux-4.9.27/Documentation/arm/SA1100/FreeBird: Cannot change mode to rw-rw-r--: Stale file handle tar: linux-4.9.27/Documentation/arm/SA1100/FreeBird: Cannot close: Bad file descriptor linux-4.9.27/Documentation/arm/SA1100/GraphicsClient tar: linux-4.9.27/Documentation/arm/SA1100/GraphicsClient: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/SA1100/GraphicsMaster tar: linux-4.9.27/Documentation/arm/SA1100/GraphicsMaster: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/SA1100/HUW_WEBPANEL tar: linux-4.9.27/Documentation/arm/SA1100/HUW_WEBPANEL: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/SA1100/Itsy tar: linux-4.9.27/Documentation/arm/SA1100/Itsy: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/SA1100/LART tar: linux-4.9.27/Documentation/arm/SA1100/LART: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/SA1100/PLEB tar: linux-4.9.27/Documentation/arm/SA1100/PLEB: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/SA1100/Pangolin tar: linux-4.9.27/Documentation/arm/SA1100/Pangolin: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/SA1100/Tifon tar: linux-4.9.27/Documentation/arm/SA1100/Tifon: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/SA1100/Yopy tar: linux-4.9.27/Documentation/arm/SA1100/Yopy: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/SA1100/empeg tar: linux-4.9.27/Documentation/arm/SA1100/empeg: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/SA1100/nanoEngine tar: linux-4.9.27/Documentation/arm/SA1100/nanoEngine: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/SA1100/serial_UART tar: linux-4.9.27/Documentation/arm/SA1100/serial_UART: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/SH-Mobile/ tar: linux-4.9.27/Documentation/arm/SA1100: Cannot utime: Invalid argument tar: linux-4.9.27/Documentation/arm/SA1100: Cannot change ownership to uid 0, gid 0: Invalid argument tar: linux-4.9.27/Documentation/arm/SA1100: Cannot change mode to rwxrwxr-x: Invalid argument tar: linux-4.9.27/Documentation/arm/SH-Mobile: Cannot mkdir: Invalid argument linux-4.9.27/Documentation/arm/SH-Mobile/.gitignore tar: linux-4.9.27/Documentation/arm/SH-Mobile/.gitignore: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/SPEAr/ tar: linux-4.9.27/Documentation/arm/SPEAr: Cannot mkdir: Invalid argument linux-4.9.27/Documentation/arm/SPEAr/overview.txt tar: linux-4.9.27/Documentation/arm/SPEAr/overview.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/Samsung-S3C24XX/ tar: linux-4.9.27/Documentation/arm/Samsung-S3C24XX: Cannot mkdir: Invalid argument linux-4.9.27/Documentation/arm/Samsung-S3C24XX/CPUfreq.txt tar: linux-4.9.27/Documentation/arm/Samsung-S3C24XX/CPUfreq.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/Samsung-S3C24XX/EB2410ITX.txt tar: linux-4.9.27/Documentation/arm/Samsung-S3C24XX/EB2410ITX.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/Samsung-S3C24XX/GPIO.txt tar: linux-4.9.27/Documentation/arm/Samsung-S3C24XX/GPIO.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/Samsung-S3C24XX/H1940.txt tar: linux-4.9.27/Documentation/arm/Samsung-S3C24XX/H1940.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/Samsung-S3C24XX/NAND.txt tar: linux-4.9.27/Documentation/arm/Samsung-S3C24XX/NAND.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/Samsung-S3C24XX/Overview.txt tar: linux-4.9.27/Documentation/arm/Samsung-S3C24XX/Overview.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/Samsung-S3C24XX/S3C2412.txt tar: linux-4.9.27/Documentation/arm/Samsung-S3C24XX/S3C2412.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/Samsung-S3C24XX/S3C2413.txt tar: linux-4.9.27/Documentation/arm/Samsung-S3C24XX/S3C2413.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/Samsung-S3C24XX/SMDK2440.txt tar: linux-4.9.27/Documentation/arm/Samsung-S3C24XX/SMDK2440.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/Samsung-S3C24XX/Suspend.txt tar: linux-4.9.27/Documentation/arm/Samsung-S3C24XX/Suspend.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/Samsung-S3C24XX/USB-Host.txt tar: linux-4.9.27/Documentation/arm/Samsung-S3C24XX/USB-Host.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/Samsung/ gzip: stdin: Bad file descriptor tar: linux-4.9.27/Documentation/arm/Samsung: Cannot mkdir: Invalid argument linux-4.9.27/Documentation/arm/Samsung/Bootloader-interface.txt tar: linux-4.9.27/Documentation/arm/Samsung/Bootloader-interface.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/Samsung/GPIO.txt tar: linux-4.9.27/Documentation/arm/Samsung/GPIO.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/Samsung/Overview.txt tar: linux-4.9.27/Documentation/arm/Samsung/Overview.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/Samsung/clksrc-change-registers.awk tar: linux-4.9.27/Documentation/arm/Samsung/clksrc-change-registers.awk: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/Setup tar: linux-4.9.27/Documentation/arm/Setup: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/VFP/ tar: linux-4.9.27/Documentation/arm/VFP: Cannot mkdir: Invalid argument linux-4.9.27/Documentation/arm/VFP/release-notes.txt tar: linux-4.9.27/Documentation/arm/VFP/release-notes.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/cluster-pm-race-avoidance.txt tar: linux-4.9.27/Documentation/arm/cluster-pm-race-avoidance.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/firmware.txt tar: linux-4.9.27/Documentation/arm/firmware.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/kernel_mode_neon.txt tar: linux-4.9.27/Documentation/arm/kernel_mode_neon.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/kernel_user_helpers.txt tar: linux-4.9.27/Documentation/arm/kernel_user_helpers.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/keystone/ tar: linux-4.9.27/Documentation/arm/keystone: Cannot mkdir: Invalid argument linux-4.9.27/Documentation/arm/keystone/Overview.txt tar: linux-4.9.27/Documentation/arm/keystone/Overview.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/keystone/knav-qmss.txt tar: linux-4.9.27/Documentation/arm/keystone/knav-qmss.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/mem_alignment tar: linux-4.9.27/Documentation/arm/mem_alignment: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/memory.txt tar: linux-4.9.27/Documentation/arm/memory.txt: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/nwfpe/ tar: linux-4.9.27/Documentation/arm/nwfpe: Cannot mkdir: Invalid argument linux-4.9.27/Documentation/arm/nwfpe/NOTES tar: linux-4.9.27/Documentation/arm/nwfpe/NOTES: Cannot open: Invalid argument linux-4.9.27/Documentation/arm/nwfpe/README tar: Unexpected EOF in archive tar: linux-4.9.27/Documentation/arm: Cannot utime: Invalid argument tar: linux-4.9.27/Documentation/arm: Cannot change ownership to uid 0, gid 0: Invalid argument tar: linux-4.9.27/Documentation/arm: Cannot change mode to rwxrwxr-x: Invalid argument tar: linux-4.9.27/Documentation: Cannot utime: Invalid argument tar: linux-4.9.27/Documentation: Cannot change ownership to uid 0, gid 0: Invalid argument tar: linux-4.9.27/Documentation: Cannot change mode to rwxrwxr-x: Invalid argument tar: linux-4.9.27: Cannot utime: Invalid argument tar: linux-4.9.27: Cannot change ownership to uid 0, gid 0: Invalid argument tar: linux-4.9.27: Cannot change mode to rwxrwxr-x: Invalid argument tar: Error is not recoverable: exiting now
Hi Xavi, I was able to reproduce the issue mentioned in comment8 and comment13. After an add-brick operation, one of the directories doesn't have trusted.dht.mds value set on the backend. Because of this reason, the next setxattr call was failing which resulted in the error. Log snippet [2020-05-20 04:59:12.181881] I [MSGID: 109063] [dht-layout.c:650:dht_layout_normalize] 2-patchy-dht: Found anomalies in /linux-4.13.5/Documentation/arm (gfid = 54bcccd6-ecd1-4ef6-9e94-720c24c85b5a). Holes=1 overlaps=0 [2020-05-20 05:14:49.539630] E [MSGID: 109011] [dht-inode-write.c:1296:dht_setattr] 2-patchy-dht: Failed to get mds subvol for path /linux-4.13.5/Documentation/arm/SA1100 [2020-05-20 05:14:49.540622] W [fuse-bridge.c:1665:fuse_setattr_cbk] 0-glusterfs-fuse: 9191: SETATTR() /linux-4.13.5/Documentation/arm/SA1100 => -1 (Invalid argument). Regards Rafi KC
*** Bug 1848532 has been marked as a duplicate of this bug. ***
The same issue is already fixed as part of RHGS 3.5.4 with this bug - https://bugzilla.redhat.com/show_bug.cgi?id=1924044 Also there is this pending issue seen 1 out of 10 times with distributed-disperse volume only, which is tracked with the bug - https://bugzilla.redhat.com/show_bug.cgi?id=1937314 Sunil, In this case can we go ahead and move this bug out of RHGS 3.5.5 and take the required action of CLOSING this bug as DUP of other ?
*** This bug has been marked as a duplicate of bug 1924044 ***
(In reply to Mohit Agrawal from comment #33) > > *** This bug has been marked as a duplicate of bug 1924044 *** Thanks Mohit. As this bug is CLOSED, just removing the acks and flags