Description of problem: ======================= 'rm -rf <dirs>' throws Input / output errors during delete operations. [root@rhs-client29 linux]# ls -l total 0 drwxr-xr-x. 3 root root 0 Apr 14 13:48 linux.1 drwxr-xr-x. 3 root root 0 Apr 14 13:48 linux.10 drwxr-xr-x. 3 root root 0 Apr 14 13:48 linux.2 drwxr-xr-x. 3 root root 0 Apr 14 13:48 linux.3 drwxr-xr-x. 3 root root 0 Apr 14 13:48 linux.4 drwxr-xr-x. 3 root root 0 Apr 14 13:48 linux.5 drwxr-xr-x. 3 root root 0 Apr 14 13:48 linux.6 drwxr-xr-x. 3 root root 0 Apr 14 13:48 linux.7 drwxr-xr-x. 3 root root 0 Apr 14 13:48 linux.8 drwxr-xr-x. 3 root root 0 Apr 14 13:48 linux.9 [root@rhs-client29 linux]# rm -rf linux.1 rm: cannot remove `linux.1/linux-3.19/Documentation/sound/alsa/HD-Audio-Models.txt': Input/output error rm: cannot remove `linux.1/linux-3.19/Documentation/sound/alsa/HD-Audio.txt': Input/output error rm: cannot remove `linux.1/linux-3.19/Documentation/sound/alsa/Procfile.txt': Input/output error rm: cannot remove `linux.1/linux-3.19/Documentation/sound/alsa/VIA82xx-mixer.txt': Input/output error rm: cannot remove `linux.1/linux-3.19/Documentation/sound/alsa/alsa-parameters.txt': Input/output error rm: cannot remove `linux.1/linux-3.19/Documentation/sound/alsa/hdspm.txt': Input/output error rm: cannot remove `linux.1/linux-3.19/Documentation/sound/alsa/powersave.txt': Input/output error rm: cannot remove `linux.1/linux-3.19/Documentation/sound/alsa/seq_oss.html': Input/output error rm: cannot remove `linux.1/linux-3.19/Documentation/sound/alsa/serial-u16550.txt': Input/output error rm: cannot remove `linux.1/linux-3.19/Documentation/sound/alsa/soc/overview.txt': Input/output error ^C [root@rhs-client29 linux]# pwd /mnt/nfs/linux [root@rhs-client29 linux]# mount /dev/mapper/vg_rhsclient29-lv_root on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0") /dev/sda1 on /boot type ext4 (rw) /dev/mapper/vg_rhsclient29-lv_home on /home type ext4 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) gvfs-fuse-daemon on /root/.gvfs type fuse.gvfs-fuse-daemon (rw,nosuid,nodev) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) vertigo:/testvol on /mnt/nfs type nfs (rw,noac,addr=10.70.34.56) [root@rhs-client29 linux]# NFS Log file shows: =============== [2015-04-15 09:55:42.678352] I [ec-heal.c:546:ec_heal_init] 0-ec: Healing '<gfid:05cae3d6-feb7-4618-ae0d-c1c033ca0ae2>/devicetree', gfid 9bfe7338-e8cb-4223-b55b-7cae0cad31b9 [2015-04-15 09:55:42.865724] W [ec-combine.c:858:ec_combine_check] 0-testvol-disperse-0: Mismatching xdata in answers of 'LOOKUP' [2015-04-15 09:55:42.865768] W [ec-combine.c:858:ec_combine_check] 0-testvol-disperse-0: Mismatching xdata in answers of 'LOOKUP' [2015-04-15 09:55:42.865895] W [ec-combine.c:858:ec_combine_check] 0-testvol-disperse-0: Mismatching xdata in answers of 'LOOKUP' [2015-04-15 09:55:43.159871] W [ec-combine.c:858:ec_combine_check] 0-testvol-disperse-0: Mismatching xdata in answers of 'LOOKUP' [2015-04-15 09:55:43.159908] W [ec-combine.c:858:ec_combine_check] 0-testvol-disperse-0: Mismatching xdata in answers of 'LOOKUP' [2015-04-15 09:55:43.159941] W [ec-combine.c:858:ec_combine_check] 0-testvol-disperse-0: Mismatching xdata in answers of 'LOOKUP' [2015-04-15 09:55:43.459021] W [ec-common.c:162:ec_check_status] 0-testvol-disperse-0: Operation failed on some subvolumes (up=FFF, mask=FFA, remaining=0, good=FFA, bad=5) [2015-04-15 09:55:43.459084] I [ec-heal.c:546:ec_heal_init] 0-ec: Healing '<gfid:05cae3d6-feb7-4618-ae0d-c1c033ca0ae2>/devicetree', gfid 9bfe7338-e8cb-4223-b55b-7cae0cad31b9 [2015-04-15 09:55:43.459880] W [ec-common.c:407:ec_child_select] 0-testvol-disperse-0: Executing operation with some subvolumes unavailable (5) [2015-04-15 09:55:43.609652] W [ec-combine.c:858:ec_combine_check] 0-testvol-disperse-0: Mismatching xdata in answers of 'LOOKUP' [2015-04-15 09:55:43.609725] W [ec-combine.c:858:ec_combine_check] 0-testvol-disperse-0: Mismatching xdata in answers of 'LOOKUP' [2015-04-15 09:55:43.610976] W [ec-combine.c:858:ec_combine_check] 0-testvol-disperse-0: Mismatching xdata in answers of 'LOOKUP' [2015-04-15 09:55:43.611088] W [ec-combine.c:858:ec_combine_check] 0-testvol-disperse-0: Mismatching xdata in answers of 'LOOKUP' [2015-04-15 09:55:43.611203] W [ec-combine.c:858:ec_combine_check] 0-testvol-disperse-0: Mismatching xdata in answers of 'LOOKUP' ^C Version-Release number of selected component (if applicable): ============================================================= [root@vertigo ~]# gluster --version glusterfs 3.7dev built on Apr 13 2015 07:14:27 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com> GlusterFS comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of GlusterFS under the terms of the GNU General Public License. How reproducible: ================== 100% Steps to Reproduce: 1. Create a distributed-disperse (2x(8+4)) volume, enable quota, USS and epoll 2. Untar a linux tarball into multiple directories on nfs mount 3. Now, delete the directories. Actual results: Input/ Output errors Expected results: No errors should be seen Additional info: sosreports of the node will be attached.
Created attachment 1014668 [details] sosreport of client
This is seen even on fuse. [root@dhcp37-61 linux]# rm -rf dir.5 rm: cannot remove `dir.5/linux-3.19/Documentation/arm': Input/output error rm: cannot remove `dir.5/linux-3.19/Documentation/devicetree/bindings/arm/rockchip': Input/output error rm: cannot remove `dir.5/linux-3.19/Documentation/devicetree/bindings/clock': Input/output error rm: cannot remove `dir.5/linux-3.19/Documentation/devicetree/bindings/drm': Input/output error rm: cannot remove `dir.5/linux-3.19/Documentation/devicetree/bindings/iio': Input/output error rm: cannot remove `dir.5/linux-3.19/Documentation/devicetree/bindings/mipi/dsi': Is a directory rm: cannot remove `dir.5/linux-3.19/Documentation/devicetree/bindings/mips/cavium': Is a directory rm: cannot remove `dir.5/linux-3.19/Documentation/devicetree/bindings/soc/fsl': Input/output error rm: cannot remove `dir.5/linux-3.19/Documentation/filesystems': Input/output error rm: cannot remove `dir.5/linux-3.19/Documentation/i2c': Input/output error rm: cannot remove `dir.5/linux-3.19/Documentation/misc-devices': Input/output error rm: cannot remove `dir.5/linux-3.19/Documentation/virtual/kvm': Is a directory rm: cannot remove `dir.5/linux-3.19/Documentation/w1/masters': Is a directory rm: cannot remove `dir.5/linux-3.19/arch/arc': Input/output error rm: cannot remove `dir.5/linux-3.19/arch/arm/mach-gemini': Directory not empty rm: cannot remove `dir.5/linux-3.19/arch/arm/mach-iop33x/include/mach': Input/output error rm: cannot remove `dir.5/linux-3.19/arch/arm/mach-sa1100/include/mach': Is a directory rm: cannot remove `dir.5/linux-3.19/arch/arm/mach-versatile/include': Input/output error [root@dhcp37-61 linux]# df -h . Filesystem Size Used Avail Use% Mounted on ninja:/testvol 7.3T 109G 7.2T 2% /mnt/fuse1 You have new mail in /var/spool/mail/root [root@dhcp37-61 linux]# mount /dev/mapper/vg_dhcp3761-lv_root on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0") /dev/vda1 on /boot type ext4 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) ninja:/testvol on /mnt/fuse1 type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072) [root@dhcp37-61 linux]#
This seen on the latest 3.7.0.2 RHGS build too and the entries are becoming stale.
REVIEW: http://review.gluster.org/10806 (cluster/ec: Handle lookup failures while op in progress) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu)
COMMIT: http://review.gluster.org/10806 committed in master by Pranith Kumar Karampuri (pkarampu) ------ commit 64d2a5a25b64601958c0f47ec2e6356226f97d4b Author: Pranith Kumar K <pkarampu> Date: Mon May 18 14:46:55 2015 +0530 cluster/ec: Handle lookup failures while op in progress Change-Id: Ia1834ec23d5de615526d4d4e4d2e32aff155b7f7 BUG: 1211962 Signed-off-by: Pranith Kumar K <pkarampu> Reviewed-on: http://review.gluster.org/10806 Tested-by: NetBSD Build System Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Xavier Hernandez <xhernandez>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report. glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/ [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user