Description of problem: While deleting files from the fuse mount, it returned EIO errors. This was part of the stress testing exercise on AMI. Files were bring deleted from multiple clients. Before that while the files were being created, one of the servers was brought up and down at regular intervals. While deleting one the directories with a lot of image file, the rm process returned ENOTEMPTY and then it returned EIO. Version-Release number of selected component (if applicable): 3.3.0RC1 AMI Actual results: rm: cannot remove `tmp_2/Common/image-systems-fake-nas/jhegedus/si/images': Directory not empty rm: cannot remove `tmp_3': Input/output error rm: cannot remove `tmp_4': Input/output error rm: cannot remove `tmp_5': Input/output error rm: cannot remove `tmp_6': Input/output error rm: cannot remove `tmp_7': Input/output error rm: cannot remove `tmp_8': Input/output error rm: cannot remove `tmp_9': Input/output error rm: cannot remove `upload-ws': Input/output error rm: cannot remove `wrapper': Input/output error [root@ip-10-142-243-68 gfs]# [root@ip-10-142-243-68 gfs]# [root@ip-10-142-243-68 gfs]# [root@ip-10-142-243-68 gfs]# [root@ip-10-142-243-68 gfs]# ls ls: cannot open directory .: Input/output error [root@ip-10-142-243-68 gfs]# ls ls: cannot open directory .: Input/output error [root@ip-10-142-243-68 gfs]# ls ls: cannot open directory .: Input/output error [root@ip-10-142-243-68 gfs]# ls ls: cannot open directory .: Input/output error Expected results: Additional info: Client log: [2012-06-09 15:38:51.874984] I [afr-self-heal-common.c:1811:afr_sh_find_fresh_parents] 4-amitest-replicate-1: /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images: Performing conservative merge [2012-06-09 15:38:51.876060] E [afr-self-heal-common.c:1087:afr_sh_common_lookup_resp_handler] 4-amitest-replicate-1: path /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/fa65a84a05934b23ad6a547be7f14759.jpg on subvolume amitest-client-3 => -1 (No such file or directory) [2012-06-09 15:38:51.876108] E [afr-self-heal-common.c:1087:afr_sh_common_lookup_resp_handler] 4-amitest-replicate-1: path /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/fa65a84a05934b23ad6a547be7f14759.jpg on subvolume amitest-client-2 => -1 (No such file or directory) [2012-06-09 15:38:51.876206] E [afr-self-heal-common.c:1399:afr_sh_common_lookup_cbk] 4-amitest-replicate-1: Failed to lookup /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/fa65a84a05934b23ad6a547be7f14759.jpg, reason No such file or directory [2012-06-09 15:38:51.876755] E [afr-self-heal-common.c:2156:afr_self_heal_completion_cbk] 4-amitest-replicate-1: background meta-data data entry missing-entry gfid self-heal failed on /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/fa65a84a05934b23ad6a547be7f14759.jpg [2012-06-09 15:38:52.018728] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-0: remote operation failed: No such file or directory [2012-06-09 15:38:52.018948] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-1: remote operation failed: No such file or directory [2012-06-09 15:38:52.018984] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829015: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/5ecda42aa03f401f84343e56721fef29.jpg => -1 (No such file or directory) [2012-06-09 15:38:52.036803] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-3: remote operation failed: No such file or directory [2012-06-09 15:38:52.037183] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-2: remote operation failed: No such file or directory [2012-06-09 15:38:52.037217] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829017: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/5708ab2082994e479126d920d130081b.jpg => -1 (No such file or directory) [2012-06-09 15:38:52.042296] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-2: remote operation failed: No such file or directory [2012-06-09 15:38:52.042348] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-3: remote operation failed: No such file or directory [2012-06-09 15:38:52.042371] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829019: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/bde8215dcff94691b8c1f6813db3aada.jpg => -1 (No such file or directory) [2012-06-09 15:38:52.054869] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-3: remote operation failed: No such file or directory [2012-06-09 15:38:52.054917] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-2: remote operation failed: No such file or directory [2012-06-09 15:38:52.054962] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829021: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/1611721da83046c6a7260977f79b58a1.jpg => -1 (No such file or directory) [2012-06-09 15:38:52.059523] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-2: remote operation failed: No such file or directory [2012-06-09 15:38:52.059725] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-3: remote operation failed: No such file or directory [2012-06-09 15:38:52.059846] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829023: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/4f5a346c1be142a6aec57b9d45c63f5b.jpg => -1 (No such file or directory) [2012-06-09 15:38:52.064550] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-0: remote operation failed: No such file or directory [2012-06-09 15:38:52.064858] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-1: remote operation failed: No such file or directory [2012-06-09 15:38:52.065006] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829025: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/90c9160061b8411d808341a2692376da.jpg => -1 (No such file or directory) [2012-06-09 15:38:52.070688] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-3: remote operation failed: No such file or directory [2012-06-09 15:38:52.070756] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-2: remote operation failed: No such file or directory [2012-06-09 15:38:52.070781] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829027: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/dcea58191b704fcd8c10086f88faa264.jpg => -1 (No such file or directory) [2012-06-09 15:38:52.276574] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-2: remote operation failed: No such file or directory [2012-06-09 15:38:52.276735] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-3: remote operation failed: No such file or directory [2012-06-09 15:38:52.276820] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829029: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/71b434f17760483c8dd6fd2fa270d9c2.jpg => -1 (No such file or directory) [2012-06-09 15:38:52.290928] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-2: remote operation failed: No such file or directory [2012-06-09 15:38:52.291074] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-3: remote operation failed: No such file or directory [2012-06-09 15:38:52.291101] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829037: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/5109efe9e93e4e24bc9ee1384516a0c4.jpg => -1 (No such file or directory) [2012-06-09 15:38:52.864217] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-0: remote operation failed: No such file or directory [2012-06-09 15:38:52.864311] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-1: remote operation failed: No such file or directory [2012-06-09 15:38:52.864335] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829039: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/b8a9161db01c410aae9d722de608da25.jpg => -1 (No such file or directory) [2012-06-09 15:38:53.075040] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-2: remote operation failed: No such file or directory [2012-06-09 15:38:53.075101] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-3: remote operation failed: No such file or directory [2012-06-09 15:38:53.075125] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829041: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/f9c1391e8bde4f2087fdfa24b8cb1aad.jpg => -1 (No such file or directory) [2012-06-09 15:38:53.109850] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-1: remote operation failed: No such file or directory [2012-06-09 15:38:53.109901] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-0: remote operation failed: No such file or directory [2012-06-09 15:38:53.109923] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829044: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/98aa0c4f2f54489bbb144a9cd29338ea.jpg => -1 (No such file or directory) [2012-06-09 15:38:53.112629] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-3: remote operation failed: No such file or directory [2012-06-09 15:38:53.112674] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-2: remote operation failed: No such file or directory [2012-06-09 15:38:53.112696] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829046: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/5043a0124b094a0dab84bd1fd2afe848.jpg => -1 (No such file or directory) [2012-06-09 15:38:53.137059] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-1: remote operation failed: No such file or directory [2012-06-09 15:38:53.137130] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-0: remote operation failed: No such file or directory [2012-06-09 15:38:53.137169] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829048: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/963284c2f492448da5086e7f6db50b96.jpg => -1 (No such file or directory) [2012-06-09 15:38:53.148085] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-0: remote operation failed: No such file or directory [2012-06-09 15:38:53.148129] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-1: remote operation failed: No such file or directory [2012-06-09 15:38:53.148164] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829050: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/9293f9d0b6544abba1de64e4d63a11c0.jpg => -1 (No such file or directory) [2012-06-09 15:38:53.149965] I [afr-common.c:1340:afr_launch_self_heal] 4-amitest-replicate-0: background entry self-heal triggered. path: /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images, reason: lookup detected pending operations [2012-06-09 15:38:53.150891] I [afr-self-heal-entry.c:2321:afr_sh_entry_fix] 4-amitest-replicate-0: /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images: Performing conservative merge [2012-06-09 15:39:09.214759] I [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk] 4-amitest-replicate-0: background entry self-heal completed on /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images [root@ip-10-142-243-68 ~]# tail -n 100 /var/log/glusterfs/mnt-gfs-.log [2012-06-10 03:20:01.579658] I [glusterfsd-mgmt.c:1565:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
This was seen even with deletes happening from multiple fuse clients on different machines.
Avati, do you remember that in fuse driver there was a case which results in making a entry return EIO for every access. Do you think that issue still exists in fuse ?
Are you referring to this - http://sourceforge.net/apps/mediawiki/fuse/index.php?title=FAQ#What_is_the_reason_for_IO_errors.3F
yes, that link tells the cases. I suspect one of this would have been hit by above activities. Anush, can you add more of details here? like 'volume info' output? Pranith, can you follow the link and see if any of the logs point to cases which returns different inode type for a given inode?
this issue should be something similar to 856921 (patch pushed at http://review.gluster.org/3952) hence closing it