Bug 830555 - EIO errors while deleting files
EIO errors while deleting files
Status: CLOSED CURRENTRELEASE
Product: GlusterFS
Classification: Community
Component: fuse (Show other bugs)
3.3.0
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Csaba Henk
: Triaged
Depends On:
Blocks: 848335 858445
  Show dependency treegraph
 
Reported: 2012-06-10 11:40 EDT by Anush Shetty
Modified: 2013-07-24 13:35 EDT (History)
3 users (show)

See Also:
Fixed In Version: glusterfs-3.4.0
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 848335 (view as bug list)
Environment:
Last Closed: 2013-07-24 13:35:44 EDT
Type: Bug
Regression: ---
Mount Type: fuse
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Anush Shetty 2012-06-10 11:40:30 EDT
Description of problem: While deleting files from the fuse mount, it returned EIO errors. 


This was part of the stress testing exercise on AMI. Files were bring deleted from multiple clients. Before that while the files were being created, one of the servers was brought up and down at regular intervals. While deleting one the directories with a lot of image file, the rm process returned ENOTEMPTY and then it returned EIO. 

Version-Release number of selected component (if applicable): 3.3.0RC1 AMI

  
Actual results:

rm: cannot remove `tmp_2/Common/image-systems-fake-nas/jhegedus/si/images': Directory not empty
rm: cannot remove `tmp_3': Input/output error
rm: cannot remove `tmp_4': Input/output error
rm: cannot remove `tmp_5': Input/output error
rm: cannot remove `tmp_6': Input/output error
rm: cannot remove `tmp_7': Input/output error
rm: cannot remove `tmp_8': Input/output error
rm: cannot remove `tmp_9': Input/output error
rm: cannot remove `upload-ws': Input/output error
rm: cannot remove `wrapper': Input/output error
[root@ip-10-142-243-68 gfs]#
[root@ip-10-142-243-68 gfs]#
[root@ip-10-142-243-68 gfs]#
[root@ip-10-142-243-68 gfs]#
[root@ip-10-142-243-68 gfs]# ls
ls: cannot open directory .: Input/output error
[root@ip-10-142-243-68 gfs]# ls
ls: cannot open directory .: Input/output error
[root@ip-10-142-243-68 gfs]# ls
ls: cannot open directory .: Input/output error
[root@ip-10-142-243-68 gfs]# ls
ls: cannot open directory .: Input/output error



Expected results:


Additional info:

Client log:


[2012-06-09 15:38:51.874984] I [afr-self-heal-common.c:1811:afr_sh_find_fresh_parents] 4-amitest-replicate-1: /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images: Performing conservative merge
[2012-06-09 15:38:51.876060] E [afr-self-heal-common.c:1087:afr_sh_common_lookup_resp_handler] 4-amitest-replicate-1: path /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/fa65a84a05934b23ad6a547be7f14759.jpg on subvolume amitest-client-3 => -1 (No such file or directory)
[2012-06-09 15:38:51.876108] E [afr-self-heal-common.c:1087:afr_sh_common_lookup_resp_handler] 4-amitest-replicate-1: path /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/fa65a84a05934b23ad6a547be7f14759.jpg on subvolume amitest-client-2 => -1 (No such file or directory)
[2012-06-09 15:38:51.876206] E [afr-self-heal-common.c:1399:afr_sh_common_lookup_cbk] 4-amitest-replicate-1: Failed to lookup /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/fa65a84a05934b23ad6a547be7f14759.jpg, reason No such file or directory
[2012-06-09 15:38:51.876755] E [afr-self-heal-common.c:2156:afr_self_heal_completion_cbk] 4-amitest-replicate-1: background  meta-data data entry missing-entry gfid self-heal failed on /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/fa65a84a05934b23ad6a547be7f14759.jpg
[2012-06-09 15:38:52.018728] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-0: remote operation failed: No such file or directory
[2012-06-09 15:38:52.018948] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-1: remote operation failed: No such file or directory
[2012-06-09 15:38:52.018984] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829015: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/5ecda42aa03f401f84343e56721fef29.jpg => -1 (No such file or directory)
[2012-06-09 15:38:52.036803] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-3: remote operation failed: No such file or directory
[2012-06-09 15:38:52.037183] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-2: remote operation failed: No such file or directory
[2012-06-09 15:38:52.037217] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829017: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/5708ab2082994e479126d920d130081b.jpg => -1 (No such file or directory)
[2012-06-09 15:38:52.042296] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-2: remote operation failed: No such file or directory
[2012-06-09 15:38:52.042348] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-3: remote operation failed: No such file or directory
[2012-06-09 15:38:52.042371] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829019: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/bde8215dcff94691b8c1f6813db3aada.jpg => -1 (No such file or directory)
[2012-06-09 15:38:52.054869] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-3: remote operation failed: No such file or directory
[2012-06-09 15:38:52.054917] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-2: remote operation failed: No such file or directory
[2012-06-09 15:38:52.054962] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829021: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/1611721da83046c6a7260977f79b58a1.jpg => -1 (No such file or directory)
[2012-06-09 15:38:52.059523] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-2: remote operation failed: No such file or directory
[2012-06-09 15:38:52.059725] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-3: remote operation failed: No such file or directory
[2012-06-09 15:38:52.059846] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829023: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/4f5a346c1be142a6aec57b9d45c63f5b.jpg => -1 (No such file or directory)
[2012-06-09 15:38:52.064550] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-0: remote operation failed: No such file or directory
[2012-06-09 15:38:52.064858] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-1: remote operation failed: No such file or directory
[2012-06-09 15:38:52.065006] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829025: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/90c9160061b8411d808341a2692376da.jpg => -1 (No such file or directory)
[2012-06-09 15:38:52.070688] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-3: remote operation failed: No such file or directory
[2012-06-09 15:38:52.070756] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-2: remote operation failed: No such file or directory
[2012-06-09 15:38:52.070781] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829027: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/dcea58191b704fcd8c10086f88faa264.jpg => -1 (No such file or directory)
[2012-06-09 15:38:52.276574] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-2: remote operation failed: No such file or directory
[2012-06-09 15:38:52.276735] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-3: remote operation failed: No such file or directory
[2012-06-09 15:38:52.276820] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829029: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/71b434f17760483c8dd6fd2fa270d9c2.jpg => -1 (No such file or directory)
[2012-06-09 15:38:52.290928] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-2: remote operation failed: No such file or directory
[2012-06-09 15:38:52.291074] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-3: remote operation failed: No such file or directory
[2012-06-09 15:38:52.291101] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829037: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/5109efe9e93e4e24bc9ee1384516a0c4.jpg => -1 (No such file or directory)
[2012-06-09 15:38:52.864217] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-0: remote operation failed: No such file or directory
[2012-06-09 15:38:52.864311] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-1: remote operation failed: No such file or directory
[2012-06-09 15:38:52.864335] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829039: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/b8a9161db01c410aae9d722de608da25.jpg => -1 (No such file or directory)
[2012-06-09 15:38:53.075040] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-2: remote operation failed: No such file or directory
[2012-06-09 15:38:53.075101] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-3: remote operation failed: No such file or directory
[2012-06-09 15:38:53.075125] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829041: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/f9c1391e8bde4f2087fdfa24b8cb1aad.jpg => -1 (No such file or directory)
[2012-06-09 15:38:53.109850] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-1: remote operation failed: No such file or directory
[2012-06-09 15:38:53.109901] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-0: remote operation failed: No such file or directory
[2012-06-09 15:38:53.109923] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829044: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/98aa0c4f2f54489bbb144a9cd29338ea.jpg => -1 (No such file or directory)
[2012-06-09 15:38:53.112629] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-3: remote operation failed: No such file or directory
[2012-06-09 15:38:53.112674] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-2: remote operation failed: No such file or directory
[2012-06-09 15:38:53.112696] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829046: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/5043a0124b094a0dab84bd1fd2afe848.jpg => -1 (No such file or directory)
[2012-06-09 15:38:53.137059] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-1: remote operation failed: No such file or directory
[2012-06-09 15:38:53.137130] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-0: remote operation failed: No such file or directory
[2012-06-09 15:38:53.137169] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829048: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/963284c2f492448da5086e7f6db50b96.jpg => -1 (No such file or directory)
[2012-06-09 15:38:53.148085] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-0: remote operation failed: No such file or directory
[2012-06-09 15:38:53.148129] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-1: remote operation failed: No such file or directory
[2012-06-09 15:38:53.148164] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829050: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/9293f9d0b6544abba1de64e4d63a11c0.jpg => -1 (No such file or directory)
[2012-06-09 15:38:53.149965] I [afr-common.c:1340:afr_launch_self_heal] 4-amitest-replicate-0: background  entry self-heal triggered. path: /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images, reason: lookup detected pending operations
[2012-06-09 15:38:53.150891] I [afr-self-heal-entry.c:2321:afr_sh_entry_fix] 4-amitest-replicate-0: /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images: Performing conservative merge
[2012-06-09 15:39:09.214759] I [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk] 4-amitest-replicate-0: background  entry self-heal completed on /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images
[root@ip-10-142-243-68 ~]# tail -n 100 /var/log/glusterfs/mnt-gfs-.log 
[2012-06-10 03:20:01.579658] I [glusterfsd-mgmt.c:1565:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
Comment 1 Anush Shetty 2012-06-11 10:46:44 EDT
This was seen even with deletes happening from multiple fuse clients on different machines.
Comment 2 Amar Tumballi 2012-07-05 02:58:35 EDT
Avati, do you remember that in fuse driver there was a case which results in making a entry return EIO for every access. Do you think that issue still exists in fuse ?
Comment 4 Amar Tumballi 2012-07-06 03:48:38 EDT
yes, that link tells the cases. I suspect one of this would have been hit by above activities.

Anush, can you add more of details here? like 'volume info' output?

Pranith, can you follow the link and see if any of the logs point to cases which returns different inode type for a given inode?
Comment 5 Amar Tumballi 2012-09-20 02:48:57 EDT
this issue should be something similar to 856921 (patch pushed at http://review.gluster.org/3952) hence closing it

Note You need to log in before you can comment on or make changes to this bug.