Bug 830555

Summary:	EIO errors while deleting files
Product:	[Community] GlusterFS	Reporter:	Anush Shetty <ashetty>
Component:	fuse	Assignee:	Csaba Henk <csaba>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	medium	Docs Contact:
Priority:	medium
Version:	3.3.0	CC:	aavati, amarts, gluster-bugs
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.4.0	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:
Clones:	848335 (view as bug list)		Environment:
Last Closed:	2013-07-24 17:35:44 UTC	Type:	Bug
Regression:	---	Mount Type:	fuse
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	848335, 858445

Description Anush Shetty 2012-06-10 15:40:30 UTC

Description of problem: While deleting files from the fuse mount, it returned EIO errors. 


This was part of the stress testing exercise on AMI. Files were bring deleted from multiple clients. Before that while the files were being created, one of the servers was brought up and down at regular intervals. While deleting one the directories with a lot of image file, the rm process returned ENOTEMPTY and then it returned EIO. 

Version-Release number of selected component (if applicable): 3.3.0RC1 AMI

  
Actual results:

rm: cannot remove `tmp_2/Common/image-systems-fake-nas/jhegedus/si/images': Directory not empty
rm: cannot remove `tmp_3': Input/output error
rm: cannot remove `tmp_4': Input/output error
rm: cannot remove `tmp_5': Input/output error
rm: cannot remove `tmp_6': Input/output error
rm: cannot remove `tmp_7': Input/output error
rm: cannot remove `tmp_8': Input/output error
rm: cannot remove `tmp_9': Input/output error
rm: cannot remove `upload-ws': Input/output error
rm: cannot remove `wrapper': Input/output error
[root@ip-10-142-243-68 gfs]#
[root@ip-10-142-243-68 gfs]#
[root@ip-10-142-243-68 gfs]#
[root@ip-10-142-243-68 gfs]#
[root@ip-10-142-243-68 gfs]# ls
ls: cannot open directory .: Input/output error
[root@ip-10-142-243-68 gfs]# ls
ls: cannot open directory .: Input/output error
[root@ip-10-142-243-68 gfs]# ls
ls: cannot open directory .: Input/output error
[root@ip-10-142-243-68 gfs]# ls
ls: cannot open directory .: Input/output error



Expected results:


Additional info:

Client log:


[2012-06-09 15:38:51.874984] I [afr-self-heal-common.c:1811:afr_sh_find_fresh_parents] 4-amitest-replicate-1: /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images: Performing conservative merge
[2012-06-09 15:38:51.876060] E [afr-self-heal-common.c:1087:afr_sh_common_lookup_resp_handler] 4-amitest-replicate-1: path /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/fa65a84a05934b23ad6a547be7f14759.jpg on subvolume amitest-client-3 => -1 (No such file or directory)
[2012-06-09 15:38:51.876108] E [afr-self-heal-common.c:1087:afr_sh_common_lookup_resp_handler] 4-amitest-replicate-1: path /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/fa65a84a05934b23ad6a547be7f14759.jpg on subvolume amitest-client-2 => -1 (No such file or directory)
[2012-06-09 15:38:51.876206] E [afr-self-heal-common.c:1399:afr_sh_common_lookup_cbk] 4-amitest-replicate-1: Failed to lookup /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/fa65a84a05934b23ad6a547be7f14759.jpg, reason No such file or directory
[2012-06-09 15:38:51.876755] E [afr-self-heal-common.c:2156:afr_self_heal_completion_cbk] 4-amitest-replicate-1: background  meta-data data entry missing-entry gfid self-heal failed on /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/fa65a84a05934b23ad6a547be7f14759.jpg
[2012-06-09 15:38:52.018728] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-0: remote operation failed: No such file or directory
[2012-06-09 15:38:52.018948] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-1: remote operation failed: No such file or directory
[2012-06-09 15:38:52.018984] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829015: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/5ecda42aa03f401f84343e56721fef29.jpg => -1 (No such file or directory)
[2012-06-09 15:38:52.036803] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-3: remote operation failed: No such file or directory
[2012-06-09 15:38:52.037183] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-2: remote operation failed: No such file or directory
[2012-06-09 15:38:52.037217] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829017: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/5708ab2082994e479126d920d130081b.jpg => -1 (No such file or directory)
[2012-06-09 15:38:52.042296] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-2: remote operation failed: No such file or directory
[2012-06-09 15:38:52.042348] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-3: remote operation failed: No such file or directory
[2012-06-09 15:38:52.042371] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829019: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/bde8215dcff94691b8c1f6813db3aada.jpg => -1 (No such file or directory)
[2012-06-09 15:38:52.054869] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-3: remote operation failed: No such file or directory
[2012-06-09 15:38:52.054917] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-2: remote operation failed: No such file or directory
[2012-06-09 15:38:52.054962] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829021: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/1611721da83046c6a7260977f79b58a1.jpg => -1 (No such file or directory)
[2012-06-09 15:38:52.059523] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-2: remote operation failed: No such file or directory
[2012-06-09 15:38:52.059725] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-3: remote operation failed: No such file or directory
[2012-06-09 15:38:52.059846] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829023: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/4f5a346c1be142a6aec57b9d45c63f5b.jpg => -1 (No such file or directory)
[2012-06-09 15:38:52.064550] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-0: remote operation failed: No such file or directory
[2012-06-09 15:38:52.064858] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-1: remote operation failed: No such file or directory
[2012-06-09 15:38:52.065006] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829025: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/90c9160061b8411d808341a2692376da.jpg => -1 (No such file or directory)
[2012-06-09 15:38:52.070688] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-3: remote operation failed: No such file or directory
[2012-06-09 15:38:52.070756] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-2: remote operation failed: No such file or directory
[2012-06-09 15:38:52.070781] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829027: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/dcea58191b704fcd8c10086f88faa264.jpg => -1 (No such file or directory)
[2012-06-09 15:38:52.276574] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-2: remote operation failed: No such file or directory
[2012-06-09 15:38:52.276735] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-3: remote operation failed: No such file or directory
[2012-06-09 15:38:52.276820] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829029: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/71b434f17760483c8dd6fd2fa270d9c2.jpg => -1 (No such file or directory)
[2012-06-09 15:38:52.290928] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-2: remote operation failed: No such file or directory
[2012-06-09 15:38:52.291074] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-3: remote operation failed: No such file or directory
[2012-06-09 15:38:52.291101] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829037: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/5109efe9e93e4e24bc9ee1384516a0c4.jpg => -1 (No such file or directory)
[2012-06-09 15:38:52.864217] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-0: remote operation failed: No such file or directory
[2012-06-09 15:38:52.864311] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-1: remote operation failed: No such file or directory
[2012-06-09 15:38:52.864335] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829039: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/b8a9161db01c410aae9d722de608da25.jpg => -1 (No such file or directory)
[2012-06-09 15:38:53.075040] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-2: remote operation failed: No such file or directory
[2012-06-09 15:38:53.075101] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-3: remote operation failed: No such file or directory
[2012-06-09 15:38:53.075125] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829041: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/f9c1391e8bde4f2087fdfa24b8cb1aad.jpg => -1 (No such file or directory)
[2012-06-09 15:38:53.109850] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-1: remote operation failed: No such file or directory
[2012-06-09 15:38:53.109901] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-0: remote operation failed: No such file or directory
[2012-06-09 15:38:53.109923] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829044: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/98aa0c4f2f54489bbb144a9cd29338ea.jpg => -1 (No such file or directory)
[2012-06-09 15:38:53.112629] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-3: remote operation failed: No such file or directory
[2012-06-09 15:38:53.112674] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-2: remote operation failed: No such file or directory
[2012-06-09 15:38:53.112696] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829046: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/5043a0124b094a0dab84bd1fd2afe848.jpg => -1 (No such file or directory)
[2012-06-09 15:38:53.137059] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-1: remote operation failed: No such file or directory
[2012-06-09 15:38:53.137130] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-0: remote operation failed: No such file or directory
[2012-06-09 15:38:53.137169] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829048: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/963284c2f492448da5086e7f6db50b96.jpg => -1 (No such file or directory)
[2012-06-09 15:38:53.148085] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-0: remote operation failed: No such file or directory
[2012-06-09 15:38:53.148129] W [client3_1-fops.c:592:client3_1_unlink_cbk] 4-amitest-client-1: remote operation failed: No such file or directory
[2012-06-09 15:38:53.148164] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 133829050: UNLINK() /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images/9293f9d0b6544abba1de64e4d63a11c0.jpg => -1 (No such file or directory)
[2012-06-09 15:38:53.149965] I [afr-common.c:1340:afr_launch_self_heal] 4-amitest-replicate-0: background  entry self-heal triggered. path: /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images, reason: lookup detected pending operations
[2012-06-09 15:38:53.150891] I [afr-self-heal-entry.c:2321:afr_sh_entry_fix] 4-amitest-replicate-0: /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images: Performing conservative merge
[2012-06-09 15:39:09.214759] I [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk] 4-amitest-replicate-0: background  entry self-heal completed on /tmp_2/Common/image-systems-fake-nas/jhegedus/si/images
[root@ip-10-142-243-68 ~]# tail -n 100 /var/log/glusterfs/mnt-gfs-.log 
[2012-06-10 03:20:01.579658] I [glusterfsd-mgmt.c:1565:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing

Comment 1 Anush Shetty 2012-06-11 14:46:44 UTC

This was seen even with deletes happening from multiple fuse clients on different machines.

Comment 2 Amar Tumballi 2012-07-05 06:58:35 UTC

Avati, do you remember that in fuse driver there was a case which results in making a entry return EIO for every access. Do you think that issue still exists in fuse ?

Comment 3 Anand Avati 2012-07-05 17:56:20 UTC

Are you referring to this - http://sourceforge.net/apps/mediawiki/fuse/index.php?title=FAQ#What_is_the_reason_for_IO_errors.3F

Comment 4 Amar Tumballi 2012-07-06 07:48:38 UTC

yes, that link tells the cases. I suspect one of this would have been hit by above activities.

Anush, can you add more of details here? like 'volume info' output?

Pranith, can you follow the link and see if any of the logs point to cases which returns different inode type for a given inode?

Comment 5 Amar Tumballi 2012-09-20 06:48:57 UTC

this issue should be something similar to 856921 (patch pushed at http://review.gluster.org/3952) hence closing it