Created attachment 567935 [details] Statedump of the backend Description of problem: While using the perl script to do do search and replace replace, the script hung at the read call. This was only seen on distribute volumes with quick-read on. How reproducible: Consistently Steps to Reproduce: 1. echo 'test' > dot 2. perl -i -pe 's/test/somethingelse/' dot Actual results: The script was hung at the read call. Additional info: Attaching the statedump of the bricks. Client log- [2012-03-06 17:40:24.340042] D [dht-diskusage.c:88:dht_du_info_cbk] 1-test2-dht: on subvolume 'test2-client-0': avail_percent is: 94.00 and avail_space is: 96749436928 and avail_inodes is: 99.00 [2012-03-06 17:40:24.340124] D [dht-diskusage.c:88:dht_du_info_cbk] 1-test2-dht: on subvolume 'test2-client-1': avail_percent is: 94.00 and avail_space is: 96749436928 and avail_inodes is: 99.00 [2012-03-06 17:40:26.804931] W [client.c:112:client_grace_timeout] 0-test2-client-0: client grace timer expired, updating the lk-version to 2 [2012-03-06 17:40:26.805034] W [client.c:112:client_grace_timeout] 0-test2-client-1: client grace timer expired, updating the lk-version to 2 [2012-03-06 17:40:27.051189] D [dht-diskusage.c:88:dht_du_info_cbk] 1-test2-dht: on subvolume 'test2-client-1': avail_percent is: 94.00 and avail_space is: 96749436928 and avail_inodes is: 99.00 [2012-03-06 17:40:27.051255] D [dht-diskusage.c:88:dht_du_info_cbk] 1-test2-dht: on subvolume 'test2-client-0': avail_percent is: 94.00 and avail_space is: 96749436928 and avail_inodes is: 99.00 [2012-03-06 17:40:27.052061] W [client3_1-fops.c:375:client3_1_open_cbk] 1-test2-client-0: remote operation failed: No such file or directory. Path: /dot [2012-03-06 17:40:27.052437] W [client3_1-fops.c:880:client3_1_getxattr_cbk] 1-test2-client-0: remote operation failed: No such file or directory. Path: (null) [2012-03-06 17:40:27.052665] W [client3_1-fops.c:2180:client3_1_lookup_cbk] 1-test2-client-0: remote operation failed: Invalid argument. Path: /dot [2012-03-06 17:40:27.052694] D [dht-common.c:269:dht_discover_cbk] 1-test2-dht: lookup of /dot on test2-client-0 returned error (Invalid argument) [2012-03-06 17:40:27.052797] W [client3_1-fops.c:2180:client3_1_lookup_cbk] 1-test2-client-1: remote operation failed: Invalid argument. Path: /dot [2012-03-06 17:40:27.052820] D [dht-common.c:269:dht_discover_cbk] 1-test2-dht: lookup of /dot on test2-client-1 returned error (Invalid argument) [2012-03-06 17:40:27.052841] I [dht-layout.c:600:dht_layout_normalize] 1-test2-dht: found anomalies in /dot. holes=1 overlaps=0 [2012-03-06 17:40:27.052862] D [dht-layout.c:614:dht_layout_normalize] 1-test2-dht: path=/dot err=Invalid argument on subvol=test2-client-0 [2012-03-06 17:40:27.052881] D [dht-layout.c:614:dht_layout_normalize] 1-test2-dht: path=/dot err=Invalid argument on subvol=test2-client-1 [2012-03-06 17:40:27.052900] D [dht-common.c:197:dht_discover_complete] 1-test2-dht: normalizing failed on /dot
Shishir, I guess some of the outstanding patches from Raghu (Du) should already fix it. http://review.gluster.com/#dashboard,1000007 (in quick-read) Please try to review it and see for anything obvious. Avati, Can you send some thoughts on utilizing anon-fd for quick-read, which we believe should reduce the complexity involved.
This seems to be a dup of bug 765363. http://review.gluster.com/#change,506,patchset=6 patch fixes this issue
*** This bug has been marked as a duplicate of bug 765363 ***