Description of problem: Glusterfs version: 3.3.0qa45 gluster-swift version: swift-rc1-rpm glusterfs volume type: distribute-replicate(2x2) node_timeout in proxy,containter,object conf files is : 60 worker threads in proxy, container, object, account is : 4 Created 250 objects. Now tried to DELETE while issuing curl command in parallel, issue is the below error is mentioned in /var/log/messages, although the object is DELETE'd successfully, Jun 5 14:45:45 gqac022 container-server 127.0.0.1 - - [05/Jun/2012:14:45:45 +0000] "DELETE /sdb1/181472/AUTH_test/cont1/f.124" 204 - "tx84541617bbdd43dc85e9920a790a3943" "-" "-" 0.0021 Jun 5 14:45:45 gqac022 object-server 127.0.0.1 - - [05/Jun/2012:14:45:45 +0000] "DELETE /sdb1/98530/AUTH_test/cont1/f.124" 204 - "-" "tx84541617bbdd43dc85e9920a790a3943" "curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.12.9.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2" 0.0549 Jun 5 14:45:45 gqac022 proxy-server 10.16.156.3 10.16.156.3 05/Jun/2012/14/45/45 DELETE /v1/AUTH_test/cont1/f.124 HTTP/1.0 204 - curl/7.19.7%20%28x86_64-redhat-linux-gnu%29%20libcurl/7.19.7%20NSS/3.12.9.0%20zlib/1.2.3%20libidn/1.18%20libssh2/1.2.2 test%2CAUTH_tk97edb64f5c034fb08097c94b27da92e2 - - - tx84541617bbdd43dc85e9920a790a3943 - 24.9937 - Jun 5 14:45:45 gqac022 container-server ERROR __call__ error with HEAD /sdb1/181472/AUTH_test/cont1 : #012Traceback (most recent call last):#012 File "/usr/lib/python2.6/site-packages/swift/container/server.py", line 497, in __call__#012 res = getattr(self, req.method)(req)#012 File "/usr/lib/python2.6/site-packages/swift/container/server.py", line 271, in HEAD#012 None, None)#012 File "/usr/lib/python2.6/site-packages/swift/plugins/DiskDir.py", line 265, in list_objects_iter#012 objects, object_count, bytes_used = get_container_details(self.datadir)#012 File "/usr/lib/python2.6/site-packages/swift/plugins/utils.py", line 520, in get_container_details#012 object_list, object_count, bytes_used = get_container_details_from_fs(cont_path, cont_path)#012 File "/usr/lib/python2.6/site-packages/swift/plugins/utils.py", line 472, in get_container_details_from_fs#012 obj_list)#012 File "/usr/lib/python2.6/site-packages/swift/plugins/utils.py", line 452, in update_list#012 obj_list)#012 File "/usr/lib/python2.6/site-packages/swift/plugins/utils.py", line 444, in _update_list#012 bytes_used += os.path.getsize(path + '/' + i)#012 File "/usr/lib64/python2.6/genericpath.py", line 49, in getsize#012 return os.stat(filename).st_size#012OSError: [Errno 2] No such file or directory: '/mnt/gluster-object/AUTH_test/cont1/f.124' (txn: tx4fbba6382a6e438da060f4033f59a71b) Jun 5 14:45:45 gqac022 container-server 127.0.0.1 - - [05/Jun/2012:14:45:45 +0000] "HEAD /sdb1/181472/AUTH_test/cont1" 500 1205 "tx4fbba6382a6e438da060f4033f59a71b" "-" "-" 0.0857 Jun 5 14:45:45 gqac022 proxy-server 10.16.156.3 10.16.156.3 05/Jun/2012/14/45/45 DELETE /v1/AUTH_test/cont1/f.136 HTTP/1.0 404 - curl/7.19.7%20%28x86_64-redhat-linux-gnu%29%20libcurl/7.19.7%20NSS/3.12.9.0%20zlib/1.2.3%20libidn/1.18%20libssh2/1.2.2 test%2CAUTH_tk97edb64f5c034fb08097c94b27da92e2 - - - tx4fbba6382a6e438da060f4033f59a71b - 24.9989 - shell script for deleting the objects, for i in {1..250}; do curl -v -H 'X-Auth-Token: AUTH_tk97edb64f5c034fb08097c94b27da92e2' http://<server-ip>:8080/v1/AUTH_test/cont1/f.$i -X DELETE & done Version-Release number of selected component (if applicable): 3.3.0qa45 How reproducible: always Steps to Reproduce: 1.as mentioned above 2. 3. Actual results: the DELETE should not throw error Expected results: Additional info: if more tunables are required, please update.
Due to parallel deletes, when the server does readdir the file was present but by the time the server tried to do stat the file was deleted and stat returned error. Thus causing an exception.
Can you verify this in the latest gluster-swift-1.7.4 rpm's?