Bug 828901 - object-storage: parallel DELETE throws error
object-storage: parallel DELETE throws error
Status: CLOSED CURRENTRELEASE
Product: GlusterFS
Classification: Community
Component: object-storage (Show other bugs)
pre-release
Unspecified Unspecified
high Severity unspecified
: ---
: ---
Assigned To: Junaid
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-06-05 10:54 EDT by Saurabh
Modified: 2016-01-19 01:10 EST (History)
3 users (show)

See Also:
Fixed In Version: glusterfs-3.4.0
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-07-24 13:42:21 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Saurabh 2012-06-05 10:54:31 EDT
Description of problem:

Glusterfs version: 3.3.0qa45
gluster-swift version: swift-rc1-rpm
glusterfs volume type: distribute-replicate(2x2)
node_timeout in proxy,containter,object conf files is : 60
worker threads in proxy, container, object, account is : 4

Created 250 objects.
Now tried to DELETE while issuing curl command in parallel,

issue is the below error is mentioned in /var/log/messages, although the object is DELETE'd successfully,

Jun  5 14:45:45 gqac022 container-server 127.0.0.1 - - [05/Jun/2012:14:45:45 +0000] "DELETE /sdb1/181472/AUTH_test/cont1/f.124" 204 - "tx84541617bbdd43dc85e9920a790a3943" "-" "-" 0.0021
Jun  5 14:45:45 gqac022 object-server 127.0.0.1 - - [05/Jun/2012:14:45:45 +0000] "DELETE /sdb1/98530/AUTH_test/cont1/f.124" 204 - "-" "tx84541617bbdd43dc85e9920a790a3943" "curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.12.9.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2" 0.0549
Jun  5 14:45:45 gqac022 proxy-server 10.16.156.3 10.16.156.3 05/Jun/2012/14/45/45 DELETE /v1/AUTH_test/cont1/f.124 HTTP/1.0 204 - curl/7.19.7%20%28x86_64-redhat-linux-gnu%29%20libcurl/7.19.7%20NSS/3.12.9.0%20zlib/1.2.3%20libidn/1.18%20libssh2/1.2.2 test%2CAUTH_tk97edb64f5c034fb08097c94b27da92e2 - - - tx84541617bbdd43dc85e9920a790a3943 - 24.9937 -
Jun  5 14:45:45 gqac022 container-server ERROR __call__ error with HEAD /sdb1/181472/AUTH_test/cont1 : #012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/container/server.py", line 497, in __call__#012    res = getattr(self, req.method)(req)#012  File "/usr/lib/python2.6/site-packages/swift/container/server.py", line 271, in HEAD#012    None, None)#012  File "/usr/lib/python2.6/site-packages/swift/plugins/DiskDir.py", line 265, in list_objects_iter#012    objects, object_count, bytes_used = get_container_details(self.datadir)#012  File "/usr/lib/python2.6/site-packages/swift/plugins/utils.py", line 520, in get_container_details#012    object_list, object_count, bytes_used = get_container_details_from_fs(cont_path, cont_path)#012  File "/usr/lib/python2.6/site-packages/swift/plugins/utils.py", line 472, in get_container_details_from_fs#012    obj_list)#012  File "/usr/lib/python2.6/site-packages/swift/plugins/utils.py", line 452, in update_list#012    obj_list)#012  File "/usr/lib/python2.6/site-packages/swift/plugins/utils.py", line 444, in _update_list#012    bytes_used += os.path.getsize(path + '/' + i)#012  File "/usr/lib64/python2.6/genericpath.py", line 49, in getsize#012    return os.stat(filename).st_size#012OSError: [Errno 2] No such file or directory: '/mnt/gluster-object/AUTH_test/cont1/f.124' (txn: tx4fbba6382a6e438da060f4033f59a71b)
Jun  5 14:45:45 gqac022 container-server 127.0.0.1 - - [05/Jun/2012:14:45:45 +0000] "HEAD /sdb1/181472/AUTH_test/cont1" 500 1205 "tx4fbba6382a6e438da060f4033f59a71b" "-" "-" 0.0857
Jun  5 14:45:45 gqac022 proxy-server 10.16.156.3 10.16.156.3 05/Jun/2012/14/45/45 DELETE /v1/AUTH_test/cont1/f.136 HTTP/1.0 404 - curl/7.19.7%20%28x86_64-redhat-linux-gnu%29%20libcurl/7.19.7%20NSS/3.12.9.0%20zlib/1.2.3%20libidn/1.18%20libssh2/1.2.2 test%2CAUTH_tk97edb64f5c034fb08097c94b27da92e2 - - - tx4fbba6382a6e438da060f4033f59a71b - 24.9989 -


shell script for deleting the objects, 

for i in {1..250}; do curl -v -H 'X-Auth-Token: AUTH_tk97edb64f5c034fb08097c94b27da92e2' http://<server-ip>:8080/v1/AUTH_test/cont1/f.$i -X DELETE & done


Version-Release number of selected component (if applicable):
3.3.0qa45

How reproducible:
always

Steps to Reproduce:
1.as mentioned above
2.
3.
  
Actual results:
the DELETE should not throw error

Expected results:


Additional info:

if more tunables are required, please update.
Comment 1 Junaid 2012-06-26 05:59:53 EDT
Due to parallel deletes, when the server does readdir the file was present but by the time the server tried to do stat the file was deleted and stat returned error. Thus causing an exception.
Comment 2 Junaid 2013-01-22 01:31:39 EST
Can you verify this in the latest gluster-swift-1.7.4 rpm's?

Note You need to log in before you can comment on or make changes to this bug.