Bug 828901 - object-storage: parallel DELETE throws error
Summary: object-storage: parallel DELETE throws error
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: object-storage
Version: pre-release
Hardware: Unspecified
OS: Unspecified
high
unspecified
Target Milestone: ---
Assignee: Junaid
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-06-05 14:54 UTC by Saurabh
Modified: 2016-01-19 06:10 UTC (History)
3 users (show)

Fixed In Version: glusterfs-3.4.0
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-07-24 17:42:21 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Description Saurabh 2012-06-05 14:54:31 UTC
Description of problem:

Glusterfs version: 3.3.0qa45
gluster-swift version: swift-rc1-rpm
glusterfs volume type: distribute-replicate(2x2)
node_timeout in proxy,containter,object conf files is : 60
worker threads in proxy, container, object, account is : 4

Created 250 objects.
Now tried to DELETE while issuing curl command in parallel,

issue is the below error is mentioned in /var/log/messages, although the object is DELETE'd successfully,

Jun  5 14:45:45 gqac022 container-server 127.0.0.1 - - [05/Jun/2012:14:45:45 +0000] "DELETE /sdb1/181472/AUTH_test/cont1/f.124" 204 - "tx84541617bbdd43dc85e9920a790a3943" "-" "-" 0.0021
Jun  5 14:45:45 gqac022 object-server 127.0.0.1 - - [05/Jun/2012:14:45:45 +0000] "DELETE /sdb1/98530/AUTH_test/cont1/f.124" 204 - "-" "tx84541617bbdd43dc85e9920a790a3943" "curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.12.9.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2" 0.0549
Jun  5 14:45:45 gqac022 proxy-server 10.16.156.3 10.16.156.3 05/Jun/2012/14/45/45 DELETE /v1/AUTH_test/cont1/f.124 HTTP/1.0 204 - curl/7.19.7%20%28x86_64-redhat-linux-gnu%29%20libcurl/7.19.7%20NSS/3.12.9.0%20zlib/1.2.3%20libidn/1.18%20libssh2/1.2.2 test%2CAUTH_tk97edb64f5c034fb08097c94b27da92e2 - - - tx84541617bbdd43dc85e9920a790a3943 - 24.9937 -
Jun  5 14:45:45 gqac022 container-server ERROR __call__ error with HEAD /sdb1/181472/AUTH_test/cont1 : #012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/container/server.py", line 497, in __call__#012    res = getattr(self, req.method)(req)#012  File "/usr/lib/python2.6/site-packages/swift/container/server.py", line 271, in HEAD#012    None, None)#012  File "/usr/lib/python2.6/site-packages/swift/plugins/DiskDir.py", line 265, in list_objects_iter#012    objects, object_count, bytes_used = get_container_details(self.datadir)#012  File "/usr/lib/python2.6/site-packages/swift/plugins/utils.py", line 520, in get_container_details#012    object_list, object_count, bytes_used = get_container_details_from_fs(cont_path, cont_path)#012  File "/usr/lib/python2.6/site-packages/swift/plugins/utils.py", line 472, in get_container_details_from_fs#012    obj_list)#012  File "/usr/lib/python2.6/site-packages/swift/plugins/utils.py", line 452, in update_list#012    obj_list)#012  File "/usr/lib/python2.6/site-packages/swift/plugins/utils.py", line 444, in _update_list#012    bytes_used += os.path.getsize(path + '/' + i)#012  File "/usr/lib64/python2.6/genericpath.py", line 49, in getsize#012    return os.stat(filename).st_size#012OSError: [Errno 2] No such file or directory: '/mnt/gluster-object/AUTH_test/cont1/f.124' (txn: tx4fbba6382a6e438da060f4033f59a71b)
Jun  5 14:45:45 gqac022 container-server 127.0.0.1 - - [05/Jun/2012:14:45:45 +0000] "HEAD /sdb1/181472/AUTH_test/cont1" 500 1205 "tx4fbba6382a6e438da060f4033f59a71b" "-" "-" 0.0857
Jun  5 14:45:45 gqac022 proxy-server 10.16.156.3 10.16.156.3 05/Jun/2012/14/45/45 DELETE /v1/AUTH_test/cont1/f.136 HTTP/1.0 404 - curl/7.19.7%20%28x86_64-redhat-linux-gnu%29%20libcurl/7.19.7%20NSS/3.12.9.0%20zlib/1.2.3%20libidn/1.18%20libssh2/1.2.2 test%2CAUTH_tk97edb64f5c034fb08097c94b27da92e2 - - - tx4fbba6382a6e438da060f4033f59a71b - 24.9989 -


shell script for deleting the objects, 

for i in {1..250}; do curl -v -H 'X-Auth-Token: AUTH_tk97edb64f5c034fb08097c94b27da92e2' http://<server-ip>:8080/v1/AUTH_test/cont1/f.$i -X DELETE & done


Version-Release number of selected component (if applicable):
3.3.0qa45

How reproducible:
always

Steps to Reproduce:
1.as mentioned above
2.
3.
  
Actual results:
the DELETE should not throw error

Expected results:


Additional info:

if more tunables are required, please update.

Comment 1 Junaid 2012-06-26 09:59:53 UTC
Due to parallel deletes, when the server does readdir the file was present but by the time the server tried to do stat the file was deleted and stat returned error. Thus causing an exception.

Comment 2 Junaid 2013-01-22 06:31:39 UTC
Can you verify this in the latest gluster-swift-1.7.4 rpm's?


Note You need to log in before you can comment on or make changes to this bug.