Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 782003

Summary:	object-strorage: not all objects getting created, when PUT tried in parallel
Product:	[Community] GlusterFS	Reporter:	Saurabh <saujain>
Component:	object-storage	Assignee:	Junaid <junaid>
Status:	CLOSED DUPLICATE	QA Contact:
Severity:	medium	Docs Contact:
Priority:	medium
Version:	pre-release	CC:	gluster-bugs, mzywusko, vagarwal, vbellur
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2012-04-05 09:10:14 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Saurabh 2012-01-16 10:20:26 UTC

Description of problem:

Trying to create the objects in parallel while using pound as the load balancer, 
it creates very few objects, 


I tried to create 50 empty objects but it created only 4-5 objects only, 

[root@RHSSA1 container2]# 
[root@RHSSA1 container2]# ls
1MB        ef0saurabh-laptop_4  ef3saurabh-laptop_0  f.11  f.14  f.17  f.2   f.4  f.7  index.html
5bytefile  ef1saurabh-laptop_1  f.1                  f.12  f.15  f.18  f.20  f.5  f.8
debug.log  ef2saurabh-laptop_4  f.10                 f.13  f.16  f.19  f.3   f.6  f.9


Now, result with objects getting created without using pound and using the using single URL,

the intended number of objects are created, but this is not the case in each and every run of the script creating empty objects,

because in the latest run it has just created very few,

[root@RHSSA1 parallel]# ls /mnt/gluster-object/AUTH_test/container2/
1MB        debug.log    ef0RHSSA1_1  ef0RHSSA1_3  f.1   f.11  f.13  f.15  f.17  f.19  f.20  f.4  f.6  f.8  index.html
5bytefile  ef0RHSSA1_0  ef0RHSSA1_2  ef0RHSSA1_4  f.10  f.12  f.14  f.16  f.18  f.2   f.3   f.5  f.7  f.9


Version-Release number of selected component (if applicable):


How reproducible:

Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:

Comment 1 Saurabh 2012-04-02 09:24:47 UTC

This time i tried to create the data in a different manner, for this I tried to use two different scripts, sending the curl requests to different servers of the cluster,

the result is:-
1. data gets created
2. but few objects still didn't get created and the logs reported are these,

Apr  1 20:43:39 QA-51 proxy-server 172.17.251.78 172.17.251.78 02/Apr/2012/00/43/39 PUT /v1/AUTH_test/container3/file1KB_719 HTTP/1.0 2
01 - curl/7.19.7%20%28x86_64-redhat-linux-gnu%29%20libcurl/7.19.7%20NSS/3.12.9.0%20zlib/1.2.3%20libidn/1.18%20libssh2/1.2.2 test%2CAUTH
_tk173b5b96c8164f16b47c6122cac2fcfe 1024 - - tx28fd8d0ed7ec463bb6014bc574e14fed - 1.0338 -
Apr  1 20:43:39 QA-51 container-server ERROR __call__ error with PUT /sdb1/227882/AUTH_test/container3/file1KB_720 : #012Traceback (mos
t recent call last):#012  File "/usr/lib/python2.6/site-packages/swift-1.4.9-py2.6.egg/swift/container/server.py", line 489, in __call_
_#012    res = getattr(self, req.method)(req)#012  File "/usr/lib/python2.6/site-packages/swift-1.4.9-py2.6.egg/swift/container/server.
py", line 202, in PUT#012    broker = self._get_container_broker(drive, part, account, container)#012  File "/usr/lib/python2.6/site-pa
ckages/swift-1.4.9-py2.6.egg/swift/container/server.py", line 82, in _get_container_broker#012    fs_object = self.fs_object)#012  File
 "/usr/lib/python2.6/site-packages/swift-1.4.9-py2.6.egg/swift/plugins/DiskDir.py", line 144, in __init__#012    check_valid_account(ac
count, fs_object)#012  File "/usr/lib/python2.6/site-packages/swift-1.4.9-py2.6.egg/swift/plugins/utils.py", line 356, in check_valid_a
ccount#012    return _check_valid_account(account, fs_object)#012  File "/usr/lib/python2.6/site-packages/swift-1.4.9-py2.6.egg/swift/p
lugins/utils.py", line 326, in _check_valid_account#012    if not check_account_exists(fs_object.get_export_from_account_id(account), \
#012  File "/usr/lib/python2.6/site-packages/swift-1.4.9-py2.6.egg/swift/plugins/Glusterfs.py", line 99, in get_export_from_account_id#
012    for export in self.get_export_list():#012  File "/usr/lib/python2.6/site-packages/swift-1.4.9-py2.6.egg/swift/plugins/Glusterfs.
py", line 92, in get_export_list#012    return self.get_export_list_local()#012  File "/usr/lib/python2.6/site-packages/swift-1.4.9-py2
.6.egg/swift/plugins/Glusterfs.py", line 52, in get_export_list_local#012    raise Exception('Getting volume failed %s', self.name)#012
Exception: ('Getting volume failed %s', 'glusterfs') (txn: txce2aeaf477ec40e786e91c3d3a56e210)
Apr  1 20:43:39 QA-51 container-server 127.0.0.1 - - [02/Apr/2012:00:43:39 +0000] "PUT /sdb1/227882/AUTH_test/container3/file1KB_720"  5
00 1630 "txce2aeaf477ec40e786e91c3d3a56e210" "-" "-" 0.0695
Apr  1 20:43:39 QA-51 object-server ERROR Container update failed (saving for async update later): 500 response from 127.0.0.1:6011/sdb1 (txn: txce2aeaf477ec40e786e91c3d3a56e210)
Apr  1 20:43:39 QA-51 object-server ERROR __call__ error with PUT /sdb1/190597/AUTH_test/container3/file1KB_720 : #012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift-1.4.9-py2.6.egg/swift/obj/server.py", line 859, in __call__#012    res = getattr(self, req.method)(req)#012  File "/usr/lib/python2.6/site-packages/swift-1.4.9-py2.6.egg/swift/obj/server.py", line 655, in PUT#012    device)#012  File "/usr/lib/python2.6/site-packages/swift-1.4.9-py2.6.egg/swift/obj/server.py", line 471, in container_update#012    contdevice, headers_out, objdevice)#012  File "/usr/lib/python2.6/site-packages/swift-1.4.9-py2.6.egg/swift/obj/server.py", line 449, in async_update#012    os.path.join(self.devices, objdevice, 'tmp'))#012  File "/usr/lib/python2.6/site-packages/swift-1.4.9-py2.6.egg/swift/common/utils.py", line 860, in write_pickle#012    fd, tmppath = mkstemp(dir=tmp, suffix='.tmp')#012  File "/usr/lib64/python2.6/tempfile.py", line 293, in mkstemp#012    return _mkstemp_inner(dir, prefix, suffix, flags)#012  File "/usr/lib64/python2.6/tempfile.py", line 228, in _mkstemp_inner#012    fd = _os.open(file, flags, 0600)#012OSError: [Errno 2] No such file or directory: '/mnt/gluster-object/sdb1/tmp/tmpnuLYCV.tmp' (txn: txce2aeaf477ec40e786e91c3d3a56e210)
Apr  1 20:43:39 QA-51 object-server 127.0.0.1 - - [02/Apr/2012:00:43:39 +0000] "PUT /sdb1/190597/AUTH_test/container3/file1KB_720" 500 1093 "-" "txce2aeaf477ec40e786e91c3d3a56e210" "curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.12.9.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2" 0.5085
Apr  1 20:43:39 QA-51 proxy-server ERROR 500 Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift-1.4.9-py2.6.egg/swift/obj/server.py", line 859, in __call__#012    res = getattr(self, req.method)(req)#012  File "/usr/lib/python2.6/site-packages/swift-1.4.9-py2.6.egg/swift/obj/server.py", line 655, in PUT#012    device)#012  File "/usr/lib/python2.6/site-packages/swift-1.4.9-py2.6.egg/swift/obj/server.py", line 471, in container_update#012    contdevice, headers_out, objdevice)#012  File "/usr/lib/python2.6/site-packages/swift-1.4.9-py2.6.egg/swift/obj/server.py", line 449, in async_update#012    os.path.join(self.devices, objdevice, 'tmp'))#012  File "/usr/lib/python2.6/site-packages/swift-1.4.9-py2.6.egg/swift/common/utils.py", line 860, in write_pickle#012    fd, tmppath = mkstemp(dir=tmp, suffix='.tmp')#012  File "/usr/lib64/python2.6/tempfile.py", line 293, in mkstemp#012    return _mkstemp_inner(dir, prefix, suffix, flags)#012  File "/usr/lib64/python2.6/tempfile.py", line 228, in _mkstemp_inner#012    fd = _os.open(file, flags, 0600)#012OSError: [Errno 2] No  From Object Server re: /v1/AUTH_test/container3/file1KB_720 127.0.0.1:6010 (txn: txce2aeaf477ec40e786e91c3d3a56e210) (client_ip: 172.17.251.78)
Apr  1 20:43:39 QA-51 proxy-server Object PUT returning 503 for [500] (txn: txce2aeaf477ec40e786e91c3d3a56e210) (client_ip: 172.17.251.78)

Comment 2 Junaid 2012-04-05 09:10:14 UTC


*** This bug has been marked as a duplicate of bug 806892 ***