| Summary: | Quota: gluster-swift must handle OSError: [Errno 122] Disk quota exceeded | ||
|---|---|---|---|
| Product: | Red Hat Gluster Storage | Reporter: | pushpesh sharma <psharma> |
| Component: | quota | Assignee: | Prashanth Pai <ppai> |
| Status: | CLOSED WONTFIX | QA Contact: | pushpesh sharma <psharma> |
| Severity: | low | Docs Contact: | |
| Priority: | low | ||
| Version: | 2.1 | CC: | bbandari, lpabon, rhs-bugs, storage-qa-internal, thiago, vagarwal |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Known Issue | |
| Doc Text: |
Cause: The gluster-swift object-server correctly catches EDQUOT and returns 507. But Swift proxy-server chooses to ignore 507 received and returns 503 instead.
Consequence: The user would get 503 instead of 507 as HTTP response when quota is exceeded. This is however logged.
Workaround (if any): None
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2014-05-26 12:09:03 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
Description of problem:
There is a gluster volume with quota enabled
On a directory called 'dir' a 100MB hard limit is set
This volume is mounted on a gluster-client with G4S
When user tries to create data more than the hard limit, glusterfs sends OSError: [Errno 122] Disk quota exceeded
glusters-swift is supposed to handle this error properly and return with a REST respose of 507 no space left on device instead of 500(Internal Server Error)
Version-Release number of selected component (if applicable):
[root@dhcp207-15 dummy]# rpm -qa|grep swift
gluster-swift-account-1.8.0-6.11.el6rhs.noarch
gluster-swift-plugin-1.8.0-7.el6rhs.noarch
gluster-swift-1.8.0-6.11.el6rhs.noarch
gluster-swift-proxy-1.8.0-6.11.el6rhs.noarch
gluster-swift-container-1.8.0-6.11.el6rhs.noarch
python-swiftclient-1.4.0-2.el6ost.noarch
gluster-swift-object-1.8.0-6.11.el6rhs.noarch
How reproducible:
It is always reproducible, with following steps:-
1.On gluster volume set the quota for dir
gluster volume quota test limit-usage /dir 100MB
volume quota : success
[root@dhcp207-109 ~]# gluster volume quota test list
Path Hard-limit Soft-limit Used Available
--------------------------------------------------------------------------------
/dir 100.0MB 80% 0Bytes 100.0MB
2.upload >100MB data to directory
#curl -X PUT -T 100MB -v -H 'X-Auth-Token: AUTH_tkc919529fb4f84f9699c747d92bb25133' http://10.65.207.15:8080/v1/AUTH_test/dir/100MB
#curl -X PUT -T 100MB -v -H 'X-Auth-Token: AUTH_tkc919529fb4f84f9699c747d92bb25133' http://10.65.207.15:8080/v1/AUTH_test/dir/100MB
* About to connect() to 10.65.207.15 port 8080 (#0)
* Trying 10.65.207.15... connected
* Connected to 10.65.207.15 (10.65.207.15) port 8080 (#0)
> PUT /v1/AUTH_test/dir/100MB HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.3.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: 10.65.207.15:8080
> Accept: */*
> X-Auth-Token: AUTH_tkc919529fb4f84f9699c747d92bb25133
> Content-Length: 104857600
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
< HTTP/1.1 503 Internal Server Error
< Last-Modified: Fri, 18 Oct 2013 07:30:18 GMT
< Content-Length: 118
< Content-Type: text/html; charset=UTF-8
< X-Trans-Id: tx6205a4c2f4c14607a1a4351b194a8861
< Date: Fri, 18 Oct 2013 07:30:18 GMT
<
* Connection #0 to host 10.65.207.15 left intact
* Closing connection #0
<html><h1>Service Unavailable</h1><p>The server is currently unavailable. Please try again at a later time.</p></html>[root@dhcp207-15 dummy]#
Actual results:
500 Internal server Error
Expected results:
507 No space left /Or quota exceeded
Additional info:
snaps form logs
Oct 18 13:00:18 dhcp207-15 object-server ERROR __call__ error with PUT /test/0/AUTH_test/dir/100MB : #012Traceback (most recent call last):#012 File "/usr/lib/python2.6/site-packages/swift/obj/server.py", line 928, in __call__#012 res = method(req)#012 File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 1558, in wrapped#012 return func(*a, **kw)#012 File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 520, in _timing_stats#012 resp = func(ctrl, *args, **kwargs)#012 File "/usr/lib/python2.6/site-packages/gluster/swift/obj/server.py", line 63, in PUT#012 return server.ObjectController.PUT(self, request)#012 File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 1558, in wrapped#012 return func(*a, **kw)#012 File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 520, in _timing_stats#012 resp = func(ctrl, *args, **kwargs)#012 File "/usr/lib/python2.6/site-packages/swift/obj/server.py", line 705, in PUT#012 file.put(fd, metadata)#012 File "/usr/lib64/python2.6/contextlib.py", line 34, in __exit__#012 self.gen.throw(type, value, traceback)#012 File "/usr/lib/python2.6/site-packages/gluster/swift/common/DiskFile.py", line 775, in mkstemp#012 yield fd#012 File "/usr/lib/python2.6/site-packages/swift/obj/server.py", line 661, in PUT#012 written = os.write(fd, chunk)#012OSError: [Errno 122] Disk quota exceeded (txn: tx6205a4c2f4c14607a1a4351b194a8861)
Oct 18 13:00:18 dhcp207-15 proxy-server ERROR 500 Traceback (most recent call last):#012 File "/usr/lib/python2.6/site-packages/swift/obj/server.py", line 928, in __call__#012 res = method(req)#012 File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 1558, in wrapped#012 return func(*a, **kw)#012 File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 520, in _timing_stats#012 resp = func(ctrl, *args, **kwargs)#012 File "/usr/lib/python2.6/site-packages/gluster/swift/obj/server.py", line 63, in PUT#012 return server.ObjectController.PUT(self, request)#012 File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 1558, in wrapped#012 return func(*a, **kw)#012 File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 520, in _timing_stats#012 resp = func(ctrl, *args, **kwargs)#012 File "/usr/lib/python2.6/site-packages/swift/obj/server.py", line 705, in PUT#012 file.put(fd, metadata)#012 File "/usr/lib64/python2.6/contextlib.py", line 34, in __exit__#012 self.gen.throw(type, value, traceback)#012 File "/usr/lib/python2.6/site-packa From Object Server re: /v1/AUTH_test/dir/100MB 10.65.207.15:6010 (txn: tx6205a4c2f4c14607a1a4351b194a8861)
Oct 18 13:00:18 dhcp207-15 proxy-server Object PUT returning 503 for [500] (txn: tx6205a4c2f4c14607a1a4351b194a8861) (client_ip: 10.65.207.15)
Oct 18 13:00:18 dhcp207-15 proxy-server 10.65.207.15 10.65.207.15 18/Oct/2013/07/30/18 PUT /v1/AUTH_test/dir/100MB HTTP/1.0 503 - curl/7.19.7%20%28x86_64-redhat-linux-gnu%29%20libcurl/7.19.7%20NSS/3.14.3.0%20zlib/1.2.3%20libidn/1.18%20libssh2/1.4.2 test%2CAUTH_tkc919529fb4f84f9699c747d92bb25133 104857600 118 - tx6205a4c2f4c14607a1a4351b194a8861 - 0.3334 -
Replace 500 with 503 (Internal Server Error) Again, this bug will be fixed in Havana as currently Openstack Grizzly does not honour 507. I think this is the issue with write-behind: https://bugzilla.redhat.com/show_bug.cgi?id=986812 This will remain known issue for Denali. This bug may go to 'wontfix' as the new directional changes for the project suggest this is very low priority corner case. Closing this as WONTFIX as there's nothing we can change gluster-swift to fix this. Refer to doc text for more info. Documentation tracker bug: https://bugzilla.redhat.com/show_bug.cgi?id=1101914 |
Description of problem: Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1.On gluster volume set the quota for dir gluster volume quota test limit-usage /dir 100MB volume quota : success [root@dhcp207-109 ~]# gluster volume quota test list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- /dir 100.0MB 80% 0Bytes 100.0MB 2.upload >100MB data to directory #curl -X PUT -T 100MB -v -H 'X-Auth-Token: AUTH_tkc919529fb4f84f9699c747d92bb25133' http://10.65.207.15:8080/v1/AUTH_test/dir/100MB #curl -X PUT -T 100MB -v -H 'X-Auth-Token: AUTH_tkc919529fb4f84f9699c747d92bb25133' http://10.65.207.15:8080/v1/AUTH_test/dir/100MB * About to connect() to 10.65.207.15 port 8080 (#0) * Trying 10.65.207.15... connected * Connected to 10.65.207.15 (10.65.207.15) port 8080 (#0) > PUT /v1/AUTH_test/dir/100MB HTTP/1.1 > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.3.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2 > Host: 10.65.207.15:8080 > Accept: */* > X-Auth-Token: AUTH_tkc919529fb4f84f9699c747d92bb25133 > Content-Length: 104857600 > Expect: 100-continue > < HTTP/1.1 100 Continue < HTTP/1.1 503 Internal Server Error < Last-Modified: Fri, 18 Oct 2013 07:30:18 GMT < Content-Length: 118 < Content-Type: text/html; charset=UTF-8 < X-Trans-Id: tx6205a4c2f4c14607a1a4351b194a8861 < Date: Fri, 18 Oct 2013 07:30:18 GMT < * Connection #0 to host 10.65.207.15 left intact * Closing connection #0 <html><h1>Service Unavailable</h1><p>The server is currently unavailable. Please try again at a later time.</p></html>[root@dhcp207-15 dummy]# 3. snaps form logs Oct 18 13:00:18 dhcp207-15 object-server ERROR __call__ error with PUT /test/0/AUTH_test/dir/100MB : #012Traceback (most recent call last):#012 File "/usr/lib/python2.6/site-packages/swift/obj/server.py", line 928, in __call__#012 res = method(req)#012 File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 1558, in wrapped#012 return func(*a, **kw)#012 File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 520, in _timing_stats#012 resp = func(ctrl, *args, **kwargs)#012 File "/usr/lib/python2.6/site-packages/gluster/swift/obj/server.py", line 63, in PUT#012 return server.ObjectController.PUT(self, request)#012 File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 1558, in wrapped#012 return func(*a, **kw)#012 File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 520, in _timing_stats#012 resp = func(ctrl, *args, **kwargs)#012 File "/usr/lib/python2.6/site-packages/swift/obj/server.py", line 705, in PUT#012 file.put(fd, metadata)#012 File "/usr/lib64/python2.6/contextlib.py", line 34, in __exit__#012 self.gen.throw(type, value, traceback)#012 File "/usr/lib/python2.6/site-packages/gluster/swift/common/DiskFile.py", line 775, in mkstemp#012 yield fd#012 File "/usr/lib/python2.6/site-packages/swift/obj/server.py", line 661, in PUT#012 written = os.write(fd, chunk)#012OSError: [Errno 122] Disk quota exceeded (txn: tx6205a4c2f4c14607a1a4351b194a8861) Oct 18 13:00:18 dhcp207-15 proxy-server ERROR 500 Traceback (most recent call last):#012 File "/usr/lib/python2.6/site-packages/swift/obj/server.py", line 928, in __call__#012 res = method(req)#012 File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 1558, in wrapped#012 return func(*a, **kw)#012 File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 520, in _timing_stats#012 resp = func(ctrl, *args, **kwargs)#012 File "/usr/lib/python2.6/site-packages/gluster/swift/obj/server.py", line 63, in PUT#012 return server.ObjectController.PUT(self, request)#012 File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 1558, in wrapped#012 return func(*a, **kw)#012 File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 520, in _timing_stats#012 resp = func(ctrl, *args, **kwargs)#012 File "/usr/lib/python2.6/site-packages/swift/obj/server.py", line 705, in PUT#012 file.put(fd, metadata)#012 File "/usr/lib64/python2.6/contextlib.py", line 34, in __exit__#012 self.gen.throw(type, value, traceback)#012 File "/usr/lib/python2.6/site-packa From Object Server re: /v1/AUTH_test/dir/100MB 10.65.207.15:6010 (txn: tx6205a4c2f4c14607a1a4351b194a8861) Oct 18 13:00:18 dhcp207-15 proxy-server Object PUT returning 503 for [500] (txn: tx6205a4c2f4c14607a1a4351b194a8861) (client_ip: 10.65.207.15) Oct 18 13:00:18 dhcp207-15 proxy-server 10.65.207.15 10.65.207.15 18/Oct/2013/07/30/18 PUT /v1/AUTH_test/dir/100MB HTTP/1.0 503 - curl/7.19.7%20%28x86_64-redhat-linux-gnu%29%20libcurl/7.19.7%20NSS/3.14.3.0%20zlib/1.2.3%20libidn/1.18%20libssh2/1.4.2 test%2CAUTH_tkc919529fb4f84f9699c747d92bb25133 104857600 118 - tx6205a4c2f4c14607a1a4351b194a8861 - 0.3334 - Actual results: Expected results: [root@dhcp207-15 dummy]# rpm -qa|grep swift gluster-swift-account-1.8.0-6.11.el6rhs.noarch gluster-swift-plugin-1.8.0-7.el6rhs.noarch gluster-swift-1.8.0-6.11.el6rhs.noarch gluster-swift-proxy-1.8.0-6.11.el6rhs.noarch gluster-swift-container-1.8.0-6.11.el6rhs.noarch python-swiftclient-1.4.0-2.el6ost.noarch gluster-swift-object-1.8.0-6.11.el6rhs.noarch Additional info: