1020724 – Quota: gluster-swift must handle OSError: [Errno 122] Disk quota exceeded

Bug 1020724 - Quota: gluster-swift must handle OSError: [Errno 122] Disk quota exceeded

Summary: Quota: gluster-swift must handle OSError: [Errno 122] Disk quota exceeded

Keywords:
Status:	CLOSED WONTFIX
Alias:	None
Product:	Red Hat Gluster Storage
Classification:	Red Hat
Component:	quota
Sub Component:
Version:	2.1
Hardware:	x86_64
OS:	Linux
Priority:	low
Severity:	low
Target Milestone:	---
Target Release:	---
Assignee:	Prashanth Pai
QA Contact:	pushpesh sharma
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2013-10-18 08:31 UTC by pushpesh sharma
Modified:	2016-09-17 12:41 UTC (History)
CC List:	6 users (show)
Fixed In Version:
Doc Type:	Known Issue
Doc Text:	Cause: The gluster-swift object-server correctly catches EDQUOT and returns 507. But Swift proxy-server chooses to ignore 507 received and returns 503 instead. Consequence: The user would get 503 instead of 507 as HTTP response when quota is exceeded. This is however logged. Workaround (if any): None
Clone Of:
Environment:
Last Closed:	2014-05-26 12:09:03 UTC
Target Upstream Version:
Dependent Products:

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

Description pushpesh sharma 2013-10-18 08:31:15 UTC

Description of problem:


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.On gluster volume set the quota for dir
gluster volume quota test limit-usage /dir 100MB
volume quota : success
[root@dhcp207-109 ~]# gluster volume quota test list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/dir                                     100.0MB       80%      0Bytes 100.0MB
2.upload >100MB data to directory 

#curl -X PUT -T 100MB -v -H 'X-Auth-Token: AUTH_tkc919529fb4f84f9699c747d92bb25133' http://10.65.207.15:8080/v1/AUTH_test/dir/100MB

#curl -X PUT -T 100MB -v -H 'X-Auth-Token: AUTH_tkc919529fb4f84f9699c747d92bb25133' http://10.65.207.15:8080/v1/AUTH_test/dir/100MB
* About to connect() to 10.65.207.15 port 8080 (#0)
*   Trying 10.65.207.15... connected
* Connected to 10.65.207.15 (10.65.207.15) port 8080 (#0)
> PUT /v1/AUTH_test/dir/100MB HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.3.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: 10.65.207.15:8080
> Accept: */*
> X-Auth-Token: AUTH_tkc919529fb4f84f9699c747d92bb25133
> Content-Length: 104857600
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
< HTTP/1.1 503 Internal Server Error
< Last-Modified: Fri, 18 Oct 2013 07:30:18 GMT
< Content-Length: 118
< Content-Type: text/html; charset=UTF-8
< X-Trans-Id: tx6205a4c2f4c14607a1a4351b194a8861
< Date: Fri, 18 Oct 2013 07:30:18 GMT
< 
* Connection #0 to host 10.65.207.15 left intact
* Closing connection #0
<html><h1>Service Unavailable</h1><p>The server is currently unavailable. Please try again at a later time.</p></html>[root@dhcp207-15 dummy]# 

3. snaps form logs 
Oct 18 13:00:18 dhcp207-15 object-server ERROR __call__ error with PUT /test/0/AUTH_test/dir/100MB : #012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/obj/server.py", line 928, in __call__#012    res = method(req)#012  File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 1558, in wrapped#012    return func(*a, **kw)#012  File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 520, in _timing_stats#012    resp = func(ctrl, *args, **kwargs)#012  File "/usr/lib/python2.6/site-packages/gluster/swift/obj/server.py", line 63, in PUT#012    return server.ObjectController.PUT(self, request)#012  File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 1558, in wrapped#012    return func(*a, **kw)#012  File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 520, in _timing_stats#012    resp = func(ctrl, *args, **kwargs)#012  File "/usr/lib/python2.6/site-packages/swift/obj/server.py", line 705, in PUT#012    file.put(fd, metadata)#012  File "/usr/lib64/python2.6/contextlib.py", line 34, in __exit__#012    self.gen.throw(type, value, traceback)#012  File "/usr/lib/python2.6/site-packages/gluster/swift/common/DiskFile.py", line 775, in mkstemp#012    yield fd#012  File "/usr/lib/python2.6/site-packages/swift/obj/server.py", line 661, in PUT#012    written = os.write(fd, chunk)#012OSError: [Errno 122] Disk quota exceeded (txn: tx6205a4c2f4c14607a1a4351b194a8861)
Oct 18 13:00:18 dhcp207-15 proxy-server ERROR 500 Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/obj/server.py", line 928, in __call__#012    res = method(req)#012  File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 1558, in wrapped#012    return func(*a, **kw)#012  File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 520, in _timing_stats#012    resp = func(ctrl, *args, **kwargs)#012  File "/usr/lib/python2.6/site-packages/gluster/swift/obj/server.py", line 63, in PUT#012    return server.ObjectController.PUT(self, request)#012  File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 1558, in wrapped#012    return func(*a, **kw)#012  File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 520, in _timing_stats#012    resp = func(ctrl, *args, **kwargs)#012  File "/usr/lib/python2.6/site-packages/swift/obj/server.py", line 705, in PUT#012    file.put(fd, metadata)#012  File "/usr/lib64/python2.6/contextlib.py", line 34, in __exit__#012    self.gen.throw(type, value, traceback)#012  File "/usr/lib/python2.6/site-packa From Object Server re: /v1/AUTH_test/dir/100MB 10.65.207.15:6010 (txn: tx6205a4c2f4c14607a1a4351b194a8861)
Oct 18 13:00:18 dhcp207-15 proxy-server Object PUT returning 503 for [500] (txn: tx6205a4c2f4c14607a1a4351b194a8861) (client_ip: 10.65.207.15)
Oct 18 13:00:18 dhcp207-15 proxy-server 10.65.207.15 10.65.207.15 18/Oct/2013/07/30/18 PUT /v1/AUTH_test/dir/100MB HTTP/1.0 503 - curl/7.19.7%20%28x86_64-redhat-linux-gnu%29%20libcurl/7.19.7%20NSS/3.14.3.0%20zlib/1.2.3%20libidn/1.18%20libssh2/1.4.2 test%2CAUTH_tkc919529fb4f84f9699c747d92bb25133 104857600 118 - tx6205a4c2f4c14607a1a4351b194a8861 - 0.3334 -

Actual results:


Expected results:


[root@dhcp207-15 dummy]# rpm -qa|grep swift
gluster-swift-account-1.8.0-6.11.el6rhs.noarch
gluster-swift-plugin-1.8.0-7.el6rhs.noarch
gluster-swift-1.8.0-6.11.el6rhs.noarch
gluster-swift-proxy-1.8.0-6.11.el6rhs.noarch
gluster-swift-container-1.8.0-6.11.el6rhs.noarch
python-swiftclient-1.4.0-2.el6ost.noarch
gluster-swift-object-1.8.0-6.11.el6rhs.noarch


Additional info:

Comment 1 pushpesh sharma 2013-10-18 08:47:28 UTC

Description of problem:

There is a gluster volume with quota enabled
On a directory called 'dir' a 100MB hard limit is set
This volume is mounted on a gluster-client with G4S
When user tries to create data more than the hard limit, glusterfs sends OSError: [Errno 122] Disk quota exceeded  

glusters-swift is supposed to handle this error properly and return with a REST respose of 507 no space left on device instead of 500(Internal Server Error) 

Version-Release number of selected component (if applicable):

[root@dhcp207-15 dummy]# rpm -qa|grep swift
gluster-swift-account-1.8.0-6.11.el6rhs.noarch
gluster-swift-plugin-1.8.0-7.el6rhs.noarch
gluster-swift-1.8.0-6.11.el6rhs.noarch
gluster-swift-proxy-1.8.0-6.11.el6rhs.noarch
gluster-swift-container-1.8.0-6.11.el6rhs.noarch
python-swiftclient-1.4.0-2.el6ost.noarch
gluster-swift-object-1.8.0-6.11.el6rhs.noarch


How reproducible:

It is always reproducible, with following steps:-

 1.On gluster volume set the quota for dir
gluster volume quota test limit-usage /dir 100MB
volume quota : success
[root@dhcp207-109 ~]# gluster volume quota test list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/dir                                     100.0MB       80%      0Bytes 100.0MB
2.upload >100MB data to directory 

#curl -X PUT -T 100MB -v -H 'X-Auth-Token: AUTH_tkc919529fb4f84f9699c747d92bb25133' http://10.65.207.15:8080/v1/AUTH_test/dir/100MB

#curl -X PUT -T 100MB -v -H 'X-Auth-Token: AUTH_tkc919529fb4f84f9699c747d92bb25133' http://10.65.207.15:8080/v1/AUTH_test/dir/100MB
* About to connect() to 10.65.207.15 port 8080 (#0)
*   Trying 10.65.207.15... connected
* Connected to 10.65.207.15 (10.65.207.15) port 8080 (#0)
> PUT /v1/AUTH_test/dir/100MB HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.3.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: 10.65.207.15:8080
> Accept: */*
> X-Auth-Token: AUTH_tkc919529fb4f84f9699c747d92bb25133
> Content-Length: 104857600
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
< HTTP/1.1 503 Internal Server Error
< Last-Modified: Fri, 18 Oct 2013 07:30:18 GMT
< Content-Length: 118
< Content-Type: text/html; charset=UTF-8
< X-Trans-Id: tx6205a4c2f4c14607a1a4351b194a8861
< Date: Fri, 18 Oct 2013 07:30:18 GMT
< 
* Connection #0 to host 10.65.207.15 left intact
* Closing connection #0
<html><h1>Service Unavailable</h1><p>The server is currently unavailable. Please try again at a later time.</p></html>[root@dhcp207-15 dummy]#

Actual results:

500 Internal server Error

Expected results:

507 No space left /Or quota exceeded 
 
Additional info:

snaps form logs 
Oct 18 13:00:18 dhcp207-15 object-server ERROR __call__ error with PUT /test/0/AUTH_test/dir/100MB : #012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/obj/server.py", line 928, in __call__#012    res = method(req)#012  File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 1558, in wrapped#012    return func(*a, **kw)#012  File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 520, in _timing_stats#012    resp = func(ctrl, *args, **kwargs)#012  File "/usr/lib/python2.6/site-packages/gluster/swift/obj/server.py", line 63, in PUT#012    return server.ObjectController.PUT(self, request)#012  File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 1558, in wrapped#012    return func(*a, **kw)#012  File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 520, in _timing_stats#012    resp = func(ctrl, *args, **kwargs)#012  File "/usr/lib/python2.6/site-packages/swift/obj/server.py", line 705, in PUT#012    file.put(fd, metadata)#012  File "/usr/lib64/python2.6/contextlib.py", line 34, in __exit__#012    self.gen.throw(type, value, traceback)#012  File "/usr/lib/python2.6/site-packages/gluster/swift/common/DiskFile.py", line 775, in mkstemp#012    yield fd#012  File "/usr/lib/python2.6/site-packages/swift/obj/server.py", line 661, in PUT#012    written = os.write(fd, chunk)#012OSError: [Errno 122] Disk quota exceeded (txn: tx6205a4c2f4c14607a1a4351b194a8861)
Oct 18 13:00:18 dhcp207-15 proxy-server ERROR 500 Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/swift/obj/server.py", line 928, in __call__#012    res = method(req)#012  File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 1558, in wrapped#012    return func(*a, **kw)#012  File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 520, in _timing_stats#012    resp = func(ctrl, *args, **kwargs)#012  File "/usr/lib/python2.6/site-packages/gluster/swift/obj/server.py", line 63, in PUT#012    return server.ObjectController.PUT(self, request)#012  File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 1558, in wrapped#012    return func(*a, **kw)#012  File "/usr/lib/python2.6/site-packages/swift/common/utils.py", line 520, in _timing_stats#012    resp = func(ctrl, *args, **kwargs)#012  File "/usr/lib/python2.6/site-packages/swift/obj/server.py", line 705, in PUT#012    file.put(fd, metadata)#012  File "/usr/lib64/python2.6/contextlib.py", line 34, in __exit__#012    self.gen.throw(type, value, traceback)#012  File "/usr/lib/python2.6/site-packa From Object Server re: /v1/AUTH_test/dir/100MB 10.65.207.15:6010 (txn: tx6205a4c2f4c14607a1a4351b194a8861)
Oct 18 13:00:18 dhcp207-15 proxy-server Object PUT returning 503 for [500] (txn: tx6205a4c2f4c14607a1a4351b194a8861) (client_ip: 10.65.207.15)
Oct 18 13:00:18 dhcp207-15 proxy-server 10.65.207.15 10.65.207.15 18/Oct/2013/07/30/18 PUT /v1/AUTH_test/dir/100MB HTTP/1.0 503 - curl/7.19.7%20%28x86_64-redhat-linux-gnu%29%20libcurl/7.19.7%20NSS/3.14.3.0%20zlib/1.2.3%20libidn/1.18%20libssh2/1.4.2 test%2CAUTH_tkc919529fb4f84f9699c747d92bb25133 104857600 118 - tx6205a4c2f4c14607a1a4351b194a8861 - 0.3334 -

Comment 2 pushpesh sharma 2013-10-18 08:49:41 UTC

Replace 500 with 503 (Internal Server Error)

Comment 4 Prashanth Pai 2013-10-22 12:48:16 UTC

Again, this bug will be fixed in Havana as currently Openstack Grizzly does not honour 507.

Comment 5 Prashanth Pai 2013-11-19 05:42:06 UTC

http://review.gluster.org/#/c/6199/

Comment 6 Prashanth Pai 2014-02-28 09:54:40 UTC

I think this is the issue with write-behind: https://bugzilla.redhat.com/show_bug.cgi?id=986812

Comment 8 pushpesh sharma 2014-05-26 10:30:05 UTC

This will remain known issue for Denali. This bug may go to 'wontfix' as the new directional changes for the project suggest this is very low priority corner case.

Comment 9 Prashanth Pai 2014-05-26 12:09:03 UTC

Closing this as WONTFIX as there's nothing we can change gluster-swift to fix this. Refer to doc text for more info.

Comment 10 pushpesh sharma 2014-05-28 07:30:19 UTC

Documentation tracker bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1101914

Note You need to log in before you can comment on or make changes to this bug.