Bug 998383

Summary:	[REST]: An Incomplete PUT request results in a dangling hidden file on the gluster volume
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	pushpesh sharma <psharma>
Component:	gluster-swift	Assignee:	Luis Pabón <lpabon>
Status:	CLOSED CANTFIX	QA Contact:	amainkar
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	2.1	CC:	david.macdonald, lpabon, madam, ppai, pportant, rhs-bugs, vagarwal
Target Milestone:	---	Keywords:	ZStream
Target Release:	---
Hardware:	aarch64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:
Clones:	1024608 (view as bug list)		Environment:
Last Closed:	2015-02-20 06:30:48 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1024608

Description pushpesh sharma 2013-08-19 08:10:31 UTC

Description of problem:


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.On a plane G4S setup, issue a PUT curl requests for a considerable bigger file. and interrupt the request (ctrl^+c) in between.
  
2.

curl -v -X PUT -T 10g.img -H 'X-Auth-Token: AUTH_tke4a76ae4646e48b8ac626771cd1d6578' -k http://localhost:8080/v1/AUTH_test/dir/10.img
* About to connect() to localhost port 8080 (#0)
*   Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 8080 (#0)
> PUT /v1/AUTH_test/dir/10.img HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.3.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: localhost:8080
> Accept: */*
> X-Auth-Token: AUTH_tke4a76ae4646e48b8ac626771cd1d6578
> Content-Length: 10000000000
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
^C
[root@mater ~]# 


3. Look for the files on volume using mount point. 

[root@mater ~]# ls -ahl /mnt/gluster-object/test/dir/
total 1.7G
drwxr-xr-x  2 root root   65 Aug 18 15:15 .
drwxr-xr-x 10 root root  24K Aug 15 16:56 ..
-rwxr-xr-x  1 root root 1.7G Aug 18 15:15 .10.img.ec135219c994552721fcf6b29f836181
[root@mater ~]# 

Aug 18 15:15:51 mater proxy-server Client disconnected without sending enough data (txn: tx12da5da3632c4b3ca20c47a53f8a941f) (client_ip: 127.0.0.1)
Aug 18 15:15:51 mater proxy-server 127.0.0.1 127.0.0.1 18/Aug/2013/19/15/51 PUT /v1/AUTH_test/dir/10.img HTTP/1.0 499 - curl/7.19.7%20%28x86_64-redhat-linux-gnu%29%20libcurl/7.19.7%20NSS/3.14.3.0%20zlib/1.2.3%20libidn/1.18%20libssh2/1.4.2 test%2CAUTH_tke4a76ae4646e48b8ac626771cd1d6578 1777876992 89 - tx12da5da3632c4b3ca20c47a53f8a941f - 24.9707 -



Actual results:


Expected results:

Any dangling file should be removed, immediately/or even after a time frame if we are providing some kind of recovery mode.
 
Additional info:

 rpm -qa|grep swift
gluster-swift-container-1.8.0-6.11.el6rhs.noarch
python-swiftclient-1.4.0-1.el6.noarch
gluster-swift-plugin-1.8.0-6.el6rhs.noarch
gluster-swift-proxy-1.8.0-6.11.el6rhs.noarch
gluster-swift-account-1.8.0-6.11.el6rhs.noarch
gluster-swift-1.8.0-6.11.el6rhs.noarch
gluster-swift-object-1.8.0-6.11.el6rhs.noarch
[root@mater ~]#

Comment 2 Luis Pabón 2013-08-19 12:47:49 UTC

Please fix in release-1.8.0 branch first, then after it is merged, we will move it to master branch.

Comment 3 Prashanth Pai 2013-10-28 14:11:57 UTC

openstack-swift behavior : On doing a Ctrl+C during upload, the temp file generated is not automatically deleted. It is cleared when "swift-init main restart" is executed.

gluster-swift behavior: gluster-swift follows swift behavior ONLY when "catch_errors" filter is removed from pipeline in /etc/swift/proxy-server.conf

Comment 4 Prashanth Pai 2013-10-30 05:47:04 UTC

Removing "catch_errors" was a false positive. The bug still persists regardless of catch_errors is present in pipeline or not.

Comment 5 Luis Pabón 2013-11-07 03:19:39 UTC

I was able to spend some time investigating this issue and it looks like we may need some changes in the proxy object controller in the OpenStack Swift code for gluster-swift code to work correctly.  One of the issues is that in the PUT code, if the function detects that the client connection has timed out, then it just returns.  It returns without notifying the object server and closing the Proxy<-->Object server connection.

More information to come.

Comment 6 Luis Pabón 2013-11-11 16:13:01 UTC

This fix requires fixes in OpenStack Swift Icehouse.  The main issue is in swift/proxy/controllers/obj.py:PUT.  It first creates connections 'conns' to the object servers.  Once the connections are in process of transferring data, it may get a timeout in ChunkReadTimeout(..) which makes it fall to a 'break'.  Once it hits the break, then it falls to the code:
        if req.content_length and bytes_transferred < req.content_length:
            req.client_disconnect = True
            self.app.logger.warn(
                _('Client disconnected without sending enough data'))
            self.app.logger.increment('client_disconnects')
            return HTTPClientDisconnect(request=req)

Returns to the caller without notifying the object server that the channel has been terminated.  Actually if you notice, this happens with quite a few lines of code just above the code state previously.  Returning to the caller without notifying the underlying servers can cause major issues.

Comment 10 Prashanth Pai 2015-02-20 06:30:48 UTC

This has to be fixed in Swift, there's nothing that can be done in gluster-swift.