Description of problem: Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1.On a plane G4S setup, issue a PUT curl requests for a considerable bigger file. and interrupt the request (ctrl^+c) in between. 2. curl -v -X PUT -T 10g.img -H 'X-Auth-Token: AUTH_tke4a76ae4646e48b8ac626771cd1d6578' -k http://localhost:8080/v1/AUTH_test/dir/10.img * About to connect() to localhost port 8080 (#0) * Trying 127.0.0.1... connected * Connected to localhost (127.0.0.1) port 8080 (#0) > PUT /v1/AUTH_test/dir/10.img HTTP/1.1 > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.3.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2 > Host: localhost:8080 > Accept: */* > X-Auth-Token: AUTH_tke4a76ae4646e48b8ac626771cd1d6578 > Content-Length: 10000000000 > Expect: 100-continue > < HTTP/1.1 100 Continue ^C [root@mater ~]# 3. Look for the files on volume using mount point. [root@mater ~]# ls -ahl /mnt/gluster-object/test/dir/ total 1.7G drwxr-xr-x 2 root root 65 Aug 18 15:15 . drwxr-xr-x 10 root root 24K Aug 15 16:56 .. -rwxr-xr-x 1 root root 1.7G Aug 18 15:15 .10.img.ec135219c994552721fcf6b29f836181 [root@mater ~]# Aug 18 15:15:51 mater proxy-server Client disconnected without sending enough data (txn: tx12da5da3632c4b3ca20c47a53f8a941f) (client_ip: 127.0.0.1) Aug 18 15:15:51 mater proxy-server 127.0.0.1 127.0.0.1 18/Aug/2013/19/15/51 PUT /v1/AUTH_test/dir/10.img HTTP/1.0 499 - curl/7.19.7%20%28x86_64-redhat-linux-gnu%29%20libcurl/7.19.7%20NSS/3.14.3.0%20zlib/1.2.3%20libidn/1.18%20libssh2/1.4.2 test%2CAUTH_tke4a76ae4646e48b8ac626771cd1d6578 1777876992 89 - tx12da5da3632c4b3ca20c47a53f8a941f - 24.9707 - Actual results: Expected results: Any dangling file should be removed, immediately/or even after a time frame if we are providing some kind of recovery mode. Additional info: rpm -qa|grep swift gluster-swift-container-1.8.0-6.11.el6rhs.noarch python-swiftclient-1.4.0-1.el6.noarch gluster-swift-plugin-1.8.0-6.el6rhs.noarch gluster-swift-proxy-1.8.0-6.11.el6rhs.noarch gluster-swift-account-1.8.0-6.11.el6rhs.noarch gluster-swift-1.8.0-6.11.el6rhs.noarch gluster-swift-object-1.8.0-6.11.el6rhs.noarch [root@mater ~]#
Please fix in release-1.8.0 branch first, then after it is merged, we will move it to master branch.
openstack-swift behavior : On doing a Ctrl+C during upload, the temp file generated is not automatically deleted. It is cleared when "swift-init main restart" is executed. gluster-swift behavior: gluster-swift follows swift behavior ONLY when "catch_errors" filter is removed from pipeline in /etc/swift/proxy-server.conf
Removing "catch_errors" was a false positive. The bug still persists regardless of catch_errors is present in pipeline or not.
I was able to spend some time investigating this issue and it looks like we may need some changes in the proxy object controller in the OpenStack Swift code for gluster-swift code to work correctly. One of the issues is that in the PUT code, if the function detects that the client connection has timed out, then it just returns. It returns without notifying the object server and closing the Proxy<-->Object server connection. More information to come.
This fix requires fixes in OpenStack Swift Icehouse. The main issue is in swift/proxy/controllers/obj.py:PUT. It first creates connections 'conns' to the object servers. Once the connections are in process of transferring data, it may get a timeout in ChunkReadTimeout(..) which makes it fall to a 'break'. Once it hits the break, then it falls to the code: if req.content_length and bytes_transferred < req.content_length: req.client_disconnect = True self.app.logger.warn( _('Client disconnected without sending enough data')) self.app.logger.increment('client_disconnects') return HTTPClientDisconnect(request=req) Returns to the caller without notifying the object server that the channel has been terminated. Actually if you notice, this happens with quite a few lines of code just above the code state previously. Returning to the caller without notifying the underlying servers can cause major issues.
This has to be fixed in Swift, there's nothing that can be done in gluster-swift.