Bug 987963 - G4S: HEAD Request for a file,leave the connection open forever
Summary: G4S: HEAD Request for a file,leave the connection open forever
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: gluster-swift
Version: 2.1
Hardware: x86_64
OS: Linux
low
low
Target Milestone: ---
: ---
Assignee: Luis Pabón
QA Contact: pushpesh sharma
URL:
Whiteboard:
Depends On: 960985
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-07-24 13:22 UTC by pushpesh sharma
Modified: 2016-11-08 22:25 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Known Issue
Doc Text:
HEAD Request for an object leaves the connection open. This can be observed when a tool like curl is used to send HTTP requests. Workaround (if any): Terminate/Close the request/command.
Clone Of: 960985
Environment:
Last Closed: 2014-06-26 06:11:54 UTC
Embargoed:


Attachments (Terms of Use)

Description pushpesh sharma 2013-07-24 13:22:59 UTC
+++ This bug was initially created as a clone of Bug #960985 +++

1. Actual file size:-

[root@dhcp207-5 swift]# ls -lh /etc/swift/proxy-server.conf 
-rw-r--r--. 1 root root 1.2K May  8 21:09 /etc/swift/proxy-server.conf

2. Upload the file to a container (success 201, but content length mentioned is not correct i.e. 108, Etag matches md5sum  )

[root@dhcp207-5 swift]# curl -v -X PUT -T proxy-server.conf -H 'X-Auth-Token: AUTH_tk0efc970d210444c09ccfa5209ef0f963' -k https://10.65.207.5:443/v1/AUTH_test2/dir2/test1K.txt
* About to connect() to 10.65.207.5 port 443 (#0)
*   Trying 10.65.207.5...
* connected
* Connected to 10.65.207.5 (10.65.207.5) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* warning: ignoring value of ssl.verifyhost
* skipping SSL peer certificate verification
* SSL connection using TLS_RSA_WITH_AES_256_CBC_SHA
* Server certificate:
* 	subject: O=Default Company Ltd,L=Default City,C=XX
* 	start date: May 08 15:24:51 2013 GMT
* 	expire date: Jun 07 15:24:51 2013 GMT
* 	common name: (nil)
* 	issuer: O=Default Company Ltd,L=Default City,C=XX
> PUT /v1/AUTH_test2/dir2/test1K.txt HTTP/1.1
> User-Agent: curl/7.27.0
> Host: 10.65.207.5
> Accept: */*
> X-Auth-Token: AUTH_tk0efc970d210444c09ccfa5209ef0f963
> Content-Length: 1214
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 201 Created
< Content-Length: 118
< Content-Type: text/html; charset=UTF-8
< Etag: d55c7dbeb35f435f522c299ba91a60f9
< Last-Modified: Wed, 08 May 2013 18:25:12 GMT
< Date: Wed, 08 May 2013 18:25:12 GMT
< 
<html>
 <head>
  <title>201 Created</title>
 </head>
 <body>
  <h1>201 Created</h1>
  <br /><br />



 </body>
* Connection #0 to host 10.65.207.5 left intact
</html>* Closing connection #0

3.) HEAD Request for the same file results in incorrect content-length
(connection is also not closed , WHY ?)
 
[root@dhcp207-5 swift]# curl -v -X HEAD -H 'X-Auth-Token: AUTH_tk0efc970d210444c09ccfa5209ef0f963' -k https://10.65.207.5:443/v1/AUTH_test2/dir2/test1K.txt
* About to connect() to 10.65.207.5 port 443 (#0)
*   Trying 10.65.207.5...
* connected
* Connected to 10.65.207.5 (10.65.207.5) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* warning: ignoring value of ssl.verifyhost
* skipping SSL peer certificate verification
* SSL connection using TLS_RSA_WITH_AES_256_CBC_SHA
* Server certificate:
* 	subject: O=Default Company Ltd,L=Default City,C=XX
* 	start date: May 08 15:24:51 2013 GMT
* 	expire date: Jun 07 15:24:51 2013 GMT
* 	common name: (nil)
* 	issuer: O=Default Company Ltd,L=Default City,C=XX
> HEAD /v1/AUTH_test2/dir2/test1K.txt HTTP/1.1
> User-Agent: curl/7.27.0
> Host: 10.65.207.5
> Accept: */*
> X-Auth-Token: AUTH_tk0efc970d210444c09ccfa5209ef0f963
> 
< HTTP/1.1 200 OK
< Last-Modified: Wed, 08 May 2013 18:25:12 GMT
< Etag: d55c7dbeb35f435f522c299ba91a60f9
< X-Timestamp: 1368037512.85567
< Accept-Ranges: bytes
< Content-Length: 108
< Content-Type: text/html
< Date: Wed, 08 May 2013 18:25:37 GMT
< 
^C

4.) GET request for the file displays correct content and metadata of file 

[root@dhcp207-5 swift]# curl -v -X GET -H 'X-Auth-Token: AUTH_tk0efc970d210444c09ccfa5209ef0f963' -k https://10.65.207.5:443/v1/AUTH_test2/dir2/test1K.txt
* About to connect() to 10.65.207.5 port 443 (#0)
*   Trying 10.65.207.5...
* connected
* Connected to 10.65.207.5 (10.65.207.5) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* warning: ignoring value of ssl.verifyhost
* skipping SSL peer certificate verification
* SSL connection using TLS_RSA_WITH_AES_256_CBC_SHA
* Server certificate:
* 	subject: O=Default Company Ltd,L=Default City,C=XX
* 	start date: May 08 15:24:51 2013 GMT
* 	expire date: Jun 07 15:24:51 2013 GMT
* 	common name: (nil)
* 	issuer: O=Default Company Ltd,L=Default City,C=XX
> GET /v1/AUTH_test2/dir2/test1K.txt HTTP/1.1
> User-Agent: curl/7.27.0
> Host: 10.65.207.5
> Accept: */*
> X-Auth-Token: AUTH_tk0efc970d210444c09ccfa5209ef0f963
> 
< HTTP/1.1 200 OK
< Last-Modified: Wed, 08 May 2013 18:25:12 GMT
< Etag: d55c7dbeb35f435f522c299ba91a60f9
< X-Timestamp: 1368037512.85567
< Accept-Ranges: bytes
< Content-Length: 1214
< Content-Type: text/plain
< Date: Wed, 08 May 2013 18:26:33 GMT
< 
[DEFAULT]
#bind_port = 8080
user = root
log_facility = LOG_LOCAL1
bind_port = 443
cert_file = /etc/swift/cert.crt
key_file = /etc/swift/cert.key
bind_ip = 10.65.207.5

[pipeline:main]
pipeline = healthcheck cache tempauth proxy-server

[app:proxy-server]
use = egg:gluster_swift_ufo#proxy
allow_account_management = true
account_autocreate = true

[filter:tempauth]
use = egg:swift#tempauth
# Here you need to add users explicitly. See the OpenStack Swift Deployment
# Guide for more information. The user and user64 directives take the
# following form:
#     user_<account>_<username> = <key> [group] [group] [...] [storage_url]
#     user64_<account_b64>_<username_b64> = <key> [group] [group] [...] [storage_url]
# Where you use user64 for accounts and/or usernames that include underscores.
#
# NOTE (and WARNING): The account name must match the device name specified
# when generating the account, container, and object build rings.
#
# E.g.
#     user_ufo0_admin = abc123 .admin
user_test_tester = testing .admin
user_test2_tester2 = testing2 .admin
user_test_tester3 = testing3

[filter:healthcheck]
use = egg:swift#healthcheck

[filter:cache]
use = egg:swift#memcache
memcache_servers = 127.0.0.1:11211

* Connection #0 to host 10.65.207.5 left intact
* Closing connection #0
[root@dhcp207-5 swift]# curl -v -X HEAD -H 'X-Auth-Token: AUTH_tk0efc970d210444c09ccfa5209ef0f963' -k https://10.65.207.5:443/v1/AUTH_test2/dir2/test1K.txt
* About to connect() to 10.65.207.5 port 443 (#0)
*   Trying 10.65.207.5...
* connected
* Connected to 10.65.207.5 (10.65.207.5) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* warning: ignoring value of ssl.verifyhost
* skipping SSL peer certificate verification
* SSL connection using TLS_RSA_WITH_AES_256_CBC_SHA
* Server certificate:
* 	subject: O=Default Company Ltd,L=Default City,C=XX
* 	start date: May 08 15:24:51 2013 GMT
* 	expire date: Jun 07 15:24:51 2013 GMT
* 	common name: (nil)
* 	issuer: O=Default Company Ltd,L=Default City,C=XX
> HEAD /v1/AUTH_test2/dir2/test1K.txt HTTP/1.1
> User-Agent: curl/7.27.0
> Host: 10.65.207.5
> Accept: */*
> X-Auth-Token: AUTH_tk0efc970d210444c09ccfa5209ef0f963
> 
< HTTP/1.1 200 OK
< Last-Modified: Wed, 08 May 2013 18:25:12 GMT
< Etag: d55c7dbeb35f435f522c299ba91a60f9
< X-Timestamp: 1368037512.85567
< Accept-Ranges: bytes
< Content-Length: 108
< Content-Type: text/html
< Date: Wed, 08 May 2013 18:26:51 GMT
< 
^C
[root@dhcp207-5 swift]#

--- Additional comment from pushpesh sharma on 2013-05-08 09:03:22 EDT ---

Swift Functional Test stack trace:-

FAIL: testHead (test.functional.tests.TestFile)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/psharma/git/swift/test/functional/tests.py", line 1309, in testHead
    self.assertEquals(info['content_length'], self.env.file_size)
AssertionError: 13 != 128

Test Code :-

 def testHead(self):
        file_name = Utils.create_name()
        content_type = Utils.create_name()

        file = self.env.container.file(file_name)
        file.content_type = content_type
        file.write_random(self.env.file_size)

        md5 = file.md5

        file = self.env.container.file(file_name)
        info = file.info()

        self.assert_status(200)
        self.assertEquals(info['content_length'], self.env.file_size)
        self.assertEquals(info['etag'], md5)
        self.assertEquals(info['content_type'], content_type)
        self.assert_('last_modified' in info)

--- Additional comment from pushpesh sharma on 2013-05-08 09:03:46 EDT ---

[root@dhcp207-5 swift]# rpm -qa |grep swift 
glusterfs-swift-3.3.1-13.fc18.noarch
glusterfs-swift-container-3.3.1-13.fc18.noarch
glusterfs-swift-account-3.3.1-13.fc18.noarch
glusterfs-swift-object-3.3.1-13.fc18.noarch
glusterfs-swift-proxy-3.3.1-13.fc18.noarch
python-swiftclient-1.2.0-3.fc18.noarch
[root@dhcp207-5 swift]#

--- Additional comment from Junaid on 2013-06-04 02:44:50 EDT ---

Tried the similar commands on RHEL with the latest master 
commit 40c313378e5fffb846f1fd143573a687e2774b28

1. File of size 14785

[root@unused swift]# ls -l 0001-Backport-of-patch-e1ff51c04554d51616d2845f92ab726cb0.patch
-rw-r--r--. 1 root root 14785 Jan 16 12:42 0001-Backport-of-patch-e1ff51c04554d51616d2845f92ab726cb0.patch


2. Upload a file of size 14785

[root@unused swift]# curl -v -H 'X-Storage-Token: AUTH_tk472dbbe21e2a40cd969d80cb15074e55' -X PUT -T 0001-Backport-of-patch-e1ff51c04554d51616d2845f92ab726cb0.patch http://192.168.122.60:8080/v1/AUTH_test/c1/d1/d1/t.txt
* About to connect() to 192.168.122.60 port 8080 (#0)
*   Trying 192.168.122.60... connected
* Connected to 192.168.122.60 (192.168.122.60) port 8080 (#0)
> PUT /v1/AUTH_test/c1/d1/d1/t.txt HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.13.1.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2
> Host: 192.168.122.60:8080
> Accept: */*
> X-Storage-Token: AUTH_tk472dbbe21e2a40cd969d80cb15074e55
> Content-Length: 14785
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
< HTTP/1.1 201 Created
< Last-Modified: Tue, 04 Jun 2013 01:03:33 GMT
< Content-Length: 0
< Etag: 4086e0328521e858a9f3bde3e55ad45b
< Content-Type: text/html; charset=UTF-8
< Date: Tue, 04 Jun 2013 01:03:33 GMT
< 
* Connection #0 to host 192.168.122.60 left intact
* Closing connection #0

2. A HEAD operation on the same file displays the correct file size.

[root@unused swift]# curl -v -H 'X-Storage-Token: AUTH_tk472dbbe21e2a40cd969d80cb15074e55' -X HEAD http://192.168.122.60:8080/v1/AUTH_test/c1/d1/d1/t.txt
* About to connect() to 192.168.122.60 port 8080 (#0)
*   Trying 192.168.122.60... connected
* Connected to 192.168.122.60 (192.168.122.60) port 8080 (#0)
> HEAD /v1/AUTH_test/c1/d1/d1/t.txt HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.13.1.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2
> Host: 192.168.122.60:8080
> Accept: */*
> X-Storage-Token: AUTH_tk472dbbe21e2a40cd969d80cb15074e55
> 
< HTTP/1.1 200 OK
< Content-Length: 14785
< Accept-Ranges: bytes
< Last-Modified: Tue, 04 Jun 2013 01:03:33 GMT
< Etag: 4086e0328521e858a9f3bde3e55ad45b
< X-Timestamp: 1370307813.80901
< Content-Type: text/plain
< Date: Tue, 04 Jun 2013 01:03:59 GMT
< 
^C

--- Additional comment from Luis Pabon on 2013-06-19 23:30:09 EDT ---

Issue has been resolved.  Using http://build.gluster.org/job/gluster-swift-builds-cent6/lastSuccessfulBuild/artifact/build/glusterfs-openstack-swift-1.8.0-29.el6.noarch.rpm. 

[root@heka-client-10 swift]# nosetests --exe test/functional/tests.py:TestFile.testHead
testHead (test.functional.tests.TestFile) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.743s

OK

--- Additional comment from pushpesh sharma on 2013-07-24 09:18:43 EDT ---

nosetests --exe ~/gluster-swift/test/functional/tests.py:TestFile.testHead
.
----------------------------------------------------------------------
Ran 1 test in 0.935s

OK
[psharma@dhcp193-181 ~]$ curl -v -X HEAD -H 'X-Auth-Token: AUTH_tk44b09617995d4305a19676857a86b86e' -k http://10.65.207.210:8080/v1/AUTH_test/container/dir1-l0/file.txt
* About to connect() to 10.65.207.210 port 8080 (#0)
*   Trying 10.65.207.210...
* Connected to 10.65.207.210 (10.65.207.210) port 8080 (#0)
> HEAD /v1/AUTH_test/container/dir1-l0/file.txt HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 10.65.207.210:8080
> Accept: */*
> X-Auth-Token: AUTH_tk44b09617995d4305a19676857a86b86e
> 
< HTTP/1.1 200 OK
< Content-Length: 1024
< Accept-Ranges: bytes
< Last-Modified: Wed, 24 Jul 2013 12:42:57 GMT
< Etag: 568ee44fdfc5763002295ac7f732aada
< X-Timestamp: 1374669777.37519
< Content-Type: text/plain
< Date: Wed, 24 Jul 2013 12:43:12 GMT
< 
^C
[psharma@dhcp193-181 ~]$ 


Curl request to the file does not close the connection as it closes the connection for a HEAD request to container and Directory.

Comment 2 Prashanth Pai 2014-06-10 09:22:54 UTC
This is a bug in Swift. (I could reproduce this on a SAIO deployment). Can go in as a known issue. 

Updated doc text accordingly.

Comment 3 pushpesh sharma 2014-06-11 07:24:12 UTC
Agree with Prashanth to make it a know issue.

Comment 4 Prashanth Pai 2014-06-26 06:11:54 UTC
I did some more search on why this behavior is observed. Turns out this is the behavior of curl command line tool and can be reproduced on every URL on the internet (not just Swift). The curl command on HEAD request terminated gracefully when "-I" option is provided.

More info: http://serverfault.com/a/140977


Note You need to log in before you can comment on or make changes to this bug.