Bug 1368641

Description of problem:

Seems there is a memory leak in the swift when used with the glance as backup. And the connection for some reason is terminated during the transfer(either HAProxy) the memory stays allocated in swift proccess which get's killed by OOM afterwards.

Version-Release number of selected component (if applicable):

openstack-swift-2.5.0-2.el7ost.noarch
openstack-swift-account-2.5.0-2.el7ost.noarch
openstack-swift-container-2.5.0-2.el7ost.noarch
openstack-swift-object-2.5.0-2.el7ost.noarch
openstack-swift-plugin-swift3-1.9-1.el7ost.noarch
openstack-swift-proxy-2.5.0-2.el7ost.noarch
python-swiftclient-2.6.0-1.el7ost.noarch

openstack-glance-11.0.1-4.el7ost.noarch
python-glance-11.0.1-4.el7ost.noarch
python-glance-store-0.9.1-3.el7ost.noarch
python-glanceclient-1.1.1-1.el7ost.noarch

How reproducible:

glance -d image-download 03d98224-66ba-11e6-8b77-86f30ca893d3 --file edge.iso
--progress
DEBUG:keystoneclient.session:REQ: curl -g -i -X GET http://192.168.0.5:5000/v2.0
-H "Accept: application/json" -H "User-Agent: python-keystoneclient"
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection
(1): 192.168.0.5
DEBUG:requests.packages.urllib3.connectionpool:"GET /v2.0 HTTP/1.1" 200 335
DEBUG:keystoneclient.session:RESP: [200] Vary: X-Auth-Token Content-Type:
application/json Content-Length: 335 X-Openstack-Request-Id:
req-0f622e02-66ba-11e6-8b77-86f30ca893d3 Date: Sat, 20 Aug 2016 03:37:49 GMT
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z",
"media-types": [{"base": "application/json", "type":
"application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links":
[{"href": "http://192.168.0.5:5000/v2.0/", "rel": "self"}, {"href":
"http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}

DEBUG:keystoneclient.auth.identity.v2:Making authentication request to
http://192.168.0.5:5000/v2.0/tokens
DEBUG:requests.packages.urllib3.connectionpool:"POST /v2.0/tokens HTTP/1.1"
200 3423
DEBUG:keystoneclient.session:REQ: curl -g -i -X GET
http://192.168.0.5:9292/v2/images/03d98224-66ba-11e6-8b77-86f30ca893d3/file -H
"User-Agent: python-glanceclient" -H "Content-Type: application/octet-stream"
-H "X-Auth-Token: xxxxxxxxxxxxxxx"
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection
(1): 192.168.0.5
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/glanceclient/shell.py", line 668, in
main
    OpenStackImagesShell().main(argv)
  File "/usr/lib/python2.7/site-packages/glanceclient/shell.py", line 599, in
main
    args.func(client, args)
  File "/usr/lib/python2.7/site-packages/glanceclient/v2/shell.py", line 282,
in do_image_download
    body = gc.images.data(args.id)
  File "/usr/lib/python2.7/site-packages/glanceclient/v2/images.py", line 195,
in data
    resp, body = self.http_client.get(url)
  File "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 176,
in get
    return self.request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line
343, in request
    raise exc.CommunicationError(message=message)
CommunicationError: Error finding address for
http://192.168.0.5:9292/v2/images/03d98224-66ba-11e6-8b77-86f30ca893d3/file:
Unable to establish connection to
http://192.168.0.5:9292/v2/images/03d98224-66ba-11e6-8b77-86f30ca893d3/file
Error finding address for
http://192.168.0.5:9292/v2/images/03d98224-66ba-11e6-8b77-86f30ca893d3/file:
Unable to establish connection to
http://192.168.0.5:9292/v2/images/03d98224-66ba-11e6-8b77-86f30ca893d3/file

Right around the time glance client error out, second sub-process of
swift-proxy started to grow:
[root@overcloud-controller-0 swift]# ps awfux|grep swift-proxy
swift    13084  1.0  0.0 302536 35248 ?        Ss   03:36   0:01
/usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift    13160 35.3 20.0 13699376 13196652 ?   S    03:36   0:48  \_
/usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift    13167  3.9  2.0 1608432 1331140 ?     R    03:36   0:05  \_
/usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf

and here where it settled:
[root@overcloud-controller-0 swift]# ps awfux|grep swift-proxy
swift    13084  0.7  0.0 302536 31448 ?        Ss   03:36   0:04
/usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift    26553 58.8 52.5 34980264 34553208 ?   S    03:41   3:06  \_
/usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift      455  0.0  0.0 302836 31556 ?        S    03:43   0:00  \_
/usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf

Steps to Reproduce:
1.
2.
3.

Actual results:



Expected results:


Additional info: