Bug 1300431

Summary: Unable to upload glance image on IPv6 and SSL enabled overcloud
Product: Red Hat OpenStack Reporter: Marius Cornea <mcornea>
Component: rhosp-directorAssignee: Emilien Macchi <emacchi>
Status: CLOSED NOTABUG QA Contact: yeylon <yeylon>
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.0 (Kilo)CC: emacchi, mburns, rhel-osp-director-maint, srevivo
Target Milestone: y3   
Target Release: 7.0 (Kilo)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-01-21 11:33:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Marius Cornea 2016-01-20 19:21:21 UTC
Description of problem:
Uploading a Glance image in a 1 controller + 1 compute IPv6 and SSL enabled deployment fails with:
400 Bad Request: Client disconnected before sending all data to backend (HTTP 400) 

Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-0.8.6-108.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy a 1 controller + 1 compute IPv6 and SSL overcloud 
2. source overcloudrc
3.  glance image-create --name Fedora22 --file Fedora-Cloud-Base-22-20150521.x86_64.qcow2  --disk-format qcow2 --container-format bare --is-public true --progress

Actual results:
400 Bad Request: Client disconnected before sending all data to backend (HTTP 400)

Expected results:
Image gets uploaded.

Comment 1 Marius Cornea 2016-01-20 19:36:04 UTC
I can't find much useful information on this. I enabled debug for glance-api and glance-registry. Here are the logs:

==> /var/log/glance/registry.log <==
2016-01-20 14:34:44.201 25784 INFO glance.registry.api.v1.images [req-d41ad37b-be43-42f7-a7bc-d7a504639b61 d85d403387ae499ebedef88c8943e682 6047f9e7eeb142db80da1665cf10a8b6 - - -] Successfully created image b89ba072-f0e8-4f8f-b828-0813c46da5f4
2016-01-20 14:34:44.202 25784 INFO eventlet.wsgi.server [req-d41ad37b-be43-42f7-a7bc-d7a504639b61 d85d403387ae499ebedef88c8943e682 6047f9e7eeb142db80da1665cf10a8b6 - - -] fd00:fd00:fd00:2000:f816:3eff:fed3:33a2 - - [20/Jan/2016 14:34:44] "POST /images HTTP/1.1" 200 694 0.294029
2016-01-20 14:34:44.469 25783 INFO eventlet.wsgi.server [req-420d74f2-a464-4d72-97f9-1955de6af221 d85d403387ae499ebedef88c8943e682 6047f9e7eeb142db80da1665cf10a8b6 - - -] fd00:fd00:fd00:2000:f816:3eff:fed3:33a2 - - [20/Jan/2016 14:34:44] "GET /images/b89ba072-f0e8-4f8f-b828-0813c46da5f4 HTTP/1.1" 200 694 0.263663
2016-01-20 14:34:44.554 25790 INFO glance.registry.api.v1.images [req-2b267143-6573-4fad-b84c-8cdc39ebaec3 d85d403387ae499ebedef88c8943e682 6047f9e7eeb142db80da1665cf10a8b6 - - -] Updating metadata for image b89ba072-f0e8-4f8f-b828-0813c46da5f4
2016-01-20 14:34:44.555 25790 INFO eventlet.wsgi.server [req-2b267143-6573-4fad-b84c-8cdc39ebaec3 d85d403387ae499ebedef88c8943e682 6047f9e7eeb142db80da1665cf10a8b6 - - -] fd00:fd00:fd00:2000:f816:3eff:fed3:33a2 - - [20/Jan/2016 14:34:44] "PUT /images/b89ba072-f0e8-4f8f-b828-0813c46da5f4 HTTP/1.1" 200 694 0.082013

==> /var/log/glance/api.log <==
2016-01-20 14:34:44.658 26936 WARNING glance.api.v1.upload_utils [req-0d6a6378-d73c-4c20-b124-d30cf944ca5c d85d403387ae499ebedef88c8943e682 6047f9e7eeb142db80da1665cf10a8b6 - - -] Client disconnected before sending all data to backend

==> /var/log/glance/registry.log <==
2016-01-20 14:34:44.947 25789 INFO glance.registry.api.v1.images [req-983d56c6-7831-4fbb-a6b6-5fed449d49dd d85d403387ae499ebedef88c8943e682 6047f9e7eeb142db80da1665cf10a8b6 - - -] Updating metadata for image b89ba072-f0e8-4f8f-b828-0813c46da5f4
2016-01-20 14:34:44.948 25789 INFO eventlet.wsgi.server [req-983d56c6-7831-4fbb-a6b6-5fed449d49dd d85d403387ae499ebedef88c8943e682 6047f9e7eeb142db80da1665cf10a8b6 - - -] fd00:fd00:fd00:2000:f816:3eff:fed3:33a2 - - [20/Jan/2016 14:34:44] "PUT /images/b89ba072-f0e8-4f8f-b828-0813c46da5f4 HTTP/1.1" 200 694 0.287550

==> /var/log/glance/api.log <==
2016-01-20 14:34:49.006 26936 INFO eventlet.wsgi.server [req-0d6a6378-d73c-4c20-b124-d30cf944ca5c d85d403387ae499ebedef88c8943e682 6047f9e7eeb142db80da1665cf10a8b6 - - -] fd00:fd00:fd00:3000:f816:3eff:fe05:c5bd - - [20/Jan/2016 14:34:49] "POST /v1/images HTTP/1.1" 400 396 5.174109


stack@instack:~>>> glance --debug image-create --name Fedora22 --file Fedora-Cloud-Base-22-20150521.x86_64.qcow2  --disk-format qcow2 --container-format bare --is-public true --progress 
curl -g -i -X POST -H 'Accept-Encoding: gzip, deflate' -H 'x-image-meta-container_format: bare' -H 'Accept: */*' -H 'X-Auth-Token: {SHA1}21e1730f948f9c8447be5c32585b8d83b466e011' -H 'x-image-meta-size: 228599296' -H 'Connection: keep-alive' -H 'x-image-meta-is_public: True' -H 'User-Agent: python-glanceclient' -H 'Content-Type: application/octet-stream' -H 'x-image-meta-disk_format: qcow2' -H 'x-image-meta-name: Fedora22'  --cert None --key None https://myovercloud.net:13292/v1/images
[=============================>] 100%
Request returned failure status 400.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/glanceclient/shell.py", line 652, in main
    args.func(client, args)
  File "/usr/lib/python2.7/site-packages/glanceclient/v1/shell.py", line 245, in do_image_create
    image = gc.images.create(**fields)
  File "/usr/lib/python2.7/site-packages/glanceclient/v1/images.py", line 319, in create
    data=image_data)
  File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 265, in post
    return self._request('POST', url, **kwargs)
  File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 230, in _request
    raise exc.from_response(resp, resp.text)
HTTPBadRequest: 400 Bad Request: Client disconnected before sending all data to backend (HTTP 400)
400 Bad Request: Client disconnected before sending all data to backend (HTTP 400)

Comment 2 Emilien Macchi 2016-01-20 20:45:35 UTC
So I debugged and all endpoints were missing brackets when running ipv6.
That's something we did in os-cloud-config 0.2.8-13.
The machine where you ran the issue was using 0.2.8-12.

So endpoints did not have brackets, Glance client using swiftclient in backend to contact Swift endpoint could not establish connection so image failed to be uploaded.

I'm testing again now with 0.2.8-13 and I'll close the bug if it works.

Comment 3 Marius Cornea 2016-01-21 11:33:23 UTC
(In reply to Emilien Macchi from comment #2)
> So I debugged and all endpoints were missing brackets when running ipv6.
> That's something we did in os-cloud-config 0.2.8-13.
> The machine where you ran the issue was using 0.2.8-12.
> 
> So endpoints did not have brackets, Glance client using swiftclient in
> backend to contact Swift endpoint could not establish connection so image
> failed to be uploaded.
> 
> I'm testing again now with 0.2.8-13 and I'll close the bug if it works.

This worked with os-cloud-config-0.2.8-13.el7ost.noarch. Closing this as not a bug. Thanks!