Bug 1299855

Summary: Glance client error using IPv6 - ValueError: invalid literal for int() with base 10: 'fd00'
Product: Red Hat OpenStack Reporter: Attila Darazs <adarazs>
Component: openstack-tripleo-heat-templatesAssignee: Erno Kuvaja <ekuvaja>
Status: CLOSED NOTABUG QA Contact: Attila Darazs <adarazs>
Severity: urgent Docs Contact:
Priority: high    
Version: 7.0 (Kilo)CC: adarazs, cschwede, cyril, eglynn, fpercoco, jcoufal, jpena, jruzicka, jschluet, mburns, pgrist, rhel-osp-director-maint, scohen, srevivo
Target Milestone: ---Keywords: Automation
Target Release: 11.0 (Ocata)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1299953 (view as bug list) Environment:
Last Closed: 2017-02-09 22:48:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1299953, 1312983    

Description Attila Darazs 2016-01-19 12:20:35 UTC
Description of problem:

On an IPv6 deployment any glance command throws back an HTTP 500 error.

[stack@instack ~]$ cat overcloudrc 
export OS_NO_CACHE=True
export OS_CLOUDNAME=overcloud
export OS_AUTH_URL=http://[2001:db8:fd00:1000:f816:3eff:fe1d:2f22]:5000/v2.0
export NOVA_VERSION=1.1
export COMPUTE_API_VERSION=1.1
export OS_USERNAME=admin
export no_proxy=,2001:db8:fd00:1000:f816:3eff:fe1d:2f22
export OS_PASSWORD=V9eHtk3RJZusTm898mWYG7XTw
export PYTHONWARNINGS="ignore:Certificate has no, ignore:A true SSLContext object is not available"
export OS_TENANT_NAME=admin
unset OS_CACERT
[stack@instack ~]$ source overcloudrc
[stack@instack ~]$ date; glance image-list
Tue Jan 19 07:11:56 EST 2016
500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500)
[stack@instack ~]$ ssh heat-admin.2.7
Last login: Tue Jan 19 07:07:12 2016 from 192.0.2.1
[heat-admin@overcloud-controller-0 ~]$ sudo -i
[root@overcloud-controller-0 ~]# grep "07:11" /var/log/glance/api.log 
2016-01-19 07:11:56.515 17724 ERROR glance.registry.client.v1.client [req-c18946ba-b696-4169-b0ec-9f859f2201d8 329b970d22014155ac5c96387ef52918 56ad9efb11534a72be4f03fc28e1a46a - - -] Registry client request GET /images/detail raised ValueError
2016-01-19 07:11:56.515 17724 TRACE glance.registry.client.v1.client Traceback (most recent call last):
2016-01-19 07:11:56.515 17724 TRACE glance.registry.client.v1.client   File "/usr/lib/python2.7/site-packages/glance/registry/client/v1/client.py", line 117, in do_request
2016-01-19 07:11:56.515 17724 TRACE glance.registry.client.v1.client     **kwargs)
2016-01-19 07:11:56.515 17724 TRACE glance.registry.client.v1.client   File "/usr/lib/python2.7/site-packages/glance/common/client.py", line 71, in wrapped
2016-01-19 07:11:56.515 17724 TRACE glance.registry.client.v1.client     return func(self, *args, **kwargs)
2016-01-19 07:11:56.515 17724 TRACE glance.registry.client.v1.client   File "/usr/lib/python2.7/site-packages/glance/common/client.py", line 376, in do_request
2016-01-19 07:11:56.515 17724 TRACE glance.registry.client.v1.client     headers=copy.deepcopy(headers))
2016-01-19 07:11:56.515 17724 TRACE glance.registry.client.v1.client   File "/usr/lib/python2.7/site-packages/glance/common/client.py", line 88, in wrapped
2016-01-19 07:11:56.515 17724 TRACE glance.registry.client.v1.client     return func(self, method, url, body, headers)
2016-01-19 07:11:56.515 17724 TRACE glance.registry.client.v1.client   File "/usr/lib/python2.7/site-packages/glance/common/client.py", line 455, in _do_request
2016-01-19 07:11:56.515 17724 TRACE glance.registry.client.v1.client     c = connection_type(url.hostname, url.port, **self.connect_kwargs)
2016-01-19 07:11:56.515 17724 TRACE glance.registry.client.v1.client   File "/usr/lib64/python2.7/urlparse.py", line 112, in port
2016-01-19 07:11:56.515 17724 TRACE glance.registry.client.v1.client     port = int(port, 10)
2016-01-19 07:11:56.515 17724 TRACE glance.registry.client.v1.client ValueError: invalid literal for int() with base 10: 'fd00'
2016-01-19 07:11:56.515 17724 TRACE glance.registry.client.v1.client 
2016-01-19 07:11:56.515 17724 ERROR glance.common.wsgi [req-c18946ba-b696-4169-b0ec-9f859f2201d8 329b970d22014155ac5c96387ef52918 56ad9efb11534a72be4f03fc28e1a46a - - -] Caught error: invalid literal for int() with base 10: 'fd00'
2016-01-19 07:11:56.515 17724 TRACE glance.common.wsgi Traceback (most recent call last):
2016-01-19 07:11:56.515 17724 TRACE glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 881, in __call__
2016-01-19 07:11:56.515 17724 TRACE glance.common.wsgi     request, **action_args)
2016-01-19 07:11:56.515 17724 TRACE glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 909, in dispatch
2016-01-19 07:11:56.515 17724 TRACE glance.common.wsgi     return method(*args, **kwargs)
2016-01-19 07:11:56.515 17724 TRACE glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/api/v1/images.py", line 348, in detail
2016-01-19 07:11:56.515 17724 TRACE glance.common.wsgi     images = registry.get_images_detail(req.context, **params)
2016-01-19 07:11:56.515 17724 TRACE glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/registry/client/v1/api.py", line 152, in get_images_detail
2016-01-19 07:11:56.515 17724 TRACE glance.common.wsgi     return c.get_images_detailed(**kwargs)
2016-01-19 07:11:56.515 17724 TRACE glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/registry/client/v1/client.py", line 146, in get_images_detailed
2016-01-19 07:11:56.515 17724 TRACE glance.common.wsgi     res = self.do_request("GET", "/images/detail", params=params)
2016-01-19 07:11:56.515 17724 TRACE glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/registry/client/v1/client.py", line 132, in do_request
2016-01-19 07:11:56.515 17724 TRACE glance.common.wsgi     'exc_name': exc_name})
2016-01-19 07:11:56.515 17724 TRACE glance.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2016-01-19 07:11:56.515 17724 TRACE glance.common.wsgi     six.reraise(self.type_, self.value, self.tb)
2016-01-19 07:11:56.515 17724 TRACE glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/registry/client/v1/client.py", line 117, in do_request
2016-01-19 07:11:56.515 17724 TRACE glance.common.wsgi     **kwargs)
2016-01-19 07:11:56.515 17724 TRACE glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/common/client.py", line 71, in wrapped
2016-01-19 07:11:56.515 17724 TRACE glance.common.wsgi     return func(self, *args, **kwargs)
2016-01-19 07:11:56.515 17724 TRACE glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/common/client.py", line 376, in do_request
2016-01-19 07:11:56.515 17724 TRACE glance.common.wsgi     headers=copy.deepcopy(headers))
2016-01-19 07:11:56.515 17724 TRACE glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/common/client.py", line 88, in wrapped
2016-01-19 07:11:56.515 17724 TRACE glance.common.wsgi     return func(self, method, url, body, headers)
2016-01-19 07:11:56.515 17724 TRACE glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/common/client.py", line 455, in _do_request
2016-01-19 07:11:56.515 17724 TRACE glance.common.wsgi     c = connection_type(url.hostname, url.port, **self.connect_kwargs)
2016-01-19 07:11:56.515 17724 TRACE glance.common.wsgi   File "/usr/lib64/python2.7/urlparse.py", line 112, in port
2016-01-19 07:11:56.515 17724 TRACE glance.common.wsgi     port = int(port, 10)
2016-01-19 07:11:56.515 17724 TRACE glance.common.wsgi ValueError: invalid literal for int() with base 10: 'fd00'
2016-01-19 07:11:56.515 17724 TRACE glance.common.wsgi 


Version-Release number of selected component (if applicable):
ospd poodle: 7-director/2016-01-18.6
core poodle: 7.0/2016-01-18.2

[stack@instack ~]$ rpm -qa|grep glance
python-glance-2015.1.2-1.el7ost.noarch
python-glanceclient-0.17.3-2.el7ost.noarch
python-glance-store-0.4.0-3.el7ost.noarch
openstack-glance-2015.1.2-1.el7ost.noarch

(same on the overcloud)

How reproducible:
100%

Steps to Reproduce:
1. deploy ipv6 overcloud
2. source overcloudrc
3. glance image-list

Actual results:
HTTP 500

Expected results:
list images

Additional info:
Might be an old, non IPv6 compatible python glance client?

Comment 2 Attila Darazs 2016-01-19 14:07:47 UTC
Using the latest upstream master of python-glanceclient (https://github.com/openstack/python-glanceclient) both the image-list and image-create worked, so the problem is definitely in the old glanceclient.

Comment 3 Flavio Percoco 2016-01-19 15:46:22 UTC
The code failing is in `glance/common/client.py` which is not glanceclient. It's an old http client used to talk to the registry service. Therefore, the communication between glance-api and glance-registry doesn't support IPv6.

Cyril, I think this could be tackled with a lite spec upstream in the Mitaka timeframe. Could you please write one?

Comment 4 Javier Peña 2016-01-19 17:20:42 UTC
FWIW, I have a manually set up environment running on IPv6, and glanceclient works fine for me in this case. Could it be a configuration issue?

In terms of packages, the only difference is that I have python-glanceclient-0.17.3-1.el7ost.noarch vs python-glanceclient-0.17.3-2.el7ost.noarch in the report.

Comment 5 Flavio Percoco 2016-01-19 18:30:50 UTC
I took a look at the environment and it seems to be a config issue. The registry_host, unlike the bind_host has to be a valid, parsable, ipv6 as it's a string read by glance-api to talk to glance registry. The value in the environment was missing the `[]` which are required by ipv6. Changing it to the one below seemed to have fixed the issue:

registry_host=[fd00:fd00:fd00:2000:f816:3eff:fedb:a219]

Comment 6 Cyril Roelandt 2016-01-20 14:11:26 UTC
I can confirm Flavio's analysis (shared by Mike):

>>> pr = urlparse("http://2001:db8:fd00:1000:f816:3eff:fe1d:2f22:5000")
>>> pr.port
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/urlparse.py", line 113, in port
    port = int(port, 10)
ValueError: invalid literal for int() with base 10: 'db8'


>>> pr = urlparse("http://[2001:db8:fd00:1000:f816:3eff:fe1d:2f22]:5000")
>>> pr.port
5000

Also, RFC 2732 (http://www.ietf.org/rfc/rfc2732.txt) states that 'to use a literal IPv6 address in a URL, the literal address should be enclosed in "[" and "]" characters.'

Comment 7 Mike Burns 2016-04-07 21:03:37 UTC
This bug did not make the OSP 8.0 release.  It is being deferred to OSP 10.

Comment 9 Paul Grist 2016-10-14 01:36:13 UTC
This one is coming over late and moving to OSP11 as something we should fix soon. If this in progress or could be done sooner let's bring it back sooner, but not sure IPV6 is common enough to not leave this as something to work around for now.

Comment 10 Sean Cohen 2016-12-21 14:57:33 UTC
Attila,
Can you confirm that this is still reproduces on OSP10?
Sean

Comment 11 Paul Grist 2017-02-09 22:48:33 UTC
Based on #c5 and #c6, closing this out as not a bug, please re-open if that's not the case or the issue persists without the [].

Comment 12 Attila Darazs 2017-04-27 10:27:03 UTC
I missed this one... clearing the needinfo. Not an issue any more.