1344148 – RDO mitaka openstack-tempest build requires updated python-urllib3

RDO tickets are now tracked in Jira https://issues.redhat.com/projects/RDO/issues/

Bug 1344148 - RDO mitaka openstack-tempest build requires updated python-urllib3

Summary: RDO mitaka openstack-tempest build requires updated python-urllib3

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	None
Product:	RDO
Classification:	Community
Component:	distribution
Sub Component:
Version:	Mitaka
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	---
Target Release:	Mitaka
Assignee:	Alan Pevec
QA Contact:	Shai Revivo
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1345736
TreeView+	depends on / blocked

Reported:	2016-06-08 21:43 UTC by Steve Linabery
Modified:	2017-06-18 07:36 UTC (History)
CC List:	7 users (show)
Fixed In Version:
Clone Of:
Environment:
Last Closed:	2017-06-18 07:36:20 UTC
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Description Steve Linabery 2016-06-08 21:43:26 UTC

Description of problem:

1) installed latest RDO mitaka build of openstack-tempest [0]
2) applied patch to openstack-zaqar source to allow config_tempest.py script to work [1]
3) invoked config_tempest.py and got the following stacktrace:

2016-06-08 17:37:46.437 15614 CRITICAL tempest [-] TypeError: __init__() got an unexpected keyword argument 'chunked'
2016-06-08 17:37:46.437 15614 ERROR tempest Traceback (most recent call last):
2016-06-08 17:37:46.437 15614 ERROR tempest   File "./tools/config_tempest.py", line 847, in <module>
2016-06-08 17:37:46.437 15614 ERROR tempest     main()
2016-06-08 17:37:46.437 15614 ERROR tempest   File "./tools/config_tempest.py", line 155, in main
2016-06-08 17:37:46.437 15614 ERROR tempest     clients = ClientManager(conf, not args.non_admin)
2016-06-08 17:37:46.437 15614 ERROR tempest   File "./tools/config_tempest.py", line 417, in __init__
2016-06-08 17:37:46.437 15614 ERROR tempest     tenant_name)['id']
2016-06-08 17:37:46.437 15614 ERROR tempest   File "/home/stack/tempest/tempest/common/identity.py", line 20, in get_tenant_by_name
2016-06-08 17:37:46.437 15614 ERROR tempest     tenants = client.list_tenants()['tenants']
2016-06-08 17:37:46.437 15614 ERROR tempest   File "/home/stack/tempest/tempest/services/identity/v2/json/tenants_client.py", line 56, in list_tenants
2016-06-08 17:37:46.437 15614 ERROR tempest     resp, body = self.get('tenants')
2016-06-08 17:37:46.437 15614 ERROR tempest   File "/home/stack/tempest/tempest/lib/common/rest_client.py", line 281, in get
2016-06-08 17:37:46.437 15614 ERROR tempest     return self.request('GET', url, extra_headers, headers)
2016-06-08 17:37:46.437 15614 ERROR tempest   File "/home/stack/tempest/tempest/lib/common/rest_client.py", line 644, in request
2016-06-08 17:37:46.437 15614 ERROR tempest     body=body, chunked=chunked)
2016-06-08 17:37:46.437 15614 ERROR tempest   File "/home/stack/tempest/tempest/lib/common/rest_client.py", line 534, in _request
2016-06-08 17:37:46.437 15614 ERROR tempest     method, url, headers, body, self.filters)
2016-06-08 17:37:46.437 15614 ERROR tempest   File "/home/stack/tempest/tempest/lib/auth.py", line 187, in auth_request
2016-06-08 17:37:46.437 15614 ERROR tempest     filters, method, url, headers, body)
2016-06-08 17:37:46.437 15614 ERROR tempest   File "/home/stack/tempest/tempest/lib/auth.py", line 272, in _decorate_request
2016-06-08 17:37:46.437 15614 ERROR tempest     auth_data = self.get_auth()
2016-06-08 17:37:46.437 15614 ERROR tempest   File "/home/stack/tempest/tempest/lib/auth.py", line 149, in get_auth
2016-06-08 17:37:46.437 15614 ERROR tempest     self.set_auth()
2016-06-08 17:37:46.437 15614 ERROR tempest   File "/home/stack/tempest/tempest/lib/auth.py", line 158, in set_auth
2016-06-08 17:37:46.437 15614 ERROR tempest     self.cache = self._get_auth()
2016-06-08 17:37:46.437 15614 ERROR tempest   File "/home/stack/tempest/tempest/lib/auth.py", line 309, in _get_auth
2016-06-08 17:37:46.437 15614 ERROR tempest     token, auth_data = auth_func(**auth_params)
2016-06-08 17:37:46.437 15614 ERROR tempest   File "/home/stack/tempest/tempest/lib/services/identity/v2/token_client.py", line 108, in get_token
2016-06-08 17:37:46.437 15614 ERROR tempest     body = self.auth(user, password, tenant)
2016-06-08 17:37:46.437 15614 ERROR tempest   File "/home/stack/tempest/tempest/lib/services/identity/v2/token_client.py", line 54, in auth
2016-06-08 17:37:46.437 15614 ERROR tempest     resp, body = self.post(self.auth_url, body=body)
2016-06-08 17:37:46.437 15614 ERROR tempest   File "/home/stack/tempest/tempest/lib/common/rest_client.py", line 266, in post
2016-06-08 17:37:46.437 15614 ERROR tempest     return self.request('POST', url, extra_headers, headers, body, chunked)
2016-06-08 17:37:46.437 15614 ERROR tempest   File "/home/stack/tempest/tempest/lib/services/identity/v2/token_client.py", line 93, in request
2016-06-08 17:37:46.437 15614 ERROR tempest     headers=headers, body=body)
2016-06-08 17:37:46.437 15614 ERROR tempest   File "/home/stack/tempest/tempest/lib/common/rest_client.py", line 574, in raw_request
2016-06-08 17:37:46.437 15614 ERROR tempest     body=body, chunked=chunked)
2016-06-08 17:37:46.437 15614 ERROR tempest   File "/home/stack/tempest/tempest/lib/common/http.py", line 54, in request
2016-06-08 17:37:46.437 15614 ERROR tempest     *args, **new_kwargs)
2016-06-08 17:37:46.437 15614 ERROR tempest   File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 73, in request
2016-06-08 17:37:46.437 15614 ERROR tempest     **urlopen_kw)
2016-06-08 17:37:46.437 15614 ERROR tempest   File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 151, in request_encode_body
2016-06-08 17:37:46.437 15614 ERROR tempest     return self.urlopen(method, url, **extra_kw)
2016-06-08 17:37:46.437 15614 ERROR tempest   File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 162, in urlopen
2016-06-08 17:37:46.437 15614 ERROR tempest     response = conn.urlopen(method, u.request_uri, **kw)
2016-06-08 17:37:46.437 15614 ERROR tempest   File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 577, in urlopen
2016-06-08 17:37:46.437 15614 ERROR tempest     **response_kw)
2016-06-08 17:37:46.437 15614 ERROR tempest   File "/usr/lib/python2.7/site-packages/urllib3/response.py", line 375, in from_httplib
2016-06-08 17:37:46.437 15614 ERROR tempest     **response_kw)
2016-06-08 17:37:46.437 15614 ERROR tempest TypeError: __init__() got an unexpected keyword argument 'chunked'
2016-06-08 17:37:46.437 15614 ERROR tempest 



4) pip install --upgrade urllib3==1.15.1 (version in redhat-openstack/tempest master branch requirements.txt) removes the stacktrace

[0] https://trunk.rdoproject.org/centos7-mitaka/current/openstack-tempest-12.0.1-0.20160608094305.ebe375c.el7.centos.noarch.rpm
[1] https://review.openstack.org/#/c/327268/

Comment 1 Daniel Mellado 2016-06-09 07:08:05 UTC

Verified. Would it be possible to upgrade that requirement? I don't see that many other options on this...

Comment 2 Daniel Mellado 2016-06-09 08:48:57 UTC

After some investigation, this needs for the upgrade comes from having chuned transfer encoding enabled in tempest's rest-clinet fom urllib3 1.15

From urllib3 release notes:
===========================
Chunked transfer encoding when requesting with chunked=True. (Issue #790)

So we've two options here:

1) Upgrade urllib3 to 1.15 => That'd be the most straightforward and preferred one

2) Add a patch to our downstream tempest rest-client (tempest/lib/common/rest_client.py) removing all the chunked options from GET.POST,PUT and so on methods. I'd consider this as an 'emergency' option if the former one is absolutely not possible, as it could (most probably would) create issues later on

Comment 3 Daniel Mellado 2016-06-09 08:51:18 UTC

Also, this 2) approach would make us have to revert additional patches, such as:

commit 9a573d914836e8b77f07571a4ceddf776cb8b0d1
Author: Jordan Pittier <jordan.pittier>
Date:   Fri Apr 29 17:04:39 2016 +0200

    Remove the Glance HTTP client. Use the common Rest Client instead.
    
    No need for this anymore, now that we have urllib3 which brings
    support for chunked transfer encoding, the whole point of
    tempest/common/glance_http.py is gone.
    
    Change-Id: Idd18ab2cdf09024f83795eaa904ec3f2815f1fc8


So the downstream tempest would go far and far away from upstream.

@apevec, would it be possible to have a bump in reqs and install urllib3==1.15?

Thanks!

Comment 4 Alan Pevec 2016-06-09 13:39:53 UTC

requests+urllib3 must be upgraded together, there are updates scheduled for Newton
http://cbs.centos.org/repos/cloud7-openstack-newton-testing/x86_64/os/Packages/

Mitaka and older have not been tested aginst those yet, so first we need to verify they work with the rest of openstack services and clients.

Comment 5 Alan Pevec 2016-06-09 13:51:09 UTC

Tested by upstream CI:

* Mitaka[1] u-c:  requests===2.9.1 urllib3===1.14 
* Liberty[2] u-c: requests===2.8.1 urllib3===1.12

[1] https://github.com/openstack/requirements/blob/stable/mitaka/upper-constraints.txt

[2] https://github.com/openstack/requirements/blob/stable/liberty/upper-constraints.txt

Comment 6 Alan Pevec 2016-06-09 13:55:38 UTC

Liberty global requirements:
  requests>=2.5.2,!=2.8.0,!=2.9.0
  urllib3>=1.8.3

Mitaka g-r:
  requests>=2.8.1,!=2.9.0
  urllib3>=1.8.3

Comment 7 Alan Pevec 2016-06-09 14:08:20 UTC

We also need python-pysocks, new urllib3 dependency.
Total proposed update is following three builds, tagged into cloud7-openstack-common-pending for CI verification:
  python-pysocks-1.5.6-3.el7
  python-requests-2.10.0-1.el7
  python-urllib3-1.15.1-1.el7

These can be tested by pointing CI jobs to use https://apevec.fedorapeople.org/openstack/testing/rdo-mitaka-pending.repo

Comment 8 Alan Pevec 2016-06-17 19:34:24 UTC

>   python-pysocks-1.5.6-3.el7
>   python-requests-2.10.0-1.el7
>   python-urllib3-1.15.1-1.el7

These are now tagged openstack-common-testing and available in RDO Mitaka testing repository.

Note You need to log in before you can comment on or make changes to this bug.