Bug 1253823 - python-requests doesn't require appropriate urllib3
python-requests doesn't require appropriate urllib3
Status: CLOSED RAWHIDE
Product: Fedora
Classification: Fedora
Component: python-requests (Show other bugs)
rawhide
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Ralph Bean
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-08-14 15:40 EDT by Matthew Edmonds
Modified: 2015-09-04 22:27 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-09-04 22:27:51 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Matthew Edmonds 2015-08-14 15:40:08 EDT
Description of problem:
python-requests 2.7.0 spec requires python-urllib3 >= 1.10. This is not restrictive enough, as python-requests 2.7.0 is coded specific to 1.10.4 (see https://github.com/kennethreitz/requests/commit/ee7389da98092f6e63f68a490e7f71651c9ec047). This is leading to issues where python-requests 2.7.0 is installed along with python-urllib3 1.11.

Version-Release number of selected component (if applicable):
2.7.0

How reproducible:
100%

Steps to Reproduce:
1. Check the version of urllib3 that requests 2.7.0 includes when installed via pip (answer is 1.10.4)
2. Try to use a different version of urllib3, e.g. 1.11

Actual results:
yum allows the installation of python-urllib3 1.11 with python-requests 2.7.0

Expected results:
yum should not allow these two incompatible versions together.
Comment 1 Matthew Edmonds 2015-08-14 15:44:28 EDT
E.g. issue that comes from using requests 2.7.0 with urllib3 1.11:  https://bugs.launchpad.net/glance/+bug/1476770
Comment 2 Robert Kuska 2015-08-19 02:17:07 EDT
The newest version of urllib in Fedora is 1.10.4 (because of requests dependency) can you explain please how can yum install v1.11?
Comment 3 Matthew Edmonds 2015-08-19 09:15:11 EDT
created from source, there being a fix in 1.11 that I wanted. I don't think that should matter, though. Whether there is a v1.11 rpm in Fedora today or not, there will be in future.
Comment 4 Attila Fazekas 2015-09-01 09:00:26 EDT
I think the real issue is related to the fact, when you source install the requests it will bundle the urllib3, and when you use the packaged version it will not be bundled.

I do not know is this packaging practice discussed with upstream or not, but looks like other distros does similar kind of nuking.
Ubuntu/Debian does the same kind of modification.
gentoo just removes the bigger chardet library (In our case it wouldn't cause an issue) , OpenSuse bundles the libraries as it is in the upstream git repo.

The rpm spec file does not contains references to upstream issues or bugs related to the bundled package nukes. According to the Fedora packaging guideline this kind of things should have an upstream bug or pull request noted in the spec file.

Please discuss the packaging practices with the requests maintainers,
either have the upstream repo to unbundle the extra library and properly mark the dependencies, or bundle them into the rpm package as well.
Comment 5 Ralph Bean 2015-09-01 10:16:49 EDT
(In reply to Attila Fazekas from comment #4)
> I do not know is this packaging practice discussed with upstream or not, but
> looks like other distros does similar kind of nuking.

Yes, we've been in conversation with them about it for quite some time.  They're unfriendly to unbundling themselves, but we're "at peace" over the unbundling that the Fedora and Debian maintainers do.

> The rpm spec file does not contains references to upstream issues or bugs
> related to the bundled package nukes. According to the Fedora packaging
> guideline this kind of things should have an upstream bug or pull request
> noted in the spec file.

I just added a few links to the spec:  http://pkgs.fedoraproject.org/cgit/python-requests.git/commit/?id=c802119e775e99b17a0a286a70fc4488331a676c  Thanks for the tip.
Comment 6 Mark Hamzy 2015-09-03 16:42:36 EDT
Ralph,

The upstream python-request package goes through great lengths to bundle a known tested and working version of urllib3:

[hamzy@hamzy-tp-w540 tmp]$ grep -r import /lib/python2.7/site-packages/requests/ | grep urllib3 | sed -e '/^Binary file/d'
/lib/python2.7/site-packages/requests/adapters.py:from .packages.urllib3.poolmanager import PoolManager, proxy_from_url
/lib/python2.7/site-packages/requests/adapters.py:from .packages.urllib3.response import HTTPResponse
/lib/python2.7/site-packages/requests/adapters.py:from .packages.urllib3.util import Timeout as TimeoutSauce
/lib/python2.7/site-packages/requests/adapters.py:from .packages.urllib3.util.retry import Retry
/lib/python2.7/site-packages/requests/adapters.py:from .packages.urllib3.exceptions import ConnectTimeoutError
/lib/python2.7/site-packages/requests/adapters.py:from .packages.urllib3.exceptions import HTTPError as _HTTPError
/lib/python2.7/site-packages/requests/adapters.py:from .packages.urllib3.exceptions import MaxRetryError
/lib/python2.7/site-packages/requests/adapters.py:from .packages.urllib3.exceptions import ProxyError as _ProxyError
/lib/python2.7/site-packages/requests/adapters.py:from .packages.urllib3.exceptions import ProtocolError
/lib/python2.7/site-packages/requests/adapters.py:from .packages.urllib3.exceptions import ReadTimeoutError
/lib/python2.7/site-packages/requests/adapters.py:from .packages.urllib3.exceptions import SSLError as _SSLError
/lib/python2.7/site-packages/requests/adapters.py:from .packages.urllib3.exceptions import ResponseError
/lib/python2.7/site-packages/requests/adapters.py:        which we retry a request, import urllib3's ``Retry`` class and pass
/lib/python2.7/site-packages/requests/packages/__init__.py:from . import urllib3
/lib/python2.7/site-packages/requests/models.py:from .packages.urllib3.fields import RequestField
/lib/python2.7/site-packages/requests/models.py:from .packages.urllib3.filepost import encode_multipart_formdata
/lib/python2.7/site-packages/requests/models.py:from .packages.urllib3.util import parse_url
/lib/python2.7/site-packages/requests/models.py:from .packages.urllib3.exceptions import (
/lib/python2.7/site-packages/requests/sessions.py:from .packages.urllib3._collections import RecentlyUsedContainer
/lib/python2.7/site-packages/requests/__init__.py:    from .packages.urllib3.contrib import pyopenssl
/lib/python2.7/site-packages/requests/exceptions.py:from .packages.urllib3.exceptions import HTTPError as BaseHTTPError


But this is then removed by Fedora:

[hamzy@hamzy-tp-w540 tmp]$ ls -ld /usr/lib/python2.7/site-packages/requests/packages/*
lrwxrwxrwx. 1 root root  13 May 28 12:50 /usr/lib/python2.7/site-packages/requests/packages/chardet -> ../../chardet
-rw-r--r--. 1 root root  62 Apr 23 01:22 /usr/lib/python2.7/site-packages/requests/packages/__init__.py
-rw-r--r--. 2 root root 265 May 28 12:50 /usr/lib/python2.7/site-packages/requests/packages/__init__.pyc
-rw-r--r--. 2 root root 265 May 28 12:50 /usr/lib/python2.7/site-packages/requests/packages/__init__.pyo
lrwxrwxrwx. 1 root root  13 May 28 12:50 /usr/lib/python2.7/site-packages/requests/packages/urllib3 -> ../../urllib3

Which breaks Open Stack for example:

So, could you please explain in some more detail about being "at peace" with purposely unbundling?
Comment 7 Matthew Edmonds 2015-09-03 16:52:58 EDT
Fedora should either:
1) stop unbundling urllib3 from requests. This is the safest approach, since requests may have customized urllib3 to their needs...
2) continue to unbundle urllib3, but have the rpm spec require the specific version of urllib3 that would otherwise have been bundled with this version of requests.

Either of those would fix this issue. I was expecting #2, because I assume (perhaps incorrectly), that #1 has already been discussed and discarded elsewhere. But again... EITHER of these would solve this issue.
Comment 8 Ralph Bean 2015-09-04 22:19:50 EDT
(In reply to Mark Hamzy from comment #6)
> So, could you please explain in some more detail about being "at peace" with
> purposely unbundling?

Sure thing.  For starters, bundled libraries are against the Fedora packaging policy in general:  https://fedoraproject.org/wiki/Packaging:No_Bundled_Libraries

In short, we need to unbundle said libraries in order to ensure that we can fix security vulnerabilities across the distribution in a sane way.  There are other reasons listed in the above link.

By "at peace" I meant to suggest contrast with our previous mode of engagement which felt hostile for a time.  They bundled a library, we provided a patch to unbundle it, they rejected it, we didn't like that, they didn't like that we didn't like that, etc.

These days we have good, regular dialogue with upstream.  They understand our need to provide a system copy of python-requests to other system programs that need it (and the requirement that we place on the packaging (unbundling) of those system libraries), and we in turn understand their need to ship a package that they have full control over -- a "product" as Kenneth has called it.

When you say that this "breaks Open Stack"... that warrants its own ticket if there are details beyond what's being reported here.

(In reply to Matthew Edmonds from comment #7)
> Fedora should either:
> 1) stop unbundling urllib3 from requests. This is the safest approach, since
> requests may have customized urllib3 to their needs...
> 2) continue to unbundle urllib3, but have the rpm spec require the specific
> version of urllib3 that would otherwise have been bundled with this version
> of requests.
> 
> Either of those would fix this issue. I was expecting #2, because I assume
> (perhaps incorrectly), that #1 has already been discussed and discarded
> elsewhere. But again... EITHER of these would solve this issue.

Yes.  #2 is good.  Let's do that.

Note You need to log in before you can comment on or make changes to this bug.