Bug 1253823

E.g. issue that comes from using requests 2.7.0 with urllib3 1.11:  https://bugs.launchpad.net/glance/+bug/1476770

The newest version of urllib in Fedora is 1.10.4 (because of requests dependency) can you explain please how can yum install v1.11?

created from source, there being a fix in 1.11 that I wanted. I don't think that should matter, though. Whether there is a v1.11 rpm in Fedora today or not, there will be in future.

I think the real issue is related to the fact, when you source install the requests it will bundle the urllib3, and when you use the packaged version it will not be bundled.

I do not know is this packaging practice discussed with upstream or not, but looks like other distros does similar kind of nuking.
Ubuntu/Debian does the same kind of modification.
gentoo just removes the bigger chardet library (In our case it wouldn't cause an issue) , OpenSuse bundles the libraries as it is in the upstream git repo.

The rpm spec file does not contains references to upstream issues or bugs related to the bundled package nukes. According to the Fedora packaging guideline this kind of things should have an upstream bug or pull request noted in the spec file.

Please discuss the packaging practices with the requests maintainers,
either have the upstream repo to unbundle the extra library and properly mark the dependencies, or bundle them into the rpm package as well.

(In reply to Attila Fazekas from comment #4)
> I do not know is this packaging practice discussed with upstream or not, but
> looks like other distros does similar kind of nuking.

Yes, we've been in conversation with them about it for quite some time.  They're unfriendly to unbundling themselves, but we're "at peace" over the unbundling that the Fedora and Debian maintainers do.

> The rpm spec file does not contains references to upstream issues or bugs
> related to the bundled package nukes. According to the Fedora packaging
> guideline this kind of things should have an upstream bug or pull request
> noted in the spec file.

I just added a few links to the spec:  http://pkgs.fedoraproject.org/cgit/python-requests.git/commit/?id=c802119e775e99b17a0a286a70fc4488331a676c  Thanks for the tip.

Ralph,

The upstream python-request package goes through great lengths to bundle a known tested and working version of urllib3:

[hamzy@hamzy-tp-w540 tmp]$ grep -r import /lib/python2.7/site-packages/requests/ | grep urllib3 | sed -e '/^Binary file/d'
/lib/python2.7/site-packages/requests/adapters.py:from .packages.urllib3.poolmanager import PoolManager, proxy_from_url
/lib/python2.7/site-packages/requests/adapters.py:from .packages.urllib3.response import HTTPResponse
/lib/python2.7/site-packages/requests/adapters.py:from .packages.urllib3.util import Timeout as TimeoutSauce
/lib/python2.7/site-packages/requests/adapters.py:from .packages.urllib3.util.retry import Retry
/lib/python2.7/site-packages/requests/adapters.py:from .packages.urllib3.exceptions import ConnectTimeoutError
/lib/python2.7/site-packages/requests/adapters.py:from .packages.urllib3.exceptions import HTTPError as _HTTPError
/lib/python2.7/site-packages/requests/adapters.py:from .packages.urllib3.exceptions import MaxRetryError
/lib/python2.7/site-packages/requests/adapters.py:from .packages.urllib3.exceptions import ProxyError as _ProxyError
/lib/python2.7/site-packages/requests/adapters.py:from .packages.urllib3.exceptions import ProtocolError
/lib/python2.7/site-packages/requests/adapters.py:from .packages.urllib3.exceptions import ReadTimeoutError
/lib/python2.7/site-packages/requests/adapters.py:from .packages.urllib3.exceptions import SSLError as _SSLError
/lib/python2.7/site-packages/requests/adapters.py:from .packages.urllib3.exceptions import ResponseError
/lib/python2.7/site-packages/requests/adapters.py:        which we retry a request, import urllib3's ``Retry`` class and pass
/lib/python2.7/site-packages/requests/packages/__init__.py:from . import urllib3
/lib/python2.7/site-packages/requests/models.py:from .packages.urllib3.fields import RequestField
/lib/python2.7/site-packages/requests/models.py:from .packages.urllib3.filepost import encode_multipart_formdata
/lib/python2.7/site-packages/requests/models.py:from .packages.urllib3.util import parse_url
/lib/python2.7/site-packages/requests/models.py:from .packages.urllib3.exceptions import (
/lib/python2.7/site-packages/requests/sessions.py:from .packages.urllib3._collections import RecentlyUsedContainer
/lib/python2.7/site-packages/requests/__init__.py:    from .packages.urllib3.contrib import pyopenssl
/lib/python2.7/site-packages/requests/exceptions.py:from .packages.urllib3.exceptions import HTTPError as BaseHTTPError


But this is then removed by Fedora:

[hamzy@hamzy-tp-w540 tmp]$ ls -ld /usr/lib/python2.7/site-packages/requests/packages/*
lrwxrwxrwx. 1 root root  13 May 28 12:50 /usr/lib/python2.7/site-packages/requests/packages/chardet -> ../../chardet
-rw-r--r--. 1 root root  62 Apr 23 01:22 /usr/lib/python2.7/site-packages/requests/packages/__init__.py
-rw-r--r--. 2 root root 265 May 28 12:50 /usr/lib/python2.7/site-packages/requests/packages/__init__.pyc
-rw-r--r--. 2 root root 265 May 28 12:50 /usr/lib/python2.7/site-packages/requests/packages/__init__.pyo
lrwxrwxrwx. 1 root root  13 May 28 12:50 /usr/lib/python2.7/site-packages/requests/packages/urllib3 -> ../../urllib3

Which breaks Open Stack for example:

So, could you please explain in some more detail about being "at peace" with purposely unbundling?

Fedora should either:
1) stop unbundling urllib3 from requests. This is the safest approach, since requests may have customized urllib3 to their needs...
2) continue to unbundle urllib3, but have the rpm spec require the specific version of urllib3 that would otherwise have been bundled with this version of requests.

Either of those would fix this issue. I was expecting #2, because I assume (perhaps incorrectly), that #1 has already been discussed and discarded elsewhere. But again... EITHER of these would solve this issue.

(In reply to Mark Hamzy from comment #6)
> So, could you please explain in some more detail about being "at peace" with
> purposely unbundling?

Sure thing.  For starters, bundled libraries are against the Fedora packaging policy in general:  https://fedoraproject.org/wiki/Packaging:No_Bundled_Libraries

In short, we need to unbundle said libraries in order to ensure that we can fix security vulnerabilities across the distribution in a sane way.  There are other reasons listed in the above link.

By "at peace" I meant to suggest contrast with our previous mode of engagement which felt hostile for a time.  They bundled a library, we provided a patch to unbundle it, they rejected it, we didn't like that, they didn't like that we didn't like that, etc.

These days we have good, regular dialogue with upstream.  They understand our need to provide a system copy of python-requests to other system programs that need it (and the requirement that we place on the packaging (unbundling) of those system libraries), and we in turn understand their need to ship a package that they have full control over -- a "product" as Kenneth has called it.

When you say that this "breaks Open Stack"... that warrants its own ticket if there are details beyond what's being reported here.

(In reply to Matthew Edmonds from comment #7)
> Fedora should either:
> 1) stop unbundling urllib3 from requests. This is the safest approach, since
> requests may have customized urllib3 to their needs...
> 2) continue to unbundle urllib3, but have the rpm spec require the specific
> version of urllib3 that would otherwise have been bundled with this version
> of requests.
> 
> Either of those would fix this issue. I was expecting #2, because I assume
> (perhaps incorrectly), that #1 has already been discussed and discarded
> elsewhere. But again... EITHER of these would solve this issue.

Yes.  #2 is good.  Let's do that.

Applied here:  http://pkgs.fedoraproject.org/cgit/python-requests.git/commit/?id=026230bb9c69e7b76f8fb8eb5618b518a5d8dfe4

Thanks for the report.