Description of problem: Looks like someone prematurely removed the python36 runtime RPM from the repositories Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. Launch a new, CentOS 7 system (7.6 is currently latest version available) 2. Install/activate EPEL repositories 3. Attempt to install python36 Actual results: Runtime RPM not found Expected results: Installation of python36 RPM succeeds Additional info: While the python36 runtime RPM is obsoleted in RHEL 7.7, CentOS 7.7 has yet to drop. Inability to install python36 from EPEL is a premature and breaking change to the ability to use Python 3.6 via RPM under CentOS 7.x
I assumed EPEL is driven by RHEL version and not CentOS. Sorry, but I do not know much abotu how EPEL works and there was nobody else who was driving this on the EPEL side. I'll ask on the EPEL mailing list for advise. For existing CentOS installations that already have python36 installed, does it somewhat get removed?
To clarify the severity of this issue: this breaks the automatic deployment of any python36 based services on centos7 (for autoscaling and such). I would be in favor of reverting this change, asap. This is causing us minor operational issues, but I expect many people to be in for a surprise on Monday.
As a quick workaround before the EPEL people say they answer, fetch the lastest (and last) package from Koji: yum install https://kojipkgs.fedoraproject.org/packages/python36/3.6.8/1.el7/$(rpm --eval %_arch)/python36{,-libs}-3.6.8-1.el7.$(rpm --eval %_arch).rpm
As of Friday, our CI testing-jobs - which were written for RHEL and CentOS 7 - were failing on the RHEL jobs but not the CentOS ones: RHEL 7.7's python3 RPM has an "Obsoletes" RPM option that ensures that, if someone selects the python36 package, the RHEL-packaged python3 RPM gets installed, instead. Today, those same jobs - which were updated to OR test for python3 and python36 - are succeeding on RHEL 7 (release 7) but failing on on CentOS 7 (release 6, since that's the latest available). Similarly, our organization's EC2s that are managed via AWS AutoScaling policies are failing to rebuild due to inability to instally the python36 package (or, because it's not yet available, the CentOS python3 RPM). So, my Monday looks like it's going to be spent hand-jamming deployments and explaining to CentOS-using partners, "you need to re-architect your deployment automation until this problem is fixed" (either by CentOS releasing 7.7 or EPEL putting back the python36 runtime-RPM.
This comes up sometimes with point releases. Technically by policy [0] the EPEL package must be removed once it conflicts with something in RHEL. That said, it's common for EPEL maintainers to delay complying with this policy until the corresponding CentOS rebuild is released. One way to compromise is to split the difference and retire the conflicting package once the new RHEL packages show up in the CentOS CR repository [1]. That hasn't happened yet for RHEL 7.7. I imagine CentOS is fully focused on 8 at the moment. I'd be fine with temporarily un-retiring python36, but I would suggest checking in the the CentOS folks to see if the CR rebuilds of the RHEL 7.7 python3 packages are happening soon. If they are then it may be worthwhile to just let people enable CR if needed, and not bother with un-retiring and retiring python36 again. [0]: https://fedoraproject.org/wiki/EPEL/GuidelinesAndPolicies#Policy [1]: https://wiki.centos.org/AdditionalResources/Repositories/CR
As a point of reference, here is the list of retired epel7 packages: python36 python-rpm-macros python3-setuptools python-pip python-wheel
Down-side to "just point to CR" is that if your build-to platforms don't activate CR by default (I know ours don't because, the vast majority of the time, CR is either empty, or, when not empty, can result in unintended upgrades happening), you need to alter deployment-automation to pull from that repo. Given that the RHEL 7.7 packaging was already properly handling the presence of the Python36 runtime on EPEL, it seems like the less breakage-inducing path would be unretirement.
An alternate solution that can be put in place *right now* for users of CentOS and RHEL 7.6- is to use the SCL package Python 3.6. Just adjust your systems/CI jobs to perform the following steps: # CentOS $ yum -y install centos-release-scl $ yum -y install rh-python36 $ . /opt/rh/rh-python36/enable # or scl enable rh-python36 bash [if interactive] # RHEL 7 $ subscription-manager repos --enable rhel-server-rhscl-7-rpms $ yum -y install rh-python36 The benefit is that this Python is directly supported by Red Hat, thus shouldn't be going anywhere during its supported lifecycle (I can still access it on a RHEL 7.7 system).
Oof... Switching to SCL is semi-fine for interactive uses of python, considerably less so for automated uses (can require significantly greater re-tooling and testing of that retooling).
Another alternate solution is to build the RPMs yourself from CentOS's source. The python3 package has already been checked into git so it should be simple enough to retrieve and rebuild the RPMs on your own. It is another hassle, but one that will be solved once CentOS 7.7 is released. A COPR repo could also be set-up in the interim to act as a middle ground unless the retirement is reversed. https://git.centos.org/rpms/python3/tree/c7
We are working on unretiring the package as quickly as possible.
In the meantime, there is an archived version of epel7 python36 in /pub/archive/epel/7/
https://pagure.io/releng/issue/8607
*** Bug 1739833 has been marked as a duplicate of this bug. ***
The packages were unretired. I'm tagging them back to epel7.
*** Bug 1739977 has been marked as a duplicate of this bug. ***
*** Bug 1740053 has been marked as a duplicate of this bug. ***
(In reply to Miro Hrončok from comment #15) > The packages were unretired. > > I'm tagging them back to epel7. Thank you. Verified that I'm able to see it in the repo as https://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/p/python36-3.6.8-1.el7.x86_64.rpm. Going to re-run my CI tests ("belt and suspenders"!).
Glad they are back. I've done some testing as well and I'm able to install python36 into mock (and I believe mock uses CentOS as well). Slight problem is of course this: Problem: package python34-pip-8.1.2-9.el7.noarch conflicts with python36-pip < 8.1.2-9.el7 provided by python36-pip-8.1.2-8.el7.noarch I'll remove the conflict, it is no longer needed for the original purpose.
(In reply to Miro Hrončok from comment #19) > Glad they are back. I've done some testing as well and I'm able to install > python36 into mock (and I believe mock uses CentOS as well). > > Slight problem is of course this: > > Problem: package python34-pip-8.1.2-9.el7.noarch conflicts with python36-pip > < 8.1.2-9.el7 provided by python36-pip-8.1.2-8.el7.noarch > > I'll remove the conflict, it is no longer needed for the original purpose. Sorry, but I can see neither python36 nor python36-devel in https://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/p/ same regards python36-pip.
They are currently in the koki buildroot and can be gotten directly from https://koji.fedoraproject.org/koji/buildinfo?buildID=1258296 . We are waiting for a successful compose to get them back into dl.fedoraproject.org/pub/epel/7/
(In reply to Miro Hrončok from comment #19) > Slight problem is of course this: > > Problem: package python34-pip-8.1.2-9.el7.noarch conflicts with python36-pip > < 8.1.2-9.el7 provided by python36-pip-8.1.2-8.el7.noarch > > I'll remove the conflict, it is no longer needed for the original purpose. I cannot rebuild the package: https://koji.fedoraproject.org/koji/taskinfo?taskID=36953871 DEBUG util.py:585: BUILDSTDERR: Error: DEBUG util.py:585: BUILDSTDERR: Problem: conflicting requests DEBUG util.py:585: BUILDSTDERR: - nothing provides python-rpm-macros > 3-30 needed by python-devel-2.7.5-86.el7.ppc64le DEBUG util.py:585: BUILDSTDERR: - nothing provides python2-rpm-macros > 3-30 needed by python-devel-2.7.5-86.el7.ppc64le Smooge, any idea how to fix this? python-devel-2.7.5-86.el7 is a RHEL 7.7 package, so is python{,2}-rpm-macros > 3-30.
heh, a x86_64 scratchbuild passed.
Alright. One of my CI tests still seems to be failing (still tweaking to properly handle the new RPM-name divergence across EL7s), but am at least seeing the unretired RPM: # yum --showduplicates list available python36 Loaded plugins: fastestmirror Determining fastest mirrors epel/x86_64/metalink | 14 kB 00:00:00 * base: centos4.zswap.net * centos-sclo-rh: centos4.zswap.net * centos-sclo-sclo: centos4.zswap.net * epel: iad.mirror.rackspace.com * extras: centos4.zswap.net * updates: centos4.zswap.net base | 3.6 kB 00:00:00 epel | 5.4 kB 00:00:00 extras | 3.4 kB 00:00:00 updates | 3.4 kB 00:00:00 (1/7): base/7/x86_64/group_gz | 166 kB 00:00:00 (2/7): epel/x86_64/group_gz | 88 kB 00:00:00 (3/7): base/7/x86_64/primary_db | 6.0 MB 00:00:00 (4/7): epel/x86_64/updateinfo | 994 kB 00:00:00 (5/7): extras/7/x86_64/primary_db | 215 kB 00:00:00 (6/7): epel/x86_64/primary_db | 6.8 MB 00:00:00 (7/7): updates/7/x86_64/primary_db | 7.4 MB 00:00:00 Available Packages python36.x86_64 3.6.8-1.el7 epel
FEDORA-EPEL-2019-e9b441e14c has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2019-e9b441e14c
I see python36-3.6.8-1.el7.x86_64.rpm in https://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/p/
python-pip-epel-8.1.2-10.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2019-e9b441e14c
python-pip-epel-8.1.2-10.el7 has been pushed to the Fedora EPEL 7 stable repository. If problems still persist, please make note of it in this bug report.
Now that CentOS 7.7 is out, can we retire these again?
I would say we could do so by Friday? It is still only getting out to mirrors.
As a point of reference, mock gives me: $ mock -r epel-7-x86_64 install python3 python3-setuptools python3-pip python3-wheel python3-rpm-macros python3-pip 9.0.3-5.el7 from base python3-rpm-macros 3-32.el7 from base python3-setuptools 39.2.0-10.el7 from base python3-wheel 0.31.1-4.el7 from base python3 3.6.8-10.el7 from base So we should be able to retire python36, python-rpm-macros, python3-setuptools, python-pip and python-wheel again. However waiting an extra week or so shouldn't hurt things. Users will get the RHEL packages anyway (they obsolete or naturally upgrade the EPEL ones).
Yeah, lets wait a week...
you cannot simply obsolete it without providing alternatives to users that are working with centos7.x when x < 7.
(In reply to Ilia Meerovich from comment #33) > you cannot simply obsolete it without providing alternatives to users that > are working with centos7.x when x < 7. I'm afraid I don't understand your terminology. What exactly do you mean by "obsolete"? And what exactly do you mena by "cannot"?
We have dependencies on python36 packages and we are using older centos7 version. We are using epel repo in order to install python36. I assume that we are not the unique user. If I understand correctly - python36 packages will be removed from epel repo. It will break stuff to all users that are using previous centos 7 with python36. So what will be the alternative? Will python36 remain in epel? will it be renamed?
Thanks for explaining that. > So what will be the alternative? Upgrade to CentOS 7.7. Unfortunately, EPEL 7 only supports the latest RHEL 7.x release. That is 7.7. > Will python36 remain in epel? No. > will it be renamed? No.
OTOH I just got: Problem: conflicting requests - nothing provides python-rpm-macros > 3-30 needed by python-devel-2.7.5-86.el7.x86_64 - nothing provides python2-rpm-macros > 3-30 needed by python-devel-2.7.5-86.el7.x86_64 In koschei when https://bodhi.fedoraproject.org/overrides/python-rpm-macros-3-31.el7 expired. While mock gives me: $ rpm -q --root ~/rpmbuild/mock/epel-7-x86_64/root/ -i python2-rpm-macros Name : python2-rpm-macros Version : 3 Release : 32.el7 Architecture: noarch Install Date: St 18. září 2019, 01:17:01 CEST Group : Unspecified Size : 1495 License : MIT Signature : RSA/SHA256, Čt 22. srpna 2019, 23:59:46 CEST, Key ID 24c6a8a7f4a80eb5 Source RPM : python-rpm-macros-3-32.el7.src.rpm Build Date : St 7. srpna 2019, 12:11:00 CEST Build Host : x86-02.bsys.centos.org Relocations : (not relocatable) Packager : CentOS BuildSystem <http://bugs.centos.org> Vendor : CentOS Summary : RPM macros for building Python 2 packages Description : RPM macros for building Python 2 packages. $ rpm -q --root ~/rpmbuild/mock/epel-7-x86_64/root/ -i python-rpm-macros Name : python-rpm-macros Version : 3 Release : 32.el7 Architecture: noarch Install Date: St 18. září 2019, 01:17:00 CEST Group : Unspecified Size : 4235 License : MIT Signature : RSA/SHA256, Pá 23. srpna 2019, 00:00:07 CEST, Key ID 24c6a8a7f4a80eb5 Source RPM : python-rpm-macros-3-32.el7.src.rpm Build Date : St 7. srpna 2019, 12:11:00 CEST Build Host : x86-02.bsys.centos.org Relocations : (not relocatable) Packager : CentOS BuildSystem <http://bugs.centos.org> Vendor : CentOS Summary : The unversioned Python RPM macros Description : ... I wonder, whether it's just Koji blocking the RHEL packages that collide with EPEL. I've extended the override lifetime.