Bug 1739804 - Retired python36 breaks CentOS
Summary: Retired python36 breaks CentOS
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Fedora EPEL
Classification: Fedora
Component: python36
Version: epel7
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Charalampos Stratakis
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1739833 1739977 1740053 (view as bug list)
Depends On:
Blocks: 1734572
TreeView+ depends on / blocked
 
Reported: 2019-08-10 17:39 UTC by Thomas Jones
Modified: 2020-10-13 09:37 UTC (History)
17 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-13 09:37:26 UTC
Type: Bug


Attachments (Terms of Use)

Description Thomas Jones 2019-08-10 17:39:01 UTC
Description of problem:

Looks like someone prematurely removed the python36 runtime RPM from the repositories

Version-Release number of selected component (if applicable):

How reproducible:


Steps to Reproduce:
1. Launch a new, CentOS 7 system (7.6 is currently latest version available)
2. Install/activate EPEL repositories
3. Attempt to install python36

Actual results:

Runtime RPM not found

Expected results:

Installation of python36 RPM succeeds

Additional info:

While the python36 runtime RPM is obsoleted in RHEL 7.7, CentOS 7.7 has yet to drop. Inability to install python36 from EPEL is a premature and breaking change to the ability to use Python 3.6 via RPM under CentOS 7.x

Comment 1 Miro Hrončok 2019-08-10 19:20:44 UTC
I assumed EPEL is driven by RHEL version and not CentOS. Sorry, but I do not know much abotu how EPEL works and there was nobody else who was driving this on the EPEL side.

I'll ask on the EPEL mailing list for advise.

For existing CentOS installations that already have python36 installed, does it somewhat get removed?

Comment 2 Wouter De Borger 2019-08-10 19:51:42 UTC
To clarify the severity of this issue: this breaks the automatic deployment of any python36 based services on centos7 (for autoscaling and such).

I would be in favor of reverting this change, asap. This is causing us minor operational issues, but I expect many people to be in for a surprise on Monday.

Comment 3 Miro Hrončok 2019-08-10 20:39:17 UTC
As a quick workaround before the EPEL people say they answer, fetch the lastest (and last) package from Koji:

yum install https://kojipkgs.fedoraproject.org/packages/python36/3.6.8/1.el7/$(rpm --eval %_arch)/python36{,-libs}-3.6.8-1.el7.$(rpm --eval %_arch).rpm

Comment 4 Thomas Jones 2019-08-10 21:46:34 UTC
As of Friday, our CI testing-jobs - which were written for RHEL and CentOS 7 - were failing on the RHEL jobs but not the CentOS ones: RHEL 7.7's python3 RPM has an "Obsoletes" RPM option that ensures that, if someone selects the python36 package, the RHEL-packaged python3 RPM gets installed, instead. Today, those same jobs - which were updated to OR test for python3 and python36 - are succeeding on RHEL 7 (release 7) but failing on on CentOS 7 (release 6, since that's the latest available).

Similarly, our organization's EC2s that are managed via AWS AutoScaling policies are failing to rebuild due to inability to instally the python36 package (or, because it's not yet available, the CentOS python3 RPM). So, my Monday looks like it's going to be spent hand-jamming deployments and explaining to CentOS-using partners, "you need to re-architect your deployment automation until this problem is fixed" (either by CentOS releasing 7.7 or EPEL putting back the python36 runtime-RPM.

Comment 5 Carl George 2019-08-10 22:28:15 UTC
This comes up sometimes with point releases.  Technically by policy [0] the EPEL package must be removed once it conflicts with something in RHEL.  That said, it's common for EPEL maintainers to delay complying with this policy until the corresponding CentOS rebuild is released.  One way to compromise is to split the difference and retire the conflicting package once the new RHEL packages show up in the CentOS CR repository [1].  That hasn't happened yet for RHEL 7.7.  I imagine CentOS is fully focused on 8 at the moment.

I'd be fine with temporarily un-retiring python36, but I would suggest checking in the the CentOS folks to see if the CR rebuilds of the RHEL 7.7 python3 packages are happening soon.  If they are then it may be worthwhile to just let people enable CR if needed, and not bother with un-retiring and retiring python36 again.

[0]: https://fedoraproject.org/wiki/EPEL/GuidelinesAndPolicies#Policy
[1]: https://wiki.centos.org/AdditionalResources/Repositories/CR

Comment 6 Miro Hrončok 2019-08-10 22:55:53 UTC
As a point of reference, here is the list of retired epel7 packages:

python36
python-rpm-macros
python3-setuptools
python-pip
python-wheel

Comment 7 Thomas Jones 2019-08-10 22:56:42 UTC
Down-side to "just point to CR" is that if your build-to platforms don't activate CR by default (I know ours don't because, the vast majority of the time, CR is either empty, or, when not empty, can result in unintended upgrades happening), you need to alter deployment-automation to pull from that repo.

Given that the RHEL 7.7 packaging was already properly handling the presence of the Python36 runtime on EPEL, it seems like the less breakage-inducing path would be unretirement.

Comment 8 Michael Rochefort 2019-08-10 23:18:30 UTC
An alternate solution that can be put in place *right now* for users of CentOS and RHEL 7.6- is to use the SCL package Python 3.6. Just adjust your systems/CI jobs to perform the following steps:

# CentOS
$ yum -y install centos-release-scl
$ yum -y install rh-python36
$ . /opt/rh/rh-python36/enable # or scl enable rh-python36 bash [if interactive]

# RHEL 7
$ subscription-manager repos --enable rhel-server-rhscl-7-rpms
$ yum -y install rh-python36

The benefit is that this Python is directly supported by Red Hat, thus shouldn't be going anywhere during its supported lifecycle (I can still access it on a RHEL 7.7 system).

Comment 9 Thomas Jones 2019-08-10 23:23:26 UTC
Oof... Switching to SCL is semi-fine for interactive uses of python, considerably less so for automated uses (can require significantly greater re-tooling and testing of that retooling).

Comment 10 Michael Rochefort 2019-08-10 23:45:51 UTC
Another alternate solution is to build the RPMs yourself from CentOS's source. The python3 package has already been checked into git so it should be simple enough to retrieve and rebuild the RPMs on your own. It is another hassle, but one that will be solved once CentOS 7.7 is released. A COPR repo could also be set-up in the interim to act as a middle ground unless the retirement is reversed.

https://git.centos.org/rpms/python3/tree/c7

Comment 11 Stephen John Smoogen 2019-08-11 01:01:50 UTC
We are working on unretiring the package as quickly as possible.

Comment 12 Stephen John Smoogen 2019-08-11 01:05:14 UTC
In the meantime, there is an archived version of epel7 python36 in /pub/archive/epel/7/

Comment 13 Miro Hrončok 2019-08-11 07:29:29 UTC
https://pagure.io/releng/issue/8607

Comment 14 Miro Hrončok 2019-08-11 08:47:32 UTC
*** Bug 1739833 has been marked as a duplicate of this bug. ***

Comment 15 Miro Hrončok 2019-08-11 08:48:18 UTC
The packages were unretired.

I'm tagging them back to epel7.

Comment 16 Esteban Fonseca 2019-08-12 03:15:57 UTC
*** Bug 1739977 has been marked as a duplicate of this bug. ***

Comment 17 Miro Hrončok 2019-08-12 08:45:48 UTC
*** Bug 1740053 has been marked as a duplicate of this bug. ***

Comment 18 Thomas Jones 2019-08-12 10:43:32 UTC
(In reply to Miro Hrončok from comment #15)
> The packages were unretired.
> 
> I'm tagging them back to epel7.

Thank you. Verified that I'm able to see it in the repo as https://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/p/python36-3.6.8-1.el7.x86_64.rpm. Going to re-run my CI tests ("belt and suspenders"!).

Comment 19 Miro Hrončok 2019-08-12 10:55:10 UTC
Glad they are back. I've done some testing as well and I'm able to install python36 into mock (and I believe mock uses CentOS as well).

Slight problem is of course this:

Problem: package python34-pip-8.1.2-9.el7.noarch conflicts with python36-pip < 8.1.2-9.el7 provided by python36-pip-8.1.2-8.el7.noarch

I'll remove the conflict, it is no longer needed for the original purpose.

Comment 20 Ilia Meerovich 2019-08-12 10:57:31 UTC
(In reply to Miro Hrončok from comment #19)
> Glad they are back. I've done some testing as well and I'm able to install
> python36 into mock (and I believe mock uses CentOS as well).
> 
> Slight problem is of course this:
> 
> Problem: package python34-pip-8.1.2-9.el7.noarch conflicts with python36-pip
> < 8.1.2-9.el7 provided by python36-pip-8.1.2-8.el7.noarch
> 
> I'll remove the conflict, it is no longer needed for the original purpose.


Sorry, but I can see neither python36 nor python36-devel in https://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/p/
same regards python36-pip.

Comment 21 Stephen John Smoogen 2019-08-12 11:03:01 UTC
They are currently in the koki buildroot and can be gotten directly from https://koji.fedoraproject.org/koji/buildinfo?buildID=1258296 . We are waiting for a successful compose to get them back into dl.fedoraproject.org/pub/epel/7/

Comment 22 Miro Hrončok 2019-08-12 11:13:03 UTC
(In reply to Miro Hrončok from comment #19)
> Slight problem is of course this:
> 
> Problem: package python34-pip-8.1.2-9.el7.noarch conflicts with python36-pip
> < 8.1.2-9.el7 provided by python36-pip-8.1.2-8.el7.noarch
> 
> I'll remove the conflict, it is no longer needed for the original purpose.

I cannot rebuild the package:

https://koji.fedoraproject.org/koji/taskinfo?taskID=36953871

DEBUG util.py:585:  BUILDSTDERR: Error: 
DEBUG util.py:585:  BUILDSTDERR:  Problem: conflicting requests
DEBUG util.py:585:  BUILDSTDERR:   - nothing provides python-rpm-macros > 3-30 needed by python-devel-2.7.5-86.el7.ppc64le
DEBUG util.py:585:  BUILDSTDERR:   - nothing provides python2-rpm-macros > 3-30 needed by python-devel-2.7.5-86.el7.ppc64le


Smooge, any idea how to fix this? python-devel-2.7.5-86.el7 is a RHEL 7.7 package, so is python{,2}-rpm-macros > 3-30.

Comment 23 Miro Hrončok 2019-08-12 11:16:23 UTC
heh, a x86_64 scratchbuild passed.

Comment 24 Thomas Jones 2019-08-12 11:22:10 UTC
Alright. One of my CI tests still seems to be failing (still tweaking to properly handle the new RPM-name divergence across EL7s), but am at least seeing the unretired RPM:

# yum --showduplicates list available python36
Loaded plugins: fastestmirror
Determining fastest mirrors
epel/x86_64/metalink                                                                                                           |  14 kB  00:00:00
 * base: centos4.zswap.net
 * centos-sclo-rh: centos4.zswap.net
 * centos-sclo-sclo: centos4.zswap.net
 * epel: iad.mirror.rackspace.com
 * extras: centos4.zswap.net
 * updates: centos4.zswap.net
base                                                                                                                           | 3.6 kB  00:00:00
epel                                                                                                                           | 5.4 kB  00:00:00
extras                                                                                                                         | 3.4 kB  00:00:00
updates                                                                                                                        | 3.4 kB  00:00:00
(1/7): base/7/x86_64/group_gz                                                                                                 | 166 kB  00:00:00
(2/7): epel/x86_64/group_gz                                                                                                   |  88 kB  00:00:00
(3/7): base/7/x86_64/primary_db                                                                                               | 6.0 MB  00:00:00
(4/7): epel/x86_64/updateinfo                                                                                                 | 994 kB  00:00:00
(5/7): extras/7/x86_64/primary_db                                                                                             | 215 kB  00:00:00
(6/7): epel/x86_64/primary_db                                                                                                | 6.8 MB  00:00:00
(7/7): updates/7/x86_64/primary_db                                                                                           | 7.4 MB  00:00:00
Available Packages
python36.x86_64                                                            3.6.8-1.el7                                                            epel

Comment 25 Fedora Update System 2019-08-12 11:23:20 UTC
FEDORA-EPEL-2019-e9b441e14c has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2019-e9b441e14c

Comment 26 Miro Hrončok 2019-08-12 12:55:35 UTC
I see python36-3.6.8-1.el7.x86_64.rpm in https://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/p/

Comment 27 Fedora Update System 2019-08-13 03:49:26 UTC
python-pip-epel-8.1.2-10.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2019-e9b441e14c

Comment 28 Fedora Update System 2019-08-15 17:36:08 UTC
python-pip-epel-8.1.2-10.el7 has been pushed to the Fedora EPEL 7 stable repository. If problems still persist, please make note of it in this bug report.

Comment 29 Kevin Fenzi 2019-09-17 22:55:29 UTC
Now that CentOS 7.7 is out, can we retire these again?

Comment 30 Stephen John Smoogen 2019-09-17 23:20:17 UTC
I would say we could do so by Friday? It is still only getting out to mirrors.

Comment 31 Miro Hrončok 2019-09-17 23:22:24 UTC
As a point of reference, mock gives me:

$ mock -r epel-7-x86_64 install python3 python3-setuptools python3-pip python3-wheel python3-rpm-macros

python3-pip 9.0.3-5.el7 from base
python3-rpm-macros 3-32.el7 from base
python3-setuptools 39.2.0-10.el7 from base
python3-wheel 0.31.1-4.el7 from base
python3 3.6.8-10.el7 from base

So we should be able to retire python36, python-rpm-macros, python3-setuptools, python-pip and python-wheel again.

However waiting an extra week or so shouldn't hurt things. Users will get the RHEL packages anyway (they obsolete or naturally upgrade the EPEL ones).

Comment 32 Kevin Fenzi 2019-09-17 23:28:15 UTC
Yeah, lets wait a week...

Comment 33 Ilia Meerovich 2019-09-18 15:02:21 UTC
you cannot simply obsolete it without providing alternatives to users that are working with centos7.x when x < 7.

Comment 34 Miro Hrončok 2019-09-18 15:16:56 UTC
(In reply to Ilia Meerovich from comment #33)
> you cannot simply obsolete it without providing alternatives to users that
> are working with centos7.x when x < 7.

I'm afraid I don't understand your terminology. What exactly do you mean by "obsolete"? And what exactly do you mena by "cannot"?

Comment 35 Ilia Meerovich 2019-09-18 15:26:06 UTC
We have dependencies on python36 packages and we are using older centos7 version.
We are using epel repo in order to install python36.
I assume that we are not the unique user.

If I understand correctly - python36 packages will be removed from epel repo.
It will break stuff to all users that are using previous centos 7 with python36.

So what will be the alternative? Will python36 remain in epel? will it be renamed?

Comment 36 Miro Hrončok 2019-09-18 15:46:58 UTC
Thanks for explaining that.

> So what will be the alternative?

Upgrade to CentOS 7.7. Unfortunately, EPEL 7 only supports the latest RHEL 7.x release. That is 7.7.

> Will python36 remain in epel?

No.

> will it be renamed?

No.

Comment 37 Miro Hrončok 2019-09-25 05:29:23 UTC
OTOH I just got:

Problem: conflicting requests
- nothing provides python-rpm-macros > 3-30 needed by python-devel-2.7.5-86.el7.x86_64
- nothing provides python2-rpm-macros > 3-30 needed by python-devel-2.7.5-86.el7.x86_64

In koschei when https://bodhi.fedoraproject.org/overrides/python-rpm-macros-3-31.el7 expired.



While mock gives me:


$ rpm -q --root ~/rpmbuild/mock/epel-7-x86_64/root/ -i python2-rpm-macros
Name        : python2-rpm-macros
Version     : 3
Release     : 32.el7
Architecture: noarch
Install Date: St 18. září 2019, 01:17:01 CEST
Group       : Unspecified
Size        : 1495
License     : MIT
Signature   : RSA/SHA256, Čt 22. srpna 2019, 23:59:46 CEST, Key ID 24c6a8a7f4a80eb5
Source RPM  : python-rpm-macros-3-32.el7.src.rpm
Build Date  : St 7. srpna 2019, 12:11:00 CEST
Build Host  : x86-02.bsys.centos.org
Relocations : (not relocatable)
Packager    : CentOS BuildSystem <http://bugs.centos.org>
Vendor      : CentOS
Summary     : RPM macros for building Python 2 packages
Description :
RPM macros for building Python 2 packages.

$ rpm -q --root ~/rpmbuild/mock/epel-7-x86_64/root/ -i python-rpm-macros
Name        : python-rpm-macros
Version     : 3
Release     : 32.el7
Architecture: noarch
Install Date: St 18. září 2019, 01:17:00 CEST
Group       : Unspecified
Size        : 4235
License     : MIT
Signature   : RSA/SHA256, Pá 23. srpna 2019, 00:00:07 CEST, Key ID 24c6a8a7f4a80eb5
Source RPM  : python-rpm-macros-3-32.el7.src.rpm
Build Date  : St 7. srpna 2019, 12:11:00 CEST
Build Host  : x86-02.bsys.centos.org
Relocations : (not relocatable)
Packager    : CentOS BuildSystem <http://bugs.centos.org>
Vendor      : CentOS
Summary     : The unversioned Python RPM macros
Description :
...



I wonder, whether it's just Koji blocking the RHEL packages that collide with EPEL. I've extended the override lifetime.


Note You need to log in before you can comment on or make changes to this bug.