|Summary:||PYC magic number has changed, all pyc files must be re-created|
|Product:||[Fedora] Fedora||Reporter:||Christian Heimes <cheimes>|
|Component:||python3||Assignee:||Miro Hrončok <mhroncok>|
|Status:||CLOSED CURRENTRELEASE||QA Contact:||Fedora Extras Quality Assurance <extras-qa>|
|Version:||32||CC:||cstratak, dmalcolm, lslebodn, mail, m.cyprian, mhroncok, pviktori, rkuska, shcherbina.iryna, slavek.kabrda, tomspur, torsava, vstinner|
|Fixed In Version:||Doc Type:||If docs needed, set a value|
|Doc Text:||Story Points:||---|
|Last Closed:||2020-05-05 12:56:05 UTC||Type:||Bug|
|oVirt Team:||---||RHEL 7.3 requirements from Atomic Host:|
|Cloudforms Team:||---||Target Upstream Version:|
|Bug Depends On:||1799511, 1799934, 1770859, 1779753, 1781891, 1792942, 1795451, 1799513, 1799928, 1799929, 1799936, 1799938, 1799951|
Description Christian Heimes 2019-09-02 13:38:09 UTC
Description of problem: Upstream commit https://github.com/python/cpython/pull/15456/commits/2247fd5900ac0904585750c3c76fae714ea616cb changed the magic number for PYC files to address upstream bug https://bugs.python.org/issue37830. All pyc files of all Python packages must be re-created. Version-Release number of selected component (if applicable): python3-3.8.0~b4-1.fc32.x86_64 How reproducible: always Steps to Reproduce: 1. dnf install python3-urllib3 2. python3 -c "import urllib3" 3. rpm --verify python3-urllib3 Actual results: All pyc files have changed and now have a mismatching hash: S.5....T. /usr/lib/python3.8/site-packages/urllib3/__pycache__/__init__.cpython-38.pyc S.5....T. /usr/lib/python3.8/site-packages/urllib3/__pycache__/_collections.cpython-38.pyc S.5....T. /usr/lib/python3.8/site-packages/urllib3/__pycache__/connection.cpython-38.pyc S.5....T. /usr/lib/python3.8/site-packages/urllib3/__pycache__/connectionpool.cpython-38.pyc S.5....T. /usr/lib/python3.8/site-packages/urllib3/__pycache__/exceptions.cpython-38.pyc S.5....T. /usr/lib/python3.8/site-packages/urllib3/__pycache__/fields.cpython-38.pyc ... Expected results: pyc files should not have changed Additional info:
Comment 1 Miro Hrončok 2019-09-02 13:47:00 UTC
We could do one of the following: A) targeted rebuild of everything with the old bytecode version B) wait for the mass rebuild Mass rebuild happens in January 2020, hence quite late from now. What are the chances that this will happen again between b4 and rc1? I suggest we wait for rc1 and rebuild then.
Comment 2 Christian Heimes 2019-09-02 13:48:10 UTC
The magic number for byte code is stable for a minor release. Since Python 3.8 is still in beta, byte code changes can rarely occur. The byte code has changed between 3.8b3 and 3.8b4. Virtually all python3-* packages have been created with 3.8b3. This causes three different kinds of problems: 1) If a non-root user executes Python code, then startup will be slower because Python cannot use the pyc files with the old magic number. 2) If a root user with restricted SELinux context executes Python code, then write operation will fail and the audit log will be pumped with AVC violations. The result is (1) + lots of noise. 3) If a root user with unrestricted SELinux context runs Python code, Python is able to regenerate and store the updated PYC files. Afterwards rpm --verify will fail for all touched files. I highly recommend to rebuild all Python packages before Fedora 32 is released. I don't expect any PYC format changes now. To be safe you could wait until 3.8.0 final is out.
Comment 3 Miro Hrončok 2019-09-02 13:56:57 UTC
> I highly recommend to rebuild all Python packages before Fedora 32 is released. This will happen either way, the mass rebuild is scheduled before the Fedora 32 release date. Packages that fail to build will however still be affected. I suggest we wait **at least** for rc1 and then try to rebuild everything.
Comment 4 Christian Heimes 2019-09-02 13:58:52 UTC
> I suggest we wait **at least** for rc1 and then try to rebuild everything. +1
Comment 5 Christian Heimes 2019-09-02 20:16:56 UTC
*** Bug 1748005 has been marked as a duplicate of this bug. ***
Comment 6 Miro Hrončok 2019-09-30 09:07:39 UTC
The rc1 is almost upon us: https://email@example.com/message/ACMKEQNGK4FVUIZ6TYD5H26OSPIO5GSN/ The magic number is still 3413: https://github.com/python/cpython/blob/3.8/Lib/importlib/_bootstrap_external.py I'll draft a script that should tell us what packages need rebuilding. ### Notes: This lists all source RPMs that ship Python 3.8 bytecode: $ repoquery --repo=rawhide -f '*.cpython-38.pyc' --source This lists all successful builds in Koji of a source package <pkg>, after python3-3.8.0~b4-1.fc32 finished: $ koji list-builds --package <pkg> --after '2019-08-31 16:11:41' --state=COMPLETE --quiet It took an arbitrary amount of extra time before python3-3.8.0~b4-1.fc32 was present in the buildroot. I suggest that we check builds from 2019-08-31 manually.
Comment 7 Miro Hrončok 2019-09-30 09:37:05 UTC
Comment 8 Miro Hrončok 2019-09-30 12:55:56 UTC
The following packages were built on 2019-08-31, after 16:11:41. With Python 3.8.0b3: - python-efel - python-jep - python-fastavro - python-lfpy - python-hdmf With Python 3.8.0b4: - duplicity - python-molecule - python-django-cors-headers In theory, some package may have started the build with 3.8.0b3 but finished the next day, however given the nature of this bug, such packages will be taken care of during the mass rebuild.
Comment 9 Miro Hrončok 2019-10-03 11:22:32 UTC
This was announced on devel yesterday: https://firstname.lastname@example.org/message/ESAVCLLOGXOLUX2V6BQWDCH5TBUEO4C5/ Rebuilds have started.
Comment 11 Miro Hrončok 2019-10-03 21:04:09 UTC
The following packages either haven't finished building or failed: PyGreSQL bodhi diffoscope enki file future gcc gmsh gtts hatch hovercraft httpwatcher kicad lammps mir otf2 pcs protobuf pyOpenSSL pybluez pylast pyparted pytest python-PyPDF2 python-ansi2html python-arpeggio python-ase python-astunparse python-autobahn python-basemap python-colorclass python-compreffor python-conda-package-handling python-cpuinfo python-dictdiffer python-diff-cover python-django-post_office python-f5-icontrol-rest python-flake8-polyfill python-google-apputils python-hug python-inotify python-jaraco-packaging python-keystoneauth1 python-mapnik python-mechanize python-mimeparse python-openshift python-pdfminer python-pip-shims python-pluggy python-pocketlint python-portend python-progressbar2 python-prometheus_client python-rosinstall python-ryu python-setuptools-lint python-shortuuid python-stem python-subliminal python-tw2-forms python-txaio python-urllib3 python-verboselogs python-watchdog python-webassets python-xmp-toolkit python3-pytest-asyncio pyusb qutebrowser scipy sshuttle targetcli torbrowser-launcher veusz vigra virt-bootstrap The remaining 2585 is fine. I'll recheck after couple days.
Comment 14 Miro Hrončok 2019-10-08 13:41:09 UTC
Current list: PyGreSQL bodhi diffoscope enki future gtts hatch hovercraft httpwatcher lammps mir otf2 pcs pyOpenSSL pybluez pylast pyparted pytest python-PyPDF2 python-arpeggio python-ase python-astunparse python-autobahn python-basemap python-colorclass python-conda-package-handling python-cpuinfo python-dictdiffer python-diff-cover python-django-post_office python-f5-icontrol-rest python-flake8-polyfill python-google-apputils python-hug python-inotify python-jaraco-packaging python-keystoneauth1 python-mapnik python-mechanize python-pdfminer python-pip-shims python-pluggy python-pocketlint python-portend python-progressbar2 python-prometheus_client python-rosinstall python-ryu python-setuptools-lint python-shortuuid python-stem python-subliminal python-tw2-forms python-txaio python-urllib3 python-verboselogs python-watchdog python-webassets python-xmp-toolkit python3-pytest-asyncio pyusb sshuttle targetcli torbrowser-launcher veusz vigra virt-bootstrap
Comment 17 Miro Hrončok 2019-11-08 23:33:53 UTC
Current list: bodhi diffoscope enki gtts hovercraft httpwatcher lammps otf2 pcs pylast python-PyPDF2 python-arpeggio python-ase python-basemap python-conda-package-handling python-cpuinfo python-diff-cover python-django-post_office python-f5-icontrol-rest python-flake8-polyfill python-hug python-inotify python-jaraco-packaging python-mapnik python-mechanize python-pdfminer python-pip-shims python-pluggy python-pocketlint python-progressbar2 python-rosinstall python-ryu python-setuptools-lint python-shortuuid python-stem python-subliminal python-tw2-forms python-verboselogs python-watchdog python-webassets python-xmp-toolkit python3-pytest-asyncio sshuttle targetcli vigra virt-bootstrap
Comment 18 Miro Hrončok 2019-12-01 22:30:38 UTC
Current list: bodhi diffoscope hovercraft httpwatcher otf2 pylast python-arpeggio python-basemap python-django-post_office python-f5-icontrol-rest python-flake8-polyfill python-hug python-jaraco-packaging python-mapnik python-pdfminer python-pip-shims python-pluggy python-progressbar2 python-ryu python-shortuuid python-stem python-subliminal python-tw2-forms python-watchdog python-xmp-toolkit vigra I've resubmitted the builds.
Comment 19 Miro Hrončok 2019-12-04 16:36:30 UTC
Current list: bodhi hovercraft httpwatcher otf2 pylast python-arpeggio python-basemap python-django-post_office python-f5-icontrol-rest python-flake8-polyfill python-hug python-jaraco-packaging python-mapnik python-pip-shims python-pluggy python-progressbar2 python-ryu python-shortuuid python-stem python-subliminal python-watchdog python-xmp-toolkit
Comment 30 Miro Hrončok 2020-01-07 18:03:52 UTC
Fabian, could you please stop associating EPEL updates with his bugzilla? Thanks.
Comment 31 Fabian Affolter 2020-01-07 18:44:43 UTC
That's your own changelog entry that links to this report, isn't it? * Thu Oct 03 2019 Miro Hrončok <email@example.com> - 2.17.2-4 - Rebuilt for Python 3.8.0rc1 (#1748018) This looks like a processing issue to me and should be handled like that. If a package is built for EPEL then there is no match for the product and the version (this report is for Fedora Rawhide). The bug no. simply ends in the template and the updates shows up here. I can keep an eye on the updates but no promises as there are a lot of packages on the way.
Comment 32 Miro Hrončok 2020-01-07 18:50:45 UTC
When you file the package update, you are supposed to fill in the bugs and notes according to reality. fedpkg tries to help you by taking the info from the changelog, but in this case, it requires a human to verify the information. You can use fedpkg update without the template by providing options on the command line: $ fedpkg update --type newpackage --bugs "" --notes "Introduce this package to EPEL 8 🥳" --stable-karma 1 -unstable-karma -1 (untested)
Comment 41 Fabian Affolter 2020-01-12 10:13:56 UTC
Comment 42 Fabian Affolter 2020-01-12 10:22:45 UTC
This bug was now manually removed from all updates. To my understanding they shouldn't close this bug.
Comment 43 Miro Hrončok 2020-01-12 19:24:16 UTC
(In reply to Fabian Affolter from comment #42) > This bug was now manually removed from all updates. To my understanding they > shouldn't close this bug. Thanks.
Comment 45 Miro Hrončok 2020-01-28 12:50:14 UTC
The mass rebuild of Fedora 32 is in progress. I will run the script that collects the information once the mass rebuild is over. All the remaining packages will later have open FTBFS bugzillas and we can use the bugzilla depends-on/blocks thing to rack them.
Comment 46 Miro Hrončok 2020-02-06 18:22:17 UTC
After the mass rebuild: hovercraft httpwatcher pylast python-django-post_office python-flake8-polyfill python-hug python-jaraco-packaging python-mapnik python-pip-shims python-progressbar2 python-watchdog python-xmp-toolkit
Comment 47 Miro Hrončok 2020-02-07 00:06:15 UTC
(I've marked all the unrelated EPEL Bodhi comments as private to make this bugzilla more readable for those who cannot see them.)
Comment 48 Ben Cotton 2020-02-11 16:39:29 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 32 development cycle. Changing version to 32.
Comment 49 Miro Hrončok 2020-05-05 12:56:05 UTC
I don't think this needs to be tracked any more. The remaining packages are either orphaned or being taken care of.