Bug 1748018 - PYC magic number has changed, all pyc files must be re-created
Summary: PYC magic number has changed, all pyc files must be re-created
Keywords:
Status: ON_QA
Alias: None
Product: Fedora
Classification: Fedora
Component: python3
Version: 32
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Miro Hrončok
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1748005 (view as bug list)
Depends On: 1799511 1799928 1799934 1799938 1799951 1770859 1779753 1781891 1792942 1795451 1799513 1799929 1799936
Blocks: PYTHON38
TreeView+ depends on / blocked
 
Reported: 2019-09-02 13:38 UTC by Christian Heimes
Modified: 2020-02-11 16:39 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-01-19 00:17:34 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Python 37830 None None None 2019-09-02 13:38:09 UTC

Description Christian Heimes 2019-09-02 13:38:09 UTC
Description of problem:
Upstream commit https://github.com/python/cpython/pull/15456/commits/2247fd5900ac0904585750c3c76fae714ea616cb changed the magic number for PYC files to address upstream bug https://bugs.python.org/issue37830. All pyc files of all Python packages must be re-created.

Version-Release number of selected component (if applicable):
python3-3.8.0~b4-1.fc32.x86_64

How reproducible:
always

Steps to Reproduce:
1. dnf install python3-urllib3
2. python3 -c "import urllib3"
3. rpm --verify python3-urllib3

Actual results:
All pyc files have changed and now have a mismatching hash:
S.5....T.    /usr/lib/python3.8/site-packages/urllib3/__pycache__/__init__.cpython-38.pyc
S.5....T.    /usr/lib/python3.8/site-packages/urllib3/__pycache__/_collections.cpython-38.pyc
S.5....T.    /usr/lib/python3.8/site-packages/urllib3/__pycache__/connection.cpython-38.pyc
S.5....T.    /usr/lib/python3.8/site-packages/urllib3/__pycache__/connectionpool.cpython-38.pyc
S.5....T.    /usr/lib/python3.8/site-packages/urllib3/__pycache__/exceptions.cpython-38.pyc
S.5....T.    /usr/lib/python3.8/site-packages/urllib3/__pycache__/fields.cpython-38.pyc
...

Expected results:
pyc files should not have changed

Additional info:

Comment 1 Miro Hrončok 2019-09-02 13:47:00 UTC
We could do one of the following:

 A) targeted rebuild of everything with the old bytecode version
 B) wait for the mass rebuild

Mass rebuild happens in January 2020, hence quite late from now.

What are the chances that this will happen again between b4 and rc1? I suggest we wait for rc1 and rebuild then.

Comment 2 Christian Heimes 2019-09-02 13:48:10 UTC
The magic number for byte code is stable for a minor release. Since Python 3.8 is still in beta, byte code changes can rarely occur. The byte code has changed between 3.8b3 and 3.8b4.

Virtually all python3-* packages have been created with 3.8b3. This causes three different kinds of problems:

1) If a non-root user executes Python code, then startup will be slower because Python cannot use the pyc files with the old magic number.
2) If a root user with restricted SELinux context executes Python code, then write operation will fail and the audit log will be pumped with AVC violations. The result is (1) + lots of noise.
3) If a root user with unrestricted SELinux context runs Python code, Python is able to regenerate and store the updated PYC files. Afterwards rpm --verify will fail for all touched files.

I highly recommend to rebuild all Python packages before Fedora 32 is released. I don't expect any PYC format changes now. To be safe you could wait until 3.8.0 final is out.

Comment 3 Miro Hrončok 2019-09-02 13:56:57 UTC
> I highly recommend to rebuild all Python packages before Fedora 32 is released.

This will happen either way, the mass rebuild is scheduled before the Fedora 32 release date. Packages that fail to build will however still be affected.

I suggest we wait **at least** for rc1 and then try to rebuild everything.

Comment 4 Christian Heimes 2019-09-02 13:58:52 UTC
> I suggest we wait **at least** for rc1 and then try to rebuild everything.

+1

Comment 5 Christian Heimes 2019-09-02 20:16:56 UTC
*** Bug 1748005 has been marked as a duplicate of this bug. ***

Comment 6 Miro Hrončok 2019-09-30 09:07:39 UTC
The rc1 is almost upon us: https://mail.python.org/archives/list/python-dev@python.org/message/ACMKEQNGK4FVUIZ6TYD5H26OSPIO5GSN/

The magic number is still 3413: https://github.com/python/cpython/blob/3.8/Lib/importlib/_bootstrap_external.py

I'll draft a script that should tell us what packages need rebuilding.


### Notes:

This lists all source RPMs that ship Python 3.8 bytecode:

    $ repoquery --repo=rawhide -f '*.cpython-38.pyc' --source


This lists all successful builds in Koji of a source package <pkg>, after python3-3.8.0~b4-1.fc32 finished:

    $ koji list-builds --package <pkg> --after '2019-08-31 16:11:41' --state=COMPLETE --quiet


It took an arbitrary amount of extra time before python3-3.8.0~b4-1.fc32 was present in the buildroot. I suggest that we check builds from 2019-08-31 manually.

Comment 8 Miro Hrončok 2019-09-30 12:55:56 UTC
The following packages were built on 2019-08-31, after 16:11:41.


With Python 3.8.0b3:
- python-efel
- python-jep
- python-fastavro
- python-lfpy
- python-hdmf


With Python 3.8.0b4:
- duplicity
- python-molecule
- python-django-cors-headers



In theory, some package may have started the build with 3.8.0b3 but finished the next day, however given the nature of this bug, such packages will be taken care of during the mass rebuild.

Comment 9 Miro Hrončok 2019-10-03 11:22:32 UTC
This was announced on devel yesterday: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/ESAVCLLOGXOLUX2V6BQWDCH5TBUEO4C5/

Rebuilds have started.

Comment 11 Miro Hrončok 2019-10-03 21:04:09 UTC
The following packages either haven't finished building or failed:

PyGreSQL
bodhi
diffoscope
enki
file
future
gcc
gmsh
gtts
hatch
hovercraft
httpwatcher
kicad
lammps
mir
otf2
pcs
protobuf
pyOpenSSL
pybluez
pylast
pyparted
pytest
python-PyPDF2
python-ansi2html
python-arpeggio
python-ase
python-astunparse
python-autobahn
python-basemap
python-colorclass
python-compreffor
python-conda-package-handling
python-cpuinfo
python-dictdiffer
python-diff-cover
python-django-post_office
python-f5-icontrol-rest
python-flake8-polyfill
python-google-apputils
python-hug
python-inotify
python-jaraco-packaging
python-keystoneauth1
python-mapnik
python-mechanize
python-mimeparse
python-openshift
python-pdfminer
python-pip-shims
python-pluggy
python-pocketlint
python-portend
python-progressbar2
python-prometheus_client
python-rosinstall
python-ryu
python-setuptools-lint
python-shortuuid
python-stem
python-subliminal
python-tw2-forms
python-txaio
python-urllib3
python-verboselogs
python-watchdog
python-webassets
python-xmp-toolkit
python3-pytest-asyncio
pyusb
qutebrowser
scipy
sshuttle
targetcli
torbrowser-launcher
veusz
vigra
virt-bootstrap

The remaining 2585 is fine.

I'll recheck after couple days.

Comment 14 Miro Hrončok 2019-10-08 13:41:09 UTC
Current list:

PyGreSQL
bodhi
diffoscope
enki
future
gtts
hatch
hovercraft
httpwatcher
lammps
mir
otf2
pcs
pyOpenSSL
pybluez
pylast
pyparted
pytest
python-PyPDF2
python-arpeggio
python-ase
python-astunparse
python-autobahn
python-basemap
python-colorclass
python-conda-package-handling
python-cpuinfo
python-dictdiffer
python-diff-cover
python-django-post_office
python-f5-icontrol-rest
python-flake8-polyfill
python-google-apputils
python-hug
python-inotify
python-jaraco-packaging
python-keystoneauth1
python-mapnik
python-mechanize
python-pdfminer
python-pip-shims
python-pluggy
python-pocketlint
python-portend
python-progressbar2
python-prometheus_client
python-rosinstall
python-ryu
python-setuptools-lint
python-shortuuid
python-stem
python-subliminal
python-tw2-forms
python-txaio
python-urllib3
python-verboselogs
python-watchdog
python-webassets
python-xmp-toolkit
python3-pytest-asyncio
pyusb
sshuttle
targetcli
torbrowser-launcher
veusz
vigra
virt-bootstrap

Comment 17 Miro Hrončok 2019-11-08 23:33:53 UTC
Current list:


bodhi
diffoscope
enki
gtts
hovercraft
httpwatcher
lammps
otf2
pcs
pylast
python-PyPDF2
python-arpeggio
python-ase
python-basemap
python-conda-package-handling
python-cpuinfo
python-diff-cover
python-django-post_office
python-f5-icontrol-rest
python-flake8-polyfill
python-hug
python-inotify
python-jaraco-packaging
python-mapnik
python-mechanize
python-pdfminer
python-pip-shims
python-pluggy
python-pocketlint
python-progressbar2
python-rosinstall
python-ryu
python-setuptools-lint
python-shortuuid
python-stem
python-subliminal
python-tw2-forms
python-verboselogs
python-watchdog
python-webassets
python-xmp-toolkit
python3-pytest-asyncio
sshuttle
targetcli
vigra
virt-bootstrap

Comment 18 Miro Hrončok 2019-12-01 22:30:38 UTC
Current list:

bodhi
diffoscope
hovercraft
httpwatcher
otf2
pylast
python-arpeggio
python-basemap
python-django-post_office
python-f5-icontrol-rest
python-flake8-polyfill
python-hug
python-jaraco-packaging
python-mapnik
python-pdfminer
python-pip-shims
python-pluggy
python-progressbar2
python-ryu
python-shortuuid
python-stem
python-subliminal
python-tw2-forms
python-watchdog
python-xmp-toolkit
vigra


I've resubmitted the builds.

Comment 19 Miro Hrončok 2019-12-04 16:36:30 UTC
Current list:

bodhi
hovercraft
httpwatcher
otf2
pylast
python-arpeggio
python-basemap
python-django-post_office
python-f5-icontrol-rest
python-flake8-polyfill
python-hug
python-jaraco-packaging
python-mapnik
python-pip-shims
python-pluggy
python-progressbar2
python-ryu
python-shortuuid
python-stem
python-subliminal
python-watchdog
python-xmp-toolkit

Comment 30 Miro Hrončok 2020-01-07 18:03:52 UTC
Fabian, could you please stop associating EPEL updates with his bugzilla? Thanks.

Comment 31 Fabian Affolter 2020-01-07 18:44:43 UTC
That's your own changelog entry that links to this report, isn't it?

* Thu Oct 03 2019 Miro Hrončok <mhroncok@redhat.com> - 2.17.2-4
- Rebuilt for Python 3.8.0rc1 (#1748018)

This looks like a processing issue to me and should be handled like that. If a package is built for EPEL then there is no match for the product and the version (this report is for Fedora Rawhide). The bug no. simply ends in the template and the updates shows up here.

I can keep an eye on the updates but no promises as there are a lot of packages on the way.

Comment 32 Miro Hrončok 2020-01-07 18:50:45 UTC
When you file the package update, you are supposed to fill in the bugs and notes according to reality. fedpkg tries to help you by taking the info from the changelog, but in this case, it requires a human to verify the information. You can use fedpkg update without the template by providing options on the command line:

$ fedpkg update --type newpackage --bugs "" --notes "Introduce this package to EPEL 8 🥳" --stable-karma 1 -unstable-karma -1

(untested)

Comment 41 Fabian Affolter 2020-01-12 10:13:56 UTC
Aha, sorry.

Comment 42 Fabian Affolter 2020-01-12 10:22:45 UTC
This bug was now manually removed from all updates. To my understanding they shouldn't close this bug.

Comment 43 Miro Hrončok 2020-01-12 19:24:16 UTC
(In reply to Fabian Affolter from comment #42)
> This bug was now manually removed from all updates. To my understanding they
> shouldn't close this bug.

Thanks.

Comment 45 Miro Hrončok 2020-01-28 12:50:14 UTC
The mass rebuild of Fedora 32 is in progress. I will run the script that collects the information once the mass rebuild is over. All the remaining packages will later have open FTBFS bugzillas and we can use the bugzilla depends-on/blocks thing to rack them.

Comment 46 Miro Hrončok 2020-02-06 18:22:17 UTC
After the mass rebuild:

hovercraft
httpwatcher
pylast
python-django-post_office
python-flake8-polyfill
python-hug
python-jaraco-packaging
python-mapnik
python-pip-shims
python-progressbar2
python-watchdog
python-xmp-toolkit

Comment 47 Miro Hrončok 2020-02-07 00:06:15 UTC
(I've marked all the unrelated EPEL Bodhi comments as private to make this bugzilla more readable for those who cannot see them.)

Comment 48 Ben Cotton 2020-02-11 16:39:29 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 32 development cycle.
Changing version to 32.


Note You need to log in before you can comment on or make changes to this bug.