Bug 1748018

We could do one of the following:

 A) targeted rebuild of everything with the old bytecode version
 B) wait for the mass rebuild

Mass rebuild happens in January 2020, hence quite late from now.

What are the chances that this will happen again between b4 and rc1? I suggest we wait for rc1 and rebuild then.

The magic number for byte code is stable for a minor release. Since Python 3.8 is still in beta, byte code changes can rarely occur. The byte code has changed between 3.8b3 and 3.8b4.

Virtually all python3-* packages have been created with 3.8b3. This causes three different kinds of problems:

1) If a non-root user executes Python code, then startup will be slower because Python cannot use the pyc files with the old magic number.
2) If a root user with restricted SELinux context executes Python code, then write operation will fail and the audit log will be pumped with AVC violations. The result is (1) + lots of noise.
3) If a root user with unrestricted SELinux context runs Python code, Python is able to regenerate and store the updated PYC files. Afterwards rpm --verify will fail for all touched files.

I highly recommend to rebuild all Python packages before Fedora 32 is released. I don't expect any PYC format changes now. To be safe you could wait until 3.8.0 final is out.

> I highly recommend to rebuild all Python packages before Fedora 32 is released.

This will happen either way, the mass rebuild is scheduled before the Fedora 32 release date. Packages that fail to build will however still be affected.

I suggest we wait **at least** for rc1 and then try to rebuild everything.

> I suggest we wait **at least** for rc1 and then try to rebuild everything.

+1

*** Bug 1748005 has been marked as a duplicate of this bug. ***

The rc1 is almost upon us: https://mail.python.org/archives/list/python-dev@python.org/message/ACMKEQNGK4FVUIZ6TYD5H26OSPIO5GSN/

The magic number is still 3413: https://github.com/python/cpython/blob/3.8/Lib/importlib/_bootstrap_external.py

I'll draft a script that should tell us what packages need rebuilding.


### Notes:

This lists all source RPMs that ship Python 3.8 bytecode:

    $ repoquery --repo=rawhide -f '*.cpython-38.pyc' --source


This lists all successful builds in Koji of a source package <pkg>, after python3-3.8.0~b4-1.fc32 finished:

    $ koji list-builds --package <pkg> --after '2019-08-31 16:11:41' --state=COMPLETE --quiet


It took an arbitrary amount of extra time before python3-3.8.0~b4-1.fc32 was present in the buildroot. I suggest that we check builds from 2019-08-31 manually.

https://github.com/hroncok/mini-mass-rebuild/blob/master/bytecodes.py

The following packages were built on 2019-08-31, after 16:11:41.


With Python 3.8.0b3:
- python-efel
- python-jep
- python-fastavro
- python-lfpy
- python-hdmf


With Python 3.8.0b4:
- duplicity
- python-molecule
- python-django-cors-headers



In theory, some package may have started the build with 3.8.0b3 but finished the next day, however given the nature of this bug, such packages will be taken care of during the mass rebuild.

This was announced on devel yesterday: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/ESAVCLLOGXOLUX2V6BQWDCH5TBUEO4C5/

Rebuilds have started.

The following packages either haven't finished building or failed:

PyGreSQL
bodhi
diffoscope
enki
file
future
gcc
gmsh
gtts
hatch
hovercraft
httpwatcher
kicad
lammps
mir
otf2
pcs
protobuf
pyOpenSSL
pybluez
pylast
pyparted
pytest
python-PyPDF2
python-ansi2html
python-arpeggio
python-ase
python-astunparse
python-autobahn
python-basemap
python-colorclass
python-compreffor
python-conda-package-handling
python-cpuinfo
python-dictdiffer
python-diff-cover
python-django-post_office
python-f5-icontrol-rest
python-flake8-polyfill
python-google-apputils
python-hug
python-inotify
python-jaraco-packaging
python-keystoneauth1
python-mapnik
python-mechanize
python-mimeparse
python-openshift
python-pdfminer
python-pip-shims
python-pluggy
python-pocketlint
python-portend
python-progressbar2
python-prometheus_client
python-rosinstall
python-ryu
python-setuptools-lint
python-shortuuid
python-stem
python-subliminal
python-tw2-forms
python-txaio
python-urllib3
python-verboselogs
python-watchdog
python-webassets
python-xmp-toolkit
python3-pytest-asyncio
pyusb
qutebrowser
scipy
sshuttle
targetcli
torbrowser-launcher
veusz
vigra
virt-bootstrap

The remaining 2585 is fine.

I'll recheck after couple days.

Current list:

PyGreSQL
bodhi
diffoscope
enki
future
gtts
hatch
hovercraft
httpwatcher
lammps
mir
otf2
pcs
pyOpenSSL
pybluez
pylast
pyparted
pytest
python-PyPDF2
python-arpeggio
python-ase
python-astunparse
python-autobahn
python-basemap
python-colorclass
python-conda-package-handling
python-cpuinfo
python-dictdiffer
python-diff-cover
python-django-post_office
python-f5-icontrol-rest
python-flake8-polyfill
python-google-apputils
python-hug
python-inotify
python-jaraco-packaging
python-keystoneauth1
python-mapnik
python-mechanize
python-pdfminer
python-pip-shims
python-pluggy
python-pocketlint
python-portend
python-progressbar2
python-prometheus_client
python-rosinstall
python-ryu
python-setuptools-lint
python-shortuuid
python-stem
python-subliminal
python-tw2-forms
python-txaio
python-urllib3
python-verboselogs
python-watchdog
python-webassets
python-xmp-toolkit
python3-pytest-asyncio
pyusb
sshuttle
targetcli
torbrowser-launcher
veusz
vigra
virt-bootstrap

Current list:


bodhi
diffoscope
enki
gtts
hovercraft
httpwatcher
lammps
otf2
pcs
pylast
python-PyPDF2
python-arpeggio
python-ase
python-basemap
python-conda-package-handling
python-cpuinfo
python-diff-cover
python-django-post_office
python-f5-icontrol-rest
python-flake8-polyfill
python-hug
python-inotify
python-jaraco-packaging
python-mapnik
python-mechanize
python-pdfminer
python-pip-shims
python-pluggy
python-pocketlint
python-progressbar2
python-rosinstall
python-ryu
python-setuptools-lint
python-shortuuid
python-stem
python-subliminal
python-tw2-forms
python-verboselogs
python-watchdog
python-webassets
python-xmp-toolkit
python3-pytest-asyncio
sshuttle
targetcli
vigra
virt-bootstrap

Current list:

bodhi
diffoscope
hovercraft
httpwatcher
otf2
pylast
python-arpeggio
python-basemap
python-django-post_office
python-f5-icontrol-rest
python-flake8-polyfill
python-hug
python-jaraco-packaging
python-mapnik
python-pdfminer
python-pip-shims
python-pluggy
python-progressbar2
python-ryu
python-shortuuid
python-stem
python-subliminal
python-tw2-forms
python-watchdog
python-xmp-toolkit
vigra


I've resubmitted the builds.

Current list:

bodhi
hovercraft
httpwatcher
otf2
pylast
python-arpeggio
python-basemap
python-django-post_office
python-f5-icontrol-rest
python-flake8-polyfill
python-hug
python-jaraco-packaging
python-mapnik
python-pip-shims
python-pluggy
python-progressbar2
python-ryu
python-shortuuid
python-stem
python-subliminal
python-watchdog
python-xmp-toolkit

Fabian, could you please stop associating EPEL updates with his bugzilla? Thanks.

That's your own changelog entry that links to this report, isn't it?

* Thu Oct 03 2019 Miro Hrončok <mhroncok> - 2.17.2-4
- Rebuilt for Python 3.8.0rc1 (#1748018)

This looks like a processing issue to me and should be handled like that. If a package is built for EPEL then there is no match for the product and the version (this report is for Fedora Rawhide). The bug no. simply ends in the template and the updates shows up here.

I can keep an eye on the updates but no promises as there are a lot of packages on the way.

When you file the package update, you are supposed to fill in the bugs and notes according to reality. fedpkg tries to help you by taking the info from the changelog, but in this case, it requires a human to verify the information. You can use fedpkg update without the template by providing options on the command line:

$ fedpkg update --type newpackage --bugs "" --notes "Introduce this package to EPEL 8 🥳" --stable-karma 1 -unstable-karma -1

(untested)

Aha, sorry.

This bug was now manually removed from all updates. To my understanding they shouldn't close this bug.

(In reply to Fabian Affolter from comment #42)
> This bug was now manually removed from all updates. To my understanding they
> shouldn't close this bug.

Thanks.

The mass rebuild of Fedora 32 is in progress. I will run the script that collects the information once the mass rebuild is over. All the remaining packages will later have open FTBFS bugzillas and we can use the bugzilla depends-on/blocks thing to rack them.

After the mass rebuild:

hovercraft
httpwatcher
pylast
python-django-post_office
python-flake8-polyfill
python-hug
python-jaraco-packaging
python-mapnik
python-pip-shims
python-progressbar2
python-watchdog
python-xmp-toolkit

(I've marked all the unrelated EPEL Bodhi comments as private to make this bugzilla more readable for those who cannot see them.)

This bug appears to have been reported against 'rawhide' during the Fedora 32 development cycle.
Changing version to 32.

I don't think this needs to be tracked any more. The remaining packages are either orphaned or being taken care of.