Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be unavailable on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1411588 - Python 3 only packages are not bytecompiled in %{__os_install_post} [fedora-all] [rhel-all]
Summary: Python 3 only packages are not bytecompiled in %{__os_install_post} [fedora-a...
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: rpm
Version: 26
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Packaging Maintenance Team
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-10 02:42 UTC by Miro Hrončok
Modified: 2017-07-24 07:12 UTC (History)
8 users (show)

Fixed In Version: rpm-4.13.0-12.fc26
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-07-24 07:12:25 UTC
Type: Bug


Attachments (Terms of Use)

Description Miro Hrončok 2017-01-10 02:42:51 UTC
tl;dr
=====

/usr/lib/rpm/brp-python-bytecompile is broken when /usr/bin/python is not there because it exists too early when detecting it.

That happens for Python 3 only packages and might produce broken packages.

Possible solution is proposed.

All Fedoras are affected. ELs as well.



Full size bug report with all the details
=========================================

When a package does not install /usr/bin/python (that's Python 2) to the buildroot (aka it does not BuildRequire python, python2-devel, python2-foo etc.),
the /usr/lib/rpm/brp-python-bytecompile script does nothing.

This script is supposed to go trough some directories (/usr/lib(64)?/python[0-9]\.[0-9]) and bytecompile files in there with the appropriate Python version.

However, the script has this section at the beginning, that disables the entire procedure when $default_python (/usr/bin/python) is not found:

    # If we don't have a python interpreter, avoid changing anything.
    default_python=${1:-/usr/bin/python}
    if [ ! -x "$default_python" ]; then
    	exit 0
    fi

BTW the script is invoked as `/usr/lib/rpm/brp-python-bytecompile /usr/bin/python 1`, so $default_python is /usr/bin/python. You can check with `rpm --eval '%{__os_install_post}'`.

This stops the execution early for python3 only packages.


Example
-------

build.log of a python3 only package:

https://koji.fedoraproject.org/koji/buildinfo?buildID=826617
https://kojipkgs.fedoraproject.org/packages/python-zeroconf/0.17.6/1.fc26/data/logs/noarch/build.log

Search for /usr/lib/rpm/brp-python-bytecompile:

    + /usr/lib/rpm/brp-python-bytecompile /usr/bin/python 1
    + /usr/lib/rpm/brp-python-hardlink

The brp-python-bytecompile script has no output.



python2 subpackage added:

https://koji.fedoraproject.org/koji/buildinfo?buildID=828568
https://kojipkgs.fedoraproject.org/packages/python-zeroconf/0.17.6/2.fc26/data/logs/noarch/build.log

    + /usr/lib/rpm/brp-python-bytecompile /usr/bin/python 1
    Bytecompiling .py files below /builddir/build/BUILDROOT/python-zeroconf-0.17.6-2.fc26.noarch/usr/lib/python3.5 using /usr/bin/python3.5
    Bytecompiling .py files below /builddir/build/BUILDROOT/python-zeroconf-0.17.6-2.fc26.noarch/usr/lib/python2.7 using /usr/bin/python2.7
    + /usr/lib/rpm/brp-python-hardlink

This might be harmless for most packages using %py3_install macro, because that should bytecompile all the files installed to site-packages anyway.
But it might be problematic for not so common packages where stuff is manually copied into site-packages. Or, when you change timestamps after %py3_install.


What can happen as a result
---------------------------

I've discovered this bug during a build of python3 only package "esptool" that had:

    %install
    %py3_install
    ln -s ./%{name}.py %{buildroot}%{_bindir}/%{name}
    
    # Shebang in site-packages
    sed -i 1d %{buildroot}%{python3_sitelib}/%{name}.py

After the build, rpmlint discovered some errors:

    esptool.noarch: E: python-bytecode-inconsistent-mtime /usr/lib/python3.6/site-packages/__pycache__/esptool.cpython-36.pyc 2017-01-10T03:02:16 /usr/lib/python3.6/site-packages/esptool.py 2017-01-10T03:02:18
    esptool.noarch: E: python-bytecode-inconsistent-mtime /usr/lib/python3.6/site-packages/__pycache__/esptool.cpython-36.opt-1.pyc 2017-01-10T03:02:16 /usr/lib/python3.6/site-packages/esptool.py 2017-01-10T03:02:18

(The time difference here is introduced by a sleep invocation at the end of %py3_install.)

I've managed to workaround the issue by moving the sed command to %build as follows:

    %build
    %py3_build
    
    # Shebang in site-packages
    sed -i 1d build/lib/%{name}.py

(Note that I was not able to do it in %prep as would be the natural choice.)

Nevertheless this should not happen.


Possible solution
-----------------

The /usr/lib/rpm/brp-python-bytecompile script does two things:

 1. bytecompiles stuff in site-packages with the appropriate Python version
 2. bytecompiles stuff elsewhere with $default_python (aka %{__python} aka mostly /usr/bin/python if not redefined in spec)

Clearly, it cannot do 2. without $default_python being there.
However, it can do 1. quite well without it.

I think (but I have not tested it), that moving the detection of $default_python presence just bellow 1. might fix the problem.
That is right above the line that says:

    # Handle other locations in the filesystem using the default python


Steps to reproduce
------------------

...not included as the result is obvious from the logs.
But if needed, it can be reproduced by building any python3 only package (such as for example esptool). Rpmlint errors can be reproduced by adding sleep and touch after %py3_install.




Version-Release number
----------------------

4.13.0-10.fc26 (manually checked the sources)
4.13.0-9.fc26 (happens to me in mock)
4.13.0-6.fc25 (on my machine)
4.13.0-1.fc24 (manually checked the sources)
4.11.3-21.el7 (tested with CentOS based EPEL7 mock)
4.8.0-55.el6 (tested with CentOS 6 docker)

Comment 1 Miro Hrončok 2017-01-10 02:44:29 UTC
s/exists/exits/ in tl;dr

Comment 2 Florian Festi 2017-02-15 10:38:09 UTC
Fixed upstream in a8e51b3bb05c6acb1d9b2e3d34f859ddda1677be

Added patch to rawhide. Please test.

Comment 3 Igor Gnatenko 2017-02-15 17:54:38 UTC
(In reply to Florian Festi from comment #2)
> Fixed upstream in a8e51b3bb05c6acb1d9b2e3d34f859ddda1677be
> 
> Added patch to rawhide. Please test.

Does fix problem.

Comment 4 Stephen Gallagher 2017-02-15 19:05:17 UTC
Yes, I can also confirm that the patch to Rawhide fixed the issue when I built rolekit.

Comment 5 Miro Hrončok 2017-02-16 11:14:26 UTC
https://koji.fedoraproject.org/koji/taskinfo?taskID=17897509

+ /usr/lib/rpm/brp-python-bytecompile /usr/bin/python 1
Bytecompiling .py files below /builddir/build/BUILDROOT/esptool-1.3-1.fc26.noarch/usr/lib/python3.6 using /usr/bin/python3.6

Fixed in rawhide.

Comment 6 Petr Viktorin 2017-02-27 12:07:18 UTC
Would it make sense to backport to older Fedoras?

Comment 7 Fedora End Of Life 2017-02-28 10:55:02 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 26 development cycle.
Changing version to '26'.

Comment 8 Igor Gnatenko 2017-07-24 07:12:25 UTC
(In reply to Petr Viktorin from comment #6)
> Would it make sense to backport to older Fedoras?

Probably not, because /usr/bin/python didn't disappear from buildroot.


Note You need to log in before you can comment on or make changes to this bug.