Bug 2266767 - meson has a non-reproducible pyc file
Summary: meson has a non-reproducible pyc file
Keywords:
Status: ASSIGNED
Alias: None
Product: Fedora
Classification: Fedora
Component: python-rpm-macros
Version: rawhide
Hardware: noarch
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Miro Hrončok
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2024-02-28 22:54 UTC by Zbigniew Jędrzejewski-Szmek
Modified: 2024-03-27 13:22 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: ---
Embargoed:


Attachments (Terms of Use)
meson-1.3.2-1.fc41.noarch.rpm (1.96 MB, application/x-rpm)
2024-02-28 22:57 UTC, Zbigniew Jędrzejewski-Szmek
no flags Details
rpm from rebuild (1.96 MB, application/x-rpm)
2024-02-28 22:59 UTC, Zbigniew Jędrzejewski-Szmek
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Fedora Pagure fedora-reproducible-builds/project issue 12 0 None None None 2024-02-29 11:57:01 UTC

Description Zbigniew Jędrzejewski-Szmek 2024-02-28 22:54:46 UTC
I rebuilt meson-1.3.2-1.fc41 using https://github.com/keszybz/fedora-repro-build,
and rpmdiff reports that the archive contents are different:

meson-1.3.2-1.fc41.noarch:
....N...... /usr/lib/python3.12/site-packages/mesonbuild/__pycache__/environment.cpython-312.opt-1.pyc
S.5.N...... /usr/lib/python3.12/site-packages/mesonbuild/__pycache__/environment.cpython-312.pyc

This is strange. "N" means "inode", and file permissions and contents are actually identical for
'environment.cpython-312.opt-1.pyc'.

OTOH, contents of 'environment.cpython-312.pyc' are indeed different:

$ sha256sum /var/tmp/meson-1.3.2-1.fc41.*/usr/lib/python3.12/site-packages/mesonbuild/environment.*  /var/tmp/meson-1.3.2-1.fc41.*/usr/lib/python3.12/site-packages/mesonbuild/__pycache__/environment.*                                                                                                    
b6f65c5720361686672edf8f3f412baea7d5e6911360cf806b3f9d69761a15c8  /var/tmp/meson-1.3.2-1.fc41.1/usr/lib/python3.12/site-packages/mesonbuild/environment.py
b6f65c5720361686672edf8f3f412baea7d5e6911360cf806b3f9d69761a15c8  /var/tmp/meson-1.3.2-1.fc41.2/usr/lib/python3.12/site-packages/mesonbuild/environment.py
12efe362d8523f9d7c93e7128ef62cabb754d47cdfc19d82d931e25baf07829b  /var/tmp/meson-1.3.2-1.fc41.1/usr/lib/python3.12/site-packages/mesonbuild/__pycache__/environment.cpython-312.opt-1.pyc
e9b83e30f9404d3de0059f701b0c1787633bf744ce0340e64940f08484c2ca3d  /var/tmp/meson-1.3.2-1.fc41.1/usr/lib/python3.12/site-packages/mesonbuild/__pycache__/environment.cpython-312.pyc
12efe362d8523f9d7c93e7128ef62cabb754d47cdfc19d82d931e25baf07829b  /var/tmp/meson-1.3.2-1.fc41.2/usr/lib/python3.12/site-packages/mesonbuild/__pycache__/environment.cpython-312.opt-1.pyc
12efe362d8523f9d7c93e7128ef62cabb754d47cdfc19d82d931e25baf07829b  /var/tmp/meson-1.3.2-1.fc41.2/usr/lib/python3.12/site-packages/mesonbuild/__pycache__/environment.cpython-312.pyc

Unfortunately, diffoscope croaks on this rpm:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1064973 

I think we expect pyc contents to be reproducible, so this seems to be
some strangeness in python bytecode generation.

Reproducible: Always

Comment 1 Zbigniew Jędrzejewski-Szmek 2024-02-28 22:57:46 UTC
Created attachment 2019358 [details]
meson-1.3.2-1.fc41.noarch.rpm

Comment 2 Zbigniew Jędrzejewski-Szmek 2024-02-28 22:58:22 UTC
Comment on attachment 2019358 [details]
meson-1.3.2-1.fc41.noarch.rpm

rpm from koji

Comment 3 Zbigniew Jędrzejewski-Szmek 2024-02-28 22:59:02 UTC
Created attachment 2019359 [details]
rpm from rebuild

Comment 4 Miro Hrončok 2024-02-28 23:39:21 UTC
Fedora's meson-1.3.2-1.fc41 was built on ppc64le. Is it possible that yours was built on your x86_64 machine? Both attached RPMs say the host is buildvm-ppc64le-31.iad2.fedoraproject.org but perhaps this was faked by https://github.com/keszybz/fedora-repro-build ?


.pyc files built on different architectures are not guaranteed to be bit-by-bit identical. They are only guaranteed to be functionally identical.

See https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_Appendix/#_byte_compilation_reproducibility for details



The .pyc files should still be reproducible if built on the same architecture. If that is not the case, let me know.

Comment 5 Zbigniew Jędrzejewski-Szmek 2024-02-29 11:51:18 UTC
(In reply to Miro Hrončok from comment #4)
> Is it possible that yours was built on your x86_64 machine?
Yes. I'm setting %_buildhost to the original value, even though I'm building
on a normal amd64 machine.

> .pyc files built on different architectures are not guaranteed to be
> bit-by-bit identical. They are only guaranteed to be functionally identical.
> 
> See
> https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_Appendix/
> #_byte_compilation_reproducibility for details
Thanks. I have forgotten about this discussion.

So there is no bug, per se. I think we might still want to do something here
to avoid the issue. It matters for reproducible build testing, but also
it would make packaging of noarch packages easier without requiring maintainers
to know about and set %py_reproducible_pyc_path.

Comment 6 Miro Hrončok 2024-02-29 12:35:42 UTC
We could make the brp script automatic for the entire buildroot. The main problem is that marshalparser has a requirement on Python itself, so the brp script would need to be changed to invoke it with the available Python (assuming there is some). It would also complicate the bootstrap sequence unless we bundle the script to python-rpm-macros and hide it in /usr/lib/rpm. 

It is possible to experiment with this approach, try rebuilding all the Fedora packages with .pyc files, and see the impact on build time and buildability. However, we are already stretched thin, so it would probably require volunteers to drive this.

Comment 7 Zbigniew Jędrzejewski-Szmek 2024-02-29 14:11:30 UTC
> We could make the brp script automatic for the entire buildroot.
+1

> The main problem is that marshalparser has a requirement on Python itself, so the brp script would need to be changed to invoke it with the available Python (assuming there is some). It would also complicate the bootstrap sequence unless we bundle the script to python-rpm-macros and hide it in /usr/lib/rpm.
I would make it optional, i.e. skip the step if the tool is not available for whatever reason.

OK, I'll leave this open for now, but put it on my TODO list.
We're still gathering various things to fix and I there are other issues which are more visible.

Comment 8 Zbigniew Jędrzejewski-Szmek 2024-03-01 09:42:55 UTC
(In reply to Miro Hrončok from comment #6)
> We could make the brp script automatic for the entire buildroot.
I took a stab at implementing this. PTAL at
https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/170.


Note You need to log in before you can comment on or make changes to this bug.