Bug 1953910 - Since rpm-4.16.90-0.git15395, python-srpm-macros in Lua are no longer found
Summary: Since rpm-4.16.90-0.git15395, python-srpm-macros in Lua are no longer found
Alias: None
Product: Fedora
Classification: Fedora
Component: rpm
Version: rawhide
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
Assignee: Packaging Maintenance Team
QA Contact: Fedora Extras Quality Assurance
Depends On:
Blocks: PYTHON3.10 RPM4.17
TreeView+ depends on / blocked
Reported: 2021-04-27 07:24 UTC by Miro Hrončok
Modified: 2021-05-05 15:15 UTC (History)
9 users (show)

Fixed In Version: rpm-4.16.90-0.git15395.3.fc35
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2021-04-27 15:00:57 UTC
Type: Bug

Attachments (Terms of Use)
A diff between old and new `rpm --showrc` in an otherwise identical mock (20.81 KB, patch)
2021-04-27 07:35 UTC, Miro Hrončok
no flags Details | Diff

System ID Private Priority Status Summary Last Updated
Github rpm-software-management rpm issues 1659 0 None open Macro file comment loading "regression" 2021-04-27 08:29:51 UTC

Description Miro Hrončok 2021-04-27 07:24:13 UTC
Install python-srpm-macros-3.9-37.fc35 on rawhide.

The macro file is /usr/lib/rpm/macros.d/macros.python-srpm

With rpm-, I get:

# rpm --eval %pypi_source

# rpm --eval '%py3_dist foo'

# rpm --eval '%py_dist_name foo_bar'

# rpm --eval %python_extras_subpkg
error: %python_extras_subpkg: missing option -n (name of the base package)
error: lua script failed: [string "<lua>"]:12: error expanding macro

With rpm-4.16.90-0.git15395.2.fc35.x86_64, I get:

# rpm --eval %pypi_source

# rpm --eval '%py3_dist foo'
%py3_dist foo

# rpm --eval '%py_dist_name foo_bar'
%py_dist_name foo_bar

# rpm --eval %python_extras_subpkg

This makes at least 1000 Fedora packages fails to build from source (most likely fail to build the SRPM) with errors like:

error: Bad source: /builddir/build/SOURCES/%{pypi_source}: No such file or directory

Comment 1 Miro Hrončok 2021-04-27 07:27:52 UTC
Non-Lua macros from the same file work as before:

# rpm --eval %python3

# rpm --eval %python3_pkgversion

Lua macros from different files still work:

# rpm --eval '%pycached %{python3_sitelib}/foo.py'  # from /usr/lib/rpm/macros.d/macros.python3

Comment 2 Miro Hrončok 2021-04-27 07:32:00 UTC
This blocks our Python 3.10 rebuild.

It can also make packages build fine, but suddenly no longer build a subpackage with https://fedoraproject.org/wiki/Changes/PythonExtras -- but only for packages that only use (conditional) %python_extras_subpkg and not %pypi_source or %py3_dist.

Comment 3 Miro Hrončok 2021-04-27 07:35:41 UTC
Created attachment 1775859 [details]
A diff between old and new `rpm --showrc` in an otherwise identical mock

I've attached a diff between old and new `rpm --showrc` in an otherwise identical mock.

Comment 4 Panu Matilainen 2021-04-27 07:50:57 UTC
Yup, easily reproduced. 

It's a regression introduced in https://github.com/rpm-software-management/rpm/commit/75275a87cff04da65d3557f2c40ea2b526528c4c and what it trips over is this comment line in that macro file:

>  Set to /bin/true to avoid %ifdefs and %{? in specfiles

I'll revert the patch in rawhide to let builds continue and look for the proper fix afterwards. Just a sec.

Comment 5 Miro Hrončok 2021-04-27 07:53:38 UTC
I've just debugged that macros above `# Set to /bin/true to avoid %ifdefs and %{? in specfiles` work fine and macros below don't. You were faster

Should I escape the % in the comment? That sound s safer.

Comment 6 Miro Hrončok 2021-04-27 08:01:25 UTC
Like this: https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/97

Comment 7 Panu Matilainen 2021-04-27 08:06:23 UTC
It's a good question.

In *spec* comments, macros are always expanded, but I don't recall offhand what should happen in macro file comments.

Feel free to escape though, it's probably the best immediate fix. There is a bug in rpm nevertheless (either the %{ should be ignored in a comment or there should be an error for missing }).

Comment 8 Panu Matilainen 2021-04-27 08:29:28 UTC
Building new rpm with the problematic patch reverted now: https://koji.fedoraproject.org/koji/taskinfo?taskID=66775049

Comment 9 Miro Hrončok 2021-04-27 15:00:57 UTC
Thanks for the revert.

I'll also land https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/97

Comment 10 Jason Tibbitts 2021-04-27 15:55:30 UTC
I wonder if it's better to just use %dnl "comments" in macro files instead of worrying about escaping rules.

Comment 11 Panu Matilainen 2021-04-28 05:18:37 UTC
%dnl will *not* work in macro files because macro files are not expanded as they're read.

Comment 12 Panu Matilainen 2021-04-30 06:26:40 UTC
So, as discussed in the upstream ticket, this wasn't so much a regression but unearthing an ancient bug - a bunch of macros in the same file were always being skipped due to the bug, but an otherwise unrelated change which permits macro body to include empty lines dragged the bug into the spotlight.

Upstream fix was to not let comments affect parsing. I've reverted the revert, and added the upstream fix instead. Was about to close this but then it already was.

Comment 13 Fedora Update System 2021-05-05 15:15:41 UTC
FEDORA-2021-831cfbf901 has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.