Created attachment 1665476 [details] python-ldap.spec Description of problem: When adapting python extension to use pyproject-rpm-macros. I have found issue at the end of build. Version-Release number of selected component (if applicable): How reproducible: Use %pyproject_wheel and %pyproject_install in specfile
Created attachment 1665477 [details] build.log
Version-Release number of pyproject-rpm-macros: pyproject-rpm-macros-0-12.fc33 Simpler reproducer: Name: python-mistune Version: 0.8.3 Release: 11%{?dist} Summary: Markdown parser for Python License: BSD URL: https://github.com/lepture/mistune Source0: %{url}/archive/v%{version}.tar.gz BuildRequires: gcc BuildRequires: pyproject-rpm-macros # optional dependency, listed explicitly to have the extension module: BuildRequires: python3-Cython %description ... %package -n python3-mistune Summary: %summary %{?python_provide:%python_provide python3-mistune} %description -n python3-mistune ... %prep %autosetup -n mistune-%{version} %generate_buildrequires %pyproject_buildrequires %build %pyproject_wheel %install %pyproject_install %files -n python3-mistune %doc README.rst %license LICENSE %{python3_sitearch}/mistune-%{version}.dist-info/ %pycached %{python3_sitearch}/mistune.py %{python3_sitearch}/mistune.cpython-*.so Error: RPM build errors: error: Empty %files file /builddir/build/BUILD/mistune-0.8.3/debugsourcefiles.list Empty %files file /builddir/build/BUILD/mistune-0.8.3/debugsourcefiles.list
There are two main causes I can think of: - the sources are located somewhere weird - the so files are built without the proper flags
We do: CFLAGS="${CFLAGS:-${RPM_OPT_FLAGS}}" LDFLAGS="${LDFLAGS:-${RPM_LD_FLAGS}}" \\\ %{__python3} -m pip wheel And wheel utilizes subprocess, not forwarding the flags inside. We might just need to export the shell variables.
OK, this was not the problem. Even in current build log: Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.8rqofx + umask 022 + cd /builddir/build/BUILD + cd mistune-0.8.3 + CFLAGS='-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' + LDFLAGS='-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld' + /usr/bin/python3 -m pip wheel --wheel-dir ./pyproject-macros-wheeldir --no-deps --use-pep517 --no-build-isolation --disable-pip-version-check --progress-bar off --verbose . ... gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fPIC -I/usr/include/python3.8 -c mistune.c -o build/temp.linux-x86_64-3.8/mistune.o gcc -pthread -shared -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -g -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -g -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection build/temp.linux-x86_64-3.8/mistune.o -L/usr/lib64 -o build/lib.linux-x86_64-3.8/mistune.cpython-38-x86_64-linux-gnu.so installing to build/bdist.linux-x86_64/wheel I have verified that the gcc command contains the CFLAGS and LDFLAGS (note that the LDFLAGS contain double space but the gcc command only has one). So, no idea for now why debugsourcefiles.list is empty :(
The find-debuginfo script is looking for source files in /usr/src/debug in the buildroot. No idea yet what should put be putting them there.
AFAIK It is /usr/lib/rpm/find-debuginfo.sh itself to put it there. FTR: %build %py3_build %install %py3_install NOT YET HERE: /usr/src/debug/ %check PRESENT HERE: /usr/src/debug/ %{__python3} setup.py test
Guess I found it. pip copies the sources to a temporary directory (/tmp/pip-req-build-*), builds there, and then removes the temp directory -- see build.log So, rpmbuild can't find the sources and generates an empty source list. This makes things fail later. But it seems that even if the temporary directory wasn't removed, rpmbuild wouldn't find the sources: it expects them to be in %{_builddir}! Details: /usr/src/debug/ is indeed populated later in find-debuginfo, from debugsources.list (SOURCEFILE). debugsources.list is generated by debugedit: https://github.com/rpm-software-management/rpm/blob/af06db1d5558870f7fb8f5c502572c2d27af5c71/scripts/find-debuginfo.sh#L389 I managed to run debugedit under gdb in (hopefully) the same way find-debuginfo runs it, near the point where an entry should be written into SOURCEFILE (here called list_file): https://github.com/rpm-software-management/rpm/blob/af06db1d5558870f7fb8f5c502572c2d27af5c71/tools/debugedit.c#L1592 at this point, `s` points to "/tmp/pip-req-build-148fqu74/Modules/LDAPObject.c" This begins with neither base_dir ("/builddir/build/BUILD/python-ldap-3.1.0") nor dest_dir ("/usr/src/debug/python-ldap-3.1.0-9.fc33.x86_64"). `skip_dir_prefix` returns NULL if the prefix doesn't match, so `p` ends up NULL and nothing is written to the file. Nothing is then copied to /usr/src/debug/: https://github.com/rpm-software-management/rpm/blob/af06db1d5558870f7fb8f5c502572c2d27af5c71/scripts/find-debuginfo.sh#L578-L579 and nothing is listed in debugsourcefiles.list (a.k.a. srcout): https://github.com/rpm-software-management/rpm/blob/af06db1d5558870f7fb8f5c502572c2d27af5c71/scripts/find-debuginfo.sh#L601-L605
So if I'm right, I guess I have two reproducers to make and two upstream issues to file: - rpmbuild blows up if C stuff is compiled from only files outside %{_builddir} - pip's copying of C sources results in temporary filenames being recorded in debug info
> it expects them to be in %{_builddir} Yes, we've been hit by this quite often when a py2+py3 package with %py3dir and a native extension removed the py2 package (%py3dir is outside of %builddir). I don't think this was ever considered a bug of rpmbuild: build should happen in %{_builddir}. Can we workaround this by setting the TMPDIR, TEMP or TMP environment variables to %{_builddir}/.tmp ?
> and a native extension removed the py2 package and a native extension, and when we removed the py2 package
> Can we workaround this by setting the TMPDIR, TEMP or TMP environment variables to %{_builddir}/.tmp ? It's probably close (and yet so far) to working with: mkdir -p %{_builddir}/.tmp export TMPDIR=%{_builddir}/.tmp and adding --no-clean to the pip build and install options. Unfortunately, rpmbuild passes --unique-debug-src-base to find-debuginfo, so the sources must be in $BUILDDIR ($PWD, %{_builddir}/%{name}-%{version}), not just in %{_builddir} ($RPM_BUILD_DIR): https://github.com/rpm-software-management/rpm/blob/af06db1d5558870f7fb8f5c502572c2d27af5c71/scripts/find-debuginfo.sh#L379-L384 If I use `$PWD/.tmp` instead, then pip tries to copy the sources ($PWD) recursively to `$PWD/.tmp/pip-req-build-XXXXXXXX`, which recurses inifinitely (until it blows up on an overlong filename).
Your findings are consistent with mine: shutil.Error: [('/builddir/build/BUILD/mistune-0.8.3/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/...
One thing I learned today is that there is a tool that can rewrite directory names in debuginfo. It could be used for a hacky workaround: add --no-clean to the pip build and install options; set up the temp directory: mkdir -p %{_builddir}/.tmp export TMPDIR=%{_builddir}/.tmp and after %pyproject_install: find %{buildroot} -iname '*.so' -print0 | xargs --no-run-if-empty -0 -n1 /usr/lib/rpm/debugedit -b "%{_builddir}/.tmp/pip-req-build-"* -d "$PWD" ... which actualy gives me a debugsource RPM with the sources.
Nice! Are the sources copied with full paths? I mean, does this work properly with src/ directories?
And a second concern: does this leave a mess? Where/when can we cleanup?
I assume src/ is OK, didn't check. I guess cleanup can be right after the hack; the temporary directory is only needed for the glob to determine how it was named. The big issue is that this depends on lots of implementation details: debugedit API and location (and existence), the "pip-req-build" name, and the mechanism with which pip does builds. Maybe we can argue with pip upstream that bogus debuginfo filenames are an issue, but I still don't see a good solution. --- The ASSIGNED → NEW was a mistake, but I guess it was right: unfortunately I won't be able to give this another big chunk of time until next week, at earliest. All I know about the issue is written here.
I'll try to chat with pip people about this to see what can be improved on pip's side.
https://github.com/pypa/pip/issues/7555#issuecomment-595180864 Reading that issue, it is likely that pip will eventually build in-place. Until then, we can go with that hack and add some integration tests to ensure the fragile thing is not blowing up.
I am puzzled but I was not able to put this into the macros. Possibly I am just tired. %pyproject_wheel() %{expand:\\\ +# for proper debugsource packages, we do a tmpdir hack described in +# https://bugzilla.redhat.com/show_bug.cgi?id=1806625 +export TMPDIR="%{_builddir}/.tmp" +rm -rfv "${TMPDIR}" +mkdir -p "${TMPDIR}" CFLAGS="${CFLAGS:-${RPM_OPT_FLAGS}}" LDFLAGS="${LDFLAGS:-${RPM_LD_FLAGS}}" \\\ -%{__python3} -m pip wheel --wheel-dir %{_pyproject_wheeldir} --no-deps --use-pep517 --no-build-isolation --disable-pip-version-check --progress-bar off --verbose . +%{__python3} -m pip wheel --wheel-dir %{_pyproject_wheeldir} --no-deps --use-pep517 --no-build-isolation --disable-pip-version-check --no-clean --progress-bar off --verbose . } @@ -18,6 +23,8 @@ fi if [ -d %{buildroot}%{python3_sitearch} ]; then sed -i 's/pip/rpm/' %{buildroot}%{python3_sitearch}/*.dist-info/INSTALLER fi +# debugsource hack from %%pyproject_wheel continues here +find %{buildroot} -iname '*.so' -print0 | xargs --no-run-if-empty -0 -n1 /usr/lib/rpm/debugedit -b "%{_builddir}/.tmp/pip-req-build-"* -d "${PWD}" }
It's.... the macro in the comment: # debugsource hack from %%pyproject_wheel continues here the %% is apparently not enough to escape this, so %pyproject_install runs %pyproject_wheel again, deleting the old directory and creating a new one. Without the % in comment, python-ldap built for me. This patch worked for me: --- a/macros.pyproject +++ b/macros.pyproject @@ -1,8 +1,13 @@ %_pyproject_wheeldir ./pyproject-macros-wheeldir %pyproject_wheel() %{expand:\\\ +# for proper debugsource packages, we do a tmpdir hack described in +# https://bugzilla.redhat.com/show_bug.cgi?id=1806625 +export TMPDIR="%{_builddir}/.tmp-%{name}" +rm -rfv "${TMPDIR}" +mkdir -p "${TMPDIR}" CFLAGS="${CFLAGS:-${RPM_OPT_FLAGS}}" LDFLAGS="${LDFLAGS:-${RPM_LD_FLAGS}}" \\\ -%{__python3} -m pip wheel --wheel-dir %{_pyproject_wheeldir} --no-deps --use-pep517 --no-build-isolation --disable-pip-version-check --progress-bar off --verbose . +%{__python3} -m pip wheel --wheel-dir %{_pyproject_wheeldir} --no-deps --use-pep517 --no-build-isolation --disable-pip-version-check --no-clean --progress-bar off --verbose . } @@ -18,6 +23,8 @@ fi if [ -d %{buildroot}%{python3_sitearch} ]; then sed -i 's/pip/rpm/' %{buildroot}%{python3_sitearch}/*.dist-info/INSTALLER fi +# debugsource hack from the pyproject_wheel macro continues here +find %{buildroot} -iname '*.so' -print0 | xargs --no-run-if-empty -0 -n1 /usr/lib/rpm/debugedit -b "%{_builddir}/.tmp-%{name}/pip-req-build-"* -d "${PWD}" } %default_toxenv py%{python3_version_nodots}
Thanks!
https://src.fedoraproject.org/rpms/pyproject-rpm-macros/pull-request/43
With this patch, python-cryptography (a package with extensions and src layout) built successfully, but not correctly: the debuginfo-source package only included a directory, no files. I'm not sure if this is specific to cryptography. Alas, I'm out of time again for a while. If you'd like to investigate, the "macros" branch in my fork has a pyproject_macros-ized spec for cryptography: https://src.fedoraproject.org/fork/pviktori/rpms/python-cryptography/blob/macros/f/python-cryptography.spec
mistune behaves the same: $ rpm -qlp /var/lib/mock/fedora-rawhide-x86_64/result/python-mistune-debugsource-0.8.3-1.fc33.x86_64.rpm /usr/src/debug/python-mistune-0.8.3-1.fc33.x86_64
So, now the culprit is generated sources, as you get with CFFI (cryptography) or Cython (mistune). A build-in-place mode for pip would be nice. In the mean time, we could copy back the intermediate files at the end of %pyproject_wheel: # The build directory could contain generated sources # from e.g. Cython and CFFI, so copy new files back to builddir cp -rv --no-clobber %{_builddir}/.tmp-%{name}/pip-req-build-*/* ./ I wonder how many more hacks this will need.
I wonder whether we should explore the option to change pip (upstream would likely accept it) and backport the change to our packaged pip.
Or, make pip not recurse itself in here: > If I use `$PWD/.tmp` instead, then pip tries to copy the sources ($PWD) recursively to `$PWD/.tmp/pip-req-build-XXXXXXXX`, which recurses inifinitely (until it blows up on an overlong filename).
https://github.com/pypa/pip/pull/7873
We're giving upstream time to merge. If that takes too long or the issue starts blocking something, we can add a Fedora patch.
FEDORA-2020-d0700e9fb2 has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2020-d0700e9fb2
FEDORA-2020-8386117468 has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2020-8386117468
FEDORA-2020-0f078e7280 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2020-0f078e7280
FEDORA-2020-8386117468 has been pushed to the Fedora 31 testing repository. In short time you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-8386117468` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-8386117468 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2020-0f078e7280 has been pushed to the Fedora 30 testing repository. In short time you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-0f078e7280` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-0f078e7280 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2020-d0700e9fb2 has been pushed to the Fedora 32 testing repository. In short time you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-d0700e9fb2` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-d0700e9fb2 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2020-d0700e9fb2 has been pushed to the Fedora 32 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2020-0f078e7280 has been pushed to the Fedora 30 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2020-8386117468 has been pushed to the Fedora 31 stable repository. If problem still persists, please make note of it in this bug report.