Bug 1806625 - Empty debugsourcefiles.list when using %pyproject_wheel/install with extension modules
Summary: Empty debugsourcefiles.list when using %pyproject_wheel/install with extensio...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: pyproject-rpm-macros
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Miro Hrončok
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-02-24 16:21 UTC by Patrik Kopkan
Modified: 2020-05-10 04:50 UTC (History)
5 users (show)

Fixed In Version: pyproject-rpm-macros-0-14.fc32 pyproject-rpm-macros-0-14.fc30 pyproject-rpm-macros-0-14.fc31
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-05-06 04:29:22 UTC
Type: Bug


Attachments (Terms of Use)
python-ldap.spec (2.96 KB, text/plain)
2020-02-24 16:21 UTC, Patrik Kopkan
no flags Details
build.log (156.34 KB, text/plain)
2020-02-24 16:30 UTC, Patrik Kopkan
no flags Details

Description Patrik Kopkan 2020-02-24 16:21:08 UTC
Created attachment 1665476 [details]
python-ldap.spec

Description of problem:
When adapting python extension to use pyproject-rpm-macros. I have found issue at the end of build. 

Version-Release number of selected component (if applicable):


How reproducible:
Use %pyproject_wheel and %pyproject_install in specfile

Comment 1 Patrik Kopkan 2020-02-24 16:30:43 UTC
Created attachment 1665477 [details]
build.log

Comment 2 Miro Hrončok 2020-02-24 16:34:22 UTC
Version-Release number of pyproject-rpm-macros:  pyproject-rpm-macros-0-12.fc33


Simpler reproducer:



Name:           python-mistune
Version:        0.8.3
Release:        11%{?dist}
Summary:        Markdown parser for Python 

License:        BSD
URL:            https://github.com/lepture/mistune
Source0:        %{url}/archive/v%{version}.tar.gz

BuildRequires:  gcc
BuildRequires:  pyproject-rpm-macros

# optional dependency, listed explicitly to have the extension module:
BuildRequires:  python3-Cython

%description
...


%package -n python3-mistune
Summary:        %summary
%{?python_provide:%python_provide python3-mistune}

%description -n python3-mistune
...

%prep
%autosetup -n mistune-%{version}

%generate_buildrequires
%pyproject_buildrequires

%build
%pyproject_wheel

%install
%pyproject_install

%files -n python3-mistune
%doc README.rst
%license LICENSE
%{python3_sitearch}/mistune-%{version}.dist-info/
%pycached %{python3_sitearch}/mistune.py
%{python3_sitearch}/mistune.cpython-*.so




Error:

RPM build errors:
error: Empty %files file /builddir/build/BUILD/mistune-0.8.3/debugsourcefiles.list
    Empty %files file /builddir/build/BUILD/mistune-0.8.3/debugsourcefiles.list

Comment 3 Miro Hrončok 2020-02-24 16:43:57 UTC
There are two main causes I can think of:

 - the sources are located somewhere weird
 - the so files are built without the proper flags

Comment 4 Miro Hrončok 2020-02-28 13:54:11 UTC
We do:

	
CFLAGS="${CFLAGS:-${RPM_OPT_FLAGS}}" LDFLAGS="${LDFLAGS:-${RPM_LD_FLAGS}}" \\\
%{__python3} -m pip wheel


And wheel utilizes subprocess, not forwarding the flags inside. We might just need to export the shell variables.

Comment 5 Miro Hrončok 2020-02-28 14:30:35 UTC
OK, this was not the problem. Even in current build log:

Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.8rqofx
+ umask 022
+ cd /builddir/build/BUILD
+ cd mistune-0.8.3
+ CFLAGS='-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection'
+ LDFLAGS='-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld'
+ /usr/bin/python3 -m pip wheel --wheel-dir ./pyproject-macros-wheeldir --no-deps --use-pep517 --no-build-isolation --disable-pip-version-check --progress-bar off --verbose .
...
  gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fPIC -I/usr/include/python3.8 -c mistune.c -o build/temp.linux-x86_64-3.8/mistune.o
  gcc -pthread -shared -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -g -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -g -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection build/temp.linux-x86_64-3.8/mistune.o -L/usr/lib64 -o build/lib.linux-x86_64-3.8/mistune.cpython-38-x86_64-linux-gnu.so
  installing to build/bdist.linux-x86_64/wheel



I have verified that the gcc command contains the CFLAGS and LDFLAGS (note that the LDFLAGS contain double space but the gcc command only has one).

So, no idea for now why debugsourcefiles.list is empty :(

Comment 6 Petr Viktorin 2020-03-04 12:56:46 UTC
The find-debuginfo script is looking for source files in  /usr/src/debug in the buildroot. No idea yet what should put be putting them there.

Comment 7 Miro Hrončok 2020-03-04 13:14:23 UTC
AFAIK It is /usr/lib/rpm/find-debuginfo.sh itself to put it there.

FTR:

%build
%py3_build


%install
%py3_install
NOT YET HERE: /usr/src/debug/

%check
PRESENT HERE:  /usr/src/debug/
%{__python3} setup.py test

Comment 8 Petr Viktorin 2020-03-04 15:11:00 UTC
Guess I found it.
pip copies the sources to a temporary directory (/tmp/pip-req-build-*), builds there, and then removes the temp directory -- see build.log
So, rpmbuild can't find the sources and generates an empty source list. This makes things fail later.
But it seems that even if the temporary directory wasn't removed, rpmbuild wouldn't find the sources: it expects them to be in %{_builddir}!


Details:

/usr/src/debug/ is indeed populated later in find-debuginfo, from debugsources.list (SOURCEFILE).
debugsources.list is generated by debugedit:
https://github.com/rpm-software-management/rpm/blob/af06db1d5558870f7fb8f5c502572c2d27af5c71/scripts/find-debuginfo.sh#L389

I managed to run debugedit under gdb in (hopefully) the same way find-debuginfo runs it, near the point where an entry should be written into SOURCEFILE (here called list_file):
https://github.com/rpm-software-management/rpm/blob/af06db1d5558870f7fb8f5c502572c2d27af5c71/tools/debugedit.c#L1592

at this point, `s` points to "/tmp/pip-req-build-148fqu74/Modules/LDAPObject.c"
This begins with neither base_dir ("/builddir/build/BUILD/python-ldap-3.1.0") nor dest_dir ("/usr/src/debug/python-ldap-3.1.0-9.fc33.x86_64"). `skip_dir_prefix` returns NULL if the prefix doesn't match, so `p` ends up NULL and nothing is written to the file.

Nothing is then copied to /usr/src/debug/:
https://github.com/rpm-software-management/rpm/blob/af06db1d5558870f7fb8f5c502572c2d27af5c71/scripts/find-debuginfo.sh#L578-L579

and nothing is listed in debugsourcefiles.list (a.k.a. srcout):
https://github.com/rpm-software-management/rpm/blob/af06db1d5558870f7fb8f5c502572c2d27af5c71/scripts/find-debuginfo.sh#L601-L605

Comment 9 Petr Viktorin 2020-03-04 15:14:59 UTC
So if I'm right, I guess I have two reproducers to make and two upstream issues to file:

- rpmbuild blows up if C stuff is compiled from only files outside %{_builddir}

- pip's copying of C sources results in temporary filenames being recorded in debug info

Comment 10 Miro Hrončok 2020-03-04 15:20:12 UTC
> it expects them to be in %{_builddir}

Yes, we've been hit by this quite often when a py2+py3 package with %py3dir and a native extension removed the py2 package (%py3dir is outside of %builddir).

I don't think this was ever considered a bug of rpmbuild: build should happen in %{_builddir}.

Can we workaround this by setting the TMPDIR, TEMP or TMP environment variables to %{_builddir}/.tmp ?

Comment 11 Miro Hrončok 2020-03-04 15:27:48 UTC
> and a native extension removed the py2 package

and a native extension, and when we removed the py2 package

Comment 12 Petr Viktorin 2020-03-04 16:18:36 UTC
> Can we workaround this by setting the TMPDIR, TEMP or TMP environment variables to %{_builddir}/.tmp ?

It's probably close (and yet so far) to working with:

    mkdir -p %{_builddir}/.tmp
    export TMPDIR=%{_builddir}/.tmp

and adding --no-clean to the pip build and install options.
Unfortunately, rpmbuild passes --unique-debug-src-base to find-debuginfo, so the sources must be in $BUILDDIR ($PWD, %{_builddir}/%{name}-%{version}), not just in %{_builddir} ($RPM_BUILD_DIR):
https://github.com/rpm-software-management/rpm/blob/af06db1d5558870f7fb8f5c502572c2d27af5c71/scripts/find-debuginfo.sh#L379-L384


If I use `$PWD/.tmp` instead, then pip tries to copy the sources ($PWD) recursively to `$PWD/.tmp/pip-req-build-XXXXXXXX`, which recurses inifinitely (until it blows up on an overlong filename).

Comment 13 Miro Hrončok 2020-03-04 16:21:13 UTC
Your findings are consistent with mine:

shutil.Error: [('/builddir/build/BUILD/mistune-0.8.3/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/.tmp/pip-req-build-ex3kyjh4/...

Comment 14 Petr Viktorin 2020-03-04 16:39:38 UTC
One thing I learned today is that there is a tool that can rewrite directory names in debuginfo. It could be used for a hacky workaround:
add --no-clean to the pip build and install options;
set up the temp directory:
    mkdir -p %{_builddir}/.tmp
    export TMPDIR=%{_builddir}/.tmp
and after %pyproject_install:
    find %{buildroot} -iname '*.so' -print0 | xargs --no-run-if-empty -0 -n1 /usr/lib/rpm/debugedit -b "%{_builddir}/.tmp/pip-req-build-"* -d "$PWD"


... which actualy gives me a debugsource RPM with the sources.

Comment 15 Miro Hrončok 2020-03-04 16:50:08 UTC
Nice! Are the sources copied with full paths? I mean, does this work properly with src/ directories?

Comment 16 Miro Hrončok 2020-03-04 16:52:50 UTC
And a second concern: does this leave a mess? Where/when can we cleanup?

Comment 17 Petr Viktorin 2020-03-04 16:57:53 UTC
I assume src/ is OK, didn't check.

I guess cleanup can be right after the hack; the temporary directory is only needed for the glob to determine how it was named.

The big issue is that this depends on lots of implementation details: debugedit API and location (and existence), the "pip-req-build" name, and the mechanism with which pip does builds.

Maybe we can argue with pip upstream that bogus debuginfo filenames are an issue, but I still don't see a good solution.



---

The ASSIGNED → NEW was a mistake, but I guess it was right: unfortunately I won't be able to give this another big chunk of time until next week, at earliest. All I know about the issue is written here.

Comment 18 Miro Hrončok 2020-03-05 10:41:05 UTC
I'll try to chat with pip people about this to see what can be improved on pip's side.

Comment 19 Miro Hrončok 2020-03-05 11:28:44 UTC
https://github.com/pypa/pip/issues/7555#issuecomment-595180864

Reading that issue, it is likely that pip will eventually build in-place. Until then, we can go with that hack and add some integration tests to ensure the fragile thing is not blowing up.

Comment 20 Miro Hrončok 2020-03-09 22:33:41 UTC
I am puzzled but I was not able to put this into the macros. Possibly I am just tired.

  %pyproject_wheel() %{expand:\\\
+# for proper debugsource packages, we do a tmpdir hack described in
+# https://bugzilla.redhat.com/show_bug.cgi?id=1806625
+export TMPDIR="%{_builddir}/.tmp"
+rm -rfv "${TMPDIR}"
+mkdir -p "${TMPDIR}"
 CFLAGS="${CFLAGS:-${RPM_OPT_FLAGS}}" LDFLAGS="${LDFLAGS:-${RPM_LD_FLAGS}}" \\\
-%{__python3} -m pip wheel --wheel-dir %{_pyproject_wheeldir} --no-deps --use-pep517 --no-build-isolation --disable-pip-version-check --progress-bar off --verbose .
+%{__python3} -m pip wheel --wheel-dir %{_pyproject_wheeldir} --no-deps --use-pep517 --no-build-isolation --disable-pip-version-check --no-clean --progress-bar off --verbose .
 }
 
 
@@ -18,6 +23,8 @@ fi
 if [ -d %{buildroot}%{python3_sitearch} ]; then
   sed -i 's/pip/rpm/' %{buildroot}%{python3_sitearch}/*.dist-info/INSTALLER
 fi
+# debugsource hack from %%pyproject_wheel continues here
+find %{buildroot} -iname '*.so' -print0 | xargs --no-run-if-empty -0 -n1 /usr/lib/rpm/debugedit -b "%{_builddir}/.tmp/pip-req-build-"* -d "${PWD}"
 }

Comment 21 Petr Viktorin 2020-03-18 16:33:24 UTC
It's.... the macro in the comment:
    # debugsource hack from %%pyproject_wheel continues here
the %% is apparently not enough to escape this, so %pyproject_install runs %pyproject_wheel again, deleting the old directory and creating a new one.

Without the % in comment, python-ldap built for me.

This patch worked for me:

--- a/macros.pyproject
+++ b/macros.pyproject
@@ -1,8 +1,13 @@
 %_pyproject_wheeldir ./pyproject-macros-wheeldir
 
 %pyproject_wheel() %{expand:\\\
+# for proper debugsource packages, we do a tmpdir hack described in
+# https://bugzilla.redhat.com/show_bug.cgi?id=1806625
+export TMPDIR="%{_builddir}/.tmp-%{name}"
+rm -rfv "${TMPDIR}"
+mkdir -p "${TMPDIR}"
 CFLAGS="${CFLAGS:-${RPM_OPT_FLAGS}}" LDFLAGS="${LDFLAGS:-${RPM_LD_FLAGS}}" \\\
-%{__python3} -m pip wheel --wheel-dir %{_pyproject_wheeldir} --no-deps --use-pep517 --no-build-isolation --disable-pip-version-check --progress-bar off --verbose .
+%{__python3} -m pip wheel --wheel-dir %{_pyproject_wheeldir} --no-deps --use-pep517 --no-build-isolation --disable-pip-version-check --no-clean --progress-bar off --verbose .
 }
 
 
@@ -18,6 +23,8 @@ fi
 if [ -d %{buildroot}%{python3_sitearch} ]; then
   sed -i 's/pip/rpm/' %{buildroot}%{python3_sitearch}/*.dist-info/INSTALLER
 fi
+# debugsource hack from the pyproject_wheel macro continues here
+find %{buildroot} -iname '*.so' -print0 | xargs --no-run-if-empty -0 -n1 /usr/lib/rpm/debugedit -b "%{_builddir}/.tmp-%{name}/pip-req-build-"* -d "${PWD}"
 }
 
 %default_toxenv py%{python3_version_nodots}

Comment 22 Miro Hrončok 2020-03-18 16:34:49 UTC
Thanks!

Comment 24 Petr Viktorin 2020-03-18 17:49:21 UTC
With this patch, python-cryptography (a package with extensions and src layout) built successfully, but not correctly: the debuginfo-source package only included a directory, no files.

I'm not sure if this is specific to cryptography.

Alas, I'm out of time again for a while. If you'd like to investigate, the "macros" branch in my fork has a pyproject_macros-ized spec for cryptography:
https://src.fedoraproject.org/fork/pviktori/rpms/python-cryptography/blob/macros/f/python-cryptography.spec

Comment 25 Miro Hrončok 2020-03-18 18:07:17 UTC
mistune behaves the same:

$ rpm -qlp /var/lib/mock/fedora-rawhide-x86_64/result/python-mistune-debugsource-0.8.3-1.fc33.x86_64.rpm 
/usr/src/debug/python-mistune-0.8.3-1.fc33.x86_64

Comment 26 Petr Viktorin 2020-03-19 15:26:27 UTC
So, now the culprit is generated sources, as you get with CFFI (cryptography) or Cython (mistune).
A build-in-place mode for pip would be nice. In the mean time, we could copy back the intermediate files at the end of %pyproject_wheel:

    # The build directory could contain generated sources
    # from e.g. Cython and CFFI, so copy new files back to builddir
    cp  -rv --no-clobber %{_builddir}/.tmp-%{name}/pip-req-build-*/* ./

I wonder how many more hacks this will need.

Comment 27 Miro Hrončok 2020-03-19 15:45:59 UTC
I wonder whether we should explore the option to change pip (upstream would likely accept it) and backport the change to our packaged pip.

Comment 28 Miro Hrončok 2020-03-19 15:47:59 UTC
Or, make pip not recurse itself in here:

> If I use `$PWD/.tmp` instead, then pip tries to copy the sources ($PWD) recursively to `$PWD/.tmp/pip-req-build-XXXXXXXX`, which recurses inifinitely (until it blows up on an overlong filename).

Comment 29 Miro Hrončok 2020-03-19 17:04:35 UTC
https://github.com/pypa/pip/pull/7873

Comment 30 Petr Viktorin 2020-04-01 15:34:08 UTC
We're giving upstream time to merge. If that takes too long or the issue starts blocking something, we can add a Fedora patch.

Comment 31 Fedora Update System 2020-04-24 16:52:18 UTC
FEDORA-2020-d0700e9fb2 has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2020-d0700e9fb2

Comment 32 Fedora Update System 2020-04-24 16:57:47 UTC
FEDORA-2020-8386117468 has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2020-8386117468

Comment 33 Fedora Update System 2020-04-24 17:04:18 UTC
FEDORA-2020-0f078e7280 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2020-0f078e7280

Comment 34 Fedora Update System 2020-04-25 04:19:11 UTC
FEDORA-2020-8386117468 has been pushed to the Fedora 31 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-8386117468`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-8386117468

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 35 Fedora Update System 2020-04-25 04:55:36 UTC
FEDORA-2020-0f078e7280 has been pushed to the Fedora 30 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-0f078e7280`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-0f078e7280

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 36 Fedora Update System 2020-04-25 12:21:21 UTC
FEDORA-2020-d0700e9fb2 has been pushed to the Fedora 32 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-d0700e9fb2`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-d0700e9fb2

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 37 Fedora Update System 2020-05-06 04:29:22 UTC
FEDORA-2020-d0700e9fb2 has been pushed to the Fedora 32 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 38 Fedora Update System 2020-05-10 04:33:54 UTC
FEDORA-2020-0f078e7280 has been pushed to the Fedora 30 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 39 Fedora Update System 2020-05-10 04:50:40 UTC
FEDORA-2020-8386117468 has been pushed to the Fedora 31 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.