Bug 1873876

Summary: md5 mismatch result of kernel-devel drpm
Product: [Fedora] Fedora Reporter: Joerg Stippa <j.stippa>
Component: deltarpmAssignee: Orphan Owner <extras-orphan>
Status: CLOSED EOL QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: low Docs Contact:
Priority: unspecified    
Version: 37CC: acaringi, airlied, bskeggs, dev, hdegoede, ichavero, itamar, jarodwilson, jayjayjazz, jeremy, jglisse, john.j5live, jonathan, josef, kernel-maint, kevin, lgoncalv, linville, masami256, mchehab, mjg59, ngompa13, omar, omosnacek, pmatilai, richard.shadbolt, robatino, steved
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-12-05 20:59:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
screen trace as seen by user/admin
none
internal log of dnf
none
internal log of dnf
none
internal log of dnf
none
/var/log/dnf.log of 2022-01-30
none
dnf output to terminal
none
dnf log F37
none
dnf.librepo log F37 none

Description Joerg Stippa 2020-08-30 17:39:46 UTC
Description of problem:
'dnf update' cannot save bandwidth due to wrong DRPM of kernel-devel.

Version-Release number of selected component (if applicable):
deltarpm-3.6.2-5.fc32.x86_64.rpm
kernel-devel-5.7.17-200.fc32.x86_64.rpm

How reproducible:
Always with 'kernel-devel'. Other DRPMs run fine. I assume the DRPM is not created correctly.

Steps to Reproduce:
1. Download RPM for kernel-devel-5.7.17-200.fc.x86_64
2. Download DRPM for kernel-devel-5.7.17-200.fc32_5.8.4-200.fc32.x86_64.drpm
3. Check MD5s
   md5sum kernel-devel-5.7.17-200.*
   f1807238b12b4710e0f9e47ecb3ca42d  kernel-devel-5.7.17-200.fc32_5.8.4-200.fc32.x86_64.drpm
   6940e2254314300ea38bfd8fef39a224  kernel-devel-5.7.17-200.fc32.x86_64.rpm
4. applydeltarpm -vvvvv -p -r kernel-devel-5.7.17-200.fc32.x86_64.rpm kernel-devel-5.7.17-200.fc32_5.8.4-200.fc32.x86_64.drpm kernel-devel-5.8.4-200.fc32.x86_64.rpm
   (same without '-r ker...')

Actual results:
reading deltarpm
56404376 bytes source payload size
58154416 bytes target payload size
3234393 bytes internal data size
468502 bytes add data size
6886 blocks
52094 copy instructions
applying delta
100 percent finished.
used 5000 core pages
used 1827 swap pages
had to recreate 0 core pages
kernel-devel-5.7.17-200.fc32_5.8.4-200.fc32.x86_64.drpm: md5 mismatch of result

Expected results:
rpm successfully created

Additional info:
It is not the first time that a DRPM of kernel-devel cannot be applied.
dnf downloads the full RPM as recovery.

Comment 1 JayJayJazz 2020-09-11 21:21:33 UTC
I have two machines that show the same behavior as described by Joerg. All drpms are created successfully, only 'kernel-devel' fails everytime. Result is that the full rpm has to be downloaded.

Today, I tried to reinstall the 'kernel-devel' package:

======================================================================
Downloading Packages:
kernel-devel-5.8.6-201.fc32_5.8.7-200.fc32.x86_64.drpm                                                                                     15 MB/s | 3.0 MB     00:00    
/var/cache/dnf/updates-b3cb4614b6495970/packages/kernel-devel-5.8.6-201.fc32_5.8.7-200.fc32.x86_64.drpm: md5 mismatch of result
Some packages were not downloaded. Retrying.
kernel-devel-5.8.7-200.fc32.x86_64.rpm                                                                                                     33 MB/s |  13 MB     00:00    
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                     1.4 MB/s |  16 MB     00:11     
Failed Delta RPMs increased 13.3 MB of updates to 16.2 MB (-22.1% wasted)
======================================================================

Comment 2 Ondrej Mosnacek 2020-10-16 11:16:31 UTC
This is more likely a bug in deltarpm, reassigning...

Comment 3 Jonathan Dieter 2020-10-19 10:34:21 UTC
I've looked into this, and the problem is that, while running applydeltarpm creates a perfect byte-for-byte copy of the *uncompressed* payload, the *compressed* payload is different, thus failing the md5 check.  Historically, this has happened when the compression library used to compress the original rpm isn't the same version as the library used by applydeltarpm.  What's surprising about this is that it's only affecting kernel-devel.  If there's a library mismatch, I would expect to see problems with most, if not all deltarpms.  I'm cc'ing Neal as I'm hoping he might know if the kernel is built differently from everything else, which might explain this.  Neal, if you don't know, feel free to clear the NEEDINFO.

Comment 4 Ondrej Mosnacek 2020-10-19 10:46:13 UTC
Well, kernel.spec has this line in it:
https://src.fedoraproject.org/rpms/kernel/blob/master/f/kernel.spec#_21

Which means it forces the use of threaded compression to speed up the build. It might very well be the only component that does this, so that would explain why only the kernel packages seem to be affected.

I suppose the compressed result will depend on the number of threads used in case of threaded compression, which could explain the mismatch...

Comment 5 Jonathan Dieter 2020-10-19 19:22:04 UTC
Yeah, that would do it.  :(  I notice that it's also using xz compression instead of zstd.

I'm not sure what to do with this.  I wonder if there's some way to turn off deltarpm generation for the kernel rpm.

Comment 6 Ondrej Mosnacek 2020-10-19 19:53:01 UTC
(In reply to Jonathan Dieter from comment #5)
> Yeah, that would do it.  :(  I notice that it's also using xz compression
> instead of zstd.

Yeah, I forgot that zstd is the default now... I believe the reason to stay with xz was that it compresses faster (and build times do matter in case of the kernel, at least on RHEL) and that zstd RPM compression doesn't support the threaded mode (or at least it didn't at the time the decision was made - might be worth re-testing).

> I'm not sure what to do with this.  I wonder if there's some way to turn off
> deltarpm generation for the kernel rpm.

Alternatively, would it be somehow possible to compare the md5 of the uncompressed payload rather than compressed?

Comment 7 Jonathan Dieter 2020-10-19 21:31:34 UTC
(In reply to Ondrej Mosnacek from comment #6)
> (In reply to Jonathan Dieter from comment #5)
> > Yeah, that would do it.  :(  I notice that it's also using xz compression
> > instead of zstd.
> 
> Yeah, I forgot that zstd is the default now... I believe the reason to stay
> with xz was that it compresses faster (and build times do matter in case of
> the kernel, at least on RHEL) and that zstd RPM compression doesn't support
> the threaded mode (or at least it didn't at the time the decision was made -
> might be worth re-testing).

I noticed that deltarpm has grown the ability to handle threaded zstd, so I wonder if rpm can do it too now.

> > I'm not sure what to do with this.  I wonder if there's some way to turn off
> > deltarpm generation for the kernel rpm.
> 
> Alternatively, would it be somehow possible to compare the md5 of the
> uncompressed payload rather than compressed?

When we first started using deltarpms (and xz was still being changed enough that we were seeing this kind of problem on a regular basis), this was proposed.  The problem is that the RPM's signature will also fail, and there was (probably rightly) massive resistance to moving the signature onto the uncompressed payload because that would allow unverified data to be decompressed, opening the door to a malicious compressed payload exploiting a vulnerability in the compression algorithm.

I might see if rpm now supports threaded zstd, and, if so, see what the build time difference is between threaded zstd and threaded xz (and verify that deltarpm can consistently rebuild threaded zstd).

Comment 8 Neal Gompa 2020-10-21 02:34:59 UTC
RPM will support threaded zstd compression in RPM 4.17. We probably will want to have this backported so that RPMs and DeltaRPMs can use it.

Probably the following should be backported for our use:

* https://github.com/rpm-software-management/rpm/commit/aa06b9922d7990f164e241c1a33a6180bd036f36
* https://github.com/rpm-software-management/rpm/commit/c0064335f975c3664e4848596fba159a87b21104

Comment 9 Kevin Fenzi 2021-01-08 01:02:55 UTC
ok. Updated builders.

Comment 10 Fedora Program Management 2021-04-29 17:11:13 UTC
This message is a reminder that Fedora 32 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 32 on 2021-05-25.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '32'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 32 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 11 Joerg Stippa 2021-05-08 13:18:29 UTC
I just noticed it again on F34. Moving version from F32 -> F34.
Screen output:
/var/cache/dnf/updates-1eb77e9f45b4391a/packages/kernel-devel-5.11.17-300.fc34_5.11.18-300.fc34.x86_64.drpm: md5 mismatch of result
Some packages were not downloaded. Retrying.
kernel-devel-5.11.18-300.fc34.x86_64.rpm

Comment 12 Joerg Stippa 2021-05-16 07:51:29 UTC
Still valid for upgrading 5.11.19 to 5.11.20:
/var/cache/dnf/updates-1eb77e9f45b4391a/packages/kernel-devel-5.11.19-300.fc34_5.11.20-300.fc34.x86_64.drpm: md5 mismatch of result
Some packages were not downloaded. Retrying.

Comment 13 Joerg Stippa 2021-05-19 04:38:06 UTC
Not just the kernel is affected:
/var/cache/dnf/updates-1eb77e9f45b4391a/packages/cups-devel-2.3.3op2-5.fc34_2.3.3op2-7.fc34.x86_64.drpm: md5 mismatch of result
/var/cache/dnf/updates-1eb77e9f45b4391a/packages/cups-libs-2.3.3op2-5.fc34_2.3.3op2-7.fc34.x86_64.drpm: md5 mismatch of result
/var/cache/dnf/updates-1eb77e9f45b4391a/packages/cups-ipptool-2.3.3op2-5.fc34_2.3.3op2-7.fc34.x86_64.drpm: md5 mismatch of result
/var/cache/dnf/updates-1eb77e9f45b4391a/packages/coreutils-common-8.32-24.fc34_8.32-26.fc34.x86_64.drpm: md5 mismatch of result
/var/cache/dnf/updates-1eb77e9f45b4391a/packages/cups-2.3.3op2-5.fc34_2.3.3op2-7.fc34.x86_64.drpm: md5 mismatch of result

Comment 14 Joerg Stippa 2021-05-20 05:59:10 UTC
More packages to be affected:
/var/cache/dnf/updates-1eb77e9f45b4391a/packages/java-latest-openjdk-16.0.1.0.9-1.rolling.fc34_16.0.1.0.9-3.rolling.fc34.x86_64.drpm: md5 mismatch of result
/var/cache/dnf/updates-1eb77e9f45b4391a/packages/libmwaw-0.3.18-1.fc34_0.3.19-1.fc34.x86_64.drpm: md5 mismatch of result
/var/cache/dnf/updates-1eb77e9f45b4391a/packages/kernel-devel-5.11.20-300.fc34_5.11.21-300.fc34.x86_64.drpm: md5 mismatch of result

Comment 15 Jonathan Dieter 2021-05-20 19:07:55 UTC
The non-kernel packages failures are because zstd has been upgraded in F34 to 1.5.0, which is no longer byte-for-byte identical to 1.4.x.  Kevin, could you please upgrade the builders to zstd-1.5.0?

Comment 16 Kevin Fenzi 2021-05-20 22:56:21 UTC
I know we have done this dance before, but by 'the builders' you mean the bodhi-backend01 machine that makes the updates and drpms?

Or do you mean the builders themselves? (Except all builds are done in chroot with the latest / current packages for that release, so it should have the latest, right?)

Comment 17 Jonathan Dieter 2021-05-21 12:41:31 UTC
We need to update libzstd on the builders themselves since they're the servers that actually create and compress the rpms.  As for the builds being done in a chroot, rpmbuild itself isn't in the chroot, right?  Since it's what compresses the rpms, it needs to be using the latest version of libzstd.

Comment 18 Jonathan Dieter 2021-05-21 12:42:38 UTC
*** Bug 1911828 has been marked as a duplicate of this bug. ***

Comment 19 Kevin Fenzi 2021-05-22 17:39:52 UTC
I updated all the builders yesterday.

Comment 20 Jonathan Dieter 2021-05-23 15:02:46 UTC
Thanks so much, Kevin!

Comment 21 Andre Robatino 2021-05-28 02:11:33 UTC
In my latest F34 updates push, all 18 drpms built successfully. Looks like this is fixed.

Comment 22 Joerg Stippa 2022-01-01 07:53:01 UTC
Created attachment 1848504 [details]
screen trace as seen by user/admin

Comment 23 Joerg Stippa 2022-01-01 07:53:52 UTC
Created attachment 1848505 [details]
internal log of dnf

Comment 24 Joerg Stippa 2022-01-01 07:54:32 UTC
Created attachment 1848506 [details]
internal log of dnf

Comment 25 Joerg Stippa 2022-01-01 07:54:56 UTC
Created attachment 1848507 [details]
internal log of dnf

Comment 26 Joerg Stippa 2022-01-01 07:59:41 UTC
Today I encountered some other packages with failing drpms on F35. I started with a system being uptodate as of 2021-12-31. Maybe the build server needs another fix, cf. "duplicate" ticket?
deltarpm-3.6.2-10.fc35.x86_64
libzstd-1.5.1-3.fc35.x86_64.rpm
Added a screen trace (as shown to the user/admin) and the internal logs of dnf.

Comment 27 Jonathan Dieter 2022-01-08 11:58:32 UTC
It looks like the builders have been updated to libzstd-1.5.1 and I'm no longer seeing failed drpms on F35.  I think this has been sorted.

Comment 28 Joerg Stippa 2022-01-30 07:31:08 UTC
No, still appears. I attached the dnf.log from today. The system was fully up to date from yesterday evening, but the restoration of kernel*rpm from the DRPMs failed. (And this is not the first time after I reported it again on 01/01.)

Comment 29 Joerg Stippa 2022-01-30 07:32:40 UTC
Created attachment 1857793 [details]
/var/log/dnf.log of 2022-01-30

Comment 30 Joerg Stippa 2022-01-30 07:38:51 UTC
Note: Only kernel*drpms are failing. Others are running well:
dnf.librepo.log:
2022-01-30T08:12:04+0100 INFO Downloading: http://mirror2.hs-esslingen.de/fedora/linux/updates/35/Everything/x86_64/drpms/kernel-core-5.15.16-200.fc35_5.15.17-200.fc35.x86_64.drpm
2022-01-30T08:12:04+0100 INFO Downloading: http://mirror2.hs-esslingen.de/fedora/linux/updates/35/Everything/x86_64/drpms/kernel-devel-5.15.16-200.fc35_5.15.17-200.fc35.x86_64.drpm
2022-01-30T08:12:04+0100 INFO Downloading: http://mirror2.hs-esslingen.de/fedora/linux/updates/35/Everything/x86_64/drpms/kernel-modules-5.15.16-200.fc35_5.15.17-200.fc35.x86_64.drpm
2022-01-30T08:12:06+0100 INFO Downloading: http://mirror2.hs-esslingen.de/fedora/linux/updates/35/Everything/x86_64/drpms/grub2-breeze-theme-5.23.4-1.fc35_5.23.5-1.fc35.x86_64.drpm
2022-01-30T08:12:06+0100 INFO Downloading: http://mirror2.hs-esslingen.de/fedora/linux/updates/35/Everything/x86_64/drpms/kf5-filesystem-5.89.0-1.fc35_5.90.0-1.fc35.x86_64.drpm
2022-01-30T08:12:06+0100 INFO Downloading: http://mirror2.hs-esslingen.de/fedora/linux/updates/35/Everything/x86_64/drpms/kf5-syntax-highlighting-5.89.0-1.fc35_5.90.0-1.fc35.x86_64.drpm

dnf.log (see attachment https://bugzilla.redhat.com/attachment.cgi?id=1857793):
2022-01-30T08:12:04+0100 INFO DNF will only download packages for the transaction.
2022-01-30T08:12:04+0100 INFO Downloading Packages:
2022-01-30T08:12:06+0100 SUBDEBUG drpm: spawned 93578: /usr/bin/applydeltarpm -a x86_64 /var/cache/dnf/updates-7eea87b22825bc0d/packages/kernel-devel-5.15.16-200.fc35_5.15.17-200.fc35.x86_64.drpm /var/cache/dnf/updates-7eea87b22825bc0d/packages/kernel-devel-5.15.17-200.fc35.x86_64.rpm
2022-01-30T08:12:06+0100 SUBDEBUG drpm: spawned 93580: /usr/bin/applydeltarpm -a x86_64 /var/cache/dnf/updates-7eea87b22825bc0d/packages/grub2-breeze-theme-5.23.4-1.fc35_5.23.5-1.fc35.x86_64.drpm /var/cache/dnf/updates-7eea87b22825bc0d/packages/grub2-breeze-theme-5.23.5-1.fc35.x86_64.rpm
2022-01-30T08:12:06+0100 SUBDEBUG drpm: spawned 93582: /usr/bin/applydeltarpm -a x86_64 /var/cache/dnf/updates-7eea87b22825bc0d/packages/kf5-filesystem-5.89.0-1.fc35_5.90.0-1.fc35.x86_64.drpm /var/cache/dnf/updates-7eea87b22825bc0d/packages/kf5-filesystem-5.90.0-1.fc35.x86_64.rpm
2022-01-30T08:12:07+0100 SUBDEBUG drpm: 93582: return code: 0, 0
2022-01-30T08:12:07+0100 SUBDEBUG drpm: spawned 93584: /usr/bin/applydeltarpm -a x86_64 /var/cache/dnf/updates-7eea87b22825bc0d/packages/kf5-syntax-highlighting-5.89.0-1.fc35_5.90.0-1.fc35.x86_64.drpm /var/cache/dnf/updates-7eea87b22825bc0d/packages/kf5-syntax-highlighting-5.90.0-1.fc35.x86_64.rpm
2022-01-30T08:12:11+0100 SUBDEBUG drpm: 93584: return code: 0, 0
2022-01-30T08:12:11+0100 SUBDEBUG drpm: spawned 93586: /usr/bin/applydeltarpm -a x86_64 /var/cache/dnf/updates-7eea87b22825bc0d/packages/kernel-core-5.15.16-200.fc35_5.15.17-200.fc35.x86_64.drpm /var/cache/dnf/updates-7eea87b22825bc0d/packages/kernel-core-5.15.17-200.fc35.x86_64.rpm
2022-01-30T08:12:14+0100 SUBDEBUG drpm: spawned 93588: /usr/bin/applydeltarpm -a x86_64 /var/cache/dnf/updates-7eea87b22825bc0d/packages/kernel-modules-5.15.16-200.fc35_5.15.17-200.fc35.x86_64.drpm /var/cache/dnf/updates-7eea87b22825bc0d/packages/kernel-modules-5.15.17-200.fc35.x86_64.rpm
2022-01-30T08:12:17+0100 SUBDEBUG drpm: 93580: return code: 0, 0
2022-01-30T08:12:35+0100 SUBDEBUG drpm: 93586: return code: 1, 0
2022-01-30T08:12:38+0100 SUBDEBUG drpm: 93588: return code: 1, 0
2022-01-30T08:12:44+0100 SUBDEBUG drpm: 93578: return code: 1, 0
2022-01-30T08:12:44+0100 INFO Some packages were not downloaded. Retrying.

Comment 31 Omar Kotb 2022-02-17 11:09:53 UTC
I'm having the same problem. I've set deltarpm_precentage=95 in dnf and it works with all packages except kernel upgrades. md5 mismatch of result

Comment 32 Shadders 2022-02-18 19:18:21 UTC
Hi,
I always had an MD5 mismatch on kernel-devel, but for the past week, i am now getting this regularly on many packages. 

I am on F34. I cannot determine if it is a specific mirror. Others are getting this problem too. 

https://forums.fedoraforum.org/showthread.php?327986-DNF-Update-MD5-Mismatch-of-drpm-downloads

Started on the 9th February 2022.

Regards,
Shadders.

Comment 33 Shadders 2022-02-18 19:24:17 UTC
Hi,
Apologies, just to add, the update to libzstd was applied on the 8th February 2022 - so has this package been updated with an error - seems to be the lossless compression library. Unsure of this is used by DNF ?

Regards,
Shadders.

Comment 34 Shadders 2022-02-24 13:40:30 UTC
Hi,
Just to chase this up. I checked the packages that use this library (https://fedora.pkgs.org/34/fedora-updates-x86_64/libzstd-1.5.2-1.fc34.x86_64.rpm.html) required by section, and there are many packages reliant upon by this package (error version in use 1.5.2-1.fc34).

Does this not there create the problem that something compressed with this version, 1.5.2-1.fc34, will create a compressed file or other, that cannot be decompressed by another version ?

We are already seeing it in failed drpm's, so it is not as if this is a black swan event. 

Regards,
Shadders.

Comment 35 Jonathan Dieter 2022-03-06 14:01:51 UTC
Just a heads up that I've orphaned deltarpm and am moving this ticket back to NEW.

Comment 36 Ondrej Mosnacek 2022-08-26 08:51:38 UTC
Migrating my FAS address to omosnacek...

Comment 37 Ben Cotton 2022-11-29 16:49:22 UTC
This message is a reminder that Fedora Linux 35 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 35 on 2022-12-13.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '35'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 35 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.

Comment 38 Joerg Stippa 2022-12-05 13:04:50 UTC
Still exists in F37!
See attached screen-script, dnf.log, and dnf.librepo.log. before updating the kernel(-devel) all other packages are up-to-date.

Comment 39 Joerg Stippa 2022-12-05 13:07:51 UTC
Created attachment 1930100 [details]
dnf output to terminal

Comment 40 Joerg Stippa 2022-12-05 13:08:20 UTC
Created attachment 1930101 [details]
dnf log F37

Comment 41 Joerg Stippa 2022-12-05 13:08:56 UTC
Created attachment 1930102 [details]
dnf.librepo log F37

Comment 42 Aoife Moloney 2023-11-23 00:03:37 UTC
This message is a reminder that Fedora Linux 37 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 37 on 2023-12-05.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '37'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version. Note that the version field may be hidden.
Click the "Show advanced fields" button if you do not see it.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 37 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.

Comment 43 Aoife Moloney 2023-12-05 20:59:36 UTC
Fedora Linux 37 entered end-of-life (EOL) status on None.

Fedora Linux 37 is no longer maintained, which means that it
will not receive any further security or bug fix updates. As a result we
are closing this bug.

If you can reproduce this bug against a currently maintained version of Fedora Linux
please feel free to reopen this bug against that version. Note that the version
field may be hidden. Click the "Show advanced fields" button if you do not see
the version field.

If you are unable to reopen this bug, please file a new report against an
active release.

Thank you for reporting this bug and we are sorry it could not be fixed.