Bug 1873876
Summary: | md5 mismatch result of kernel-devel drpm | ||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Joerg Stippa <j.stippa> | ||||||||||||||||||
Component: | deltarpm | Assignee: | Orphan Owner <extras-orphan> | ||||||||||||||||||
Status: | CLOSED EOL | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||||||||||||||||
Severity: | low | Docs Contact: | |||||||||||||||||||
Priority: | unspecified | ||||||||||||||||||||
Version: | 37 | CC: | acaringi, airlied, bskeggs, dev, hdegoede, ichavero, itamar, jarodwilson, jayjayjazz, jeremy, jglisse, john.j5live, jonathan, josef, kernel-maint, kevin, lgoncalv, linville, masami256, mchehab, mjg59, ngompa13, omar, omosnacek, pmatilai, richard.shadbolt, robatino, steved | ||||||||||||||||||
Target Milestone: | --- | ||||||||||||||||||||
Target Release: | --- | ||||||||||||||||||||
Hardware: | x86_64 | ||||||||||||||||||||
OS: | Linux | ||||||||||||||||||||
Whiteboard: | |||||||||||||||||||||
Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||||||||||||||
Doc Text: | Story Points: | --- | |||||||||||||||||||
Clone Of: | Environment: | ||||||||||||||||||||
Last Closed: | 2023-12-05 20:59:36 UTC | Type: | Bug | ||||||||||||||||||
Regression: | --- | Mount Type: | --- | ||||||||||||||||||
Documentation: | --- | CRM: | |||||||||||||||||||
Verified Versions: | Category: | --- | |||||||||||||||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||||||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||||||||||||||
Embargoed: | |||||||||||||||||||||
Attachments: |
|
Description
Joerg Stippa
2020-08-30 17:39:46 UTC
I have two machines that show the same behavior as described by Joerg. All drpms are created successfully, only 'kernel-devel' fails everytime. Result is that the full rpm has to be downloaded. Today, I tried to reinstall the 'kernel-devel' package: ====================================================================== Downloading Packages: kernel-devel-5.8.6-201.fc32_5.8.7-200.fc32.x86_64.drpm 15 MB/s | 3.0 MB 00:00 /var/cache/dnf/updates-b3cb4614b6495970/packages/kernel-devel-5.8.6-201.fc32_5.8.7-200.fc32.x86_64.drpm: md5 mismatch of result Some packages were not downloaded. Retrying. kernel-devel-5.8.7-200.fc32.x86_64.rpm 33 MB/s | 13 MB 00:00 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 1.4 MB/s | 16 MB 00:11 Failed Delta RPMs increased 13.3 MB of updates to 16.2 MB (-22.1% wasted) ====================================================================== This is more likely a bug in deltarpm, reassigning... I've looked into this, and the problem is that, while running applydeltarpm creates a perfect byte-for-byte copy of the *uncompressed* payload, the *compressed* payload is different, thus failing the md5 check. Historically, this has happened when the compression library used to compress the original rpm isn't the same version as the library used by applydeltarpm. What's surprising about this is that it's only affecting kernel-devel. If there's a library mismatch, I would expect to see problems with most, if not all deltarpms. I'm cc'ing Neal as I'm hoping he might know if the kernel is built differently from everything else, which might explain this. Neal, if you don't know, feel free to clear the NEEDINFO. Well, kernel.spec has this line in it: https://src.fedoraproject.org/rpms/kernel/blob/master/f/kernel.spec#_21 Which means it forces the use of threaded compression to speed up the build. It might very well be the only component that does this, so that would explain why only the kernel packages seem to be affected. I suppose the compressed result will depend on the number of threads used in case of threaded compression, which could explain the mismatch... Yeah, that would do it. :( I notice that it's also using xz compression instead of zstd. I'm not sure what to do with this. I wonder if there's some way to turn off deltarpm generation for the kernel rpm. (In reply to Jonathan Dieter from comment #5) > Yeah, that would do it. :( I notice that it's also using xz compression > instead of zstd. Yeah, I forgot that zstd is the default now... I believe the reason to stay with xz was that it compresses faster (and build times do matter in case of the kernel, at least on RHEL) and that zstd RPM compression doesn't support the threaded mode (or at least it didn't at the time the decision was made - might be worth re-testing). > I'm not sure what to do with this. I wonder if there's some way to turn off > deltarpm generation for the kernel rpm. Alternatively, would it be somehow possible to compare the md5 of the uncompressed payload rather than compressed? (In reply to Ondrej Mosnacek from comment #6) > (In reply to Jonathan Dieter from comment #5) > > Yeah, that would do it. :( I notice that it's also using xz compression > > instead of zstd. > > Yeah, I forgot that zstd is the default now... I believe the reason to stay > with xz was that it compresses faster (and build times do matter in case of > the kernel, at least on RHEL) and that zstd RPM compression doesn't support > the threaded mode (or at least it didn't at the time the decision was made - > might be worth re-testing). I noticed that deltarpm has grown the ability to handle threaded zstd, so I wonder if rpm can do it too now. > > I'm not sure what to do with this. I wonder if there's some way to turn off > > deltarpm generation for the kernel rpm. > > Alternatively, would it be somehow possible to compare the md5 of the > uncompressed payload rather than compressed? When we first started using deltarpms (and xz was still being changed enough that we were seeing this kind of problem on a regular basis), this was proposed. The problem is that the RPM's signature will also fail, and there was (probably rightly) massive resistance to moving the signature onto the uncompressed payload because that would allow unverified data to be decompressed, opening the door to a malicious compressed payload exploiting a vulnerability in the compression algorithm. I might see if rpm now supports threaded zstd, and, if so, see what the build time difference is between threaded zstd and threaded xz (and verify that deltarpm can consistently rebuild threaded zstd). RPM will support threaded zstd compression in RPM 4.17. We probably will want to have this backported so that RPMs and DeltaRPMs can use it. Probably the following should be backported for our use: * https://github.com/rpm-software-management/rpm/commit/aa06b9922d7990f164e241c1a33a6180bd036f36 * https://github.com/rpm-software-management/rpm/commit/c0064335f975c3664e4848596fba159a87b21104 ok. Updated builders. This message is a reminder that Fedora 32 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora 32 on 2021-05-25. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '32'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 32 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. I just noticed it again on F34. Moving version from F32 -> F34. Screen output: /var/cache/dnf/updates-1eb77e9f45b4391a/packages/kernel-devel-5.11.17-300.fc34_5.11.18-300.fc34.x86_64.drpm: md5 mismatch of result Some packages were not downloaded. Retrying. kernel-devel-5.11.18-300.fc34.x86_64.rpm Still valid for upgrading 5.11.19 to 5.11.20: /var/cache/dnf/updates-1eb77e9f45b4391a/packages/kernel-devel-5.11.19-300.fc34_5.11.20-300.fc34.x86_64.drpm: md5 mismatch of result Some packages were not downloaded. Retrying. Not just the kernel is affected: /var/cache/dnf/updates-1eb77e9f45b4391a/packages/cups-devel-2.3.3op2-5.fc34_2.3.3op2-7.fc34.x86_64.drpm: md5 mismatch of result /var/cache/dnf/updates-1eb77e9f45b4391a/packages/cups-libs-2.3.3op2-5.fc34_2.3.3op2-7.fc34.x86_64.drpm: md5 mismatch of result /var/cache/dnf/updates-1eb77e9f45b4391a/packages/cups-ipptool-2.3.3op2-5.fc34_2.3.3op2-7.fc34.x86_64.drpm: md5 mismatch of result /var/cache/dnf/updates-1eb77e9f45b4391a/packages/coreutils-common-8.32-24.fc34_8.32-26.fc34.x86_64.drpm: md5 mismatch of result /var/cache/dnf/updates-1eb77e9f45b4391a/packages/cups-2.3.3op2-5.fc34_2.3.3op2-7.fc34.x86_64.drpm: md5 mismatch of result More packages to be affected: /var/cache/dnf/updates-1eb77e9f45b4391a/packages/java-latest-openjdk-16.0.1.0.9-1.rolling.fc34_16.0.1.0.9-3.rolling.fc34.x86_64.drpm: md5 mismatch of result /var/cache/dnf/updates-1eb77e9f45b4391a/packages/libmwaw-0.3.18-1.fc34_0.3.19-1.fc34.x86_64.drpm: md5 mismatch of result /var/cache/dnf/updates-1eb77e9f45b4391a/packages/kernel-devel-5.11.20-300.fc34_5.11.21-300.fc34.x86_64.drpm: md5 mismatch of result The non-kernel packages failures are because zstd has been upgraded in F34 to 1.5.0, which is no longer byte-for-byte identical to 1.4.x. Kevin, could you please upgrade the builders to zstd-1.5.0? I know we have done this dance before, but by 'the builders' you mean the bodhi-backend01 machine that makes the updates and drpms? Or do you mean the builders themselves? (Except all builds are done in chroot with the latest / current packages for that release, so it should have the latest, right?) We need to update libzstd on the builders themselves since they're the servers that actually create and compress the rpms. As for the builds being done in a chroot, rpmbuild itself isn't in the chroot, right? Since it's what compresses the rpms, it needs to be using the latest version of libzstd. *** Bug 1911828 has been marked as a duplicate of this bug. *** I updated all the builders yesterday. Thanks so much, Kevin! In my latest F34 updates push, all 18 drpms built successfully. Looks like this is fixed. Created attachment 1848504 [details]
screen trace as seen by user/admin
Created attachment 1848505 [details]
internal log of dnf
Created attachment 1848506 [details]
internal log of dnf
Created attachment 1848507 [details]
internal log of dnf
Today I encountered some other packages with failing drpms on F35. I started with a system being uptodate as of 2021-12-31. Maybe the build server needs another fix, cf. "duplicate" ticket? deltarpm-3.6.2-10.fc35.x86_64 libzstd-1.5.1-3.fc35.x86_64.rpm Added a screen trace (as shown to the user/admin) and the internal logs of dnf. It looks like the builders have been updated to libzstd-1.5.1 and I'm no longer seeing failed drpms on F35. I think this has been sorted. No, still appears. I attached the dnf.log from today. The system was fully up to date from yesterday evening, but the restoration of kernel*rpm from the DRPMs failed. (And this is not the first time after I reported it again on 01/01.) Created attachment 1857793 [details]
/var/log/dnf.log of 2022-01-30
Note: Only kernel*drpms are failing. Others are running well: dnf.librepo.log: 2022-01-30T08:12:04+0100 INFO Downloading: http://mirror2.hs-esslingen.de/fedora/linux/updates/35/Everything/x86_64/drpms/kernel-core-5.15.16-200.fc35_5.15.17-200.fc35.x86_64.drpm 2022-01-30T08:12:04+0100 INFO Downloading: http://mirror2.hs-esslingen.de/fedora/linux/updates/35/Everything/x86_64/drpms/kernel-devel-5.15.16-200.fc35_5.15.17-200.fc35.x86_64.drpm 2022-01-30T08:12:04+0100 INFO Downloading: http://mirror2.hs-esslingen.de/fedora/linux/updates/35/Everything/x86_64/drpms/kernel-modules-5.15.16-200.fc35_5.15.17-200.fc35.x86_64.drpm 2022-01-30T08:12:06+0100 INFO Downloading: http://mirror2.hs-esslingen.de/fedora/linux/updates/35/Everything/x86_64/drpms/grub2-breeze-theme-5.23.4-1.fc35_5.23.5-1.fc35.x86_64.drpm 2022-01-30T08:12:06+0100 INFO Downloading: http://mirror2.hs-esslingen.de/fedora/linux/updates/35/Everything/x86_64/drpms/kf5-filesystem-5.89.0-1.fc35_5.90.0-1.fc35.x86_64.drpm 2022-01-30T08:12:06+0100 INFO Downloading: http://mirror2.hs-esslingen.de/fedora/linux/updates/35/Everything/x86_64/drpms/kf5-syntax-highlighting-5.89.0-1.fc35_5.90.0-1.fc35.x86_64.drpm dnf.log (see attachment https://bugzilla.redhat.com/attachment.cgi?id=1857793): 2022-01-30T08:12:04+0100 INFO DNF will only download packages for the transaction. 2022-01-30T08:12:04+0100 INFO Downloading Packages: 2022-01-30T08:12:06+0100 SUBDEBUG drpm: spawned 93578: /usr/bin/applydeltarpm -a x86_64 /var/cache/dnf/updates-7eea87b22825bc0d/packages/kernel-devel-5.15.16-200.fc35_5.15.17-200.fc35.x86_64.drpm /var/cache/dnf/updates-7eea87b22825bc0d/packages/kernel-devel-5.15.17-200.fc35.x86_64.rpm 2022-01-30T08:12:06+0100 SUBDEBUG drpm: spawned 93580: /usr/bin/applydeltarpm -a x86_64 /var/cache/dnf/updates-7eea87b22825bc0d/packages/grub2-breeze-theme-5.23.4-1.fc35_5.23.5-1.fc35.x86_64.drpm /var/cache/dnf/updates-7eea87b22825bc0d/packages/grub2-breeze-theme-5.23.5-1.fc35.x86_64.rpm 2022-01-30T08:12:06+0100 SUBDEBUG drpm: spawned 93582: /usr/bin/applydeltarpm -a x86_64 /var/cache/dnf/updates-7eea87b22825bc0d/packages/kf5-filesystem-5.89.0-1.fc35_5.90.0-1.fc35.x86_64.drpm /var/cache/dnf/updates-7eea87b22825bc0d/packages/kf5-filesystem-5.90.0-1.fc35.x86_64.rpm 2022-01-30T08:12:07+0100 SUBDEBUG drpm: 93582: return code: 0, 0 2022-01-30T08:12:07+0100 SUBDEBUG drpm: spawned 93584: /usr/bin/applydeltarpm -a x86_64 /var/cache/dnf/updates-7eea87b22825bc0d/packages/kf5-syntax-highlighting-5.89.0-1.fc35_5.90.0-1.fc35.x86_64.drpm /var/cache/dnf/updates-7eea87b22825bc0d/packages/kf5-syntax-highlighting-5.90.0-1.fc35.x86_64.rpm 2022-01-30T08:12:11+0100 SUBDEBUG drpm: 93584: return code: 0, 0 2022-01-30T08:12:11+0100 SUBDEBUG drpm: spawned 93586: /usr/bin/applydeltarpm -a x86_64 /var/cache/dnf/updates-7eea87b22825bc0d/packages/kernel-core-5.15.16-200.fc35_5.15.17-200.fc35.x86_64.drpm /var/cache/dnf/updates-7eea87b22825bc0d/packages/kernel-core-5.15.17-200.fc35.x86_64.rpm 2022-01-30T08:12:14+0100 SUBDEBUG drpm: spawned 93588: /usr/bin/applydeltarpm -a x86_64 /var/cache/dnf/updates-7eea87b22825bc0d/packages/kernel-modules-5.15.16-200.fc35_5.15.17-200.fc35.x86_64.drpm /var/cache/dnf/updates-7eea87b22825bc0d/packages/kernel-modules-5.15.17-200.fc35.x86_64.rpm 2022-01-30T08:12:17+0100 SUBDEBUG drpm: 93580: return code: 0, 0 2022-01-30T08:12:35+0100 SUBDEBUG drpm: 93586: return code: 1, 0 2022-01-30T08:12:38+0100 SUBDEBUG drpm: 93588: return code: 1, 0 2022-01-30T08:12:44+0100 SUBDEBUG drpm: 93578: return code: 1, 0 2022-01-30T08:12:44+0100 INFO Some packages were not downloaded. Retrying. I'm having the same problem. I've set deltarpm_precentage=95 in dnf and it works with all packages except kernel upgrades. md5 mismatch of result Hi, I always had an MD5 mismatch on kernel-devel, but for the past week, i am now getting this regularly on many packages. I am on F34. I cannot determine if it is a specific mirror. Others are getting this problem too. https://forums.fedoraforum.org/showthread.php?327986-DNF-Update-MD5-Mismatch-of-drpm-downloads Started on the 9th February 2022. Regards, Shadders. Hi, Apologies, just to add, the update to libzstd was applied on the 8th February 2022 - so has this package been updated with an error - seems to be the lossless compression library. Unsure of this is used by DNF ? Regards, Shadders. Hi, Just to chase this up. I checked the packages that use this library (https://fedora.pkgs.org/34/fedora-updates-x86_64/libzstd-1.5.2-1.fc34.x86_64.rpm.html) required by section, and there are many packages reliant upon by this package (error version in use 1.5.2-1.fc34). Does this not there create the problem that something compressed with this version, 1.5.2-1.fc34, will create a compressed file or other, that cannot be decompressed by another version ? We are already seeing it in failed drpm's, so it is not as if this is a black swan event. Regards, Shadders. Just a heads up that I've orphaned deltarpm and am moving this ticket back to NEW. Migrating my FAS address to omosnacek... This message is a reminder that Fedora Linux 35 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora Linux 35 on 2022-12-13. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a 'version' of '35'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, change the 'version' to a later Fedora Linux version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora Linux 35 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora Linux, you are encouraged to change the 'version' to a later version prior to this bug being closed. Still exists in F37! See attached screen-script, dnf.log, and dnf.librepo.log. before updating the kernel(-devel) all other packages are up-to-date. Created attachment 1930100 [details]
dnf output to terminal
Created attachment 1930101 [details]
dnf log F37
Created attachment 1930102 [details]
dnf.librepo log F37
This message is a reminder that Fedora Linux 37 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora Linux 37 on 2023-12-05. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a 'version' of '37'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, change the 'version' to a later Fedora Linux version. Note that the version field may be hidden. Click the "Show advanced fields" button if you do not see it. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora Linux 37 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora Linux, you are encouraged to change the 'version' to a later version prior to this bug being closed. Fedora Linux 37 entered end-of-life (EOL) status on None. Fedora Linux 37 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora Linux please feel free to reopen this bug against that version. Note that the version field may be hidden. Click the "Show advanced fields" button if you do not see the version field. If you are unable to reopen this bug, please file a new report against an active release. Thank you for reporting this bug and we are sorry it could not be fixed. |