Description of problem:
'dnf update' cannot save bandwidth due to wrong DRPM of kernel-devel.
Version-Release number of selected component (if applicable):
Always with 'kernel-devel'. Other DRPMs run fine. I assume the DRPM is not created correctly.
Steps to Reproduce:
1. Download RPM for kernel-devel-5.7.17-200.fc.x86_64
2. Download DRPM for kernel-devel-5.7.17-200.fc32_5.8.4-200.fc32.x86_64.drpm
3. Check MD5s
4. applydeltarpm -vvvvv -p -r kernel-devel-5.7.17-200.fc32.x86_64.rpm kernel-devel-5.7.17-200.fc32_5.8.4-200.fc32.x86_64.drpm kernel-devel-5.8.4-200.fc32.x86_64.rpm
(same without '-r ker...')
56404376 bytes source payload size
58154416 bytes target payload size
3234393 bytes internal data size
468502 bytes add data size
52094 copy instructions
100 percent finished.
used 5000 core pages
used 1827 swap pages
had to recreate 0 core pages
kernel-devel-5.7.17-200.fc32_5.8.4-200.fc32.x86_64.drpm: md5 mismatch of result
rpm successfully created
It is not the first time that a DRPM of kernel-devel cannot be applied.
dnf downloads the full RPM as recovery.
I have two machines that show the same behavior as described by Joerg. All drpms are created successfully, only 'kernel-devel' fails everytime. Result is that the full rpm has to be downloaded.
Today, I tried to reinstall the 'kernel-devel' package:
kernel-devel-5.8.6-201.fc32_5.8.7-200.fc32.x86_64.drpm 15 MB/s | 3.0 MB 00:00
/var/cache/dnf/updates-b3cb4614b6495970/packages/kernel-devel-5.8.6-201.fc32_5.8.7-200.fc32.x86_64.drpm: md5 mismatch of result
Some packages were not downloaded. Retrying.
kernel-devel-5.8.7-200.fc32.x86_64.rpm 33 MB/s | 13 MB 00:00
Total 1.4 MB/s | 16 MB 00:11
Failed Delta RPMs increased 13.3 MB of updates to 16.2 MB (-22.1% wasted)
This is more likely a bug in deltarpm, reassigning...
I've looked into this, and the problem is that, while running applydeltarpm creates a perfect byte-for-byte copy of the *uncompressed* payload, the *compressed* payload is different, thus failing the md5 check. Historically, this has happened when the compression library used to compress the original rpm isn't the same version as the library used by applydeltarpm. What's surprising about this is that it's only affecting kernel-devel. If there's a library mismatch, I would expect to see problems with most, if not all deltarpms. I'm cc'ing Neal as I'm hoping he might know if the kernel is built differently from everything else, which might explain this. Neal, if you don't know, feel free to clear the NEEDINFO.
Well, kernel.spec has this line in it:
Which means it forces the use of threaded compression to speed up the build. It might very well be the only component that does this, so that would explain why only the kernel packages seem to be affected.
I suppose the compressed result will depend on the number of threads used in case of threaded compression, which could explain the mismatch...
Yeah, that would do it. :( I notice that it's also using xz compression instead of zstd.
I'm not sure what to do with this. I wonder if there's some way to turn off deltarpm generation for the kernel rpm.
(In reply to Jonathan Dieter from comment #5)
> Yeah, that would do it. :( I notice that it's also using xz compression
> instead of zstd.
Yeah, I forgot that zstd is the default now... I believe the reason to stay with xz was that it compresses faster (and build times do matter in case of the kernel, at least on RHEL) and that zstd RPM compression doesn't support the threaded mode (or at least it didn't at the time the decision was made - might be worth re-testing).
> I'm not sure what to do with this. I wonder if there's some way to turn off
> deltarpm generation for the kernel rpm.
Alternatively, would it be somehow possible to compare the md5 of the uncompressed payload rather than compressed?
(In reply to Ondrej Mosnacek from comment #6)
> (In reply to Jonathan Dieter from comment #5)
> > Yeah, that would do it. :( I notice that it's also using xz compression
> > instead of zstd.
> Yeah, I forgot that zstd is the default now... I believe the reason to stay
> with xz was that it compresses faster (and build times do matter in case of
> the kernel, at least on RHEL) and that zstd RPM compression doesn't support
> the threaded mode (or at least it didn't at the time the decision was made -
> might be worth re-testing).
I noticed that deltarpm has grown the ability to handle threaded zstd, so I wonder if rpm can do it too now.
> > I'm not sure what to do with this. I wonder if there's some way to turn off
> > deltarpm generation for the kernel rpm.
> Alternatively, would it be somehow possible to compare the md5 of the
> uncompressed payload rather than compressed?
When we first started using deltarpms (and xz was still being changed enough that we were seeing this kind of problem on a regular basis), this was proposed. The problem is that the RPM's signature will also fail, and there was (probably rightly) massive resistance to moving the signature onto the uncompressed payload because that would allow unverified data to be decompressed, opening the door to a malicious compressed payload exploiting a vulnerability in the compression algorithm.
I might see if rpm now supports threaded zstd, and, if so, see what the build time difference is between threaded zstd and threaded xz (and verify that deltarpm can consistently rebuild threaded zstd).
RPM will support threaded zstd compression in RPM 4.17. We probably will want to have this backported so that RPMs and DeltaRPMs can use it.
Probably the following should be backported for our use:
ok. Updated builders.