Bug 1873876 - md5 mismatch result of kernel-devel drpm
Summary: md5 mismatch result of kernel-devel drpm
Keywords:
Status: ASSIGNED
Alias: None
Product: Fedora
Classification: Fedora
Component: deltarpm
Version: 32
Hardware: x86_64
OS: Linux
unspecified
low
Target Milestone: ---
Assignee: Jonathan Dieter
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-08-30 17:39 UTC by Joerg Stippa
Modified: 2021-01-08 01:02 UTC (History)
26 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Bug


Attachments (Terms of Use)

Description Joerg Stippa 2020-08-30 17:39:46 UTC
Description of problem:
'dnf update' cannot save bandwidth due to wrong DRPM of kernel-devel.

Version-Release number of selected component (if applicable):
deltarpm-3.6.2-5.fc32.x86_64.rpm
kernel-devel-5.7.17-200.fc32.x86_64.rpm

How reproducible:
Always with 'kernel-devel'. Other DRPMs run fine. I assume the DRPM is not created correctly.

Steps to Reproduce:
1. Download RPM for kernel-devel-5.7.17-200.fc.x86_64
2. Download DRPM for kernel-devel-5.7.17-200.fc32_5.8.4-200.fc32.x86_64.drpm
3. Check MD5s
   md5sum kernel-devel-5.7.17-200.*
   f1807238b12b4710e0f9e47ecb3ca42d  kernel-devel-5.7.17-200.fc32_5.8.4-200.fc32.x86_64.drpm
   6940e2254314300ea38bfd8fef39a224  kernel-devel-5.7.17-200.fc32.x86_64.rpm
4. applydeltarpm -vvvvv -p -r kernel-devel-5.7.17-200.fc32.x86_64.rpm kernel-devel-5.7.17-200.fc32_5.8.4-200.fc32.x86_64.drpm kernel-devel-5.8.4-200.fc32.x86_64.rpm
   (same without '-r ker...')

Actual results:
reading deltarpm
56404376 bytes source payload size
58154416 bytes target payload size
3234393 bytes internal data size
468502 bytes add data size
6886 blocks
52094 copy instructions
applying delta
100 percent finished.
used 5000 core pages
used 1827 swap pages
had to recreate 0 core pages
kernel-devel-5.7.17-200.fc32_5.8.4-200.fc32.x86_64.drpm: md5 mismatch of result

Expected results:
rpm successfully created

Additional info:
It is not the first time that a DRPM of kernel-devel cannot be applied.
dnf downloads the full RPM as recovery.

Comment 1 JayJayJazz 2020-09-11 21:21:33 UTC
I have two machines that show the same behavior as described by Joerg. All drpms are created successfully, only 'kernel-devel' fails everytime. Result is that the full rpm has to be downloaded.

Today, I tried to reinstall the 'kernel-devel' package:

======================================================================
Downloading Packages:
kernel-devel-5.8.6-201.fc32_5.8.7-200.fc32.x86_64.drpm                                                                                     15 MB/s | 3.0 MB     00:00    
/var/cache/dnf/updates-b3cb4614b6495970/packages/kernel-devel-5.8.6-201.fc32_5.8.7-200.fc32.x86_64.drpm: md5 mismatch of result
Some packages were not downloaded. Retrying.
kernel-devel-5.8.7-200.fc32.x86_64.rpm                                                                                                     33 MB/s |  13 MB     00:00    
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                     1.4 MB/s |  16 MB     00:11     
Failed Delta RPMs increased 13.3 MB of updates to 16.2 MB (-22.1% wasted)
======================================================================

Comment 2 Ondrej Mosnacek 2020-10-16 11:16:31 UTC
This is more likely a bug in deltarpm, reassigning...

Comment 3 Jonathan Dieter 2020-10-19 10:34:21 UTC
I've looked into this, and the problem is that, while running applydeltarpm creates a perfect byte-for-byte copy of the *uncompressed* payload, the *compressed* payload is different, thus failing the md5 check.  Historically, this has happened when the compression library used to compress the original rpm isn't the same version as the library used by applydeltarpm.  What's surprising about this is that it's only affecting kernel-devel.  If there's a library mismatch, I would expect to see problems with most, if not all deltarpms.  I'm cc'ing Neal as I'm hoping he might know if the kernel is built differently from everything else, which might explain this.  Neal, if you don't know, feel free to clear the NEEDINFO.

Comment 4 Ondrej Mosnacek 2020-10-19 10:46:13 UTC
Well, kernel.spec has this line in it:
https://src.fedoraproject.org/rpms/kernel/blob/master/f/kernel.spec#_21

Which means it forces the use of threaded compression to speed up the build. It might very well be the only component that does this, so that would explain why only the kernel packages seem to be affected.

I suppose the compressed result will depend on the number of threads used in case of threaded compression, which could explain the mismatch...

Comment 5 Jonathan Dieter 2020-10-19 19:22:04 UTC
Yeah, that would do it.  :(  I notice that it's also using xz compression instead of zstd.

I'm not sure what to do with this.  I wonder if there's some way to turn off deltarpm generation for the kernel rpm.

Comment 6 Ondrej Mosnacek 2020-10-19 19:53:01 UTC
(In reply to Jonathan Dieter from comment #5)
> Yeah, that would do it.  :(  I notice that it's also using xz compression
> instead of zstd.

Yeah, I forgot that zstd is the default now... I believe the reason to stay with xz was that it compresses faster (and build times do matter in case of the kernel, at least on RHEL) and that zstd RPM compression doesn't support the threaded mode (or at least it didn't at the time the decision was made - might be worth re-testing).

> I'm not sure what to do with this.  I wonder if there's some way to turn off
> deltarpm generation for the kernel rpm.

Alternatively, would it be somehow possible to compare the md5 of the uncompressed payload rather than compressed?

Comment 7 Jonathan Dieter 2020-10-19 21:31:34 UTC
(In reply to Ondrej Mosnacek from comment #6)
> (In reply to Jonathan Dieter from comment #5)
> > Yeah, that would do it.  :(  I notice that it's also using xz compression
> > instead of zstd.
> 
> Yeah, I forgot that zstd is the default now... I believe the reason to stay
> with xz was that it compresses faster (and build times do matter in case of
> the kernel, at least on RHEL) and that zstd RPM compression doesn't support
> the threaded mode (or at least it didn't at the time the decision was made -
> might be worth re-testing).

I noticed that deltarpm has grown the ability to handle threaded zstd, so I wonder if rpm can do it too now.

> > I'm not sure what to do with this.  I wonder if there's some way to turn off
> > deltarpm generation for the kernel rpm.
> 
> Alternatively, would it be somehow possible to compare the md5 of the
> uncompressed payload rather than compressed?

When we first started using deltarpms (and xz was still being changed enough that we were seeing this kind of problem on a regular basis), this was proposed.  The problem is that the RPM's signature will also fail, and there was (probably rightly) massive resistance to moving the signature onto the uncompressed payload because that would allow unverified data to be decompressed, opening the door to a malicious compressed payload exploiting a vulnerability in the compression algorithm.

I might see if rpm now supports threaded zstd, and, if so, see what the build time difference is between threaded zstd and threaded xz (and verify that deltarpm can consistently rebuild threaded zstd).

Comment 8 Neal Gompa 2020-10-21 02:34:59 UTC
RPM will support threaded zstd compression in RPM 4.17. We probably will want to have this backported so that RPMs and DeltaRPMs can use it.

Probably the following should be backported for our use:

* https://github.com/rpm-software-management/rpm/commit/aa06b9922d7990f164e241c1a33a6180bd036f36
* https://github.com/rpm-software-management/rpm/commit/c0064335f975c3664e4848596fba159a87b21104

Comment 9 Kevin Fenzi 2021-01-08 01:02:55 UTC
ok. Updated builders.


Note You need to log in before you can comment on or make changes to this bug.