Bug 1715799

Summary: Re-enable zstd compression support in RHEL-8 rpm
Product: Red Hat Enterprise Linux 8 Reporter: Tomas Mraz <tmraz>
Component: rpmAssignee: Packaging Maintenance Team <packaging-team-maint>
Status: CLOSED ERRATA QA Contact: Luca Berton <lberton>
Severity: unspecified Docs Contact:
Priority: medium    
Version: 8.0CC: bcotton, carl, devrim, dgilmore, dgregor, dustymabe, fedoraproject, freddy, fweimer, hancockrwd, heinz, jberan, jblazek, j.ramanujam, jwboyer, kdudka, kparal, lberton, mastyk, mbasti, mdomonko, mkolman, mmcgrath, mstevens, msuchy, ngompa13, nkadel, orion, ovasik, pasik, p, philwyett, pmatilai, praiskup, psppsn96, redhat, samuel-rhbugs, tbowling, tdawson, tis
Target Milestone: rcKeywords: FutureFeature
Target Release: 8.2   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: rpm-4.14.2-26.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-04-28 16:51:12 UTC Type: Feature Request
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 1750648    
Bug Blocks: 1755139    

Description Tomas Mraz 2019-05-31 10:08:20 UTC
As Fedora plans to eventually switch to zstd compression it would be unfortunate if RHEL-8 would not be able to unpack the rpms produced with the default settings on Fedora.

Please consider re-enabling of zstd compression support. I understand that it means adding zstd compressor to RHEL-8, but it might be useful anyway.

Comment 1 Panu Matilainen 2019-06-04 10:32:14 UTC
Re-enabling is a non-issue from rpm point of view. It's the inclusion of zstd that needs to be accounted for.

Comment 2 Dennis Gilmore 2019-07-05 20:03:57 UTC
I will note that you can not build fedora rawhide rpms on RHEL 8 due to the change https://fedoraproject.org/wiki/Changes/Switch_RPMs_to_zstd_compression

Running transaction check
Error: transaction check vs depsolve:
rpmlib(PayloadIsZstd) <= 5.4.18-1 is needed by binutils-2.32-17.fc31.s390x
rpmlib(PayloadIsZstd) <= 5.4.18-1 is needed by binutils-gold-2.32-17.fc31.s390x
rpmlib(PayloadIsZstd) <= 5.4.18-1 is needed by gdb-minimal-8.3.50.20190702-19.fc31.s390x
rpmlib(PayloadIsZstd) <= 5.4.18-1 is needed by guile22-2.2.6-1.fc31.s390x
rpmlib(PayloadIsZstd) <= 5.4.18-1 is needed by krb5-libs-1.17-34.fc31.s390x
rpmlib(PayloadIsZstd) <= 5.4.18-1 is needed by libssh-0.9.0-2.fc31.s390x
rpmlib(PayloadIsZstd) <= 5.4.18-1 is needed by pcre2-10.33-6.fc31.s390x
To diagnose the problem, try running: 'rpm -Va --nofiles --nodigest'.
You probably have corrupted RPMDB, running 'rpm --rebuilddb' might fix the issue.

Comment 3 Jaroslav Mracek 2019-08-27 07:25:41 UTC
From comment 1 it is not clear how to resolve the issue.

Please could you provide additional information how to build RPMs on RHEL-8 for Fedora 31+ (resolve the problem from Comment 2)?

I would like to ask you for a procedure how to build RPM package with support of rpmlib(PayloadIsZstd) for RHEL-8? 

Could you also provide a build of RPM package with rpmlib(PayloadIsZstd) for RHEL-8?

Comment 10 Neal Gompa 2019-08-28 04:38:37 UTC
(In reply to Jaroslav Mracek from comment #3)
> From comment 1 it is not clear how to resolve the issue.
> 
> Please could you provide additional information how to build RPMs on RHEL-8
> for Fedora 31+ (resolve the problem from Comment 2)?
> 
> I would like to ask you for a procedure how to build RPM package with
> support of rpmlib(PayloadIsZstd) for RHEL-8? 
> 
> Could you also provide a build of RPM package with rpmlib(PayloadIsZstd) for
> RHEL-8?

I would have figured it's obvious? The zstd package needs to be imported into RHEL, rpm's build condition switch for zstd support needs to be flipped on, then rpm needs to be rebuilt. That would make it so rpm in RHEL 8 would support zstd payloads in RPMs, and restore compatibility.

Comment 20 Troy Dawson 2019-08-29 13:41:16 UTC
I want to point out that before switching, Fedora did a study on the installation time of gzip and zstd rpm's.
https://fedoraproject.org/wiki/Changes/Switch_RPMs_to_zstd_compression

The speed increase is not trivial.

- 4x faster on tmpfs (8 seconds vs 2 seconds)
- 2.5x faster on ext4 (11 seconds vs 4 seconds)

Three of their four benefits to Fedora, would also be benefitial to RHEL.
- Faster installations/upgrades of user systems
- Faster koji builds (installations in build roots)
- Faster container builds

Have you every tested out Ubuntu or Debian and wondered why their package installs seem so snappy and fast.  This is why.

Faster package installs make the user feel like their system is faster.*

* - This is an opinion, but I have seen it before, so I believe it.

Comment 22 Neal Gompa 2019-08-29 18:53:49 UTC
> Have you every tested out Ubuntu or Debian and wondered why their package installs seem so snappy and fast.  This is why.

Debian/Ubuntu systems are not using zstd compression. They started using xz compression with Debian 9 / Ubuntu 18.04, and prior to that, they used gzip compression.

They are exploring the usage of zstd compression for Debian 11 / Ubuntu 20.04 for similar reasons, though.

Comment 23 Neal Gompa 2019-08-29 18:57:06 UTC
(In reply to Neal Gompa from comment #22)
> 
> Debian/Ubuntu systems are not using zstd compression. They started using xz
> compression with Debian 9 / Ubuntu 18.04, and prior to that, they used gzip
> compression.
> 

To clarify, they used gzip compression for the longest time *because* it's faster than xz and cheaper memory-wise.

Comment 24 Phil Wyett 2019-08-29 19:58:24 UTC
We seem to be drifting off topic here.

What is I believe is desired is the adding of zstd package(s) to RHEL 8 and enabling of zstd in rpm of the distro. This will allow others and myself the ability to build/unpack etc. fedora packages that use zstd package compression on RHEL 8.

I do not believe anyone is pushing for use of zstd package compression, well not for RHEL 8 anyway, but future possibly. ;-)

Comment 25 Pádraig Brady 2019-09-03 09:58:50 UTC
I think it would make a lot of sense to add zstd to RHEL8 to increase compat with Fedora and due to its general usefulness.

For reference bug 1743983 is tracking addition to EPEL, but that would be much less desirable.

Comment 38 Pádraig Brady 2019-10-18 13:56:54 UTC
I don't have access to bugs 1750648 and 1755139.
Am I correct in assuming zstd is not included in the main RHEL repos?
I guess I should deprecate the EPEL one so?

Comment 39 Nico Kadel-Garcia 2019-11-03 18:37:53 UTC
zstd is an added compilation time capability for RPM itself. *Exaxtly* the same problem happened about 10 years ago, when RPM switched from using gzip to using xz.

There is some discussion of updating RPM on RHEL 8.2 and thus on CentOS 8.2 to include zstd. Until then, there is a workaround for "mock", at least.

* yum install podman
* Add these lines to /etc/mock/templates/fedoa-31.tpl
* * config_opts['use_bootstrap_container'] = True
* * config_opts['use_bootstrap_images'] = True

The bug over in the "mock" issues is discussed at: https://github.com/rpm-software-management/mock/issues/390 .

This works for mock, but does not solve the issue for "rpm2cpio" on RHEL 7, RHEL 8, or the CentOS equivalents.

Comment 40 Neal Gompa 2019-11-03 18:48:49 UTC
(In reply to Nico Kadel-Garcia from comment #39)
> zstd is an added compilation time capability for RPM itself. *Exaxtly* the
> same problem happened about 10 years ago, when RPM switched from using gzip
> to using xz.

Well, it wasn't exactly the same. Back then, RPM in RHEL 5 didn't support xz at all. There was the experimental lzma thing, but when Fedora switched, we had the finalized xz payload. That had to be backported to RHEL so that our Koji builders (which ran on RHEL back then) would function properly.

Nowadays, the Fedora Koji almost entirely runs on Fedora now, so most of those problems don't exist anymore. It was forced to switch because of the introduction of mandatory rich dependencies in the core package set.

I guess we could technically go back to using RHEL for builders with RHEL 8, but I hope we don't. As a project, it makes life a lot better when we dogfood our own distribution, especially since the community can control and improve things on demand, unlike with RHEL (as evidenced by this ticket...).

Comment 41 Nico Kadel-Garcia 2019-11-03 20:38:22 UTC
"Older versions of RPM cannot handle the new compression algorithm used in new RPM releases and in new operating systems" sounds like exactly the same issue to me.

If the builders in koji are Fedora based, I'd see no reason to roll them back to RHEL 8. That would trigger precisely this problem where it's not currently an issue.

Comment 42 Panu Matilainen 2019-11-04 09:38:09 UTC
*** Bug 1768147 has been marked as a duplicate of this bug. ***

Comment 46 Robert Hancock 2020-01-09 01:32:56 UTC
Another reason this should be a priority is that the retrace server for Fedora bug reporting apparently runs on CentOS 8, and is unable to unpack Fedora 31 RPMs, which means it is entirely broken for Fedora 31: https://github.com/abrt/retrace-server/issues/258

Comment 47 Miroslav Suchý 2020-01-09 12:44:42 UTC
@hancockrwd retrace server runs on RHEL 7. We plan to upgrade to RHEL 8. But yeah, the issue on retrace server is there. We are aware of that. Retrace server use mock for retracing and there was one issue in Mock which did not allow retrace to use Bootstrap Image feature. In upcoming release retrace will use podman directly.

To sum it up, yes, it does not work now, but will be fixed soon and zstd presence in RHEL8 will have no impact on this.

Comment 48 Troy Dawson 2020-01-22 22:13:15 UTC
CentOS Stream was just updated with rpm-4.14.2-36.el8.x86_64 and libzstd-1.4.2-2.el8.x86_64.

I have verified that a Fedora 32 rpm, that I was not able to open before, I can now open.

# rpm -q rpm
rpm-4.14.2-25.el8.x86_64
# rpm2cpio marble-19.08.3-1.fc32.x86_64.rpm | cpio -idmv
<garbage on the screen>
cpio: premature end of file

# dnf update rpm
<Updated rpm as well as installed libzstd>
# rpm -q rpm libzstd
rpm-4.14.2-36.el8.x86_64
libzstd-1.4.2-2.el8.x86_64
# rpm2cpio marble-19.08.3-1.fc32.x86_64.rpm | cpio -idmv
<files in marble are listed and written>
1190 blocks
#

I think that is why these are in CentOS Stream, so they can be tested before this hits the next RHEL release (hoping 8.2).
Anyway, just want to say, success.  And thank you for getting this done.
I know it will be a few months before this get's released, if it's going out with 8.2, but I'm glad I can test it now.

Comment 50 errata-xmlrpc 2020-04-28 16:51:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:1835

Comment 51 Jagannathan Ramanujam 2020-05-15 01:01:59 UTC
Hi, should zstd be supported to be installed through microdnf as well? (in a RHEL8 based docker image)
 https://download-ib01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/z/  - doesn't look like it exists under epel 8 but it does under 7 - https://download-ib01.fedoraproject.org/pub/epel/7/x86_64/Packages/z/

Comment 52 Miroslav Suchý 2020-05-15 14:05:54 UTC
zstd is implemented in rpm. It does not matter which package manager you use. Both DNF and microdnf just call rpmlib.

Comment 53 Devrim Gündüz 2020-05-18 14:42:25 UTC
So, why doesn't this exist in CentOS 8? zstd is obsoleted in EPEL8 with the following note:

"zstd is added to the BaseOS repository"

However, neither zstd is included in CentOS 8, nor rpm links to this library.

That is breaking one of my packages (GDAL) between RHEL 8 and CentOS 8. :(

Comment 54 Troy Dawson 2020-05-18 16:01:25 UTC
CentOS releases always lag behind RHEL releases by a month or so.
If you have a project that expects them to always be the same, it would be good to take that into consideration.

EPEL had to take zstd out of EPEL8 quicker than normal due to the zstd version in EPEL8 was higher than was released in RHEL 8.2.
But, we did take a snapshot of EPEL8 right before RHEL 8.2 came out, and put it in our archives.
If you need to use EPEL8 packages, built for RHEL/CentOS 8.1, you can get them at
https://archives.fedoraproject.org/pub/archive/epel/8.1/

Comment 55 Nico Kadel-Garcia 2020-05-18 18:36:23 UTC
Then Red Hat should have set an Epoch on the library. This is not the first time components have appeared in EPEL and later in Red Hat, we've been having that with the python3 updates in RHEL 7 and when ansible.com was taken over by Red Hat and ansible packaging hilarity has ensued.

Comment 56 Troy Dawson 2020-05-18 18:55:35 UTC
"Quicker than normal" = within the first week instead of within two weeks.
We would still be having this discussion even if we were as slow as we usually are.

With RHEL 8.2 there were 8 packages we took out of EPEL8 because they are now in RHEL 8.2.
We, the EPEL team, have provided a way for you to access those packages.

With RHEL 7.8, we were a little slow because we were still figuring our the procedure, and it's possible we missed some packages.
But we have a policy in place.  It will hopefully help in the future for when these things happen.