Bug 1715799

Re-enabling is a non-issue from rpm point of view. It's the inclusion of zstd that needs to be accounted for.

I will note that you can not build fedora rawhide rpms on RHEL 8 due to the change https://fedoraproject.org/wiki/Changes/Switch_RPMs_to_zstd_compression

Running transaction check
Error: transaction check vs depsolve:
rpmlib(PayloadIsZstd) <= 5.4.18-1 is needed by binutils-2.32-17.fc31.s390x
rpmlib(PayloadIsZstd) <= 5.4.18-1 is needed by binutils-gold-2.32-17.fc31.s390x
rpmlib(PayloadIsZstd) <= 5.4.18-1 is needed by gdb-minimal-8.3.50.20190702-19.fc31.s390x
rpmlib(PayloadIsZstd) <= 5.4.18-1 is needed by guile22-2.2.6-1.fc31.s390x
rpmlib(PayloadIsZstd) <= 5.4.18-1 is needed by krb5-libs-1.17-34.fc31.s390x
rpmlib(PayloadIsZstd) <= 5.4.18-1 is needed by libssh-0.9.0-2.fc31.s390x
rpmlib(PayloadIsZstd) <= 5.4.18-1 is needed by pcre2-10.33-6.fc31.s390x
To diagnose the problem, try running: 'rpm -Va --nofiles --nodigest'.
You probably have corrupted RPMDB, running 'rpm --rebuilddb' might fix the issue.

From comment 1 it is not clear how to resolve the issue.

Please could you provide additional information how to build RPMs on RHEL-8 for Fedora 31+ (resolve the problem from Comment 2)?

I would like to ask you for a procedure how to build RPM package with support of rpmlib(PayloadIsZstd) for RHEL-8? 

Could you also provide a build of RPM package with rpmlib(PayloadIsZstd) for RHEL-8?

(In reply to Jaroslav Mracek from comment #3)
> From comment 1 it is not clear how to resolve the issue.
> 
> Please could you provide additional information how to build RPMs on RHEL-8
> for Fedora 31+ (resolve the problem from Comment 2)?
> 
> I would like to ask you for a procedure how to build RPM package with
> support of rpmlib(PayloadIsZstd) for RHEL-8? 
> 
> Could you also provide a build of RPM package with rpmlib(PayloadIsZstd) for
> RHEL-8?

I would have figured it's obvious? The zstd package needs to be imported into RHEL, rpm's build condition switch for zstd support needs to be flipped on, then rpm needs to be rebuilt. That would make it so rpm in RHEL 8 would support zstd payloads in RPMs, and restore compatibility.

I want to point out that before switching, Fedora did a study on the installation time of gzip and zstd rpm's.
https://fedoraproject.org/wiki/Changes/Switch_RPMs_to_zstd_compression

The speed increase is not trivial.

- 4x faster on tmpfs (8 seconds vs 2 seconds)
- 2.5x faster on ext4 (11 seconds vs 4 seconds)

Three of their four benefits to Fedora, would also be benefitial to RHEL.
- Faster installations/upgrades of user systems
- Faster koji builds (installations in build roots)
- Faster container builds

Have you every tested out Ubuntu or Debian and wondered why their package installs seem so snappy and fast.  This is why.

Faster package installs make the user feel like their system is faster.*

* - This is an opinion, but I have seen it before, so I believe it.

> Have you every tested out Ubuntu or Debian and wondered why their package installs seem so snappy and fast.  This is why.

Debian/Ubuntu systems are not using zstd compression. They started using xz compression with Debian 9 / Ubuntu 18.04, and prior to that, they used gzip compression.

They are exploring the usage of zstd compression for Debian 11 / Ubuntu 20.04 for similar reasons, though.

(In reply to Neal Gompa from comment #22)
> 
> Debian/Ubuntu systems are not using zstd compression. They started using xz
> compression with Debian 9 / Ubuntu 18.04, and prior to that, they used gzip
> compression.
> 

To clarify, they used gzip compression for the longest time *because* it's faster than xz and cheaper memory-wise.

We seem to be drifting off topic here.

What is I believe is desired is the adding of zstd package(s) to RHEL 8 and enabling of zstd in rpm of the distro. This will allow others and myself the ability to build/unpack etc. fedora packages that use zstd package compression on RHEL 8.

I do not believe anyone is pushing for use of zstd package compression, well not for RHEL 8 anyway, but future possibly. ;-)

I think it would make a lot of sense to add zstd to RHEL8 to increase compat with Fedora and due to its general usefulness.

For reference bug 1743983 is tracking addition to EPEL, but that would be much less desirable.

I don't have access to bugs 1750648 and 1755139.
Am I correct in assuming zstd is not included in the main RHEL repos?
I guess I should deprecate the EPEL one so?

zstd is an added compilation time capability for RPM itself. *Exaxtly* the same problem happened about 10 years ago, when RPM switched from using gzip to using xz.

There is some discussion of updating RPM on RHEL 8.2 and thus on CentOS 8.2 to include zstd. Until then, there is a workaround for "mock", at least.

* yum install podman
* Add these lines to /etc/mock/templates/fedoa-31.tpl
* * config_opts['use_bootstrap_container'] = True
* * config_opts['use_bootstrap_images'] = True

The bug over in the "mock" issues is discussed at: https://github.com/rpm-software-management/mock/issues/390 .

This works for mock, but does not solve the issue for "rpm2cpio" on RHEL 7, RHEL 8, or the CentOS equivalents.

(In reply to Nico Kadel-Garcia from comment #39)
> zstd is an added compilation time capability for RPM itself. *Exaxtly* the
> same problem happened about 10 years ago, when RPM switched from using gzip
> to using xz.

Well, it wasn't exactly the same. Back then, RPM in RHEL 5 didn't support xz at all. There was the experimental lzma thing, but when Fedora switched, we had the finalized xz payload. That had to be backported to RHEL so that our Koji builders (which ran on RHEL back then) would function properly.

Nowadays, the Fedora Koji almost entirely runs on Fedora now, so most of those problems don't exist anymore. It was forced to switch because of the introduction of mandatory rich dependencies in the core package set.

I guess we could technically go back to using RHEL for builders with RHEL 8, but I hope we don't. As a project, it makes life a lot better when we dogfood our own distribution, especially since the community can control and improve things on demand, unlike with RHEL (as evidenced by this ticket...).

"Older versions of RPM cannot handle the new compression algorithm used in new RPM releases and in new operating systems" sounds like exactly the same issue to me.

If the builders in koji are Fedora based, I'd see no reason to roll them back to RHEL 8. That would trigger precisely this problem where it's not currently an issue.

*** Bug 1768147 has been marked as a duplicate of this bug. ***

Another reason this should be a priority is that the retrace server for Fedora bug reporting apparently runs on CentOS 8, and is unable to unpack Fedora 31 RPMs, which means it is entirely broken for Fedora 31: https://github.com/abrt/retrace-server/issues/258

@hancockrwd retrace server runs on RHEL 7. We plan to upgrade to RHEL 8. But yeah, the issue on retrace server is there. We are aware of that. Retrace server use mock for retracing and there was one issue in Mock which did not allow retrace to use Bootstrap Image feature. In upcoming release retrace will use podman directly.

To sum it up, yes, it does not work now, but will be fixed soon and zstd presence in RHEL8 will have no impact on this.

CentOS Stream was just updated with rpm-4.14.2-36.el8.x86_64 and libzstd-1.4.2-2.el8.x86_64.

I have verified that a Fedora 32 rpm, that I was not able to open before, I can now open.

# rpm -q rpm
rpm-4.14.2-25.el8.x86_64
# rpm2cpio marble-19.08.3-1.fc32.x86_64.rpm | cpio -idmv
<garbage on the screen>
cpio: premature end of file

# dnf update rpm
<Updated rpm as well as installed libzstd>
# rpm -q rpm libzstd
rpm-4.14.2-36.el8.x86_64
libzstd-1.4.2-2.el8.x86_64
# rpm2cpio marble-19.08.3-1.fc32.x86_64.rpm | cpio -idmv
<files in marble are listed and written>
1190 blocks
#

I think that is why these are in CentOS Stream, so they can be tested before this hits the next RHEL release (hoping 8.2).
Anyway, just want to say, success.  And thank you for getting this done.
I know it will be a few months before this get's released, if it's going out with 8.2, but I'm glad I can test it now.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:1835

Hi, should zstd be supported to be installed through microdnf as well? (in a RHEL8 based docker image)
 https://download-ib01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/z/  - doesn't look like it exists under epel 8 but it does under 7 - https://download-ib01.fedoraproject.org/pub/epel/7/x86_64/Packages/z/

zstd is implemented in rpm. It does not matter which package manager you use. Both DNF and microdnf just call rpmlib.

So, why doesn't this exist in CentOS 8? zstd is obsoleted in EPEL8 with the following note:

"zstd is added to the BaseOS repository"

However, neither zstd is included in CentOS 8, nor rpm links to this library.

That is breaking one of my packages (GDAL) between RHEL 8 and CentOS 8. :(

CentOS releases always lag behind RHEL releases by a month or so.
If you have a project that expects them to always be the same, it would be good to take that into consideration.

EPEL had to take zstd out of EPEL8 quicker than normal due to the zstd version in EPEL8 was higher than was released in RHEL 8.2.
But, we did take a snapshot of EPEL8 right before RHEL 8.2 came out, and put it in our archives.
If you need to use EPEL8 packages, built for RHEL/CentOS 8.1, you can get them at
https://archives.fedoraproject.org/pub/archive/epel/8.1/

Then Red Hat should have set an Epoch on the library. This is not the first time components have appeared in EPEL and later in Red Hat, we've been having that with the python3 updates in RHEL 7 and when ansible.com was taken over by Red Hat and ansible packaging hilarity has ensued.

"Quicker than normal" = within the first week instead of within two weeks.
We would still be having this discussion even if we were as slow as we usually are.

With RHEL 8.2 there were 8 packages we took out of EPEL8 because they are now in RHEL 8.2.
We, the EPEL team, have provided a way for you to access those packages.

With RHEL 7.8, we were a little slow because we were still figuring our the procedure, and it's possible we missed some packages.
But we have a policy in place.  It will hopefully help in the future for when these things happen.

Anyone who wanders across this and wonders how to workaround this for whatever reason (like you build on a fedora machine and have to deploy to RHEL8.1), add the following to your spec and rebuild:

%define _source_payload w0.gzdio
%define _binary_payload w0.gzdio

Sorry but that's not a very good advice, w0.gzdio is a really poor compression setting which practically disables compression, and there's no need to touch _source_payload at all. If you're building for RHEL 8, might as well use what the rest of the content there is built with. So for that case of building packages on recent Fedora to deploy on RHEL < 8.2, this is all you need:

%define _binary_payload w2.xzdio

If the goal is to maximize compatibility across all RHEL and other distros, gzdio is certainly the most compatible but with that you'd want to use w9.gzdio for best compression.

Fair enough, though it's a workaround and not a best practice. I don't believe it should be specified unless absolutely necessary. Thanks for the correction.

> like you build on a fedora machine and have to deploy to RHEL8.1

The best practice is to build using Mock:

mock -r epel-8-x86_64 your.src.rpm

I can attest that "mock" works well from Fedora 32 to build for RHEL 8 or CentOS 8. I publish tools for complex multi-platform building for tools like Samba, at https://github.com/nkadel/samba4repo/ The lack of zstd on RHEL 8 rpm has meant difficulties using "mock" to build for Fedora, which I've encountered quite painfully. There are workarounds, but they involve using "podman" to build Fedora root cages, and it's quite awkward.