RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2184640 - Unable to verify package signature in latest centos/centos:stream9 image
Summary: Unable to verify package signature in latest centos/centos:stream9 image
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: gnupg2
Version: CentOS Stream
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: rc
: ---
Assignee: Jakub Jelen
QA Contact: Stanislav Zidek
URL:
Whiteboard:
: 2184659 2186332 2228185 (view as bug list)
Depends On:
Blocks: 2185380
TreeView+ depends on / blocked
 
Reported: 2023-04-05 10:06 UTC by Cédric Jeanneret
Modified: 2023-11-07 10:39 UTC (History)
19 users (show)

Fixed In Version: gnupg2-2.3.3-4.el9
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 2185380 (view as bug list)
Environment:
Last Closed: 2023-11-07 08:43:37 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 2015309 0 None None None 2023-04-05 11:04:05 UTC
Red Hat Bugzilla 2070722 0 medium CLOSED SHA-1 signatures work in DEFAULT 2024-01-04 08:37:07 UTC
Red Hat Issue Tracker CRYPTO-10356 0 None None None 2023-04-19 08:45:13 UTC
Red Hat Issue Tracker CS-1527 0 None None None 2023-04-05 12:10:52 UTC
Red Hat Issue Tracker RHELPLAN-154070 0 None None None 2023-04-05 10:07:51 UTC
Red Hat Product Errata RHBA-2023:6597 0 None None None 2023-11-07 08:43:44 UTC

Description Cédric Jeanneret 2023-04-05 10:06:47 UTC
Description of problem:
Starting today, it's impossible to install any package in said image:

dnf install git -y
[...]
The GPG keys listed for the "CentOS Stream 9 - BaseOS" repository are already installed but they are not correct for this package.
Check that the correct key URLs are configured for this repository.. Failing package is: cracklib-2.9.6-27.el9.x86_6
 GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial

[...]

and it goes the same for all the packages to install.

The key file exists:
-rw-r--r--. 1 root root 1683 Feb 21 15:30 /etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial


It seems to be linked to gpg, since it seems to reject the SHA1 algorithm:
[root@e40833e9c636 yum.repos.d]# gpg --dry-run --import /etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
gpg: keyblock resource '/root/.gnupg/pubring.kbx': No such file or directory
gpg: Note: signatures using the SHA1 algorithm are rejected
gpg: key 05B555B38483C65D: 1 bad signature
gpg: Total number processed: 1

dnf --version
4.14.0
  Installed: dnf-0:4.14.0-4.el9.noarch at Mon Apr  3 10:26:09 2023
  Built    : builder at Fri Jan  6 14:23:17 2023

  Installed: rpm-0:4.16.1.3-22.el9.x86_64 at Mon Apr  3 10:26:07 2023
  Built    : builder at Mon Dec 19 23:57:50 2022

gpg --version
gpg (GnuPG) 2.3.3
libgcrypt 1.10.0-unknown
Copyright (C) 2021 Free Software Foundation, Inc.
License GNU GPL-3.0-or-later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: /root/.gnupg
Supported algorithms:
Pubkey: RSA, ELG, DSA, ECDH, ECDSA, EDDSA
Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,
        CAMELLIA128, CAMELLIA192, CAMELLIA256
AEAD: EAX, OCB
Hash: SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB, BZIP2

rpm --version
RPM version 4.16.1.3


Note that the same issue seems to happen when checking the container image signature from a centos-9 host, we reported here:
https://bugs.launchpad.net/tripleo/+bug/2015309

Comment 1 Cédric Jeanneret 2023-04-05 11:05:52 UTC
The issue is introduced by gnupg2 marking "sha1" as weak, hence breaking the whole signature system in centos-9....

So we'd need:
- revert https://kojihub.stream.centos.org/koji/buildinfo?buildID=31391
- update keys/signatures
- re-introduce "Mark SHA-1 digest as weak to follow SHA-1 disablement in RHEL9 (#2070722)" once no more sha-1 is in use

Comment 2 Nikita Uvarov 2023-04-05 11:26:52 UTC
*** Bug 2184659 has been marked as a duplicate of this bug. ***

Comment 3 Hubert Kario 2023-04-05 11:38:27 UTC
SHA-1 is thoroughly broken since 2017. This first priority is to start working on making sure that SHA-1 is not used.

Comment 4 Clemens Lang 2023-04-05 12:02:02 UTC
Does rpm on CentOS stream use the gpg command line to verify signatures? I assumed it was using either the custom verifier based on OpenSSL implemented in rpm, or sequoia (as recently merged on Fedora), so whether the gpg CLI accepts SHA-1 should not matter.

Can somebody from dnf or rpm shed a light on what situations would print this error message?

Comment 5 Petr Pisar 2023-04-05 12:10:52 UTC
I tasked CS team to change the PGP certificate <https://issues.redhat.com/browse/CS-1527>.

Comment 6 Cédric Jeanneret 2023-04-05 12:12:10 UTC
From what I could see, GPG is used (maybe just a lib affected by the SHA-1 deprecation) for, at least:
- package signature verification (it's a GPG key, really)
- container signature verification (the ones provided by our Red Hat registry, for instance the ubi9 image)

Reproducers are easy, and can be self-contained:

podman pull centos/centos:stream9 # ensure latest image
podman run --rm -ti centos/centos:stream9 bash # start a container interactively
dnf install -y git # or any other package

This will show the breakage of dnf.

For the container signature verification:
*from an up-to-date CentOS Stream 9*: podman pull registry.access.redhat.com/ubi9:latest

We may do this from within a container (provided you're running it with proper privileges), or some virtual machine - as long as it's up-to-date.

This is more than probably because the public keys shipped on the system are still using the SHA-1 digest. So afaik, we'd "just" need to ship updated pub key using sha256 algorithm instead, and we should be fine... Unless we need to re-key the whole thing, but in this case, we're facing some major issues.

Comment 7 Petr Pisar 2023-04-05 12:21:35 UTC
DNF uses GnuPG via gpgme library.
RPM links only to libcrypto, thus OpenSSL. Though rpm-sign-libs package for signing packages requires /usr/bin/gpg2 program. I'm not sure how RPM verifies signatures.

Comment 8 Clemens Lang 2023-04-05 12:28:14 UTC
(In reply to Cédric Jeanneret from comment #6)
> From what I could see, GPG is used (maybe just a lib affected by the SHA-1
> deprecation) for, at least:
> - package signature verification (it's a GPG key, really)
> - container signature verification (the ones provided by our Red Hat
> registry, for instance the ubi9 image)

Libraries and tools based on OpenSSL have rejected SHA-1 signatures for a while now. In this instance, it does not really matter that the signature was made by a GPG key—every signature on an RPM is made by a GPG key, and yet they started failing in RHEL 9.0. It is only relevant which tool or library is used to verify the signature.


(In reply to Petr Pisar from comment #7)
> DNF uses GnuPG via gpgme library.

Thanks, that's helpful.

> RPM links only to libcrypto, thus OpenSSL. Though rpm-sign-libs package for
> signing packages requires /usr/bin/gpg2 program. I'm not sure how RPM
> verifies signatures.

RPM has a custom implementation based on OpenSSL to verify signatures, which stopped accepting SHA-1 signatures in 9.0.
Signing packages should no longer use SHA-1 anyway, so in this particular case it doesn't matter that it uses /usr/bin/gpg2.

Comment 9 Adam Samalik 2023-04-05 12:49:22 UTC
Quick workaround:

rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial

Comment 10 Clemens Lang 2023-04-05 12:50:44 UTC
As a workaround until we decide how to resolve this, apparently running

  rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial

fixes the issue. I verified that package installation starts working after that command was run.

So it seems the issue here is that

  (a) the CentOS signing key uses a SHA-1 binding signature; it should use SHA-256 or newer
  (b) dnf uses gpgme to initially add the rpm key to the rpmdb; gpgme uses gpg2, which now rejects the SHA-1 binding signature; if the key is already in the rpmdb, this code path no longer runs

Comment 12 Alan Pevec 2023-04-08 17:59:35 UTC
> Note that the same issue seems to happen when checking the container image
> signature from a centos-9 host, we reported here:
> https://bugs.launchpad.net/tripleo/+bug/2015309

Follow up on that, when verifying container signatures, keys are configured per registry in /etc/containers/policy.json
for registry.access.redhat.com/ubi9:latest in the failing CI job, the key is /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release provided by containers-common on CS9
which was not updated yet:

# curl https://gitlab.com/redhat/centos-stream/rpms/containers-common/-/raw/c9s/RPM-GPG-KEY-redhat-release | gpg2 --list-packets

# off=0 ctb=99 tag=6 hlen=3 plen=525
:public key packet:
	version 4, algo 1, created 1256212795, expires 0
	pkey[0]: [4096 bits]
	pkey[1]: [17 bits]
	keyid: 199E2F91FD431D51
# off=528 ctb=b4 tag=13 hlen=2 plen=51
:user ID packet: "Red Hat, Inc. (release key 2) <security>"
# off=581 ctb=89 tag=2 hlen=3 plen=566
:signature packet: algo 1, keyid 199E2F91FD431D51
	version 4, created 1256212795, md5len 0, sigclass 0x13
	digest algo 2, begin of digest 6c e9
	hashed subpkt 2 len 4 (sig created 2009-10-22)
	hashed subpkt 27 len 1 (key flags: 03)
	hashed subpkt 11 len 5 (pref-sym-algos: 9 8 7 3 2)
	hashed subpkt 21 len 3 (pref-hash-algos: 2 8 3)
	hashed subpkt 22 len 3 (pref-zip-algos: 2 3 1)
	hashed subpkt 30 len 1 (features: 01)
	hashed subpkt 23 len 1 (keyserver preferences: 80)
	subpkt 16 len 8 (issuer key ID 199E2F91FD431D51)
	data: [4095 bits]

digest algo 2 = SHA-1 https://www.rfc-editor.org/rfc/rfc4880#section-9.4

Comment 13 Josh Boyer 2023-04-13 11:50:10 UTC
*** Bug 2186332 has been marked as a duplicate of this bug. ***

Comment 14 Jakub Jelen 2023-04-19 08:40:52 UTC
The revert of the weak SHA-1 change is available in centos9stream PR: https://gitlab.com/redhat/centos-stream/rpms/gnupg2/-/merge_requests/8

Comment 19 lejeczek 2023-05-03 14:24:57 UTC
Just so you/people know that other bits are affected - the same/similar problems are encounter when 'mock' is used/ran:

-> $ mock -r centos-stream+epel-9-x86_64 --rootdir=~/mock --localrepo=/devs/var/www/dnf.repo --chain --continue rpmbuild/rpm.src/pass-1.7.4-6.el9.src.rpm
...
CentOS Stream 9 - BaseOS                                                             1.6 MB/s | 1.6 kB     00:00    
The GPG keys listed for the "CentOS Stream 9 - BaseOS" repository are already installed but they are not correct for this package.
Check that the correct key URLs are configured for this repository.. Failing package is: alternatives-1.20-2.el9.x86_64
 GPG Keys are configured as: file:///usr/share/distribution-gpg-keys/centos/RPM-GPG-KEY-CentOS-Official
Public key for audit-libs-3.0.7-103.el9.x86_64.rpm is not installed. Failing package is: audit-libs-3.0.7-103.el9.x86_64
...

thanks, L.

Comment 20 Clemens Lang 2023-05-03 14:56:55 UTC
The change that caused this was reverted in gnupg2-2.3.3-4, so if you are still seeing this, make sure you are using the latest versions.

Comment 23 Martin Osvald 🛹 2023-08-02 05:05:12 UTC
*** Bug 2228185 has been marked as a duplicate of this bug. ***

Comment 24 Andrew Schorr 2023-08-02 17:03:04 UTC
Hi, I'm running CentOS Stream 9. We never received gnupg2-2.3.3-4, and
even worse, gnupg2-2.3.3-3 was somehow magically retracted and disappeared
from the repo. So how was a downstream system that had already upgraded
to the botched 2.3.3-3 release supposed to get fixed? As far as I can tell,
dnf is not smart enough to remove a release that has been retracted.
What's the update model here?

Regards,
Andy

Comment 25 Clemens Lang 2023-08-02 17:46:39 UTC
I can confirm that gnupg2-2.3.3-2 is currently the latest available version on CentOS Stream 9:

:) cllang@frootmig:~$ podman pull quay.io/centos/centos:stream9
:) cllang@frootmig:~$ podman run --rm -it quay.io/centos/centos:stream9 bash
[root@1311aa454244 /]# dnf update
[root@1311aa454244 /]# rpm -q gnupg2
gnupg2-2.3.3-2.el9.x86_64

Something seems to have changed in the tags for the relevant builds on July 27th, although I don't understand why that would make gnupg2-2.3.3-4.el9 disappear from the repository:

:) cllang@frootmig:~$ brew -p stream list-history --build=gnupg2-2.3.3-2.el9
Mon Aug  8 17:50:39 2022 gnupg2-2.3.3-2.el9 tagged into c9s-gate by jjelen [still active]
Tue Aug  9 09:10:19 2022 gnupg2-2.3.3-2.el9 tagged into c9s-candidate by distrolibrium [still active]
Tue Aug  9 09:10:40 2022 gnupg2-2.3.3-2.el9 tagged into c9s-pending by distrolibrium
Wed Apr  5 16:48:48 2023 gnupg2-2.3.3-2.el9 tagged into c9s-override by asamalik [still active]
Wed Apr  5 19:11:51 2023 gnupg2-2.3.3-2.el9 untagged from c9s-pending by distrolibrium
Wed Apr  5 19:11:51 2023 gnupg2-2.3.3-2.el9 tagged into c9s-pending by distrolibrium [still active]
Thu Jul 27 17:43:48 2023 gnupg2-2.3.3-2.el9 tagged into c9s-pending-signed by asamalik [still active]
:) cllang@frootmig:~$ brew -p stream list-history --build=gnupg2-2.3.3-3.el9
Thu Mar 30 17:54:27 2023 gnupg2-2.3.3-3.el9 tagged into c9s-gate by jjelen [still active]
Thu Mar 30 21:02:14 2023 gnupg2-2.3.3-3.el9 tagged into c9s-candidate by distrolibrium [still active]
Thu Mar 30 21:13:25 2023 gnupg2-2.3.3-3.el9 tagged into c9s-pending by distrolibrium
Wed Apr  5 19:13:10 2023 gnupg2-2.3.3-3.el9 untagged from c9s-pending by bstinson
:) cllang@frootmig:~$ brew -p stream list-history --build=gnupg2-2.3.3-4.el9
Wed Apr 26 13:58:23 2023 gnupg2-2.3.3-4.el9 tagged into c9s-gate by jjelen [still active]
Fri Apr 28 19:42:32 2023 gnupg2-2.3.3-4.el9 tagged into c9s-candidate by distrolibrium [still active]
Fri Apr 28 20:39:54 2023 gnupg2-2.3.3-4.el9 tagged into c9s-pending by distrolibrium [still active]
Thu Jul 27 17:43:48 2023 gnupg2-2.3.3-4.el9 tagged into c9s-pending-signed by asamalik [still active]

Needinfo on asamalik, who did the latest tag change on July 27th, maybe you can help figure out what's going on and restore gnupg2-2.3.3-4.el9?

Comment 26 Clemens Lang 2023-08-02 17:48:34 UTC
In the meantime, you should be able to dnf downgrade or dnf reinstall the package.

Comment 27 Andrew Schorr 2023-08-02 18:26:57 UTC
I'm not sure whether gnupg2-2.3.3-4 ever appeared, but I can confirm that gnupg2-2.3.3-3 was
there briefly and then disappeared. I unfortunately downloaded and installed gnupg2-2.3.3-3
during that time period:

bash-5.1$ rpm -qip gnupg2-2.3.3-3.el9.x86_64.rpm 
Name        : gnupg2
Version     : 2.3.3
Release     : 3.el9
Architecture: x86_64
Install Date: (not installed)
Group       : Unspecified
Size        : 9227453
License     : GPLv3+
Signature   : RSA/SHA256, Thu 30 Mar 2023 03:13:37 PM EDT, Key ID 05b555b38483c65d
Source RPM  : gnupg2-2.3.3-3.el9.src.rpm
Build Date  : Thu 30 Mar 2023 11:50:37 AM EDT
Build Host  : x86-02.stream.rdu2.redhat.com
Packager    : builder
Vendor      : CentOS
URL         : https://www.gnupg.org/
Summary     : Utility for secure communication and data storage
Description :
GnuPG is GNU's tool for secure communication and data storage.  It can
be used to encrypt data and to create digital signatures.  It includes
an advanced key management facility and is compliant with the proposed
OpenPGP Internet standard as described in RFC2440 and the S/MIME
standard as described by several RFCs.

GnuPG 2.0 is a newer version of GnuPG with additional support for
S/MIME.  It has a different design philosophy that splits
functionality up into several modules. The S/MIME and smartcard functionality
is provided by the gnupg2-smime package.




How does an rpm get pushed to a repository and then rolled back? I didn't
know that was a valid concept. How can downstream users maintain their
systems properly when that happens? I was left stuck on 2.3.3-3 and
lots of things were breaking...

Comment 28 Petr Pisar 2023-08-03 07:21:33 UTC
(In reply to Andrew Schorr from comment #24)
> Hi, I'm running CentOS Stream 9. We never received gnupg2-2.3.3-4, and
> even worse, gnupg2-2.3.3-3 was somehow magically retracted and disappeared
> from the repo. So how was a downstream system that had already upgraded
> to the botched 2.3.3-3 release supposed to get fixed? As far as I can tell,
> dnf is not smart enough to remove a release that has been retracted.
> What's the update model here?
> 
You are completely right. The correct approach to fix this issue is to build and distribute gnupg2-2.3.3-4 for CentOS Stream. Releasing a new package build is the correct way of noticing user's system to apply a fix.

Retracting gnupg2-2.3.3-3 from repositories was probably performed to prevent from damaging system that has not yet updated to gnupg2-2.3.3-3.

Though if systems with gnupg2-2.3.3-3 cannot verify a signature of gnupg2-2.3.3-4, the systems won't either update automatically and will need an administrator's attention to install the update forcefully without checking signatures.

Looking at gnupg2-2.3.3-4 in CentOS Stream, I can see it was built <https://kojihub.stream.centos.org/koji/buildinfo?buildID=32017> on 2023-04-26. It was signed at 2024-07-27 17:43:48. It's the latest build in all four tags. So it should appear in the repository.

The repository was last updated at 2023-07-27 17:39. So it was signed just few minutes after creating the repository.

It should appear in next repository update. I believe that CentOS Stream repositories are updated with 2-week period. That means in 7 days since now.

Comment 29 Clemens Lang 2023-08-03 10:22:35 UTC
(In reply to Petr Pisar from comment #28)
> Retracting gnupg2-2.3.3-3 from repositories was probably performed to
> prevent from damaging system that has not yet updated to gnupg2-2.3.3-3.

Correct. Brian Stinson did that.


> Though if systems with gnupg2-2.3.3-3 cannot verify a signature of
> gnupg2-2.3.3-4, the systems won't either update automatically and will need
> an administrator's attention to install the update forcefully without
> checking signatures.

The problem only occurs when importing the signature keys into the keyring, which will already have happened for most users.


> Looking at gnupg2-2.3.3-4 in CentOS Stream, I can see it was built
> <https://kojihub.stream.centos.org/koji/buildinfo?buildID=32017> on
> 2023-04-26. It was signed at 2024-07-27 17:43:48. It's the latest build in
> all four tags. So it should appear in the repository.

I'm assuming 2024 was a typo here. However, I'm still surprised that this wasn't signed until 2023-07-27. It should have been signed and released to c9s months earlier, since it was verified on  2023-05-11. I have no way of verifying this now, but I'm reasonably sure that this package had already been published.


> The repository was last updated at 2023-07-27 17:39. So it was signed just
> few minutes after creating the repository.

Then why is gnupg2-2.3.3-2.el9 in the repository, when it was tagged into c9s-pending-signed in the exact same second? See:

  Thu Jul 27 17:43:48 2023 gnupg2-2.3.3-2.el9 tagged into c9s-pending-signed by asamalik [still active]


> It should appear in next repository update. I believe that CentOS Stream
> repositories are updated with 2-week period. That means in 7 days since now.

Can we speed this up?

Comment 30 Petr Pisar 2023-08-03 12:31:47 UTC
(In reply to Clemens Lang from comment #29)
> > Looking at gnupg2-2.3.3-4 in CentOS Stream, I can see it was built
> > <https://kojihub.stream.centos.org/koji/buildinfo?buildID=32017> on
> > 2023-04-26. It was signed at 2024-07-27 17:43:48. It's the latest build in
> > all four tags. So it should appear in the repository.
> 
> I'm assuming 2024 was a typo here.

Yes. Sorry. Year 2023.

> However, I'm still surprised that this
> wasn't signed until 2023-07-27. It should have been signed and released to
> c9s months earlier, since it was verified on  2023-05-11.

Yes. I was also surprised. I don't see into release process of CentOS Stream. Maybe CentOS people were occupied with creating a new signing key which would not use SHA-1 and they delayed the signing.

> > The repository was last updated at 2023-07-27 17:39. So it was signed just
> > few minutes after creating the repository.
> 
> Then why is gnupg2-2.3.3-2.el9 in the repository, when it was tagged into
> c9s-pending-signed in the exact same second? See:
> 
>   Thu Jul 27 17:43:48 2023 gnupg2-2.3.3-2.el9 tagged into c9s-pending-signed
> by asamalik [still active]
> 

I don't know. gnupg2-2.3.3-2.el9 has a more complicated history:

$ koji -p stream list-history --build gnupg2-2.3.3-2.el9
Mon Aug  8 17:50:39 2022 gnupg2-2.3.3-2.el9 tagged into c9s-gate by jjelen [still active]
Tue Aug  9 09:10:19 2022 gnupg2-2.3.3-2.el9 tagged into c9s-candidate by distrolibrium [still active]
Tue Aug  9 09:10:40 2022 gnupg2-2.3.3-2.el9 tagged into c9s-pending by distrolibrium
Wed Apr  5 16:48:48 2023 gnupg2-2.3.3-2.el9 tagged into c9s-override by asamalik [still active]
Wed Apr  5 19:11:51 2023 gnupg2-2.3.3-2.el9 untagged from c9s-pending by distrolibrium
Wed Apr  5 19:11:51 2023 gnupg2-2.3.3-2.el9 tagged into c9s-pending by distrolibrium [still active]
Thu Jul 27 17:43:48 2023 gnupg2-2.3.3-2.el9 tagged into c9s-pending-signed by asamalik [still active]

Maybe packages in c9s-override are erroneously also shipped.

> > It should appear in next repository update. I believe that CentOS Stream
> > repositories are updated with 2-week period. That means in 7 days since now.
> 
> Can we speed this up?

I asked release engineers <https://issues.redhat.com/browse/CS-1685>

Comment 31 Andrew Schorr 2023-08-03 16:10:00 UTC
(In reply to Petr Pisar from comment #28)
> Retracting gnupg2-2.3.3-3 from repositories was probably performed to
> prevent from damaging system that has not yet updated to gnupg2-2.3.3-3.
> 
> Though if systems with gnupg2-2.3.3-3 cannot verify a signature of
> gnupg2-2.3.3-4, the systems won't either update automatically and will need
> an administrator's attention to install the update forcefully without
> checking signatures.

Thanks for the explanation. I did have some issues with dnf that I was able
to fix by adding the --nogpgcheck option. 

But instead of retracting gnupg2-2.3.3-3, wouldn't the better solution have been
to overwrite it immediately with gnupg2-2.3.3-4 containing the fix? Or
is it somehow impossible to push out a patched rpm that quickly? It seems to
me that if it's possible to retract a botched rpm, then it ought to be possible
to overwrite it immediately with a fixed rpm, since that's a preferable solution
for those of us downstream who already applied the broken rpm.

Thanks,
Andy

Comment 32 Clemens Lang 2023-08-03 17:10:32 UTC
(In reply to Andrew Schorr from comment #31)
> But instead of retracting gnupg2-2.3.3-3, wouldn't the better solution have
> been to overwrite it immediately with gnupg2-2.3.3-4 containing the fix? Or
> is it somehow impossible to push out a patched rpm that quickly? It seems to
> me that if it's possible to retract a botched rpm, then it ought to be
> possible to overwrite it immediately with a fixed rpm, since that's a preferable
> solution for those of us downstream who already applied the broken rpm.

Look at the timeline:

Thu Mar 30 21:13:25 2023 gnupg2-2.3.3-3.el9 tagged into c9s-pending
Wed Apr  5 10:06:47 2023 bug 2184640 filed
Wed Apr  5 16:48:48 2023 gnupg2-2.3.3-2.el9 tagged into c9s-override
Wed Apr  5 19:11:51 2023 gnupg2-2.3.3-2.el9 untagged from c9s-pending
Wed Apr  5 19:11:51 2023 gnupg2-2.3.3-2.el9 tagged into c9s-pending
Wed Apr  5 19:13:10 2023 gnupg2-2.3.3-3.el9 untagged from c9s-pending
Wed Apr 19 08:40:52 2023 merge request for the revert (https://bugzilla.redhat.com/show_bug.cgi?id=2184640#c14)
Wed Apr 26 13:58:23 2023 gnupg2-2.3.3-4.el9 tagged into c9s-gate
Fri Apr 28 19:42:32 2023 gnupg2-2.3.3-4.el9 tagged into c9s-candidate
Fri Apr 28 20:39:54 2023 gnupg2-2.3.3-4.el9 tagged into c9s-pending

Untagging is just much much faster. Any new build must go through various quality assurance steps that will take longer than that.

I honestly thought we had shipped the fix for this on April 28th when it was tagged into c9s-pending, since there's nothing to do for the package maintainer or QE engineer after this. However, I just checked a c9s system that I installed on 2023-06-26, and that indeed has gnupg2-2.3.3-2.el9 installed and never seems to have received gnupg2-2.3.3-4.

The c9s composes are built using pungi from https://gitlab.com/redhat/centos-stream/release-engineering/pungi-centos. `pkgset_koji_tag` is the tag packages must have in koji to be included in the compose: https://gitlab.com/redhat/centos-stream/release-engineering/pungi-centos/-/blob/centos-9-stream/shared/pkgset.conf?ref_type=heads#L2. c9s configures that tag to be "c9s-compose": https://gitlab.com/redhat/centos-stream/release-engineering/pungi-centos/-/blob/centos-9-stream/centos/variables.conf?ref_type=heads#L8.

According to CentOS Stream Koji, c9s-compose inherits from c9s-override, c9s and c9s-pending-signed. That explains why gnupg2-2.3.3-2.el9 is in the compose and repositories: it has c9s-override. It doesn't explain why gnupg2-2.3.3-2.el9 was in the compose between 2022-08-09 (when it was tagged c9s-pending) and 2023-04-05 (when it was tagged c9s-override). I'm guessing the inheritance rules for c9s-compose did at some point include c9s-pending.

If we look at the tag history of the latest openssl build, we can also see that this must be the case:

:) cllang@frootmig:~$ brew -p stream list-history --build=openssl-3.0.7-24.el9
Thu Jul 13 10:32:15 2023 openssl-3.0.7-24.el9 tagged into c9s-gate by dbelyavs [still active]
Thu Jul 13 12:20:23 2023 openssl-3.0.7-24.el9 tagged into c9s-candidate by distrolibrium [still active]
Thu Jul 13 12:40:26 2023 openssl-3.0.7-24.el9 tagged into c9s-pending by distrolibrium [still active]
Thu Jul 27 17:43:56 2023 openssl-3.0.7-24.el9 tagged into c9s-pending-signed by asamalik [still active]

My CentOS Stream 9 test system installed this package on 2023-07-22, according to dnf history openssl-libs, but it didn't receive the c9s-pending-signed tag that is now required to get into a compose until 2023-07-27.

So that leaves me puzzled: Why did gnupg2-2.3.3-4.el9, which was in c9s-pending since 2023-04-28 not ship, but openssl-3.0.7-24.el9 in the same tag did ship on or before 2023-07-22? Maybe pungi prefers packages in c9s-override over those in c9s-pending even if they are newer, and gnupg2-2.3.3-2.el9 should have been untagged from c9s-override on 2023-04-28?

Comment 33 Petr Pisar 2023-08-04 06:39:46 UTC
I believe that order of the inheritance matters. Also Koji does not compare builds by version. It compares builds by time of tagging.

$ koji -p stream latest-build c9s-compose gnupg2
Build                                     Tag                   Built by
----------------------------------------  --------------------  ----------------
gnupg2-2.3.3-2.el9                        c9s-override          jjelen

$ koji -p stream latest-build c9s-compose openssl
Build                                     Tag                   Built by
----------------------------------------  --------------------  ----------------
openssl-3.0.7-24.el9                      c9s-pending-signed    dbelyavs

Comment 34 Adam Samalik 2023-08-04 14:16:00 UTC
Sorry this is fixed now, will appear in the next compose.

To workaround the original issue we tagged the older gnupg2-2.3.3-2.el9 into the -override tag temporarily. For various reasons we couldn't untag the gnupg2-2.3.3-3.el9 version fast enough. And the idea was to untag it when a proper fix lands, which didn't happen. The -override tag is very rarely used, we'll make sure to put better reminders in place next time we use it. The newest gnupg2-2.3.3-4.el9 is now the latest as it should be.

Comment 36 errata-xmlrpc 2023-11-07 08:43:37 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (gnupg2 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:6597


Note You need to log in before you can comment on or make changes to this bug.