RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2059101 - rpm checksig fails on valid rpm
Summary: rpm checksig fails on valid rpm
Keywords:
Status: CLOSED UPSTREAM
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: leapp-repository
Version: 8.7
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: rc
: ---
Assignee: Petr Stodulka
QA Contact: Martin Klusoň
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-02-28 08:30 UTC by Sandro Bonazzola
Modified: 2022-07-13 18:16 UTC (History)
20 users (show)

Fixed In Version: leapp-repository-0.16.0-1.el8_6
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-28 10:20:32 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 2058497 1 urgent CLOSED openssl-3.0.1-12.el9 breaks compose (rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release) 2023-09-30 06:11:35 UTC
Red Hat Bugzilla 2059424 1 unspecified CLOSED rpm compiled with openssl fails to import RPM-GPG-KEY-CentOS-SIG-Extras 2022-03-10 19:15:58 UTC
Red Hat Issue Tracker OAMG-6587 0 None None None 2022-03-01 20:09:49 UTC
Red Hat Issue Tracker RHELPLAN-113959 0 None None None 2022-02-28 08:33:14 UTC

Description Sandro Bonazzola 2022-02-28 08:30:35 UTC
With rpm-4.16.1.3-11.el9.x86_64 on CentOS Stream 9:

rpm --checksig -v https://download.copr.fedorainfracloud.org/results/ovirt/ovirt-master-snapshot/centos-stream-9-x86_64/03580141-ovirt-release-master/ovirt-release-master-4.5.0-0.0.master.20220228074945.git0f867a9.el9.noarch.rpm
https://download.copr.fedorainfracloud.org/results/ovirt/ovirt-master-snapshot/centos-stream-9-x86_64/03580141-ovirt-release-master/ovirt-release-master-4.5.0-0.0.master.20220228074945.git0f867a9.el9.noarch.rpm:
    Header V3 RSA/SHA1 Signature, key ID 40991906: BAD
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
    V3 RSA/SHA1 Signature, key ID 40991906: BAD
    MD5 digest: OK

rpm -qa |grep 40991906
gpg-pubkey-40991906-61703d9f


with rpm-4.14.3-14.el8_4.x86_64 on RHEL 8:

rpm --checksig -v https://download.copr.fedorainfracloud.org/results/ovirt/ovirt-master-snapshot/centos-stream-9-x86_64/03580141-ovirt-release-master/ovirt-release-master-4.5.0-0.0.master.20220228074945.git0f867a9.el9.noarch.rpm
https://download.copr.fedorainfracloud.org/results/ovirt/ovirt-master-snapshot/centos-stream-9-x86_64/03580141-ovirt-release-master/ovirt-release-master-4.5.0-0.0.master.20220228074945.git0f867a9.el9.noarch.rpm:
    Header V3 RSA/SHA1 Signature, key ID 40991906: OK
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
    V3 RSA/SHA1 Signature, key ID 40991906: OK
    MD5 digest: OK

rpm -qa |grep 40991906
gpg-pubkey-40991906-61703d9f

Comment 2 Sandro Bonazzola 2022-02-28 08:37:11 UTC
On CentOS Stream 9:

Downgraded openssl:
openssl-3.0.1-5.el9.x86_64.rpm
openssl-libs-3.0.1-5.el9.x86_64.rpm

# rpm --checksig -v https://download.copr.fedorainfracloud.org/results/ovirt/ovirt-master-snapshot/centos-stream-9-x86_64/03580141-ovirt-release-master/ovirt-release-master-4.5.0-0.0.master.20220228074945.git0f867a9.el9.noarch.rpm
https://download.copr.fedorainfracloud.org/results/ovirt/ovirt-master-snapshot/centos-stream-9-x86_64/03580141-ovirt-release-master/ovirt-release-master-4.5.0-0.0.master.20220228074945.git0f867a9.el9.noarch.rpm:
    Header V3 RSA/SHA1 Signature, key ID 40991906: OK
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
    V3 RSA/SHA1 Signature, key ID 40991906: OK
    MD5 digest: OK

upgraded again:
openssl-3.0.1-12.el9.x86_64.rpm
openssl-libs-3.0.1-12.el9.x86_64.rpm

rpm --checksig -v https://download.copr.fedorainfracloud.org/results/ovirt/ovirt-master-snapshot/centos-stream-9-x86_64/03580141-ovirt-release-master/ovirt-release-master-4.5.0-0.0.master.20220228074945.git0f867a9.el9.noarch.rpm
https://download.copr.fedorainfracloud.org/results/ovirt/ovirt-master-snapshot/centos-stream-9-x86_64/03580141-ovirt-release-master/ovirt-release-master-4.5.0-0.0.master.20220228074945.git0f867a9.el9.noarch.rpm:
    Header V3 RSA/SHA1 Signature, key ID 40991906: BAD
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
    V3 RSA/SHA1 Signature, key ID 40991906: BAD
    MD5 digest: OK


So it seems to be openssl issue

Comment 3 Sandro Bonazzola 2022-02-28 08:39:18 UTC
Seems to be caused by:
* Tue Feb 22 2022 Clemens Lang <cllang> - 1:3.0.1-8
- Disable SHA1 signature creation and verification by default
- Set rh-allow-sha1-signatures = yes to re-enable
- Resolves: rhbz#2031742

Comment 4 Clemens Lang 2022-02-28 09:05:42 UTC
> Header V3 RSA/SHA1 Signature, key ID 40991906: BAD
> V3 RSA/SHA1 Signature, key ID 40991906: BAD

These signatures are no longer considered secure. Which key was used to create this signature and what tooling? The tooling should be updated to switch to a modern hash algorithm that is not considered insecure.

Comment 5 Panu Matilainen 2022-02-28 09:56:32 UTC
Disabling SHA1-based signatures is likely to affect any number of 3rd party/vendor packages over which neither us or the customers have any control. Many users ran into similar issues with RHEL 8 FIPS-mode already due to vendor provided rpm's being built and signed with ancient toolchains, and were forced to either use --nosignature or repackage said material with modern tools to get around it when vendors were unresponsive.

As the change was accompanied by an option to re-enable it on per-system basis I don't see any problem here, the change and the workaround just need sufficient documentation.

Comment 6 Clemens Lang 2022-02-28 10:13:53 UTC
For the record, here are the signatures on that package:

$> rpm -q --qf '%{SIGPGP:armor}' ovirt-release-master-4.5.0-0.0.master.20220228074945.git0f867a9.el9.noarch.rpm
-----BEGIN PGP SIGNATURE-----
Version: rpm-4.17.0 (NSS-3)

iQEVAwUAYhx/NVfI5DlAmRkGAQJPhAgAvQg9K072O5qINX84rfo2UegCEIwHhZX8
JjY7qs77TG2kffl2uDgm2T9QjVldb0U7QkXWzE4hhM2t250md/V6Bcsz/97w06Nm
kaBh+e5H1Ny4B4SHlO8PdOlyRjKBqldh8TDxFMw3gMSbkZXOcRnXCaaRqsZ9LQ5A
2RJy2MHF0lWwegTlLq/Fhe3jN4DEO2Ax0AF6jGtDDOOzGOM5WfUxDmQYAelGGXF0
T17n40bav+aaraWVdRD0tR9+VC9eXh4raG/rXGqkmubrdTfopzHNN3IQ4A5vu0W+
je29ux8jR0frGEwQqGcK1xS4FtLWKwckcORrt0HmuDuw8vr3BB0o8w==
=6oow
-----END PGP SIGNATURE-----
$> rpm -q --qf '%{RSAHEADER:armor}' ovirt-release-master-4.5.0-0.0.master.20220228074945.git0f867a9.el9.noarch.rpm
-----BEGIN PGP SIGNATURE-----
Version: rpm-4.17.0 (NSS-3)

iQEVAwUAYhx/NVfI5DlAmRkGAQIZswgAnjPzh7eAUvBYI/yrSMVR69ykw+yBIx7O
okIZobu5lPZdZSKKpzHQXFQax9UvcBli4tEIJGcOBxB6xrlEfAT3bW2LGSCSwRug
DamWGyU6WqQ/ggxzrhHzhxgZGz4OyAQL43E9bllSFF3s+XlKw2Lx5HQ2kp5LEixo
JRkieUI/s+IZxkr0C/7BYKdL2utS96XtTM1AHStXmHTj/6T9Udc6EJcVNd4g7vEZ
oOEYKG31Gskx3zXxRe+NKRedadCytJZeq3u3GJF+/IlSERvQyR3cqQjT2AhoS1wL
MFCGBuiLyblmVEOk7glDmSp6+mmjTI7ziQJqEc4Be5c1WC6XAHdnJw==
=4RSV
-----END PGP SIGNATURE-----


Pipe those to pgpdump to see the use of SHA1:

$> rpm -q --qf '%{SIGPGP:armor}' ovirt-release-master-4.5.0-0.0.master.20220228074945.git0f867a9.el9.noarch.rpm | pgpdump
Old: Signature Packet(tag 2)(277 bytes)
        Ver 3 - old
        Hash material(5 bytes):
                Sig type - Signature of a binary document(0x00).
                Creation time - Mon Feb 28 08:52:21 CET 2022
        Key ID - 0x57C8E43940991906
        Pub alg - RSA Encrypt or Sign(pub 1)
        Hash alg - SHA1(hash 2)
        Hash left 2 bytes - 4f 84
        RSA m^d mod n(2048 bits) - ...
                -> PKCS-1

$> rpm -q --qf '%{RSAHEADER:armor}' ovirt-release-master-4.5.0-0.0.master.20220228074945.git0f867a9.el9.noarch.rpm | pgpdump
Old: Signature Packet(tag 2)(277 bytes)
        Ver 3 - old
        Hash material(5 bytes):
                Sig type - Signature of a binary document(0x00).
                Creation time - Mon Feb 28 08:52:21 CET 2022
        Key ID - 0x57C8E43940991906
        Pub alg - RSA Encrypt or Sign(pub 1)
        Hash alg - SHA1(hash 2)
        Hash left 2 bytes - 19 b3
        RSA m^d mod n(2048 bits) - ...
                -> PKCS-1

Comment 7 Clemens Lang 2022-02-28 10:42:46 UTC
This specific file seems to have been signed by fedora COPR using /bin/sign, see https://download.copr.fedorainfracloud.org/results/ovirt/ovirt-master-snapshot/centos-stream-9-x86_64/03580141-ovirt-release-master/backend.log.gz.

It seems /bin/sign may originate from the obs-sign package. There seem to be a few ways to get its sign utility to create SHA-256 hashes:
 - the self-signature of the GPG key used for the signature is SHA256, SHA224, SHA384, or SHA512
 - a config file sets the "hash" key to "sha256"
 - the command line specifies -h sha256

The default is SHA1. Given that the default will soon no longer be useful, maybe we should switch the default. I'm CC'ing Miroslav Suchy <msuchy>, who maintains the obs-signd package in fedora: https://src.fedoraproject.org/rpms/obs-signd.

Comment 11 Miroslav Suchý 2022-02-28 23:42:47 UTC
FYI obs-sign supports SHA256. It just defaults to SHA1 
   https://github.com/openSUSE/obs-sign/blob/master/sign.c#L50
I opened an issue about changing the default
  https://github.com/openSUSE/obs-sign/issues/34
We can start using SHA256 in Copr using `sign -h sha256` 
  https://pagure.io/copr/copr/issue/2106

I agree with Panu in comment #5 that this need just propper documentation. Maybe even a check in in-place upgrade tool. As it may surprise lots of customers in the middle of upgrade.

No change in RHEL is needed. The change in Copr and obs-sign is out of scope of RHEL.

Comment 12 Panu Matilainen 2022-03-01 09:27:39 UTC
This is a system-wide policy change in openssl so any explaining (aka documentation) belongs there. 
This will certainly require a release note mention, including one on rpm which is one of the affected users.

Comment 13 Panu Matilainen 2022-03-01 09:29:25 UTC
*** Bug 2059424 has been marked as a duplicate of this bug. ***

Comment 14 Sandro Bonazzola 2022-03-01 11:14:44 UTC
For those who tries to figure out how to workaround:

dnf upgrade -y \
 https://kojihub.stream.centos.org/kojifiles/packages/crypto-policies/20220223/1.git5203b41.el9/noarch/crypto-policies-20220223-1.git5203b41.el9.noarch.rpm \
 https://kojihub.stream.centos.org/kojifiles/packages/crypto-policies/20220223/1.git5203b41.el9/noarch/crypto-policies-scripts-20220223-1.git5203b41.el9.noarch.rpm

as the needed crypto-policies didn't land on mirrors yet, then:

update-crypto-policies --set LEGACY

solves.

Comment 17 Miroslav Suchý 2022-03-01 20:07:09 UTC
openssl already will track documentation part under bug 2031742.
I suggest (and I done it) to convert this to leapp bug under RHEL 8.
Leapp should create an actor which will check whether repositories (can be 3rd party) used for upgrade are using rpm signatures with SHA1 checksum. And in such a case warn the user before the in-place upgrade.

Comment 25 Petr Stodulka 2022-03-14 10:03:50 UTC
Only thing we can do here is to provide good report / error msg to users with hint what to do:
 * remove any problematic packages prior the upgrade / contact vendors for new builds of packages
   with good signature
 * apply the LEGACY system wide crypto policies

The upstream PR:
  https://github.com/oamg/leapp-repository/pull/854

Comment 26 Petr Stodulka 2022-03-16 17:27:28 UTC
The PR has been merged.

Comment 27 Alexander Todorov 2022-03-28 08:00:16 UTC
(In reply to Petr Stodulka from comment #26)
> The PR has been merged.

Was there a new version of leapp released meanwhile ?

Comment 28 Petr Stodulka 2022-03-28 10:20:32 UTC
Yes, and new version of leapp-repository as well. The fix is already part of the leapp-repository-0.16.0-1.el8_6 build. As this is the first build that is going to be released in RHEL 8, closing as upstream.

Comment 29 Petr Stodulka 2022-03-28 10:21:32 UTC
For more information track: https://bugzilla.redhat.com/show_bug.cgi?id=1997076


Note You need to log in before you can comment on or make changes to this bug.