Bug 2154924
| Summary: | FIPS_INTEGRITY_CHECKS breaks pacemaker_remote | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | Luca Miccini <lmiccini> |
| Component: | gnutls | Assignee: | Zoltan Fridrich <zfridric> |
| Status: | CLOSED NOTABUG | QA Contact: | BaseOS QE Security Team <qe-baseos-security> |
| Severity: | high | Docs Contact: | |
| Priority: | medium | ||
| Version: | 9.0 | CC: | dviroel, zfridric |
| Target Milestone: | rc | Keywords: | Triaged |
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-01-26 14:22:33 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Luca Miccini
2022-12-19 16:30:13 UTC
Hello, from the debug output it is clear to me that hmac of nettle library does not match the one in .gnutls.hmac file. I believe that upgrading gnutls to the latest release will help: https://kojihub.stream.centos.org/koji/buildinfo?buildID=27862 We still see upstream CentOS-Stream-9 FIPS periodic jobs failing on pacemaker execution, even with latest gnutls: 2022-12-30T05:00:23.833534158+00:00 stderr F <13>Dec 30 05:00:23 puppet-user: Error in GnuTLS initialization: Error while performing self checks. From base container image: Upgraded: ... gnutls-3.7.6-15.el9.x86_64 [1] https://8306a512d7988e55c3e5-a728c1b2dd1da2990e73fc543a396156.ssl.cf1.rackcdn.com/periodic/opendev.org/openstack/tripleo-ci/master/tripleo-ci-centos-9-scenario001-standalone-fips/8fc681f/logs/undercloud/var/log/containers/stdouts/mysql_wait_bundle.log [2] https://e0ead3b76fa065866748-594c8dbcc2892c36fc68c95ad460eff0.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/tripleo-ci/master/tripleo-ci-centos-9-content-provider/544d7b8/logs/container-builds/048eb71e-86c0-4603-ad8c-ddb3e6941538/base/base-build.log This is very strange. I am testing it on a centos-stream-9 virtual machine and it works perfectly fine on my side: # head -n2 /etc/os-release NAME="CentOS Stream" VERSION="9" # rpm -qa | grep pacemaker pacemaker-schemas-2.1.5-4.el9.noarch pacemaker-libs-2.1.5-4.el9.x86_64 pacemaker-cluster-libs-2.1.5-4.el9.x86_64 pacemaker-2.1.5-4.el9.x86_64 pacemaker-cli-2.1.5-4.el9.x86_64 pacemaker-remote-2.1.5-4.el9.x86_64 # rpm -qa | grep gnutls gnutls-3.7.6-15.el9.x86_64 gnutls-dane-3.7.6-15.el9.x86_64 gnutls-utils-3.7.6-15.el9.x86_64 # rpm -qa | grep nettle nettle-3.8-3.el9.x86_64 # cat /proc/sys/crypto/fips_enabled 1 # pacemaker_remoted -V (main) notice: Starting Pacemaker remote executor (qb_ipcs_us_publish) info: server name: lrmd (pcmk__init_tls_dh) info: Generating Diffie-Hellman parameters with 2048-bit prime for TLS (lrmd__init_remote_key) warning: Could not read Pacemaker Remote key from default location /etc/pacemaker/authkey (or fallback location /etc/corosync/authkey): No such file or directory (lrmd_init_remote_tls_server) warning: A cluster connection will not be possible until the key is available (qb_ipcs_us_publish) info: server name: cib_ro (qb_ipcs_us_publish) info: server name: cib_rw (qb_ipcs_us_publish) info: server name: cib_shm (qb_ipcs_us_publish) info: server name: attrd (qb_ipcs_us_publish) info: server name: stonith-ng (qb_ipcs_us_publish) info: server name: pacemakerd (qb_ipcs_us_publish) info: server name: crmd (main) notice: Pacemaker remote executor successfully started and accepting connections (main) notice: OCF resource agent search path is /usr/lib/ocf/resource.d As you can see, no errors are happening on my side. This is the content of gnutls hmac file that is used during fips integrity check: # cat /usr/lib64/.libgnutls.so.30.hmac [global] format-version = 1 [libgnutls.so.30] path = /lib64/libgnutls.so.30.33.1 hmac = 44f46cbc92ba7fc78bf267b27c956378ebdb7bcd86f555e8884c495ef763b517 [libnettle.so.8] path = /usr/lib64/libnettle.so.8.5 hmac = c1e1655f7072ab0afe5eb860f742763bfbeca26c8cce310a6e6d618b00f86501 [libhogweed.so.6] path = /usr/lib64/libhogweed.so.6.5 hmac = 5f6db195066f4b6d81875dcf9918bbb87bc63e43e652b296a65f85abab138773 And when I manually calculate the hmacs of the libraries they match with the gnutls hmac file # openssl mac -digest sha256 -macopt key:orboDeJITITejsirpADONivirpUkvarP -in /lib64/libgnutls.so.30.33.1 HMAC 44F46CBC92BA7FC78BF267B27C956378EBDB7BCD86F555E8884C495EF763B517 # openssl mac -digest sha256 -macopt key:orboDeJITITejsirpADONivirpUkvarP -in /usr/lib64/libnettle.so.8.5 HMAC C1E1655F7072AB0AFE5EB860F742763BFBECA26C8CCE310A6E6D618B00F86501 # openssl mac -digest sha256 -macopt key:orboDeJITITejsirpADONivirpUkvarP -in /usr/lib64/libhogweed.so.6.5 HMAC 5F6DB195066F4B6D81875DCF9918BBB87BC63E43E652B296A65F85ABAB138773 Easy way to check the library integrity is by using certtool (I use GNUTLS_FORCE_FIPS_MODE=1 to be extra sure that lib integrity is checked): # dnf install -y gnutls-utils # GNUTLS_FORCE_FIPS_MODE=1 certtool --generate-privkey Generating a 3072 bit RSA private key... Doing dnf upgrade should be sufficient and I dont understand how it is possible that it still fails for you. To ensure that you have the right versions, here are the links to the correct gnutls and nettle centos9 versions: https://kojihub.stream.centos.org/koji/buildinfo?buildID=27862 https://kojihub.stream.centos.org/koji/buildinfo?buildID=24827 Let me know again if this helped. Hi Zoltan, sorry for the delay. Here are some outputs from a test using one of OpenStack service containers: $ podman run -it quay.rdoproject.org/tripleomastercentos9/openstack-mariadb:current-tripleo /bin/bash Inside the container: $ head -n2 /etc/os-release NAME="CentOS Stream" VERSION="9" $ cat /proc/sys/crypto/fips_enabled 1 $pacemaker_remoted -V Error in GnuTLS initialization: Error while performing self checks. (main) notice: Starting Pacemaker remote executor (qb_ipcs_us_publish) info: server name: lrmd (pcmk__init_tls_dh) info: Generating Diffie-Hellman parameters with 2048-bit prime for TLS (pcmk__init_tls_dh) error: Could not initialize Diffie-Hellman parameters for TLS: An error has been detected in the library and cannot continue operations. | rc=-402 (main) error: Failed to create TLS listener: shutting down and staying down (crm_exit) info: Exiting pacemaker-remoted | with status 100 $ rpm -qa | grep pacemaker pacemaker-schemas-2.1.5-4.el9.noarch pacemaker-libs-2.1.5-4.el9.x86_64 pacemaker-cluster-libs-2.1.5-4.el9.x86_64 pacemaker-cli-2.1.5-4.el9.x86_64 pacemaker-2.1.5-4.el9.x86_64 pacemaker-remote-2.1.5-4.el9.x86_64 $ rpm -qa | grep gnutls gnutls-3.7.6-15.el9.x86_64 $ rpm -qa | grep nettle nettle-3.8-3.el9_0.x86_64 $ cat /usr/lib64/.libgnutls.so.30.hmac [global] format-version = 1 [libgnutls.so.30] path = /lib64/libgnutls.so.30.33.1 hmac = 44f46cbc92ba7fc78bf267b27c956378ebdb7bcd86f555e8884c495ef763b517 [libnettle.so.8] path = /usr/lib64/libnettle.so.8.5 hmac = c1e1655f7072ab0afe5eb860f742763bfbeca26c8cce310a6e6d618b00f86501 [libhogweed.so.6] path = /usr/lib64/libhogweed.so.6.5 hmac = 5f6db195066f4b6d81875dcf9918bbb87bc63e43e652b296a65f85abab138773 # these calculations below, copies from your examples, don't match with the above $ openssl mac -digest sha256 -macopt key:orboDeJITITejsirpADONivirpUkvarP -in /lib64/libgnutls.so.30.33.1 HMAC 44F46CBC92BA7FC78BF267B27C956378EBDB7BCD86F555E8884C495EF763B517 $ openssl mac -digest sha256 -macopt key:orboDeJITITejsirpADONivirpUkvarP -in /usr/lib64/libnettle.so.8.5 HMAC B6E29F232022368FC89F6E993E128BFC1946FBC9B6BF7081CFD14B6E267B02DD $ openssl mac -digest sha256 -macopt key:orboDeJITITejsirpADONivirpUkvarP -in /usr/lib64/libhogweed.so.6.5 HMAC 0B39892AF956163B9C5CDE624B2CB2A2E07E7524377E2FA9BF067CBA17B86500 Thanks! Hello and thank you for the response. It looks like you are using rhel-9.0 version of the nettle package, notice the el9_0 instead of el9. Try to explicitly install nettle-3.8-3.el9 it should find the correct version. # dnf install nettle-3.8-3.el9 Last metadata expiration check: 0:29:06 ago on Wed 25 Jan 2023 04:59:20 AM EST. Dependencies resolved. ====================================================================================================================== Package Architecture Version Repository Size ====================================================================================================================== Downgrading: nettle x86_64 3.8-3.el9 baseos 548 k Transaction Summary ====================================================================================================================== Downgrade 1 Package The hmacs should match the following: # openssl mac -digest sha256 -macopt key:orboDeJITITejsirpADONivirpUkvarP -in /usr/lib64/libnettle.so.8.5 HMAC C1E1655F7072AB0AFE5EB860F742763BFBECA26C8CCE310A6E6D618B00F86501 # openssl mac -digest sha256 -macopt key:orboDeJITITejsirpADONivirpUkvarP -in /usr/lib64/libhogweed.so.6.5 HMAC 5F6DB195066F4B6D81875DCF9918BBB87BC63E43E652B296A65F85ABAB138773 If for some reason the package manager cannot find the correct version, here is the link where you can download the correct package version: https://kojihub.stream.centos.org/koji/buildinfo?buildID=24827 Please, let me know if this helped to solve the issue. Thanks for noticing that. Installing nettle-3.8-3.el9 solved the issue. Is it expected to have a new release of nettle? The issue is that our workaround would be to install this specific version, since 'dnf update' doesn't provide a new version of the package, and the ubi9 image comes with nettle-3.8-3.el9_0.x86_64 installed. The workaround is here: https://review.opendev.org/c/openstack/tripleo-common/+/869104/3/container-images/tcib/base/base.yaml During container build process we add CS9 repos to ubi9. This is the job that runs with FIPS enabled and was failing before: https://zuul.opendev.org/t/openstack/build/8cd889b868f54730a138484604c97d90 Thanks for all the help! There are correct versions of gnutls and nettle present on both ubi9 and cs9. ubi9: gnutls-3.7.6-12.el9_0.x86_64.rpm nettle-3.8-3.el9_0.x86_64.rpm cs9: gnutls-3.7.6-15.el9.x86_64.rpm nettle-3.8-3.el9.x86_64.rpm The problems begin whenever you start mixing these packages from different repositories. I have discussed this issue internally and the conclusion is that we do not support mixing of rhel and c9s binaries. We are looking for ways to prevent such problems in the future. However, I don't consider this being a bug. In this case I feel like 'dnf upgrade' is being confused and considers el9_0 as the higher version then el9. If you mix these repositories just make sure that both gnutls and nettle come from the same repository otherwise the hmacs wont match during the integrity check. I am closing this bz as "not a bug". Hi Zoltan, when testing with ubi9 package versions: - gnutls-3.7.6-12.el9_0.x86_64.rpm[1] nettle-3.8-3.el9_0.x86_64.rpm The issue still happens. Can you help us on getting gnutls-3.7.6-15 on ubi9 repos? [1] https://cdn-ubi.redhat.com/content/public/ubi/dist/ubi9/9/x86_64/baseos/os/Packages/g/ |