2154924 – FIPS_INTEGRITY_CHECKS breaks pacemaker_remote

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2154924 - FIPS_INTEGRITY_CHECKS breaks pacemaker_remote

Summary: FIPS_INTEGRITY_CHECKS breaks pacemaker_remote

Keywords:
Status:	CLOSED NOTABUG
Alias:	None
Product:	Red Hat Enterprise Linux 9
Classification:	Red Hat
Component:	gnutls
Sub Component:
Version:	9.0
Hardware:	x86_64
OS:	Linux
Priority:	medium
Severity:	high
Target Milestone:	rc
Target Release:	---
Assignee:	Zoltan Fridrich
QA Contact:	BaseOS QE Security Team
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2022-12-19 16:30 UTC by Luca Miccini
Modified:	2023-01-30 12:52 UTC (History)
CC List:	2 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2023-01-26 14:22:33 UTC
Type:	Bug
Target Upstream Version:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	CRYPTO-9175	0	None	None	None	2022-12-20 14:20:11 UTC
Red Hat Issue Tracker	RHELPLAN-142904	0	None	None	None	2022-12-19 16:38:42 UTC

Description Luca Miccini 2022-12-19 16:30:13 UTC

Description of problem:

This has been reproduced using centos9 stream with:

[root@81d94956b13a /]# rpm -qa |grep pacemak
pacemaker-schemas-2.1.5-2.el9.noarch
pacemaker-libs-2.1.5-2.el9.x86_64
pacemaker-cluster-libs-2.1.5-2.el9.x86_64
pacemaker-cli-2.1.5-2.el9.x86_64
pacemaker-2.1.5-2.el9.x86_64
pacemaker-remote-2.1.5-2.el9.x86_64
[root@81d94956b13a /]# rpm -qa |grep libq
libqb-2.0.6-1.el9.x86_64

[root@81d94956b13a /]# rpm -qa |grep gnutl
gnutls-3.7.6-14.el9.x86_64
root@81d94956b13a /]# rpm -qa |grep nett
nettle-3.8-3.el9_0.x86_64


we are running containers on a host configured in FIPS mode:

[root@81d94956b13a /]# cat /proc/sys/crypto/fips_enabled
1

the issue is probably reproducible outside of containers (we haven't tried it tho).

running pacemaker_remote -V fails with:


[root@81d94956b13a /]# pacemaker_remoted -V
Error in GnuTLS initialization: Error while performing self checks.
(main)     notice: Starting Pacemaker remote executor
(qb_ipcs_us_publish)     info: server name: lrmd
(pcmk__init_tls_dh)     info: Generating Diffie-Hellman parameters with 2048-bit prime for TLS
(pcmk__init_tls_dh)     error: Could not initialize Diffie-Hellman parameters for TLS: An error has been detected in the library and cannot continue operations. | rc=-402
(main)     error: Failed to create TLS listener: shutting down and staying down
(crm_exit)     info: Exiting pacemaker-remoted | with status 100


increasing gnutls debug verbosity:

[root@81d94956b13a /]# export GNUTLS_DEBUG_LEVEL=666
[root@81d94956b13a /]# pacemaker_remoted -V
gnutls[2]: Enabled GnuTLS 3.7.6 logging...
gnutls[2]: FIPS140-2 mode enabled
gnutls[2]: getrandom random generator was selected
gnutls[2]: FIPS140-2 mode: 1
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: AES-128-CBC self check succeeded
gnutls[2]: Intel SSSE3 was detected
gnutls[2]: Intel SHA was detected
gnutls[2]: Intel AES accelerator was detected
gnutls[2]: Intel GCM accelerator (AVX) was detected
gnutls[2]: running tests for: AES-128-GCM
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[3]: ASSERT: aes-gcm-x86-pclmul-avx.c[aesni_gcm_aead_decrypt]:370
gnutls[2]: FIPS140-2 context is not set
gnutls[3]: ASSERT: crypto-api.c[gnutls_aead_cipher_decrypt]:1254
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[3]: ASSERT: aes-gcm-x86-pclmul-avx.c[aesni_gcm_aead_decrypt]:370
gnutls[2]: FIPS140-2 context is not set
gnutls[3]: ASSERT: crypto-api.c[gnutls_aead_cipher_decrypt]:1254
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[3]: ASSERT: aes-gcm-x86-pclmul-avx.c[aesni_gcm_aead_decrypt]:370
gnutls[2]: FIPS140-2 context is not set
gnutls[3]: ASSERT: crypto-api.c[gnutls_aead_cipher_decrypt]:1254
gnutls[2]: AES-128-GCM self check succeeded
gnutls[2]: running scatter (iovec) tests for: AES-128-GCM
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: AES-128-GCM scatter self check succeeded
gnutls[2]: compat: running tests for: AES-128-GCM
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: AES-128-GCM compat self check succeeded
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: AES-256-CBC self check succeeded
gnutls[2]: running tests for: AES-256-GCM
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[3]: ASSERT: aes-gcm-x86-pclmul-avx.c[aesni_gcm_aead_decrypt]:370
gnutls[2]: FIPS140-2 context is not set
gnutls[3]: ASSERT: crypto-api.c[gnutls_aead_cipher_decrypt]:1254
gnutls[2]: AES-256-GCM self check succeeded
gnutls[2]: running scatter (iovec) tests for: AES-256-GCM
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: AES-256-GCM scatter self check succeeded
gnutls[2]: compat: running tests for: AES-256-GCM
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: AES-256-GCM compat self check succeeded
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: AES-256-XTS self check succeeded
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: AES-256-CFB8 self check succeeded
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: SHA3-224 self check succeeded
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: SHA3-256 self check succeeded
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: SHA3-384 self check succeeded
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: SHA3-512 self check succeeded
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: MAC-SHA1 self check succeeded
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: MAC-SHA224 self check succeeded
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: MAC-SHA256 self check succeeded
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: MAC-SHA384 self check succeeded
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: MAC-SHA512 self check succeeded
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: MAC-AES-CMAC-256 self check succeeded
gnutls[3]: ASSERT: mpi.c[wrap_nettle_mpi_print]:60
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[3]: ASSERT: pk.c[_wrap_nettle_pk_verify]:1696
gnutls[2]: FIPS140-2 context is not set
gnutls[3]: ASSERT: pubkey.c[pubkey_verify_data]:2552
gnutls[3]: ASSERT: pubkey.c[gnutls_pubkey_verify_data2]:2089
gnutls[2]: RSA-2048-known-sig self test succeeded
gnutls[3]: ASSERT: mpi.c[wrap_nettle_mpi_print]:60
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: RSA-2048-enc self test succeeded
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: ECDH self test succeeded
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[3]: ASSERT: mpi.c[wrap_nettle_mpi_print]:60
gnutls[3]: ASSERT: mpi.c[wrap_nettle_mpi_print]:60
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[3]: ASSERT: pk.c[_wrap_nettle_pk_verify]:1604
gnutls[2]: FIPS140-2 context is not set
gnutls[3]: ASSERT: pubkey.c[pubkey_verify_data]:2574
gnutls[3]: ASSERT: pubkey.c[gnutls_pubkey_verify_data2]:2089
gnutls[2]: EC/ECDSA-SECP256R1-known-sig self test succeeded
gnutls[3]: ASSERT: mpi.c[wrap_nettle_mpi_print]:60
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: DH self test succeeded
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: HKDF: MAC-SHA256 self check succeeded
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: PBKDF2: MAC-SHA256 self check succeeded
gnutls[2]: TLS-PRF: MAC-SHA256 self check succeeded
gnutls[3]: ASSERT: int/drbg-aes.c[drbg_aes_generate]:160
gnutls[3]: ASSERT: int/drbg-aes.c[drbg_aes_generate]:132
gnutls[3]: ASSERT: int/drbg-aes.c[drbg_aes_generate]:160
gnutls[2]: DRBG-AES self test succeeded
gnutls[2]: Loading: /lib64/libgnutls.so.30
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: Successfully verified MAC for /lib64/libgnutls.so.30
gnutls[2]: Loading: /lib64/libnettle.so.8
gnutls[2]: FIPS140-2 context is not set
gnutls[2]: Calculated MAC for /lib64/libnettle.so.8 does not match
gnutls[3]: ASSERT: fips.c[check_lib_hmac]:386
gnutls[3]: ASSERT: fips.c[_gnutls_fips_perform_self_checks2]:572
gnutls[1]: FIPS140-2 self testing part 2 failed
gnutls[3]: ASSERT: global.c[_gnutls_global_init]:367
Error in GnuTLS initialization: Error while performing self checks.
(main)     notice: Starting Pacemaker remote executor
(qb_ipcs_us_publish)     info: server name: lrmd
(pcmk__init_tls_dh)     info: Generating Diffie-Hellman parameters with 2048-bit prime for TLS
ASSERT: dh.c[gnutls_dh_params_generate2]:470
(pcmk__init_tls_dh)     error: Could not initialize Diffie-Hellman parameters for TLS: An error has been detected in the library and cannot continue operations. | rc=-402
(main)     error: Failed to create TLS listener: shutting down and staying down
(crm_exit)     info: Exiting pacemaker-remoted | with status 100



Setting:

# GNUTLS_SKIP_FIPS_INTEGRITY_CHECKS=1 pacemaker_remoted -V

seems to workaround this issue.

I've found a similar issue (https://bugzilla.redhat.com/show_bug.cgi?id=2099651) hence why I opened this bz vs. gnutls, please let me know if I need to change the component.

Comment 1 Zoltan Fridrich 2022-12-20 13:54:33 UTC

Hello,

from the debug output it is clear to me that hmac of nettle library does not match the one in .gnutls.hmac file.
I believe that upgrading gnutls to the latest release will help:
https://kojihub.stream.centos.org/koji/buildinfo?buildID=27862

Comment 2 Douglas Viroel 2022-12-30 14:01:27 UTC

We still see upstream CentOS-Stream-9 FIPS periodic jobs failing on pacemaker execution, even with latest gnutls:

2022-12-30T05:00:23.833534158+00:00 stderr F <13>Dec 30 05:00:23 puppet-user: Error in GnuTLS initialization: Error while performing self checks.

 
From base container image:

Upgraded:
...
  gnutls-3.7.6-15.el9.x86_64


[1] https://8306a512d7988e55c3e5-a728c1b2dd1da2990e73fc543a396156.ssl.cf1.rackcdn.com/periodic/opendev.org/openstack/tripleo-ci/master/tripleo-ci-centos-9-scenario001-standalone-fips/8fc681f/logs/undercloud/var/log/containers/stdouts/mysql_wait_bundle.log
[2] https://e0ead3b76fa065866748-594c8dbcc2892c36fc68c95ad460eff0.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/tripleo-ci/master/tripleo-ci-centos-9-content-provider/544d7b8/logs/container-builds/048eb71e-86c0-4603-ad8c-ddb3e6941538/base/base-build.log

Comment 3 Zoltan Fridrich 2023-01-06 14:28:18 UTC

This is very strange. I am testing it on a centos-stream-9 virtual machine and it works perfectly fine on my side:

# head -n2 /etc/os-release 
NAME="CentOS Stream"
VERSION="9"

# rpm -qa | grep pacemaker                                                                   
pacemaker-schemas-2.1.5-4.el9.noarch                                                                                  
pacemaker-libs-2.1.5-4.el9.x86_64                                                                                     
pacemaker-cluster-libs-2.1.5-4.el9.x86_64                                                                             
pacemaker-2.1.5-4.el9.x86_64                                                                                          
pacemaker-cli-2.1.5-4.el9.x86_64                                                                                      
pacemaker-remote-2.1.5-4.el9.x86_64
# rpm -qa | grep gnutls                                                                      
gnutls-3.7.6-15.el9.x86_64                                                                                            
gnutls-dane-3.7.6-15.el9.x86_64                                                                                       
gnutls-utils-3.7.6-15.el9.x86_64
# rpm -qa | grep nettle                                                                      
nettle-3.8-3.el9.x86_64

# cat /proc/sys/crypto/fips_enabled
1

# pacemaker_remoted -V
(main)  notice: Starting Pacemaker remote executor                                                                    
(qb_ipcs_us_publish)    info: server name: lrmd                                                                       
(pcmk__init_tls_dh)     info: Generating Diffie-Hellman parameters with 2048-bit prime for TLS                        
(lrmd__init_remote_key)         warning: Could not read Pacemaker Remote key from default location /etc/pacemaker/authkey (or fallback location /etc/corosync/authkey): No such file or directory
(lrmd_init_remote_tls_server)   warning: A cluster connection will not be possible until the key is available         
(qb_ipcs_us_publish)    info: server name: cib_ro                                                                     
(qb_ipcs_us_publish)    info: server name: cib_rw                                                                     
(qb_ipcs_us_publish)    info: server name: cib_shm                                                                    
(qb_ipcs_us_publish)    info: server name: attrd                                                                      
(qb_ipcs_us_publish)    info: server name: stonith-ng                                                                 
(qb_ipcs_us_publish)    info: server name: pacemakerd                                                                 
(qb_ipcs_us_publish)    info: server name: crmd                                                                       
(main)  notice: Pacemaker remote executor successfully started and accepting connections
(main)  notice: OCF resource agent search path is /usr/lib/ocf/resource.d

As you can see, no errors are happening on my side.
This is the content of gnutls hmac file that is used during fips integrity check:

# cat /usr/lib64/.libgnutls.so.30.hmac
[global]
format-version = 1
[libgnutls.so.30]
path = /lib64/libgnutls.so.30.33.1
hmac = 44f46cbc92ba7fc78bf267b27c956378ebdb7bcd86f555e8884c495ef763b517
[libnettle.so.8]
path = /usr/lib64/libnettle.so.8.5
hmac = c1e1655f7072ab0afe5eb860f742763bfbeca26c8cce310a6e6d618b00f86501
[libhogweed.so.6]
path = /usr/lib64/libhogweed.so.6.5
hmac = 5f6db195066f4b6d81875dcf9918bbb87bc63e43e652b296a65f85abab138773

And when I manually calculate the hmacs of the libraries they match with the gnutls hmac file

# openssl mac -digest sha256 -macopt key:orboDeJITITejsirpADONivirpUkvarP -in /lib64/libgnutls.so.30.33.1 HMAC
44F46CBC92BA7FC78BF267B27C956378EBDB7BCD86F555E8884C495EF763B517
# openssl mac -digest sha256 -macopt key:orboDeJITITejsirpADONivirpUkvarP -in /usr/lib64/libnettle.so.8.5 HMAC
C1E1655F7072AB0AFE5EB860F742763BFBECA26C8CCE310A6E6D618B00F86501
# openssl mac -digest sha256 -macopt key:orboDeJITITejsirpADONivirpUkvarP -in /usr/lib64/libhogweed.so.6.5 HMAC
5F6DB195066F4B6D81875DCF9918BBB87BC63E43E652B296A65F85ABAB138773

Easy way to check the library integrity is by using certtool (I use GNUTLS_FORCE_FIPS_MODE=1 to be extra sure that lib integrity is checked):

# dnf install -y gnutls-utils
# GNUTLS_FORCE_FIPS_MODE=1 certtool --generate-privkey
Generating a 3072 bit RSA private key...

Doing dnf upgrade should be sufficient and I dont understand how it is possible that it still fails for you.
To ensure that you have the right versions, here are the links to the correct gnutls and nettle centos9 versions:
https://kojihub.stream.centos.org/koji/buildinfo?buildID=27862
https://kojihub.stream.centos.org/koji/buildinfo?buildID=24827

Let me know again if this helped.

Comment 4 Douglas Viroel 2023-01-24 20:47:29 UTC

Hi Zoltan, sorry for the delay.

Here are some outputs from a test using one of OpenStack service containers:

$ podman run -it quay.rdoproject.org/tripleomastercentos9/openstack-mariadb:current-tripleo /bin/bash

Inside the container:

$ head -n2 /etc/os-release 
NAME="CentOS Stream"
VERSION="9"

$ cat /proc/sys/crypto/fips_enabled
1

$pacemaker_remoted -V
Error in GnuTLS initialization: Error while performing self checks.
(main) 	notice: Starting Pacemaker remote executor
(qb_ipcs_us_publish) 	info: server name: lrmd
(pcmk__init_tls_dh) 	info: Generating Diffie-Hellman parameters with 2048-bit prime for TLS
(pcmk__init_tls_dh) 	error: Could not initialize Diffie-Hellman parameters for TLS: An error has been detected in the library and cannot continue operations. | rc=-402
(main) 	error: Failed to create TLS listener: shutting down and staying down
(crm_exit) 	info: Exiting pacemaker-remoted | with status 100

$ rpm -qa | grep pacemaker
pacemaker-schemas-2.1.5-4.el9.noarch
pacemaker-libs-2.1.5-4.el9.x86_64
pacemaker-cluster-libs-2.1.5-4.el9.x86_64
pacemaker-cli-2.1.5-4.el9.x86_64
pacemaker-2.1.5-4.el9.x86_64
pacemaker-remote-2.1.5-4.el9.x86_64

$ rpm -qa | grep gnutls
gnutls-3.7.6-15.el9.x86_64

$ rpm -qa | grep nettle
nettle-3.8-3.el9_0.x86_64

$ cat /usr/lib64/.libgnutls.so.30.hmac
[global]
format-version = 1
[libgnutls.so.30]
path = /lib64/libgnutls.so.30.33.1
hmac = 44f46cbc92ba7fc78bf267b27c956378ebdb7bcd86f555e8884c495ef763b517
[libnettle.so.8]
path = /usr/lib64/libnettle.so.8.5
hmac = c1e1655f7072ab0afe5eb860f742763bfbeca26c8cce310a6e6d618b00f86501
[libhogweed.so.6]
path = /usr/lib64/libhogweed.so.6.5
hmac = 5f6db195066f4b6d81875dcf9918bbb87bc63e43e652b296a65f85abab138773

# these calculations below, copies from your examples, don't match with the above
$ openssl mac -digest sha256 -macopt key:orboDeJITITejsirpADONivirpUkvarP -in /lib64/libgnutls.so.30.33.1 HMAC
44F46CBC92BA7FC78BF267B27C956378EBDB7BCD86F555E8884C495EF763B517
$ openssl mac -digest sha256 -macopt key:orboDeJITITejsirpADONivirpUkvarP -in /usr/lib64/libnettle.so.8.5 HMAC
B6E29F232022368FC89F6E993E128BFC1946FBC9B6BF7081CFD14B6E267B02DD
$ openssl mac -digest sha256 -macopt key:orboDeJITITejsirpADONivirpUkvarP -in /usr/lib64/libhogweed.so.6.5 HMAC
0B39892AF956163B9C5CDE624B2CB2A2E07E7524377E2FA9BF067CBA17B86500


Thanks!

Comment 5 Zoltan Fridrich 2023-01-25 10:44:23 UTC

Hello and thank you for the response.

It looks like you are using rhel-9.0 version of the nettle package, notice the el9_0 instead of el9.
Try to explicitly install nettle-3.8-3.el9 it should find the correct version.

# dnf install nettle-3.8-3.el9
Last metadata expiration check: 0:29:06 ago on Wed 25 Jan 2023 04:59:20 AM EST.
Dependencies resolved.
======================================================================================================================
 Package                    Architecture               Version                       Repository                  Size
======================================================================================================================
Downgrading:
 nettle                     x86_64                     3.8-3.el9                     baseos                     548 k

Transaction Summary
======================================================================================================================
Downgrade  1 Package

The hmacs should match the following:

# openssl mac -digest sha256 -macopt key:orboDeJITITejsirpADONivirpUkvarP -in /usr/lib64/libnettle.so.8.5 HMAC
C1E1655F7072AB0AFE5EB860F742763BFBECA26C8CCE310A6E6D618B00F86501
# openssl mac -digest sha256 -macopt key:orboDeJITITejsirpADONivirpUkvarP -in /usr/lib64/libhogweed.so.6.5 HMAC
5F6DB195066F4B6D81875DCF9918BBB87BC63E43E652B296A65F85ABAB138773

If for some reason the package manager cannot find the correct version, here is the link where you can download the correct package version:
https://kojihub.stream.centos.org/koji/buildinfo?buildID=24827

Please, let me know if this helped to solve the issue.

Comment 6 Douglas Viroel 2023-01-26 00:23:24 UTC

Thanks for noticing that.

Installing nettle-3.8-3.el9 solved the issue.
Is it expected to have a new release of nettle?
The issue is that our workaround would be to install this specific version, since 'dnf update' doesn't provide a new version of the package, and the ubi9 image comes with nettle-3.8-3.el9_0.x86_64 installed.
The workaround is here: https://review.opendev.org/c/openstack/tripleo-common/+/869104/3/container-images/tcib/base/base.yaml
During container build process we add CS9 repos to ubi9.
This is the job that runs with FIPS enabled and was failing before: https://zuul.opendev.org/t/openstack/build/8cd889b868f54730a138484604c97d90

Thanks for all the help!

Comment 7 Zoltan Fridrich 2023-01-26 10:04:34 UTC

There are correct versions of gnutls and nettle present on both ubi9 and cs9.

ubi9: gnutls-3.7.6-12.el9_0.x86_64.rpm  nettle-3.8-3.el9_0.x86_64.rpm
cs9:  gnutls-3.7.6-15.el9.x86_64.rpm    nettle-3.8-3.el9.x86_64.rpm

The problems begin whenever you start mixing these packages from different repositories.

Comment 8 Zoltan Fridrich 2023-01-26 14:22:33 UTC

I have discussed this issue internally and the conclusion is that we do not support mixing of rhel and c9s binaries.
We are looking for ways to prevent such problems in the future. However, I don't consider this being a bug.

In this case I feel like 'dnf upgrade' is being confused and considers el9_0 as the higher version then el9.
If you mix these repositories just make sure that both gnutls and nettle come from the same repository
otherwise the hmacs wont match during the integrity check.

I am closing this bz as "not a bug".

Comment 9 Douglas Viroel 2023-01-30 12:52:28 UTC

Hi Zoltan, when testing with ubi9 package versions:
- gnutls-3.7.6-12.el9_0.x86_64.rpm[1]  nettle-3.8-3.el9_0.x86_64.rpm
The issue still happens.

Can you help us on getting gnutls-3.7.6-15 on ubi9 repos?

[1] https://cdn-ubi.redhat.com/content/public/ubi/dist/ubi9/9/x86_64/baseos/os/Packages/g/

Note You need to log in before you can comment on or make changes to this bug.