Bug 2221813

Summary:	ESET Server Security for Linux fails to start in FIPS mode with fips.c:154: OpenSSL internal error: FATAL FIPS SELFTEST FAILURE
Product:	Red Hat Enterprise Linux 8	Reporter:	canglin
Component:	openssl	Assignee:	Clemens Lang <cllang>
Status:	CLOSED NOTABUG	QA Contact:	BaseOS QE Security Team <qe-baseos-security>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	8.8	CC:	cllang
Target Milestone:	rc	Keywords:	Triaged
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-07-17 17:10:02 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description canglin 2023-07-10 21:14:12 UTC

After updating from RHEL 8.7 to 8.8 ESET Server Security for Linux is stuck activating with crypto/fips/fips.c:154: OpenSSL internal error: FATAL FIPS SELFTEST FAILURE. ESET was working before the RHEL update.

The server is in FIPS mode.

ESET support insists it's an issue with RHEL or FIPS and not their product. If the issue is with their product is there a way to prove it?

Reproducible: Always

Steps to Reproduce:
1. Enable FIPS mode and reboot
 fips-mode-setup --enable

2. Download ESET Server Security for Linux
 curl -O https://download.eset.com/com/eset/apps/business/efs/linux/latest/efs.x86_64.bin

3. Extract RPM 
 ./efs.x86_64.bin -n

4. Install RPM using --nodigest and --nofiledigest flags since RPM is not signed with FIPS-compatible algorithm
 rpm -ivh --nodigest --nofiledigest efs-9.1.98.0.x86_64.rpm


Actual Results:  
crypto/fips/fips.c:154: OpenSSL internal error: FATAL FIPS SELFTEST FAILURE
/var/tmp/rpm-tmp.3jOv8r: line 56: 823602 Aborted                 (core dumped) /opt/eset/efs/bin/upd --compile-nups
crypto/fips/fips.c:154: OpenSSL internal error: FATAL FIPS SELFTEST FAILURE
/var/tmp/rpm-tmp.3jOv8r: line 72: 825199 Aborted                 (core dumped) '/opt/eset/efs/sbin/cfg' --eula-tag "EULA-PRODUCT-LG" --eula-version "3537.0.0" > /dev/null
crypto/fips/fips.c:154: OpenSSL internal error: FATAL FIPS SELFTEST FAILURE
/var/tmp/rpm-tmp.3jOv8r: line 85: 825239 Aborted                 (core dumped) '/opt/eset/efs/sbin/cfg' -i /dev/stdin > /dev/null  <<HERE
{
        "method": "_CE.rpc_api.screen_values_set",
        "params": {
                "values": {
                        "$LAST_PROCESSED_VERSION" : $1
                }
        }
}
HERE

crypto/fips/fips.c:154: OpenSSL internal error: FATAL FIPS SELFTEST FAILURE

Expected Results:  
ESET Server Security for Linux should install and start

Not sure if it is related but it looks like ESET is using bundled versions of libcommon and libprotobuf:
ldd /opt/eset/efs/sbin/startd
        linux-vdso.so.1 (0x00007ffc369c1000)
        libcommon.so => /opt/eset/efs/sbin/../lib/libcommon.so (0x00007f17bef9e000)
        libprotobuf.so.3.19.3.0 => /opt/eset/efs/sbin/../lib/libprotobuf.so.3.19.3.0 (0x00007f17beaf2000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f17be8ee000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f17be6e6000)
        libutil.so.1 => /lib64/libutil.so.1 (0x00007f17be4e2000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f17be160000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f17bdf48000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f17bdd28000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f17bd963000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f17bf713000)

Comment 1 Clemens Lang 2023-07-11 10:51:32 UTC

Cannot reproduce. It seems your openssl installation may have become corrupt. Have you tried re-installing the openssl-libs package using `dnf reinstall openssl-libs`?

If the FIPS selftests fail, they will usually fail for every binary that links with OpenSSL. You should be able to reproduce the same result by running a simple openssl command, such as `openssl speed aes-128-cbc`. If that also fails, the issue is with your copy of openssl.

Comment 2 canglin 2023-07-11 20:45:47 UTC

Re-installing openssl-libs package didn't help and `openssl speed aes-128-cbc` ran without any FIPS selftest failures. 

Interesting that you cannot reproduce as I was just able to with a fresh RHEL 8.8 install with FIPS mode enabled.

Comment 3 Clemens Lang 2023-07-12 08:31:02 UTC

Works fine for me on a freshly installed RHEL 8.8 system. This test is without z-stream updates, but I did try the same thing with z-stream updates installed previously, and it also worked:

$ ft redhat.os.rhel8 --release=8.8.0 + redhat.crypto.fips + ssh
[root@rhel-8-8-0 ~]# curl -O https://download.eset.com/com/eset/apps/business/efs/linux/latest/efs.x86_64.bin
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  401M  100  401M    0     0  29.7M      0  0:00:13  0:00:13 --:--:-- 30.3M
[root@rhel-8-8-0 ~]# chmod +x ./efs.x86_64.bin
[root@rhel-8-8-0 ~]# ./efs.x86_64.bin -n
[…]
[root@rhel-8-8-0 ~]# dnf install -y /usr/bin/crontab gcc kernel-devel make openssl perl sqlite
[…]
[root@rhel-8-8-0 ~]# rpm -ivh --nodigest --nofiledigest efs-9.1.98.0.x86_64.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:efs-9.1.98.0-1                   ################################# [100%]
[root@rhel-8-8-0 ~]# ps aux | grep efs
root        6507  0.6  1.6  49612 14696 ?        Ss   08:25   0:00 /opt/eset/efs/sbin/startd
eset-ef+    6509  0.6  1.7  53076 14864 ?        S    08:25   0:00 /opt/eset/efs/lib/logd
eset-ef+    6510 29.5 24.7 1130932 215308 ?      Sl   08:25   0:02 /opt/eset/efs/lib/scand
root        6511  0.7  1.9  71224 16676 ?        S    08:25   0:00 /opt/eset/efs/lib/sysinfod
eset-ef+    6512  0.8  2.2 126860 19168 ?        Sl   08:25   0:00 /opt/eset/efs/lib/updated
eset-ef+    6513  0.6  1.6  49820 14412 ?        S    08:25   0:00 /opt/eset/efs/lib/licensed
root        6514  0.4  1.6  49484 14188 ?        S    08:25   0:00 /opt/eset/efs/lib/utild
eset-ef+    6515  1.1  2.5 352108 22096 ?        Sl   08:25   0:00 /opt/eset/efs/lib/confd
root        6520  0.5  2.0 506300 17592 ?        S<l  08:25   0:00 /opt/eset/efs/lib/oaeventd
eset-ef+    6521  0.4  1.6  49444 14188 ?        S    08:25   0:00 /opt/eset/efs/lib/watchd
root        6553  0.0  0.1  12144  1152 pts/0    S+   08:25   0:00 grep --color=auto efs
[root@rhel-8-8-0 ~]# /opt/eset/efs/bin/upd --compile-nups
[root@rhel-8-8-0 ~]# echo $?
10

What is the output of
  cat /usr/lib64/.libcrypto.so.1.1.hmac /usr/lib64/.libssl.so.1.1.hmac
  rpm -q openssl-libs
  ldconfig -p | grep -E 'lib(crypto|ssl)'
on the system where this fails?

Do you have any environment variables set that might affect library loading, such as LD_LIBRARY_PATH?

Comment 4 canglin 2023-07-17 15:57:26 UTC

[root@eset-test ~]# cat /usr/lib64/.libcrypto.so.1.1.hmac /usr/lib64/.libssl.so.1.1.hmac
5c37d015873d96813809db3ccf78eb1823f28c90c4b497039a8705205d4f5515
7b24833bf4ab5d5fa8e3e4f08933a4b8b6f0ac0ba361296bd05810977d0197b5
[root@eset-test ~]# rpm -q openssl-libs
openssl-libs-1.1.1k-9.el8_7.x86_64
[root@eset-test ~]# ldconfig -p | grep -E 'lib(crypto|ssl)'
        libssl3.so (libc6,x86-64) => /lib64/libssl3.so
        libssl.so.1.1 (libc6,x86-64) => /lib64/libssl.so.1.1
        libssl.so (libc6,x86-64) => /lib64/libssl.so
        libcrypto.so.1.1 (libc6,x86-64) => /lib64/libcrypto.so.1.1
        libcrypto.so (libc6,x86-64) => /lib64/libcrypto.so

I am testing on a freshly installed system so I don't believe there are any environment variables that would affect library loading.

I was able to get ESET to work with the crypto policy set to FIPS:OSPP. FIPS:OSPP seems to be a subset of FIPS so I am unsure why it's working.

Comment 5 Clemens Lang 2023-07-17 17:10:02 UTC

I can reproduce this if the openssl-devel package is installed. It does not fail if the package is not installed.

Whether the active crypto policy is FIPS:OSPP or FIPS does not seem to matter.

Your system also shows a libssl3.so, which is definitely not provided by RHEL, so your system is very clearly not pristine: libssl3.so (libc6,x86-64) => /lib64/libssl3.so

This only seems to happen with the EFS package. /opt/eset/efs/bin/upd (which fails) isn't linked against libcrypto or libssl, so it must be dlopen(3)ing libssl or libcrypto. It seems EFS first attempts to dlopen("libcrypto.so"), and if that fails, falls back to dlopen("libcrypto.so.1.1"), which succeeds.

The openssl library determines the location of its HMAC checksum file relative to its own path, so when you dlopen("libcrypto.so"), it attempts to locate .libcrypto.so.hmac, which does not exist. On the other hand, users that expect a specific set of symbols in a binary should not dlopen("libcrypto.so"), but use a versioned name such as "libcrypto.so.1.1", or ship a copy of libcrypto compatible with their expectations. This code path has not changed since RHEL 8.1, and was present even before that, so this is not a regression.

EFS should not attempt to dlopen an unversioned library name. As a workaround, uninstall the openssl-devel package. I do not believe this is a bug in our package, and am thus closing this as NOTABUG.