After updating from RHEL 8.7 to 8.8 ESET Server Security for Linux is stuck activating with crypto/fips/fips.c:154: OpenSSL internal error: FATAL FIPS SELFTEST FAILURE. ESET was working before the RHEL update. The server is in FIPS mode. ESET support insists it's an issue with RHEL or FIPS and not their product. If the issue is with their product is there a way to prove it? Reproducible: Always Steps to Reproduce: 1. Enable FIPS mode and reboot fips-mode-setup --enable 2. Download ESET Server Security for Linux curl -O https://download.eset.com/com/eset/apps/business/efs/linux/latest/efs.x86_64.bin 3. Extract RPM ./efs.x86_64.bin -n 4. Install RPM using --nodigest and --nofiledigest flags since RPM is not signed with FIPS-compatible algorithm rpm -ivh --nodigest --nofiledigest efs-9.1.98.0.x86_64.rpm Actual Results: crypto/fips/fips.c:154: OpenSSL internal error: FATAL FIPS SELFTEST FAILURE /var/tmp/rpm-tmp.3jOv8r: line 56: 823602 Aborted (core dumped) /opt/eset/efs/bin/upd --compile-nups crypto/fips/fips.c:154: OpenSSL internal error: FATAL FIPS SELFTEST FAILURE /var/tmp/rpm-tmp.3jOv8r: line 72: 825199 Aborted (core dumped) '/opt/eset/efs/sbin/cfg' --eula-tag "EULA-PRODUCT-LG" --eula-version "3537.0.0" > /dev/null crypto/fips/fips.c:154: OpenSSL internal error: FATAL FIPS SELFTEST FAILURE /var/tmp/rpm-tmp.3jOv8r: line 85: 825239 Aborted (core dumped) '/opt/eset/efs/sbin/cfg' -i /dev/stdin > /dev/null <<HERE { "method": "_CE.rpc_api.screen_values_set", "params": { "values": { "$LAST_PROCESSED_VERSION" : $1 } } } HERE crypto/fips/fips.c:154: OpenSSL internal error: FATAL FIPS SELFTEST FAILURE Expected Results: ESET Server Security for Linux should install and start Not sure if it is related but it looks like ESET is using bundled versions of libcommon and libprotobuf: ldd /opt/eset/efs/sbin/startd linux-vdso.so.1 (0x00007ffc369c1000) libcommon.so => /opt/eset/efs/sbin/../lib/libcommon.so (0x00007f17bef9e000) libprotobuf.so.3.19.3.0 => /opt/eset/efs/sbin/../lib/libprotobuf.so.3.19.3.0 (0x00007f17beaf2000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f17be8ee000) librt.so.1 => /lib64/librt.so.1 (0x00007f17be6e6000) libutil.so.1 => /lib64/libutil.so.1 (0x00007f17be4e2000) libm.so.6 => /lib64/libm.so.6 (0x00007f17be160000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f17bdf48000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f17bdd28000) libc.so.6 => /lib64/libc.so.6 (0x00007f17bd963000) /lib64/ld-linux-x86-64.so.2 (0x00007f17bf713000)
Cannot reproduce. It seems your openssl installation may have become corrupt. Have you tried re-installing the openssl-libs package using `dnf reinstall openssl-libs`? If the FIPS selftests fail, they will usually fail for every binary that links with OpenSSL. You should be able to reproduce the same result by running a simple openssl command, such as `openssl speed aes-128-cbc`. If that also fails, the issue is with your copy of openssl.
Re-installing openssl-libs package didn't help and `openssl speed aes-128-cbc` ran without any FIPS selftest failures. Interesting that you cannot reproduce as I was just able to with a fresh RHEL 8.8 install with FIPS mode enabled.
Works fine for me on a freshly installed RHEL 8.8 system. This test is without z-stream updates, but I did try the same thing with z-stream updates installed previously, and it also worked: $ ft redhat.os.rhel8 --release=8.8.0 + redhat.crypto.fips + ssh [root@rhel-8-8-0 ~]# curl -O https://download.eset.com/com/eset/apps/business/efs/linux/latest/efs.x86_64.bin % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 401M 100 401M 0 0 29.7M 0 0:00:13 0:00:13 --:--:-- 30.3M [root@rhel-8-8-0 ~]# chmod +x ./efs.x86_64.bin [root@rhel-8-8-0 ~]# ./efs.x86_64.bin -n […] [root@rhel-8-8-0 ~]# dnf install -y /usr/bin/crontab gcc kernel-devel make openssl perl sqlite […] [root@rhel-8-8-0 ~]# rpm -ivh --nodigest --nofiledigest efs-9.1.98.0.x86_64.rpm Preparing... ################################# [100%] Updating / installing... 1:efs-9.1.98.0-1 ################################# [100%] [root@rhel-8-8-0 ~]# ps aux | grep efs root 6507 0.6 1.6 49612 14696 ? Ss 08:25 0:00 /opt/eset/efs/sbin/startd eset-ef+ 6509 0.6 1.7 53076 14864 ? S 08:25 0:00 /opt/eset/efs/lib/logd eset-ef+ 6510 29.5 24.7 1130932 215308 ? Sl 08:25 0:02 /opt/eset/efs/lib/scand root 6511 0.7 1.9 71224 16676 ? S 08:25 0:00 /opt/eset/efs/lib/sysinfod eset-ef+ 6512 0.8 2.2 126860 19168 ? Sl 08:25 0:00 /opt/eset/efs/lib/updated eset-ef+ 6513 0.6 1.6 49820 14412 ? S 08:25 0:00 /opt/eset/efs/lib/licensed root 6514 0.4 1.6 49484 14188 ? S 08:25 0:00 /opt/eset/efs/lib/utild eset-ef+ 6515 1.1 2.5 352108 22096 ? Sl 08:25 0:00 /opt/eset/efs/lib/confd root 6520 0.5 2.0 506300 17592 ? S<l 08:25 0:00 /opt/eset/efs/lib/oaeventd eset-ef+ 6521 0.4 1.6 49444 14188 ? S 08:25 0:00 /opt/eset/efs/lib/watchd root 6553 0.0 0.1 12144 1152 pts/0 S+ 08:25 0:00 grep --color=auto efs [root@rhel-8-8-0 ~]# /opt/eset/efs/bin/upd --compile-nups [root@rhel-8-8-0 ~]# echo $? 10 What is the output of cat /usr/lib64/.libcrypto.so.1.1.hmac /usr/lib64/.libssl.so.1.1.hmac rpm -q openssl-libs ldconfig -p | grep -E 'lib(crypto|ssl)' on the system where this fails? Do you have any environment variables set that might affect library loading, such as LD_LIBRARY_PATH?
[root@eset-test ~]# cat /usr/lib64/.libcrypto.so.1.1.hmac /usr/lib64/.libssl.so.1.1.hmac 5c37d015873d96813809db3ccf78eb1823f28c90c4b497039a8705205d4f5515 7b24833bf4ab5d5fa8e3e4f08933a4b8b6f0ac0ba361296bd05810977d0197b5 [root@eset-test ~]# rpm -q openssl-libs openssl-libs-1.1.1k-9.el8_7.x86_64 [root@eset-test ~]# ldconfig -p | grep -E 'lib(crypto|ssl)' libssl3.so (libc6,x86-64) => /lib64/libssl3.so libssl.so.1.1 (libc6,x86-64) => /lib64/libssl.so.1.1 libssl.so (libc6,x86-64) => /lib64/libssl.so libcrypto.so.1.1 (libc6,x86-64) => /lib64/libcrypto.so.1.1 libcrypto.so (libc6,x86-64) => /lib64/libcrypto.so I am testing on a freshly installed system so I don't believe there are any environment variables that would affect library loading. I was able to get ESET to work with the crypto policy set to FIPS:OSPP. FIPS:OSPP seems to be a subset of FIPS so I am unsure why it's working.
I can reproduce this if the openssl-devel package is installed. It does not fail if the package is not installed. Whether the active crypto policy is FIPS:OSPP or FIPS does not seem to matter. Your system also shows a libssl3.so, which is definitely not provided by RHEL, so your system is very clearly not pristine: libssl3.so (libc6,x86-64) => /lib64/libssl3.so This only seems to happen with the EFS package. /opt/eset/efs/bin/upd (which fails) isn't linked against libcrypto or libssl, so it must be dlopen(3)ing libssl or libcrypto. It seems EFS first attempts to dlopen("libcrypto.so"), and if that fails, falls back to dlopen("libcrypto.so.1.1"), which succeeds. The openssl library determines the location of its HMAC checksum file relative to its own path, so when you dlopen("libcrypto.so"), it attempts to locate .libcrypto.so.hmac, which does not exist. On the other hand, users that expect a specific set of symbols in a binary should not dlopen("libcrypto.so"), but use a versioned name such as "libcrypto.so.1.1", or ship a copy of libcrypto compatible with their expectations. This code path has not changed since RHEL 8.1, and was present even before that, so this is not a regression. EFS should not attempt to dlopen an unversioned library name. As a workaround, uninstall the openssl-devel package. I do not believe this is a bug in our package, and am thus closing this as NOTABUG.