Bug 2116610

Summary: trap invalid opcode ip:7feef81809fe sp:7fee997419c0 error:0 in libgnutls.so.30.28.2[7feef8040000+1dd000]
Product: Red Hat Enterprise Linux 8 Reporter: Ravindra Patil <ravpatil>
Component: gnutlsAssignee: Daiki Ueno <dueno>
Status: CLOSED CURRENTRELEASE QA Contact: Alexander Sosedkin <asosedki>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 8.6CC: asosedki, cllang, dareynol, dueno, ldu, mmatsuya, qguo, rhayakaw, rmetrich, sbroz
Target Milestone: rcKeywords: Regression, Triaged, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: gnutls-3.6.16-6.el8 Doc Type: Bug Fix
Doc Text:
Cause: gnutls detection of AVX instructions does not function correctly if x86 extended register state save and restore is disabled, e.g., with a `noxsave` kernel parameter Consequence: gnutls crashes with a `trap invalid opcode` error message Fix: gnutls abstains from using AVX-accelerated code if XSAVE is not detected Result: disabling XSAVE on AVX-enabled hosts doesn't cause gnutls to crash
Story Points: ---
Clone Of:
: 2131152 2136052 (view as bug list) Environment:
Last Closed: 2023-06-05 14:23:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2073863, 2131152, 2136052    

Description Ravindra Patil 2022-08-09 03:20:53 UTC
Description of problem:

When customer enables FIPS on RHEL 8.6 (VMWare ESXI Server 7.0.0, CPU- intel i9-10980XE) it leads to multiple SIGILL service failures related to libgnutls.so.30.28.2.

Multiple services are failing

  UNIT                        LOAD   ACTIVE SUB    DESCRIPTION                                        
* cups.path                   loaded failed failed CUPS Scheduler                                     
* accounts-daemon.service     loaded failed failed Accounts Service                                   
* chronyd.service             loaded failed failed NTP client/server                                  
* cups.service                loaded failed failed CUPS Scheduler                                     
* firewalld.service           loaded failed failed firewalld - dynamic firewall daemon                
* fprintd.service             loaded failed failed Fingerprint Authentication Daemon                  
* gdm.service                 loaded failed failed GNOME Display Manager                              

All services are failing with similar Illegal Instruction related to gnutls library. 

Jul 26 09:43:23 fixme kernel: traps: pool[6677] trap invalid opcode ip:7feef81809fe sp:7fee997419c0 error:0 in libgnutls.so.30.28.2[7feef8040000+1dd000]

Jul 26 09:47:34 fixme kernel: traps: fwupd[4666] trap invalid opcode ip:7ff45d02e9bb sp:7ffd2dcd6340 error:0 in libgnutls.so.30.28.2[7ff45cef3000+1dd000]

Jul 26 09:49:16 fixme kernel: traps: NetworkManager[7750] trap invalid opcode 
ip:7ff828e7b9bb sp:7ffd5c110d00 error:0 in libgnutls.so.30.28.2[7ff828d40000+1dd000]

Jul 26 09:50:11 fixme kernel: traps: nm-initrd-gener[407] trap invalid opcode ip:7fd8f313c9bb sp:7ffcd0833960 error:0 in libgnutls.so.30.28.2[7fd8f3001000+1dd000]

Jul 26 09:50:17 fixme kernel: traps: ostree-system-g[1044] trap invalid opcode ip:7f532c31e9bb sp:7fff82ee52f0 error:0 in libgnutls.so.30.28.2[7f532c1e3000+1dd000]

Jul 26 09:50:17 fixme kernel: traps: libinput-device[1117] trap invalid opcode ip:7f5b7649e9bb sp:7fff5f22d870 error:0 in libgnutls.so.30.28.2[7f5b76363000+1dd000]

Jul 26 09:50:17 fixme kernel: traps: rename_device[1122] trap invalid opcode ip:7f109da189bb sp:7ffeec7356d0 error:0 in libgnutls.so.30.28.2[7f109d8dd000+1dd000]

Jul 26 09:52:18 fixme kernel: traps: irqbalance[1264] trap invalid opcode ip:7f3bb29419bb sp:7fff61bf5cf0 error:0 in libgnutls.so.30.28.2[7f3bb2806000+1dd000]
var/log/messages:Jul 26 09:52:18 fixme kernel: traps: VGAuthService[1250] trap invalid opcode ip:7f17b56349bb sp:7ffdf1c91460 error:0 in libgnutls.so.30.28.2[7f17b54f9000+1dd000]

Jul 26 09:52:18 fixme kernel: traps: chronyd[1249] trap invalid opcode ip:7fbeed47a9bb sp:7ffe3156df10 error:0 in libgnutls.so.30.28.2[7fbeed33f000+1dd000]

Jul 26 09:52:19 fixme kernel: traps: vmtoolsd[1251] trap invalid opcode ip:7f776159d9bb sp:7ffe0dfd5740 error:0 in libgnutls.so.30.28.2[7f7761462000+1dd000]

Jul 26 09:52:19 fixme kernel: traps: polkitd[1259] trap invalid opcode ip:7f212eb7f9bb sp:7ffdbe375df0 error:0 in libgnutls.so.30.28.2[7f212ea44000+1dd000]

Application cores are dumped for every failure of the service.

$ for file in ./*; do echo "==== $file ====="; eu-unstrip -n --core=./$file | head -1; done 

==== ./core.1120 =====
0x55b383619000+0x204000 1cefa16ca358662c49252fb994385ea710882c7b@0x55b383619318 . - /usr/lib/udev/libinput-device-group
==== ./core.1123 =====
0x55eabe18b000+0x204000 16afc62b49176225a44c21534425ae24a0489f9e@0x55eabe18b318 . - /usr/lib/udev/rename_device
==== ./core.1125 =====
0x558a1ca5c000+0x204000 1cefa16ca358662c49252fb994385ea710882c7b@0x558a1ca5c318 . - /usr/lib/udev/libinput-device-group
==== ./core.1244 =====
0x555ab63d9000+0x222000 7ba8bbe851de95c165699696bfeb126e7b8e9511@0x555ab63d9318 . - /usr/bin/VGAuthService
==== ./core.1245 =====
0x55ba766ea000+0x212000 85bce56080f9749a780aa591c7a3d258d33d3184@0x55ba766ea318 . - /usr/bin/vmtoolsd
==== ./core.1250 =====
0x55561a453000+0x279000 758c98101436dfbee5e6ce4db5d68f74736ada4f@0x55561a453350 . - /usr/libexec/udisks2/udisksd
==== ./core.1269 =====
0x55836fc92000+0x203000 81906b98278d4eed5faafd38499331b4c56daa50@0x55836fc92284 . - /usr/libexec/platform-python3.6
==== ./core.1285 =====
0x55caa8314000+0x569000 e128c83e67fd41b39f0938e628d4a669bd134284@0x55caa8314350 . - /usr/sbin/NetworkManager
==== ./core.1295 =====
0x555822248000+0x206000 8b767f9bbf3814840747c2d8963ac3b745d1f188@0x555822248318 . - /usr/bin/rhsmcertd
==== ./core.1300 =====
0x5615bd7ee000+0x271000 ddfff834b1549302a0597251b8547cfdd8421bd6@0x5615bd7ee318 . - /usr/sbin/cupsd
==== ./core.1389 =====
0x558994f4e000+0x271000 ddfff834b1549302a0597251b8547cfdd8421bd6@0x558994f4e318 . - /usr/sbin/cupsd
==== ./core.1396 =====
0x55e46d6b8000+0x27c000 aaa3aec7fde3dab9bb6de50c799ab300d24ffee9@0x55e46d6b8318 . - /usr/sbin/libvirtd
==== ./core.1398 =====
0x563a68762000+0x270000 5b56d0af4627ef298324dd406d14966c60017f51@0x563a68762318 . - /usr/sbin/gdm
==== ./core.1491 =====
0x55c44da48000+0x270000 5b56d0af4627ef298324dd406d14966c60017f51@0x55c44da48318 . - /usr/sbin/gdm
0x55666f696000+0x270000 5b56d0af4627ef298324dd406d14966c60017f51@0x55666f696318 . - /usr/sbin/gdm
==== ./core.1513 =====
0x564e78466000+0x27c000 aaa3aec7fde3dab9bb6de50c799ab300d24ffee9@0x564e78466318 . - /usr/sbin/libvirtd
==== ./core.1514 =====
0x558ce2347000+0x270000 5b56d0af4627ef298324dd406d14966c60017f51@0x558ce2347318 . - /usr/sbin/gdm
==== ./core.1522 =====
0x561de0adf000+0x270000 5b56d0af4627ef298324dd406d14966c60017f51@0x561de0adf318 . - /usr/sbin/gdm
==== ./core.1523 =====
0x55f0fe451000+0x271000 ddfff834b1549302a0597251b8547cfdd8421bd6@0x55f0fe451318 . - /usr/sbin/cupsd
==== ./core.1531 =====
0x55f14440c000+0x27c000 aaa3aec7fde3dab9bb6de50c799ab300d24ffee9@0x55f14440c318 . - /usr/sbin/libvirtd
==== ./core.1539 =====
0x55ad61ec1000+0x27c000 aaa3aec7fde3dab9bb6de50c799ab300d24ffee9@0x55ad61ec1318 . - /usr/sbin/libvirtd
==== ./core.1540 =====
0x5593f0aaf000+0x271000 ddfff834b1549302a0597251b8547cfdd8421bd6@0x5593f0aaf318 . - /usr/sbin/cupsd
==== ./core.1747 =====
0x560cb7eb5000+0x391000 32087fea0eb14d63551785f4fffd2c573c82605b@0x560cb7eb5318 . - /usr/bin/flatpak

> The services are dumping core and all these are related to gnutls library with same error logs.
 
traps: service[PID] trap invalid opcode ip:7f17b56349bb sp:7ffdf1c91460 error:0 in libgnutls.so.30.28.2

Version-Release number of selected component (if applicable):
gnutls-3.6.16-4.el8.x86_64

How reproducible:

After upgrading from RHEL 8.5 to 8.6, first boot would show multiple failing services (process core dumps) and invalid opcode messages (libgnutls.so.30.28.2). 

Steps to Reproduce:
1. Upgrade RHEL 8.5 to 8.6
or
2. Install fresh RHEL 8.6
3. Enable FIPS mode 
4. Reboot the system

Actual results:
- Multiple services fails to start with core dump. 

traps: service[PID] trap invalid opcode ip:7f17b56349bb sp:7ffdf1c91460 error:0 in libgnutls.so.30.28.2

Expected results:
- There should be no such service failures in FIPS mode. 

Additional info:

Comment 24 Andreas Schneider 2023-01-10 09:03:16 UTC
*** Bug 2156261 has been marked as a duplicate of this bug. ***