Bug 2416681

Summary: gnutls leancrypto lc_kyber_768_dec uses avx instructions even when not available
Product: [Fedora] Fedora Reporter: Mark Wielaard <mjw>
Component: gnutlsAssignee: Red Hat Crypto Team <crypto-team>
Status: CLOSED DUPLICATE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 43CC: ansasaki, crypto-team, dueno, fkrenzel, nkudriavtsev, sam, tm, zfridric
Target Milestone: ---Keywords: Regression
Target Release: ---Flags: fedora-admin-xmlrpc: mirror+
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: ---
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2025-11-25 08:42:38 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Mark Wielaard 2025-11-23 21:33:04 UTC
On older x86_64 cpus (Intel Xeon E5640 here) avx might not be available.
gnutls embedded leancrypto seems to use avx instructions.
This crashes things like wget.

$ wget -q -O /tmp/ltp-full-20250930.tar.xz https://github.com/linux-test-project/ltp/releases/download/20250930/ltp-full-20250930.tar.xz
Illegal instruction        (core dumped) wget -q -O /tmp/ltp-full-20250930.tar.xz https://github.com/linux-test-project/ltp/releases/download/20250930/ltp-full-20250930.tar.xz

Under gdb it shows:

Thread 2 "wget" received signal SIGILL, Illegal instruction.
[Switching to Thread 0x7f6087ff36c0 (LWP 139540)]
Downloading 4.74 K source file /usr/src/debug/gnutls-3.8.11-1.fc43.x86_64/native_build/lib/../../bundled_leancrypto/_build/../ml-kem/src/kyber_selftest.c
0x00007f60896cf62f in lc_kyber_768_kem_dec_selftest (                           
    lc_kyber_768__dec=<optimized out>)
    at ../../bundled_leancrypto/_build/../ml-kem/src/kyber_selftest.c:137
137		struct lc_kyber_ss key_a;
(gdb) disassemble 
Dump of assembler code for function lc_kyber_768_dec:
   0x00007f60896cf5f0 <+0>:	push   %rbp
   0x00007f60896cf5f1 <+1>:	mov    %rsp,%rbp
   0x00007f60896cf5f4 <+4>:	push   %r15
   0x00007f60896cf5f6 <+6>:	push   %r14
   0x00007f60896cf5f8 <+8>:	push   %r13
   0x00007f60896cf5fa <+10>:	push   %r12
   0x00007f60896cf5fc <+12>:	mov    %rsi,%r12
   0x00007f60896cf5ff <+15>:	push   %rbx
   0x00007f60896cf600 <+16>:	mov    %rdi,%rbx
   0x00007f60896cf603 <+19>:	sub    $0x48,%rsp
   0x00007f60896cf607 <+23>:	mov    0x226ec3(%rip),%eax        # 0x7f60898f64d0 <feat.lto_priv.0>
   0x00007f60896cf60d <+29>:	mov    %fs:0x28,%r13
   0x00007f60896cf616 <+38>:	mov    %r13,0x38(%rsp)
   0x00007f60896cf61b <+43>:	mov    %rdx,%r13
   0x00007f60896cf61e <+46>:	test   $0x40000000,%eax
   0x00007f60896cf623 <+51>:	je     0x7f60896cf62a <lc_kyber_768_dec+58>
   0x00007f60896cf625 <+53>:	call   0x7f60896ed280 <lc_cpu_feature_available>
   0x00007f60896cf62a <+58>:	lea    0x10(%rsp),%r14
=> 0x00007f60896cf62f <+63>:	vpxor  %xmm0,%xmm0,%xmm0
   0x00007f60896cf633 <+67>:	mov    %r14,0x8(%rsp)
   0x00007f60896cf638 <+72>:	vmovdqu %ymm0,0x10(%rsp)
   0x00007f60896cf63e <+78>:	lock orq $0x0,(%rsp)
   0x00007f60896cf644 <+84>:	test   $0x4,%al
   0x00007f60896cf646 <+86>:	mov    0x226ea4(%rip),%eax        # 0x7f60898f64f0 <lc_alg_status_kem_pqc.lto_priv.0>
   0x00007f60896cf64c <+92>:	je     0x7f60896cf6d8 <lc_kyber_768_dec+232>
   0x00007f60896cf652 <+98>:	lock orq $0x0,(%rsp)
   0x00007f60896cf658 <+104>:	test   $0x38000,%eax
   0x00007f60896cf65d <+109>:	je     0x7f60896cf770 <lc_kyber_768_dec+384>
   0x00007f60896cf663 <+115>:	vzeroupper
   0x00007f60896cf666 <+118>:	nopl   0x0(%rax)
   0x00007f60896cf66a <+122>:	data16 cs nopw 0x0(%rax,%rax,1)
   0x00007f60896cf675 <+133>:	data16 cs nopw 0x0(%rax,%rax,1)
   0x00007f60896cf680 <+144>:	lock orq $0x0,(%rsp)
   0x00007f60896cf686 <+150>:	mov    0x226e64(%rip),%eax        # 0x7f60898f64f0 <lc_alg_status_kem_pqc.lto_priv.0>


Reproducible: Always

Steps to Reproduce:
$ wget -q -O /tmp/ltp-full-20250930.tar.xz https://github.com/linux-test-project/ltp/releases/download/20250930/ltp-full-20250930.tar.xz

Actual Results:
Illegal instruction        (core dumped) wget -q -O /tmp/ltp-full-20250930.tar.xz https://github.com/linux-test-project/ltp/releases/download/20250930/ltp-full-20250930.tar.xz

Expected Results:
download tar.xz file

Comment 1 Daiki Ueno 2025-11-25 08:42:38 UTC
Marking as a duplicate of bug 2416812, as a tentative fix (i.e., disabling assembly optimization in leancrypto) is being developed there.

*** This bug has been marked as a duplicate of bug 2416812 ***