Bug 2416681 - gnutls leancrypto lc_kyber_768_dec uses avx instructions even when not available
Summary: gnutls leancrypto lc_kyber_768_dec uses avx instructions even when not available
Keywords:
Status: CLOSED DUPLICATE of bug 2416812
Alias: None
Product: Fedora
Classification: Fedora
Component: gnutls
Version: 43
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: Red Hat Crypto Team
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2025-11-23 21:33 UTC by Mark Wielaard
Modified: 2025-12-12 04:06 UTC (History)
8 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2025-11-25 08:42:38 UTC
Type: ---
Embargoed:
fedora-admin-xmlrpc: mirror+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker FC-2685 0 None None None 2025-12-12 04:06:19 UTC

Description Mark Wielaard 2025-11-23 21:33:04 UTC
On older x86_64 cpus (Intel Xeon E5640 here) avx might not be available.
gnutls embedded leancrypto seems to use avx instructions.
This crashes things like wget.

$ wget -q -O /tmp/ltp-full-20250930.tar.xz https://github.com/linux-test-project/ltp/releases/download/20250930/ltp-full-20250930.tar.xz
Illegal instruction        (core dumped) wget -q -O /tmp/ltp-full-20250930.tar.xz https://github.com/linux-test-project/ltp/releases/download/20250930/ltp-full-20250930.tar.xz

Under gdb it shows:

Thread 2 "wget" received signal SIGILL, Illegal instruction.
[Switching to Thread 0x7f6087ff36c0 (LWP 139540)]
Downloading 4.74 K source file /usr/src/debug/gnutls-3.8.11-1.fc43.x86_64/native_build/lib/../../bundled_leancrypto/_build/../ml-kem/src/kyber_selftest.c
0x00007f60896cf62f in lc_kyber_768_kem_dec_selftest (                           
    lc_kyber_768__dec=<optimized out>)
    at ../../bundled_leancrypto/_build/../ml-kem/src/kyber_selftest.c:137
137		struct lc_kyber_ss key_a;
(gdb) disassemble 
Dump of assembler code for function lc_kyber_768_dec:
   0x00007f60896cf5f0 <+0>:	push   %rbp
   0x00007f60896cf5f1 <+1>:	mov    %rsp,%rbp
   0x00007f60896cf5f4 <+4>:	push   %r15
   0x00007f60896cf5f6 <+6>:	push   %r14
   0x00007f60896cf5f8 <+8>:	push   %r13
   0x00007f60896cf5fa <+10>:	push   %r12
   0x00007f60896cf5fc <+12>:	mov    %rsi,%r12
   0x00007f60896cf5ff <+15>:	push   %rbx
   0x00007f60896cf600 <+16>:	mov    %rdi,%rbx
   0x00007f60896cf603 <+19>:	sub    $0x48,%rsp
   0x00007f60896cf607 <+23>:	mov    0x226ec3(%rip),%eax        # 0x7f60898f64d0 <feat.lto_priv.0>
   0x00007f60896cf60d <+29>:	mov    %fs:0x28,%r13
   0x00007f60896cf616 <+38>:	mov    %r13,0x38(%rsp)
   0x00007f60896cf61b <+43>:	mov    %rdx,%r13
   0x00007f60896cf61e <+46>:	test   $0x40000000,%eax
   0x00007f60896cf623 <+51>:	je     0x7f60896cf62a <lc_kyber_768_dec+58>
   0x00007f60896cf625 <+53>:	call   0x7f60896ed280 <lc_cpu_feature_available>
   0x00007f60896cf62a <+58>:	lea    0x10(%rsp),%r14
=> 0x00007f60896cf62f <+63>:	vpxor  %xmm0,%xmm0,%xmm0
   0x00007f60896cf633 <+67>:	mov    %r14,0x8(%rsp)
   0x00007f60896cf638 <+72>:	vmovdqu %ymm0,0x10(%rsp)
   0x00007f60896cf63e <+78>:	lock orq $0x0,(%rsp)
   0x00007f60896cf644 <+84>:	test   $0x4,%al
   0x00007f60896cf646 <+86>:	mov    0x226ea4(%rip),%eax        # 0x7f60898f64f0 <lc_alg_status_kem_pqc.lto_priv.0>
   0x00007f60896cf64c <+92>:	je     0x7f60896cf6d8 <lc_kyber_768_dec+232>
   0x00007f60896cf652 <+98>:	lock orq $0x0,(%rsp)
   0x00007f60896cf658 <+104>:	test   $0x38000,%eax
   0x00007f60896cf65d <+109>:	je     0x7f60896cf770 <lc_kyber_768_dec+384>
   0x00007f60896cf663 <+115>:	vzeroupper
   0x00007f60896cf666 <+118>:	nopl   0x0(%rax)
   0x00007f60896cf66a <+122>:	data16 cs nopw 0x0(%rax,%rax,1)
   0x00007f60896cf675 <+133>:	data16 cs nopw 0x0(%rax,%rax,1)
   0x00007f60896cf680 <+144>:	lock orq $0x0,(%rsp)
   0x00007f60896cf686 <+150>:	mov    0x226e64(%rip),%eax        # 0x7f60898f64f0 <lc_alg_status_kem_pqc.lto_priv.0>


Reproducible: Always

Steps to Reproduce:
$ wget -q -O /tmp/ltp-full-20250930.tar.xz https://github.com/linux-test-project/ltp/releases/download/20250930/ltp-full-20250930.tar.xz

Actual Results:
Illegal instruction        (core dumped) wget -q -O /tmp/ltp-full-20250930.tar.xz https://github.com/linux-test-project/ltp/releases/download/20250930/ltp-full-20250930.tar.xz

Expected Results:
download tar.xz file

Comment 1 Daiki Ueno 2025-11-25 08:42:38 UTC
Marking as a duplicate of bug 2416812, as a tentative fix (i.e., disabling assembly optimization in leancrypto) is being developed there.

*** This bug has been marked as a duplicate of bug 2416812 ***


Note You need to log in before you can comment on or make changes to this bug.