Bug 2222040

Summary: /lib64/libLLVM-16.so: rebuild shared object with SHSTK support enabled
Product: Red Hat Enterprise Linux 9 Reporter: Juro Bystricky <juro.bystricky>
Component: llvmAssignee: Nikita Popov <npopov>
Status: VERIFIED --- QA Contact: Jesus Checa <jchecahi>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: CentOS StreamCC: bstinson, jchecahi, jwboyer, mprchlik, npopov, pragyansri.pathi, sipoyare
Target Milestone: rcKeywords: Triaged
Target Release: 9.3   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: llvm-16.0.6-4.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Juro Bystricky 2023-07-11 16:16:25 UTC
Description of problem:
While testing CentOS Stream 9 with a custom kernel with CET enabled (user shadow stack), we were not able to launch a VNC client. The tests were done in anticipation of CET support being merged into upstream Linux kernel in near future.

The journal shows this error:
srv-bkc gnome-session[3706]: gnome-session-is-accelerated: No hardware 3D support.
srv-bkc gnome-session[3697]: gnome-session-check-accelerated: GL Helper exited with code 256
srv-bkc gnome-session[3707]: libEGL warning: MESA-LOADER: failed to open swrast: /lib64/libLLVM-16.so: rebuild shared object with SHSTK support enabled (search paths /usr/lib64/dri, suffix _dri)
srv-bkc gnome-session-c[3707]: eglInitialize() failed
srv-bkc gnome-session[3697]: gnome-session-check-accelerated: GLES Helper exited with code 256
srv-bkc gnome-session[3651]: gnome-session-binary[3651]: WARNING: software acceleration check failed: Child process exited with code 1

Note: setting GLIBC_TUNABLES=glibc.cpu.x86_shstk=permissive did not resolve the issue.

Version-Release number of selected component (if applicable):
llvm-libs-16.0.1-1.el9.x86_64 : LLVM shared libraries

How reproducible:
100%

Steps to Reproduce:

1. Install tigervnc server + Gnome desktop, configure the server for a user
2. Start the vncserver for the user
3. Attempt to connect to the host via VNC

Actual results:

VNC client does not connect and reports an error. Host VNC server dead as a consequence:

$ sudo systemctl status vncserver@:3
○ vncserver@:3.service - Remote desktop service (VNC)
     Loaded: loaded (/usr/lib/systemd/system/vncserver@.service; enabled; preset: disabled)
     Active: inactive (dead) since Tue 2023-07-11 12:09:46 EDT; 2min 10s ago
   Duration: 38min 43.901s
    Process: 3461 ExecStartPre=/usr/libexec/vncsession-restore :3 (code=exited, status=0/SUCCESS)
    Process: 3539 ExecStart=/usr/libexec/vncsession-start :3 (code=exited, status=0/SUCCESS)
   Main PID: 3560 (code=exited, status=0/SUCCESS)
        CPU: 40ms
 

Expected results:

VNC connection working

Additional info:
The problem can avoided if disabling CET ( booting the kernel with command line argument "nousershstk")

Comment 1 Nikita Popov 2023-07-19 11:59:44 UTC
Building with -DCMAKE_SHARED_LINKER_FLAGS="-Wl,-z,cet-report=error" gives:

/opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../bin/ld: lib64/libLLVMSupport.a(blake3_sse2_x86-64_unix.S.o): error: missing IBT and SHSTK properties
/opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../bin/ld: lib64/libLLVMSupport.a(blake3_sse41_x86-64_unix.S.o): error: missing IBT and SHSTK properties
/opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../bin/ld: lib64/libLLVMSupport.a(blake3_avx2_x86-64_unix.S.o): error: missing IBT and SHSTK properties
/opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../bin/ld: lib64/libLLVMSupport.a(blake3_avx512_x86-64_unix.S.o): error: missing IBT and SHSTK properties

And looking at the build log:

[127/2534] /usr/bin/clang -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Ilib/Support/BLAKE3 -I../lib/Support/BLAKE3 -Iinclude -I../include -fPIC -O2 -g -DNDEBUG -MD -MT lib/Support/BLAKE3/CMakeFiles/LLVMSupportBlake3.dir/blake3_sse2_x86-64_unix.S.o -MF lib/Support/BLAKE3/CMakeFiles/LLVMSupportBlake3.dir/blake3_sse2_x86-64_unix.S.o.d -o lib/Support/BLAKE3/CMakeFiles/LLVMSupportBlake3.dir/blake3_sse2_x86-64_unix.S.o -c ../lib/Support/BLAKE3/blake3_sse2_x86-64_unix.S

This is indeed missing the -fcf-protection flag. This is weird because we do have

# Copy CFLAGS into ASMFLAGS, so -fcf-protection is used when compiling assembly files.
export ASMFLAGS=$CFLAGS

in the spec file, which exists to avoid exactly this problem, and that does work for our Fedora builds.

From higher up in the build log:

+ cd /builddir/build/BUILD
+ cd llvm-16.0.6.src
+ export ASMFLAGS=
+ ASMFLAGS=
+ CFLAGS='-O2 -flto=thin -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS --config /usr/lib/rpm/redhat/redhat-hardened-clang.cfg -fstack-protector-strong   -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection'
+ export CFLAGS

So apparently on c9s CFLAGS only gets assigned after we've already tried to set ASMFLAGS based on it. I assume it happens as part of the %cmake macro.

So we probably should do something like "export ASMFLAGS=%{build_cflags}" instead.