I do not think trampoline selection can be tuned in this way. Even if it worked, it would be something that could be used for debugging purposes only. For things like string function selection, not using available CPU features is fine because the outcome is identical. But trampoline selection affects how the register file is saved, and once those registers are available in the CPU, the dynamic linker has to deal with them. The XSAVE/XSAVEC trampolines are currently save and restore AVX/AVX2/AVX-512 state, so that has to be used once the supports %ymm (or %zmm) registers.

It is possible to disable OSXSAVE support globally by booting with the “noxsave” kernel option. Individual processes can be launched using the “LD_BIND_NOW=1” environment variable to disable the trampoline. Linking with the -Wl,-z,now compiler option disables trampolines for individual shared objects/main programs. (Red Hat Enterprise Linux 8 system libraries are almost all linked with -Wl,-z,now, so they will not use the trampoline internally.)

I'd be happy to talk to the customer to better understand their requirements. This will require upstream implementation first, and without understanding their needs, I cannot make a case for the change upstream.

Created attachment 1762644 [details]
Patch to fix OSXSAVE tunable support - v1

This patch borrows from the same reasoning behind the XSAVEC_Usable tunable which changes xsave_state_size to disable XSAVEC in favor of XSAVE.

Since OSXSAVE should disable both XSAVEC and XSAVE in favor of FXSAVE, it sets xsave_state_size to zero which later in the loader code will have the effect of patching the GOT with _dl_runtime_resolve_fxsave as the _dl_runtime_resolve implementation.

(In reply to Florian Weimer from comment #1)
> I do not think trampoline selection can be tuned in this way. Even if it
> worked, it would be something that could be used for debugging purposes
> only. For things like string function selection, not using available CPU
> features is fine because the outcome is identical. But trampoline selection
> affects how the register file is saved, and once those registers are
> available in the CPU, the dynamic linker has to deal with them. The
> XSAVE/XSAVEC trampolines are currently save and restore AVX/AVX2/AVX-512
> state, so that has to be used once the supports %ymm (or %zmm) registers.
> 
> It is possible to disable OSXSAVE support globally by booting with the
> “noxsave” kernel option. Individual processes can be launched using the
> “LD_BIND_NOW=1” environment variable to disable the trampoline. Linking with
> the -Wl,-z,now compiler option disables trampolines for individual shared
> objects/main programs. (Red Hat Enterprise Linux 8 system libraries are
> almost all linked with -Wl,-z,now, so they will not use the trampoline
> internally.)
> 
> I'd be happy to talk to the customer to better understand their
> requirements. This will require upstream implementation first, and without
> understanding their needs, I cannot make a case for the change upstream.

Hi Florian, thank you for the quick reply.

We need to disable XSAVE and XSAVEC in favor of FXSAVE in the dynamic linker because we need to migrate a running process from one machine to another with a different CPUs. Different CPUs might have different xsave_state_size values, which results in SIGSEGV in xsave/xsavec in the dynamic linker. Disabling xsave/xsavec in favor of fxsave on glibc fixes this problem.

Unfortunately, using LD_BIND_NOW=1 or -Wl,-z,now is not an option for us. Binaries are too big and complex.

Additionally, I do feel this is a bug. The glibc tunables allows "application authors and distribution maintainers to alter the runtime library behavior to match their workload" [1]. In this case, the -OSXSAVE tunable should alter glibc behavior by completely avoiding XSAVE/XSAVEC instructions since OSXSAVE is mandatory for those according to the Intel® 64 and IA-32 Architectures Software Developer’s Manual.

[1] https://www.gnu.org/software/libc/manual/html_node/Tunables.html

(In reply to Dante from comment #3)
> We need to disable XSAVE and XSAVEC in favor of FXSAVE in the dynamic linker
> because we need to migrate a running process from one machine to another
> with a different CPUs. Different CPUs might have different xsave_state_size
> values, which results in SIGSEGV in xsave/xsavec in the dynamic linker.
> Disabling xsave/xsavec in favor of fxsave on glibc fixes this problem.

Thanks for providing additional context.

Are you able to share more details about the kind of process migration technology being used? Is it VM-based? Or userspace checkpoint and restore, like CRIU? Or similar to the old Emacs unexec mechanism?

> Additionally, I do feel this is a bug. The glibc tunables allows
> "application authors and distribution maintainers to alter the runtime
> library behavior to match their workload" [1]. In this case, the -OSXSAVE
> tunable should alter glibc behavior by completely avoiding XSAVE/XSAVEC
> instructions since OSXSAVE is mandatory for those according to the Intel® 64
> and IA-32 Architectures Software Developer’s Manual.
> 
> [1] https://www.gnu.org/software/libc/manual/html_node/Tunables.html

I'm not sure if it is possible to avoid XSAVE/XSAVEC in glibc while preserving correctness as far as the rest of the (system) libraries are concerned. The tunables do not mask the features at the CPUID level, so the registers are still there, can be used by application code, and may need saving and restoring.

(In reply to Florian Weimer from comment #4)
> (In reply to Dante from comment #3)
> > We need to disable XSAVE and XSAVEC in favor of FXSAVE in the dynamic linker
> > because we need to migrate a running process from one machine to another
> > with a different CPUs. Different CPUs might have different xsave_state_size
> > values, which results in SIGSEGV in xsave/xsavec in the dynamic linker.
> > Disabling xsave/xsavec in favor of fxsave on glibc fixes this problem.
> 
> Thanks for providing additional context.
> 
> Are you able to share more details about the kind of process migration
> technology being used? Is it VM-based? Or userspace checkpoint and restore,
> like CRIU? Or similar to the old Emacs unexec mechanism?

Its an userspace checkpoint and restore, like CRIU.

> 
> > Additionally, I do feel this is a bug. The glibc tunables allows
> > "application authors and distribution maintainers to alter the runtime
> > library behavior to match their workload" [1]. In this case, the -OSXSAVE
> > tunable should alter glibc behavior by completely avoiding XSAVE/XSAVEC
> > instructions since OSXSAVE is mandatory for those according to the Intel® 64
> > and IA-32 Architectures Software Developer’s Manual.
> > 
> > [1] https://www.gnu.org/software/libc/manual/html_node/Tunables.html
> 
> I'm not sure if it is possible to avoid XSAVE/XSAVEC in glibc while
> preserving correctness as far as the rest of the (system) libraries are
> concerned. The tunables do not mask the features at the CPUID level, so the
> registers are still there, can be used by application code, and may need
> saving and restoring.

The glibc tunables manual page suggests that system libraries other then glibc are out of the scope of glibc tunables, i.e., tunables should only work on glibc itself. This is precisely the case with the XSAVEC_Usable tunable. Setting it does not prevent other libraries from using the XSAVEC machine instruction, but does prevent glibc itself from using it. Same behavior should apply to OSXSAVE, but currently don't.

Currently, setting GLIBC_TUNABLES="glibc.tune.hwcaps=-OSXSAVE" changes no behavior at all in glibc. A quick grep -r OSXSAVE on glibc source should be sufficient to confirm that.

Notes for QE:

1. The test Andrew details in comment 0 should be run on a reasonably new CPU with 'xsave'. As far as I understand it, xsave is available on Intel generations starting from Sandy Bridge.
It's possible to grep for xsave in the flags line of /proc/cpuinfo to verify that the instruction is available.

2. The tunable should be: GLIBC_TUNABLES="glibc.cpu.hwcaps=-XSAVE" (or -OSXSAVE, or both like "-XSAVE,-OSXSAVE").

3. On a CPU *with* xsave, setting the tunable should result in hitting the fxsave breakpoint, and *not* setting the tunable should result in hitting the *xsave* breakpoint. i.e. if xsave is not disabled, it should be preferred over fxsave.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (glibc bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2005