The assembly files in libvmaf (*.asm) get built with nasm and are not built with CET support. As a result the final library libvmaf.so does not get built with shadow stack and IBT markup. Based on skimming the two asm files, it seems likely that there is no need to modify the code to make it CET-compatible; there seem to be no indirect calls to these functions and they don't appear to be switching stacks on Linux. Hence it may be a simple matter of forcing the linker to generate the library with the correct markup[1]. However it would be useful if someone more familiar with the code takes a look. Without this, when Fedora is booted with a shadow stack enabled kernel (patches are currently in review upstream[2]), a number of php and python packages fail to build because of lacking SHSTK support in libvmaf. [1] https://src.fedoraproject.org/rpms/vmaf/pull-request/1# [2] https://lore.kernel.org/lkml/20230613001108.3040476-1-rick.p.edgecombe@intel.com/ Reproducible: Always
Thanks for the report. > As a result the final library libvmaf.so does not get built with shadow stack and IBT markup What is the problem with that ? (security? runtime miss-behavior ?) I'm not sure for what problem this is a solution ? CET looks an experimental technology and I'm not sure why vmaf eye in that respect, specially as this is used in video encoding process that are sensible to performance, I'm not sure how to bench for this change... Probably this needs a proper Fedora feature process to properly describes the expectation with this CET feature in Fedora. Also if nasm looks affected, maybe the proper fix should be held at the nasm level upstream instead of dealing with one or another nasm generated library ? Hope this helps.
The core problems are that (1) it *accidentally* misses the shadow stack markup and (2) libvmaf missing shadow stack markup results in large swathes of the php and python ecosystem breaking because they're properly built with shadow stack markup. CET is a security feature in x86 processors and support has been in Fedora for some years now (all packages are built with -fcf-protection by default), it's just that kernel support hasn't been added yet and as a result, this hasn't really been testable. Now that support is on the horizon (see [2] in comment 0), I'm testing packages to make sure that they're actually working as advertised. Also, nasm is not really buggy, it's just a deficiency in that it cannot get simple fixes from cet.h like binutils/gcc built assembly code could; one would have to write custom assembly to add the ELF markup or enforce it during linking.
This bug appears to have been reported against 'rawhide' during the Fedora Linux 39 development cycle. Changing version to 39.