Mark, I assume that the goal is to deliver a PAC-compatible userspace without actually enabling PAC. Is this correct?

That is, unwinders should keep using the xpaclri instruction.

My understanding is that we want to purge all PAC insns from all packages. The concern is that we can't control whether PAC is enabled or not in the case of RHEL containers running on non-RHEL hosts.

All userspace packages.

(In reply to Mark Salter from comment #3)
> My understanding is that we want to purge all PAC insns from all packages.
> The concern is that we can't control whether PAC is enabled or not in the
> case of RHEL containers running on non-RHEL hosts.

If we strip all the PAC instructions, then unwinding (including C++ exception handling) will not work if PAC is enabled because the unwinder (any unwinder actually) will not be able to derive the unsigned address.

This would also need a custom, downstream-only gcc patch. __builtin_return_address currently automatically demangles addresses on AArch64, and this uses the xpaclri instruction.

I think we should merely stop building with -mbranch-protection=standard:

https://gitlab.com/redhat/centos-stream/rpms/redhat-rpm-config/-/merge_requests/7

Okay, agreed.

redhat-rpm-config-186-1.el9 has the change. It currently lives in a side tag (c9s-build-side-221-stack-gate and rhel-9.0.0-beta-build-sidetag-86604-stack-gate).

The next steps are to tag annobin into the side tag and rebuild gcc and glibc.

Status update: I think we have ironed out all the immediate toolchain problems, including bug 1983727. I'm kicking off a test mass rebuild with

annobin-9.83-2.el9
gcc-11.2.1-2.el9
glibc-2.33.9000-55.el9
redhat-rpm-config-187-1.el9

on top of current CentOS 9 Stream, plus:

clang-12.0.1-1.el9
compiler-rt-12.0.1-1.el9
llvm-12.0.1-1.el9

to avoid failures in the LLVM stack related to the pending LLVM 12.0.1 transition. There will be additional known build failures due to Vulkan stack transition which is currently in progress as well.

I have rebuilt all packages in the side tag against the side tag, as scratch builds, adding the following builds to avoid build failures to transitions in flight (in other side tags):

checkpolicy-3.2-3.el9
glslang-11.5.0-1.20210623.gitae2a562.el9
jansi-2.3.3-2.el9
libselinux-3.2-5.el9
libsemanage-3.2-3.el9
libsepol-3.2-3.el9
policycoreutils-3.2-5.el9
shaderc-2020.5-3.el9
spirv-headers-1.5.4-5.20210617.git07f259e.el9
spirv-tools-2021.2-1.20210619.git5dd2f76.el9
vulkan-headers-1.2.182.0-2.el9
vulkan-loader-1.2.182.0-1.el9
vulkan-tools-1.2.182.0-1.el9
vulkan-validation-layers-1.2.182.0-1.el9

The result I installed with “dnf reinstall \*” on aarch64, ppc64, s390x, x86_64 machines.  Typical output of build times looks like this:

# rpm -qa --qf '%{buildtime} %{buildtime:day} %{name}-%{version}-%{release}\n' | sort -n | head -n 30
1533796849 Thu Aug 09 2018 beaker-core-tasks-distribution-check-install-1.0-2
1598392021 Tue Aug 25 2020 redhat-internal-cert-install-0.1-23.el7.csb
1615584576 Fri Mar 12 2021 emacs-filesystem-27.1-3.el9
1615584822 Fri Mar 12 2021 emacs-27.1-3.el9
1615584822 Fri Mar 12 2021 emacs-common-27.1-3.el9
1617738584 Tue Apr 06 2021 restraint-0.3.3-1.el9
1617738584 Tue Apr 06 2021 restraint-rhts-0.3.3-1.el9
1618532777 Thu Apr 15 2021 fuse-2.9.9-12.el9
1618532777 Thu Apr 15 2021 fuse-libs-2.9.9-12.el9
1618543166 Thu Apr 15 2021 libffi-3.1-29.el9
1623816190 Wed Jun 16 2021 openssl-pkcs11-0.4.11-6.el9
1623816677 Wed Jun 16 2021 postfix-3.5.9-9.el9
1623818273 Wed Jun 16 2021 zchunk-libs-1.1.9-4.el9
1625246368 Fri Jul 02 2021 beaker-distribution-reservesys-3.4-12
1626206556 Tue Jul 13 2021 kpartx-0.8.6-4.el9
1627373744 Tue Jul 27 2021 redhat-release-9.0-1.8.el9
1627373744 Tue Jul 27 2021 redhat-release-eula-9.0-1.8.el9
1627487207 Wed Jul 28 2021 iptables-services-1.8.7-18.el9
1627653866 Fri Jul 30 2021 libtool-ltdl-2.4.6-41.el9
1627664178 Fri Jul 30 2021 iptables-libs-1.8.7-20.el9
1627664178 Fri Jul 30 2021 iptables-nft-1.8.7-20.el9
1627664178 Fri Jul 30 2021 iptables-nft-services-1.8.7-20.el9
1627664305 Fri Jul 30 2021 device-mapper-persistent-data-0.9.0-9.el9
1627664370 Fri Jul 30 2021 usermode-1.114-1.el9
1627664422 Fri Jul 30 2021 librhsm-0.0.3-6.el9
1627664870 Fri Jul 30 2021 filesystem-3.14-8.el9
1627664872 Fri Jul 30 2021 libaio-0.3.111-12.el9
1627664873 Fri Jul 30 2021 inih-49-4.el9
1627665403 Fri Jul 30 2021 libzstd-1.5.0-1.el9
1627665466 Fri Jul 30 2021 kmod-28-4.el9

I had some issues with UEFI BootNext settings (not uncommon in Beaker unfortunately), but the results look quite good to me. The machines were still booting and some basic programs work as expected. I think the side tag is ready for merging in the sense that it shouldn't impact other teams.

I found one grub2 build issue and filed bug 1988732 for it. The other new build failures are due to other churn in the distribution.

Looking at annocheck reports, the new failures I saw are all real, more or less (autofs linking without any hardening flags, cups using -fstack-protector-all instead of -fstack-protector-strong, pcre2 reporting protection gaps due to missing annobin plugin at LTO link).

redhat-rpm-config-187-1.el9 with this change is in gating, via rhel-9.0.0-beta-build-sidetag-86604-stack-gate.

It was discovered that several packages capture compiler flags and store them in their own built packages, potentially using them for building other packages. As a result, updating redhat-rpm-config alone is not effective.

I have rebuilt the following actually packages:

  apr-1.7.0-9.5.el9
  compiler-rt-12.0.1-2.el9
  httpd-2.4.48-14.el9
  libtool-2.4.6-42.el9
  mpich-3.4.1-3.el9
  mvapich2-2.3.6-2.el9
  (ocaml-4.11.1-3.el9.2 present in c9s-build)
  perl-5.32.1-478.el9
  php-8.0.6-7.el9
  postgresql-13.3-7.el9
  python3.9-3.9.6-4.el9
  qt5-qtbase-5.15.2-24.el9
  ruby-3.0.1-151.el9
  (sip-4.19.24-5.el9 present in c9s-build)
  (tcl-8.6.10-6.el9 present in c9s-build)
  uuid-1.6.2-52.el9

The builds in parentheses have already been moved into the buildroot.