Bug 1984652
| Summary: | redhat-rpm-config: Remove -mbranch-protection=standard from aarch64 compiler flags | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | Mark Salter <msalter> |
| Component: | redhat-rpm-config | Assignee: | Florian Weimer <fweimer> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Eva Mrakova <emrakova> |
| Severity: | urgent | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 9.0 | CC: | akanksha.jain2, antorres, cstratak, drady, ed, fweimer, jamartis, jbastian, jfeeney, jlinton, jpazdziora, mas, mpolacek, pemensik, rvr, steve.capper, zdohnal |
| Target Milestone: | beta | Keywords: | Triaged |
| Target Release: | --- | ||
| Hardware: | aarch64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | redhat-rpm-config-187-1.el9 | Doc Type: | No Doc Update |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-12-07 21:35:16 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1967565, 1984995, 1988950 | ||
Mark, I assume that the goal is to deliver a PAC-compatible userspace without actually enabling PAC. Is this correct? That is, unwinders should keep using the xpaclri instruction. My understanding is that we want to purge all PAC insns from all packages. The concern is that we can't control whether PAC is enabled or not in the case of RHEL containers running on non-RHEL hosts. All userspace packages. (In reply to Mark Salter from comment #3) > My understanding is that we want to purge all PAC insns from all packages. > The concern is that we can't control whether PAC is enabled or not in the > case of RHEL containers running on non-RHEL hosts. If we strip all the PAC instructions, then unwinding (including C++ exception handling) will not work if PAC is enabled because the unwinder (any unwinder actually) will not be able to derive the unsigned address. This would also need a custom, downstream-only gcc patch. __builtin_return_address currently automatically demangles addresses on AArch64, and this uses the xpaclri instruction. I think we should merely stop building with -mbranch-protection=standard: https://gitlab.com/redhat/centos-stream/rpms/redhat-rpm-config/-/merge_requests/7 Okay, agreed. redhat-rpm-config-186-1.el9 has the change. It currently lives in a side tag (c9s-build-side-221-stack-gate and rhel-9.0.0-beta-build-sidetag-86604-stack-gate). The next steps are to tag annobin into the side tag and rebuild gcc and glibc. Status update: I think we have ironed out all the immediate toolchain problems, including bug 1983727. I'm kicking off a test mass rebuild with annobin-9.83-2.el9 gcc-11.2.1-2.el9 glibc-2.33.9000-55.el9 redhat-rpm-config-187-1.el9 on top of current CentOS 9 Stream, plus: clang-12.0.1-1.el9 compiler-rt-12.0.1-1.el9 llvm-12.0.1-1.el9 to avoid failures in the LLVM stack related to the pending LLVM 12.0.1 transition. There will be additional known build failures due to Vulkan stack transition which is currently in progress as well. I have rebuilt all packages in the side tag against the side tag, as scratch builds, adding the following builds to avoid build failures to transitions in flight (in other side tags):
checkpolicy-3.2-3.el9
glslang-11.5.0-1.20210623.gitae2a562.el9
jansi-2.3.3-2.el9
libselinux-3.2-5.el9
libsemanage-3.2-3.el9
libsepol-3.2-3.el9
policycoreutils-3.2-5.el9
shaderc-2020.5-3.el9
spirv-headers-1.5.4-5.20210617.git07f259e.el9
spirv-tools-2021.2-1.20210619.git5dd2f76.el9
vulkan-headers-1.2.182.0-2.el9
vulkan-loader-1.2.182.0-1.el9
vulkan-tools-1.2.182.0-1.el9
vulkan-validation-layers-1.2.182.0-1.el9
The result I installed with “dnf reinstall \*” on aarch64, ppc64, s390x, x86_64 machines. Typical output of build times looks like this:
# rpm -qa --qf '%{buildtime} %{buildtime:day} %{name}-%{version}-%{release}\n' | sort -n | head -n 30
1533796849 Thu Aug 09 2018 beaker-core-tasks-distribution-check-install-1.0-2
1598392021 Tue Aug 25 2020 redhat-internal-cert-install-0.1-23.el7.csb
1615584576 Fri Mar 12 2021 emacs-filesystem-27.1-3.el9
1615584822 Fri Mar 12 2021 emacs-27.1-3.el9
1615584822 Fri Mar 12 2021 emacs-common-27.1-3.el9
1617738584 Tue Apr 06 2021 restraint-0.3.3-1.el9
1617738584 Tue Apr 06 2021 restraint-rhts-0.3.3-1.el9
1618532777 Thu Apr 15 2021 fuse-2.9.9-12.el9
1618532777 Thu Apr 15 2021 fuse-libs-2.9.9-12.el9
1618543166 Thu Apr 15 2021 libffi-3.1-29.el9
1623816190 Wed Jun 16 2021 openssl-pkcs11-0.4.11-6.el9
1623816677 Wed Jun 16 2021 postfix-3.5.9-9.el9
1623818273 Wed Jun 16 2021 zchunk-libs-1.1.9-4.el9
1625246368 Fri Jul 02 2021 beaker-distribution-reservesys-3.4-12
1626206556 Tue Jul 13 2021 kpartx-0.8.6-4.el9
1627373744 Tue Jul 27 2021 redhat-release-9.0-1.8.el9
1627373744 Tue Jul 27 2021 redhat-release-eula-9.0-1.8.el9
1627487207 Wed Jul 28 2021 iptables-services-1.8.7-18.el9
1627653866 Fri Jul 30 2021 libtool-ltdl-2.4.6-41.el9
1627664178 Fri Jul 30 2021 iptables-libs-1.8.7-20.el9
1627664178 Fri Jul 30 2021 iptables-nft-1.8.7-20.el9
1627664178 Fri Jul 30 2021 iptables-nft-services-1.8.7-20.el9
1627664305 Fri Jul 30 2021 device-mapper-persistent-data-0.9.0-9.el9
1627664370 Fri Jul 30 2021 usermode-1.114-1.el9
1627664422 Fri Jul 30 2021 librhsm-0.0.3-6.el9
1627664870 Fri Jul 30 2021 filesystem-3.14-8.el9
1627664872 Fri Jul 30 2021 libaio-0.3.111-12.el9
1627664873 Fri Jul 30 2021 inih-49-4.el9
1627665403 Fri Jul 30 2021 libzstd-1.5.0-1.el9
1627665466 Fri Jul 30 2021 kmod-28-4.el9
I had some issues with UEFI BootNext settings (not uncommon in Beaker unfortunately), but the results look quite good to me. The machines were still booting and some basic programs work as expected. I think the side tag is ready for merging in the sense that it shouldn't impact other teams.
I found one grub2 build issue and filed bug 1988732 for it. The other new build failures are due to other churn in the distribution.
Looking at annocheck reports, the new failures I saw are all real, more or less (autofs linking without any hardening flags, cups using -fstack-protector-all instead of -fstack-protector-strong, pcre2 reporting protection gaps due to missing annobin plugin at LTO link).
redhat-rpm-config-187-1.el9 with this change is in gating, via rhel-9.0.0-beta-build-sidetag-86604-stack-gate. It was discovered that several packages capture compiler flags and store them in their own built packages, potentially using them for building other packages. As a result, updating redhat-rpm-config alone is not effective. I have rebuilt the following actually packages: apr-1.7.0-9.5.el9 compiler-rt-12.0.1-2.el9 httpd-2.4.48-14.el9 libtool-2.4.6-42.el9 mpich-3.4.1-3.el9 mvapich2-2.3.6-2.el9 (ocaml-4.11.1-3.el9.2 present in c9s-build) perl-5.32.1-478.el9 php-8.0.6-7.el9 postgresql-13.3-7.el9 python3.9-3.9.6-4.el9 qt5-qtbase-5.15.2-24.el9 ruby-3.0.1-151.el9 (sip-4.19.24-5.el9 present in c9s-build) (tcl-8.6.10-6.el9 present in c9s-build) uuid-1.6.2-52.el9 The builds in parentheses have already been moved into the buildroot. |
Description of problem: Current RHEL9 builds for aarch64 are creating binaries with hardware branch protection instructions in the code (BTI and PAC). These instructions are not supported on any existing hardware so they have not been fully tested. There is a potential for breakage if RHEL9 is run on future machines implementing these instructions. Therefore, the safest course is to disable the generation of these instructions until BTI and PAC can be validated on real hardware. Version-Release number of selected component (if applicable): How reproducible: 100% Steps to Reproduce: 1. objdump -d /bin/true | grep -E 'paciasp|bti' | head 2. 3. Actual results: 1118: d503233f paciasp 1140: d503245f bti c 13c0: d503233f paciasp 1840: d503245f bti c 1900: d503233f paciasp 1950: d503245f bti c 1960: d503233f paciasp 19b0: d503233f paciasp 1b04: d503233f paciasp 1c80: d503233f paciasp Expected results: No pac or bti instructions Additional info: These instructions are enabled with "-mbranch-protection=standard" in the aarch64 optflags. This should be turned off before the mass rebuild in August