Bug 2244688

Summary:	glibc-2.38.9000-13.fc40 broke rawhide buildroot on x86_64
Product:	[Fedora] Fedora	Reporter:	Fabio Valentini <decathorpe>
Component:	glibc	Assignee:	Carlos O'Donell <codonell>
Status:	CLOSED RAWHIDE	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	rawhide	CC:	arjun, codonell, dj, fberat, fweimer, jlaw, kevin, mcermak, mcoufal, mfabian, ngompa13, pfrankli, sipoyare, skolosov
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	glibc-2.38.9000-14.fc40	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-10-19 06:51:09 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Fabio Valentini 2023-10-17 18:24:15 UTC

After this update was pushed to rawhide:
https://bodhi.fedoraproject.org/updates/FEDORA-2023-44aef75db9

All x86_64 "build" and "buildSRPMfromSCM" tasks in koji started to fail due to a crash when installing the default buildroot -- apparently, because /usr/bin/p11-kit crashed with SIGILL:

"""
(...)
DEBUG util.py:448:  /usr/bin/update-ca-trust: line 35: 389976 Illegal instruction     (core dumped) /usr/bin/p11-kit extract --format=pem-bundle --filter=ca-anchors --overwrite --comment --purpose server-auth "$DEST/pem/tls-ca-bundle.pem"
DEBUG util.py:448:  warning: %post(ca-certificates-2023.2.62_v7.0.401-4.fc40.noarch) scriptlet failed, exit status 132
DEBUG util.py:446:  Error in POSTIN scriptlet in rpm package ca-certificates
(...)
DEBUG util.py:448:  /usr/bin/update-ca-trust: line 35: 390014 Illegal instruction     (core dumped) /usr/bin/p11-kit extract --format=pem-bundle --filter=ca-anchors --overwrite --comment --purpose server-auth "$DEST/pem/tls-ca-bundle.pem"
DEBUG util.py:448:  warning: %posttrans(ca-certificates-2023.2.62_v7.0.401-4.fc40.noarch) scriptlet failed, exit status 132
DEBUG util.py:446:  Error in POSTTRANS scriptlet in rpm package ca-certificates
(...)
"""

After glibc-2.38.9000-13.fc40 was untagged from rawhide, builds in koji returned to normal, so reverting to glibc-2.38.9000-12.fc40 avoids this issue.

Reproducible: Always

Comment 1 Florian Weimer 2023-10-17 19:30:07 UTC

Okay, so apparently the strace rebuild failure (which is a known FTBFS unrelated to glibc) obscured other scratch rebuild failures, leading to an incorrect test failure waiver. The opencryptoki rebuild test did catch the issue ass well.

I'll talk to to QE about splitting out the strace rebuild test because it has frequent issues that go unfixed for extended periods of time. Or perhaps we should just drop it.

Comment 2 Florian Weimer 2023-10-18 09:52:13 UTC

I'm building glibc-2.38.9000-14.fc40 with the most likely culprit reverted. Hopefully this will fix the gating tests and make it into the buildroot.

I raised the regression upstream: https://inbox.sourceware.org/libc-alpha/87fs28gug9.fsf@oldenburg.str.redhat.com/T/#t

Comment 3 Florian Weimer 2023-10-19 06:51:09 UTC

Fixed by reverting this upstream commit:

commit a3c50bf46a1ca6d9d2b7d879176d345abf95a9de
Author: Noah Goldstein <goldstein.w.n>
Date:   Thu Sep 21 09:38:37 2023 -0500

    x86: Prepare `strrchr-evex` and `strrchr-evex512` for AVX10
    
    This commit refactors `strrchr-evex` and `strrchr-evex512` to use a
    common implementation: `strrchr-evex-base.S`.
    
    The motivation is `strrchr-evex` needed to be refactored to not use
    64-bit masked registers in preperation for AVX10.
    
    Once vec-width masked register combining was removed, the EVEX and
    EVEX512 implementations can easily be implemented in the same file
    without any major overhead.
    
    The net result is performance improvements (measured on TGL) for both
    `strrchr-evex` and `strrchr-evex512`. Although, note there are some
    regressions in the test suite and it may be many of the cases that
    make the total-geomean of improvement/regression across bench-strrchr
    are cold. The point of the performance measurement is to show there
    are no major regressions, but the primary motivation is preperation
    for AVX10.
    
    Benchmarks where taken on TGL:
    https://www.intel.com/content/www/us/en/products/sku/213799/intel-core-i711850h-processor-24m-cache-up-to-4-80-ghz/specifications.html
    
    EVEX geometric_mean(N=5) of all benchmarks New / Original   : 0.74
    EVEX512 geometric_mean(N=5) of all benchmarks New / Original: 0.87
    
    Full check passes on x86.