RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2196271 - system broken after failed installation of glibc-2.34-66.el9.x86_64.rpm on xcp-ng 8.2.1 on AMD CPU
Summary: system broken after failed installation of glibc-2.34-66.el9.x86_64.rpm on xc...
Keywords:
Status: CLOSED DUPLICATE of bug 2177705
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: glibc
Version: CentOS Stream
Hardware: x86_64
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: glibc team
QA Contact: qe-baseos-tools-bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-05-08 14:10 UTC by anshockm
Modified: 2023-05-19 13:09 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-05-19 13:09:13 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
output of “ld.so --list-diagnostics” and “ld.so --list-tunables” before update (10.09 KB, text/plain)
2023-05-08 14:23 UTC, anshockm 		no flags Details
ld.so output from the failed system (10.20 KB, text/plain)
2023-05-08 15:30 UTC, anshockm 		no flags Details
ld.so output from the updated good system (10.20 KB, text/plain)
2023-05-08 15:31 UTC, anshockm 		no flags Details
ld.so output from the failed system (now the right version) (10.08 KB, text/plain)
2023-05-08 15:56 UTC, anshockm 		no flags Details
Show Obsolete (1) View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-156620 0 None None None 2023-05-08 14:13:44 UTC

Internal Links: 2166709

Description anshockm 2023-05-08 14:10:57 UTC 
Description of problem:
dnf update failed on a CentOS Stream 9 virtual machine running on XCP-ng 8.2.1 only on AMD-CPU.
After installing glibc-2.34-66.el9 new processes are failing with SIGV.


Version-Release number of selected component (if applicable):
glibc-2.34-66.el9.x86_64

How reproducible:
Fresh installation of CentOS Stream 9 

Steps to Reproduce:
1. Create new CentOS Stream 9 virtual machine on XCP-ng 8.2.1 host
2. dnf update -x glib*-2.34-66.el9 (reboot, ...)
3. dnf update

Actual results:
[root@vm839 ~]# dnf update
Last metadata expiration check: 0:04:48 ago on Mon May  8 15:12:47 2023.
Dependencies resolved.
===============================================================================================================================================================
 Package                                      Architecture                      Version                                Repository                         Size
===============================================================================================================================================================
Upgrading:
 glibc                                        x86_64                            2.34-66.el9                            baseos                            1.9 M
 glibc-common                                 x86_64                            2.34-66.el9                            baseos                            317 k
 glibc-gconv-extra                            x86_64                            2.34-66.el9                            baseos                            1.7 M
 glibc-langpack-de                            x86_64                            2.34-66.el9                            baseos                            622 k

Transaction Summary
===============================================================================================================================================================
Upgrade  4 Packages

Total download size: 4.5 M
Is this ok [y/N]: y
Downloading Packages:
(1/4): glibc-common-2.34-66.el9.x86_64.rpm                                                                                     1.1 MB/s | 317 kB     00:00    
(2/4): glibc-2.34-66.el9.x86_64.rpm                                                                                            6.0 MB/s | 1.9 MB     00:00    
(3/4): glibc-gconv-extra-2.34-66.el9.x86_64.rpm                                                                                4.7 MB/s | 1.7 MB     00:00    
(4/4): glibc-langpack-de-2.34-66.el9.x86_64.rpm                                                                                4.5 MB/s | 622 kB     00:00    
---------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                          5.3 MB/s | 4.5 MB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                       1/1 
  Upgrading        : glibc-common-2.34-66.el9.x86_64                                                                                                       1/8 
  Upgrading        : glibc-gconv-extra-2.34-66.el9.x86_64                                                                                                  2/8 
  Running scriptlet: glibc-gconv-extra-2.34-66.el9.x86_64                                                                                                  2/8 
  Upgrading        : glibc-langpack-de-2.34-66.el9.x86_64                                                                                                  3/8 
  Running scriptlet: glibc-2.34-66.el9.x86_64                                                                                                              4/8 
  Upgrading        : glibc-2.34-66.el9.x86_64                                                                                                              4/8 
  Running scriptlet: glibc-2.34-66.el9.x86_64                                                                                                              4/8 
warning: %triggerin(cronie-1.5.7-8.el9.x86_64) scriptlet failed, signal 11

Error in <unknown> scriptlet in rpm package glibc
  Cleanup          : glibc-2.34-65.el9.x86_64                                                                                                              5/8 
  Cleanup          : glibc-langpack-de-2.34-65.el9.x86_64                                                                                                  6/8 
  Cleanup          : glibc-gconv-extra-2.34-65.el9.x86_64                                                                                                  7/8 
  Running scriptlet: glibc-gconv-extra-2.34-65.el9.x86_64                                                                                                  7/8 
  Cleanup          : glibc-common-2.34-65.el9.x86_64                                                                                                       8/8 
  Running scriptlet: glibc-common-2.34-65.el9.x86_64                                                                                                       8/8 
warning: %transfiletriggerpostun(glibc-common-2.34-66.el9.x86_64) scriptlet failed, signal 11

Error in <unknown> scriptlet in rpm package glibc-common
warning: %transfiletriggerin(glibc-common-2.34-66.el9.x86_64) scriptlet failed, signal 11

Error in <unknown> scriptlet in rpm package glibc-common
  Verifying        : glibc-2.34-66.el9.x86_64                                                                                                              1/8 
  Verifying        : glibc-2.34-65.el9.x86_64                                                                                                              2/8 
  Verifying        : glibc-common-2.34-66.el9.x86_64                                                                                                       3/8 
  Verifying        : glibc-common-2.34-65.el9.x86_64                                                                                                       4/8 
  Verifying        : glibc-gconv-extra-2.34-66.el9.x86_64                                                                                                  5/8 
  Verifying        : glibc-gconv-extra-2.34-65.el9.x86_64                                                                                                  6/8 
  Verifying        : glibc-langpack-de-2.34-66.el9.x86_64                                                                                                  7/8 
  Verifying        : glibc-langpack-de-2.34-65.el9.x86_64                                                                                                  8/8 

Upgraded:
  glibc-2.34-66.el9.x86_64       glibc-common-2.34-66.el9.x86_64       glibc-gconv-extra-2.34-66.el9.x86_64       glibc-langpack-de-2.34-66.el9.x86_64      

Complete!
[root@vm839 ~]# bash
Segmentation fault (core dumped)


Expected results:
[root@vm250 ~]# dnf update
Last metadata expiration check: 0:13:11 ago on Mon May  8 15:25:30 2023.
Dependencies resolved.
===============================================================================================================================================================
 Package                                      Architecture                      Version                                Repository                         Size
===============================================================================================================================================================
Upgrading:
 glibc                                        x86_64                            2.34-66.el9                            baseos                            1.9 M
 glibc-common                                 x86_64                            2.34-66.el9                            baseos                            317 k
 glibc-gconv-extra                            x86_64                            2.34-66.el9                            baseos                            1.7 M
 glibc-langpack-de                            x86_64                            2.34-66.el9                            baseos                            622 k

Transaction Summary
===============================================================================================================================================================
Upgrade  4 Packages

Total download size: 4.5 M
Is this ok [y/N]: y
Downloading Packages:
(1/4): glibc-common-2.34-66.el9.x86_64.rpm                                                                                     1.1 MB/s | 317 kB     00:00    
(2/4): glibc-2.34-66.el9.x86_64.rpm                                                                                            5.4 MB/s | 1.9 MB     00:00    
(3/4): glibc-langpack-de-2.34-66.el9.x86_64.rpm                                                                                7.7 MB/s | 622 kB     00:00    
(4/4): glibc-gconv-extra-2.34-66.el9.x86_64.rpm                                                                                4.1 MB/s | 1.7 MB     00:00    
---------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                          7.3 MB/s | 4.5 MB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                       1/1 
  Upgrading        : glibc-common-2.34-66.el9.x86_64                                                                                                       1/8 
  Upgrading        : glibc-gconv-extra-2.34-66.el9.x86_64                                                                                                  2/8 
  Running scriptlet: glibc-gconv-extra-2.34-66.el9.x86_64                                                                                                  2/8 
  Upgrading        : glibc-langpack-de-2.34-66.el9.x86_64                                                                                                  3/8 
  Running scriptlet: glibc-2.34-66.el9.x86_64                                                                                                              4/8 
  Upgrading        : glibc-2.34-66.el9.x86_64                                                                                                              4/8 
  Running scriptlet: glibc-2.34-66.el9.x86_64                                                                                                              4/8 
  Cleanup          : glibc-2.34-65.el9.x86_64                                                                                                              5/8 
  Cleanup          : glibc-langpack-de-2.34-65.el9.x86_64                                                                                                  6/8 
  Cleanup          : glibc-gconv-extra-2.34-65.el9.x86_64                                                                                                  7/8 
  Running scriptlet: glibc-gconv-extra-2.34-65.el9.x86_64                                                                                                  7/8 
  Cleanup          : glibc-common-2.34-65.el9.x86_64                                                                                                       8/8 
  Running scriptlet: glibc-common-2.34-65.el9.x86_64                                                                                                       8/8 
  Verifying        : glibc-2.34-66.el9.x86_64                                                                                                              1/8 
  Verifying        : glibc-2.34-65.el9.x86_64                                                                                                              2/8 
  Verifying        : glibc-common-2.34-66.el9.x86_64                                                                                                       3/8 
  Verifying        : glibc-common-2.34-65.el9.x86_64                                                                                                       4/8 
  Verifying        : glibc-gconv-extra-2.34-66.el9.x86_64                                                                                                  5/8 
  Verifying        : glibc-gconv-extra-2.34-65.el9.x86_64                                                                                                  6/8 
  Verifying        : glibc-langpack-de-2.34-66.el9.x86_64                                                                                                  7/8 
  Verifying        : glibc-langpack-de-2.34-65.el9.x86_64                                                                                                  8/8 

Upgraded:
  glibc-2.34-66.el9.x86_64       glibc-common-2.34-66.el9.x86_64       glibc-gconv-extra-2.34-66.el9.x86_64       glibc-langpack-de-2.34-66.el9.x86_64      

Complete!
[root@vm250 ~]# bash
[root@vm250 ~]# 


Additional info:
The CPU
[root@vm839 ~]# lscpu 
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         48 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  8
  On-line CPU(s) list:   0-7
Vendor ID:               AuthenticAMD
  BIOS Vendor ID:        AMD
  Model name:            AMD EPYC 7552 48-Core Processor
    CPU family:          23
    Model:               49
    Thread(s) per core:  1
    Core(s) per socket:  8
    Socket(s):           1
    Stepping:            0
    BogoMIPS:            4391.80
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt pdpe1gb r
                         dtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f
                         16c rdrand hypervisor lahf_lm cr8_legacy abm sse4a misalignsse 3dnowprefetch bpext ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 sme
                         p bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr arat umip rdpid
Virtualization features: 
  Hypervisor vendor:     Xen
  Virtualization type:   full
Caches (sum of all):     
  L1d:                   256 KiB (8 instances)
  L1i:                   256 KiB (8 instances)
  L2:                    4 MiB (8 instances)
  L3:                    1.5 GiB (8 instances)
NUMA:                    
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-7
Vulnerabilities:         
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Not affected
  Retbleed:              Mitigation; untrained return thunk; SMT disabled
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Retpolines, IBPB conditional, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
  Srbds:                 Not affected
  Tsx async abort:       Not affected

The XCP-ng host version:
[15:49 k8s-1 ~]# xe host-param-get uuid=e6ae14a4-a749-4e6b-8536-c7c9f5ca189e param-name=software-version 
product_version: 8.2.1; product_version_text: 8.2; product_version_text_short: 8.2; platform_name: XCP; platform_version: 3.2.1; product_brand: XCP-ng; build_number: release/yangtze/master/58; hostname: localhost; date: 2022-10-12; dbv: 0.0.1; xapi: 1.20; xen: 4.13.5-9.30; linux: 4.19.0+1; xencenter_min: 2.16; xencenter_max: 2.16; network_backend: openvswitch; db_schema: 5.602

some more observations:

Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/bin/bash'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:835
835		LOAD_ONE_SET((%rsi), PAGE_SIZE, %VEC(4), %VEC(5), %VEC(6), %VEC(7))
(gdb) bt
#0  __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:835
#1  0x00007f1708f494d5 in msort_with_tmp (n=38, b=0x557ea3918570, p=0x7fff47bf4410) at msort.c:45
#2  msort_with_tmp (p=p@entry=0x7fff47bf4410, b=0x557ea3918570, n=n@entry=76) at msort.c:53
#3  0x00007f1708f499ac in msort_with_tmp (n=76, b=<optimized out>, p=0x7fff47bf4410) at msort.c:45
#4  __GI___qsort_r (b=<optimized out>, n=<optimized out>, s=48, cmp=<optimized out>, arg=0x0) at msort.c:254
#5  0x0000557ea2e2881f in shell_initialize () at /usr/src/debug/bash-5.1.8-6.el9.x86_64/shell.c:1919
#6  0x0000557ea2e25ef6 in main (argc=1, argv=0x7fff47bf4848, env=0x7fff47bf4858) at /usr/src/debug/bash-5.1.8-6.el9.x86_64/shell.c:582
Comment 1 Florian Weimer 2023-05-08 14:14:27 UTC 
Would you please attach “ld.so --list-diagnostics” and “ld.so --list-tunables” output? From before the update if the command does not work afterwards. Thanks.
Comment 2 anshockm 2023-05-08 14:23:45 UTC 
Created attachment 1963250 [details]
output of “ld.so --list-diagnostics” and “ld.so --list-tunables” before update
Comment 3 Florian Weimer 2023-05-08 14:32:47 UTC 
Thank you. Would you mind testing this newer build?

  https://kojihub.stream.centos.org/koji/buildinfo?buildID=32228

It contains this change from upstream, which might help here as well:

- x86: Check minimum/maximum of non_temporal_threshold [BZ #29953]

Unfortunately, I cannot easily test on such a hypervisor. Another potentially helpful data point would be whether the issue reproduces with a Fedora rawhide container image (e.g., registry.fedoraproject.org/fedora:rawhide).
Comment 4 anshockm 2023-05-08 14:50:20 UTC 
It helps!

[root@vm839 ~]# ls -l glibc-*
-rw-r--r--. 1 root root 2008067 May  8 16:47 glibc-2.34-67.el9.x86_64.rpm
-rw-r--r--. 1 root root  316188 May  8 16:47 glibc-common-2.34-67.el9.x86_64.rpm
-rw-r--r--. 1 root root 1629451 May  8 16:47 glibc-gconv-extra-2.34-67.el9.x86_64.rpm
-rw-r--r--. 1 root root  587024 May  8 16:47 glibc-langpack-de-2.34-67.el9.x86_64.rpm
[root@vm839 ~]# dnf install ./glibc*
Last metadata expiration check: 0:51:00 ago on Mon May  8 15:56:19 2023.
Dependencies resolved.
===============================================================================================================================================================
 Package                                    Architecture                    Version                                Repository                             Size
===============================================================================================================================================================
Upgrading:
 glibc                                      x86_64                          2.34-67.el9                            @commandline                          1.9 M
 glibc-common                               x86_64                          2.34-67.el9                            @commandline                          309 k
 glibc-gconv-extra                          x86_64                          2.34-67.el9                            @commandline                          1.6 M
 glibc-langpack-de                          x86_64                          2.34-67.el9                            @commandline                          573 k

Transaction Summary
===============================================================================================================================================================
Upgrade  4 Packages

Total size: 4.3 M
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                       1/1 
  Upgrading        : glibc-common-2.34-67.el9.x86_64                                                                                                       1/8 
  Upgrading        : glibc-gconv-extra-2.34-67.el9.x86_64                                                                                                  2/8 
  Running scriptlet: glibc-gconv-extra-2.34-67.el9.x86_64                                                                                                  2/8 
  Upgrading        : glibc-langpack-de-2.34-67.el9.x86_64                                                                                                  3/8 
  Running scriptlet: glibc-2.34-67.el9.x86_64                                                                                                              4/8 
  Upgrading        : glibc-2.34-67.el9.x86_64                                                                                                              4/8 
  Running scriptlet: glibc-2.34-67.el9.x86_64                                                                                                              4/8 
  Cleanup          : glibc-2.34-65.el9.x86_64                                                                                                              5/8 
  Cleanup          : glibc-langpack-de-2.34-65.el9.x86_64                                                                                                  6/8 
  Cleanup          : glibc-gconv-extra-2.34-65.el9.x86_64                                                                                                  7/8 
  Running scriptlet: glibc-gconv-extra-2.34-65.el9.x86_64                                                                                                  7/8 
  Cleanup          : glibc-common-2.34-65.el9.x86_64                                                                                                       8/8 
  Running scriptlet: glibc-common-2.34-65.el9.x86_64                                                                                                       8/8 
  Verifying        : glibc-2.34-67.el9.x86_64                                                                                                              1/8 
  Verifying        : glibc-2.34-65.el9.x86_64                                                                                                              2/8 
  Verifying        : glibc-common-2.34-67.el9.x86_64                                                                                                       3/8 
  Verifying        : glibc-common-2.34-65.el9.x86_64                                                                                                       4/8 
  Verifying        : glibc-gconv-extra-2.34-67.el9.x86_64                                                                                                  5/8 
  Verifying        : glibc-gconv-extra-2.34-65.el9.x86_64                                                                                                  6/8 
  Verifying        : glibc-langpack-de-2.34-67.el9.x86_64                                                                                                  7/8 
  Verifying        : glibc-langpack-de-2.34-65.el9.x86_64                                                                                                  8/8 

Upgraded:
  glibc-2.34-67.el9.x86_64       glibc-common-2.34-67.el9.x86_64       glibc-gconv-extra-2.34-67.el9.x86_64       glibc-langpack-de-2.34-67.el9.x86_64      

Complete!
[root@vm839 ~]#
Comment 5 Florian Weimer 2023-05-08 14:54:24 UTC 
Thank you for checking. So it was the result of an unfortunate backporting order.

Would you mind attaching the new output of “ld.so --list-diagnostics“ and “ld.so --list-tunables”? I wonder if the non-temporal threshold has changed in a drastic way. Thanks.
Comment 6 anshockm 2023-05-08 15:30:44 UTC 
Created attachment 1963254 [details]
ld.so output from the failed system
Comment 7 anshockm 2023-05-08 15:31:33 UTC 
Created attachment 1963256 [details]
ld.so output from the updated good system
Comment 8 Florian Weimer 2023-05-08 15:37:46 UTC 
(In reply to anshockm from comment #6)
> Created attachment 1963254 [details]
> ld.so output from the failed system

Are you sure this is the right file? I would expect

x86.cpu_features.non_temporal_threshold=0x4040

only after the fix was in. The computed value is likely lower than 0x4040 without the fix.
Comment 9 anshockm 2023-05-08 15:56:10 UTC 
Created attachment 1963257 [details]
ld.so output from the failed system (now the right version)

cut-and-paste error. Sorry!
Comment 10 Carlos O'Donell 2023-05-19 13:09:13 UTC 
I'm marking this CLOSED / DUPLICATE of 2177705 since the setup of a default threshold fixes this issue and is available in the -67 build.

*** This bug has been marked as a duplicate of bug 2177705 ***
Note You need to log in before you can comment on or make changes to this bug.