Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
DescriptionChristophe Besson
2023-02-28 17:28:21 UTC
Description of problem:
Customer is experiencing an ABI error during the preupgrade step:
Fatal glibc error: CPU does not support x86-64-v2
Error in PREIN scriptlet in rpm package libutempter
Error in POSTIN scriptlet in rpm package p11-kit-trust
Error in PREIN scriptlet in rpm package ca-certificates
It appears the VMware hypervisor presents a recent CPU (cascade lake) to the guest but masks some instructions (sse4*, avx* among others).
Version-Release number of selected component (if applicable):
leapp-upgrade-el8toel9-0.17.0-3.el8.noarch
How reproducible:
Always
Steps to Reproduce:
1. Using the libvirt/kVM, setup a standard VM
2. Use `virsh edit <vm-name>` to mask a CPU instruction that is required by the x86-64-v2 standard
<cpu mode='host-model' check='partial'>
<model fallback="allow"/>
<feature policy='disable' name='popcnt'/>
</cpu>
3. Install RHEL 8.6, leapp, and then try a preupgrade
Actual results:
Details: Command ['systemd-nspawn', '--register=no', '--quiet', '--keep-unit', '--capability=all', '-D', '/var/lib/leapp/scratch/mounts/root_/system_overlay', '--setenv=LEAPP_HOSTNAME=localhost.localdomain', '--setenv=LEAPP_EXPERIMENTAL=0', '--setenv=LEAPP_UNSUPPORTED=0', '--setenv=LEAPP_NO_RHSM=0', '--setenv=LEAPP_UPGRADE_PATH_TARGET_RELEASE=9.0', '--setenv=LEAPP_UPGRADE_PATH_FLAVOUR=default', '--setenv=LEAPP_IPU_IN_PROGRESS=8to9', '--setenv=LEAPP_EXECUTION_ID=c64cb486-5031-4d66-9e61-e3e9356d5a29', '--setenv=LEAPP_COMMON_TOOLS=:/etc/leapp/repos.d/system_upgrade/el8toel9/tools', '--setenv=LEAPP_COMMON_FILES=:/etc/leapp/repos.d/system_upgrade/common/files:/etc/leapp/repos.d/system_upgrade/el8toel9/files', 'dnf', 'install', '-y', '--nogpgcheck', '--setopt=module_platform_id=platform:el9', '--setopt=keepcache=1', '--releasever', '9.0', '--installroot', '/el9target', '--disablerepo', '*', '--enablerepo', 'rhel-9-for-x86_64-baseos-rpms', '--enablerepo', 'rhel-9-for-x86_64-appstream-rpms', 'dnf-command(config-manager)', 'dnf', '-v'] failed with exit code 1.
Stderr: Host and machine ids are equal (03b541e558e14fa9b3690b7fc8366e58): refusing to link journals
warning: Generating 18 missing index(es), please wait...
Fatal glibc error: CPU does not support x86-64-v2
Error in PREIN scriptlet in rpm package libutempter
Error in POSTIN scriptlet in rpm package p11-kit-trust
Error in PREIN scriptlet in rpm package ca-certificates
Error in POSTIN scriptlet in rpm package libblkid
Error in POSTIN scriptlet in rpm package systemd-libs
Error in POSTIN scriptlet in rpm package util-linux-core
Error in PREIN scriptlet in rpm package systemd
Error in POSTIN scriptlet in rpm package dbus-common
Error in PREIN scriptlet in rpm package dbus-broker
Error in POSTIN scriptlet in rpm package elfutils-default-yama-scope
Error unpacking rpm package gnupg2-2.3.3-2.el9_0.x86_64
Error in PREIN scriptlet in rpm package tpm2-tss
Error in POSTIN scriptlet in rpm package dnf
Error in POSTTRANS scriptlet in rpm package filesystem
Error in POSTTRANS scriptlet in rpm package rpm
Error in <unknown> scriptlet in rpm package glibc-common
Error in <unknown> scriptlet in rpm package glib2
Error: Transaction failed
Expected results:
An inhibitor explaining the issue would be easier to apprehend than a glibc error during the DNF transaction.
Additional info:
* Workaround is to change the CPU model from the hypervisor side.
* https://developers.redhat.com/blog/2021/01/05/building-red-hat-enterprise-linux-9-for-the-x86-64-v2-microarchitecture-level#background_of_the_x86_64_microarchitecture_levels
* as per https://gitlab.com/x86-psABIs/x86-64-ABI
(to be checked)
x86-64 baseline: cmov cx8 fpu fxsr mmx osfxsr sce sse sse2
x86-64-v2: lm pni cx16 lahf_lm popcnt sse3 sse4_1 sse4_2 ssse3
* this KCS is related:
RHEL 9 guest panic's during boot with following error "Fatal glibc error: CPU does not support x86-64-v2"
https://access.redhat.com/solutions/6833751
* lscpu from customer
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 2
Socket(s): 2
NUMA node(s): 1
Vendor ID: GenuineIntel
BIOS Vendor ID: GenuineIntel
CPU family: 6
Model: 15
Model name: Intel(R) Xeon(R) Gold 6240R CPU @ 2.40GHz
BIOS Model name: Intel(R) Xeon(R) Gold 6240R CPU @ 2.40GHz
Stepping: 1
CPU MHz: 2394.374
BogoMIPS: 4788.74
Hypervisor vendor: VMware
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 36608K
NUMA node0 CPU(s): 0-3
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc arch_perfmon nopl tsc_reliable nonstop_tsc cpuid pni ssse3 cx16 tsc_deadline_timer hypervisor lahf_lm pti tsc_adjust arat
Hi Chris, thanks for the report with the investigation and added sources!! You are right, we should add the check for the version of intel architecture. Fortunately the upgrade is failing during the preupgrade phases so the system is not negatively affected, but having the proper inhibitor will save a time for people investigating this problem in future. I will discuss the prioritisation of the fix for 8.9 release.
Comment 2Christophe Besson
2023-03-01 14:48:04 UTC
Hi Petr, indeed that would help us to have something self-explanatory.
And yes, I forgot to mention in the description there is no risk of damage (severity could be lowered).
Note for users: RHEL 9 cannot boot anyway, kernel panics directly as init/systemd crashes instantly when these CPU features are masked.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (leapp-repository bug fix and enhancement update), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2023:7013
Description of problem: Customer is experiencing an ABI error during the preupgrade step: Fatal glibc error: CPU does not support x86-64-v2 Error in PREIN scriptlet in rpm package libutempter Error in POSTIN scriptlet in rpm package p11-kit-trust Error in PREIN scriptlet in rpm package ca-certificates It appears the VMware hypervisor presents a recent CPU (cascade lake) to the guest but masks some instructions (sse4*, avx* among others). Version-Release number of selected component (if applicable): leapp-upgrade-el8toel9-0.17.0-3.el8.noarch How reproducible: Always Steps to Reproduce: 1. Using the libvirt/kVM, setup a standard VM 2. Use `virsh edit <vm-name>` to mask a CPU instruction that is required by the x86-64-v2 standard <cpu mode='host-model' check='partial'> <model fallback="allow"/> <feature policy='disable' name='popcnt'/> </cpu> 3. Install RHEL 8.6, leapp, and then try a preupgrade Actual results: Details: Command ['systemd-nspawn', '--register=no', '--quiet', '--keep-unit', '--capability=all', '-D', '/var/lib/leapp/scratch/mounts/root_/system_overlay', '--setenv=LEAPP_HOSTNAME=localhost.localdomain', '--setenv=LEAPP_EXPERIMENTAL=0', '--setenv=LEAPP_UNSUPPORTED=0', '--setenv=LEAPP_NO_RHSM=0', '--setenv=LEAPP_UPGRADE_PATH_TARGET_RELEASE=9.0', '--setenv=LEAPP_UPGRADE_PATH_FLAVOUR=default', '--setenv=LEAPP_IPU_IN_PROGRESS=8to9', '--setenv=LEAPP_EXECUTION_ID=c64cb486-5031-4d66-9e61-e3e9356d5a29', '--setenv=LEAPP_COMMON_TOOLS=:/etc/leapp/repos.d/system_upgrade/el8toel9/tools', '--setenv=LEAPP_COMMON_FILES=:/etc/leapp/repos.d/system_upgrade/common/files:/etc/leapp/repos.d/system_upgrade/el8toel9/files', 'dnf', 'install', '-y', '--nogpgcheck', '--setopt=module_platform_id=platform:el9', '--setopt=keepcache=1', '--releasever', '9.0', '--installroot', '/el9target', '--disablerepo', '*', '--enablerepo', 'rhel-9-for-x86_64-baseos-rpms', '--enablerepo', 'rhel-9-for-x86_64-appstream-rpms', 'dnf-command(config-manager)', 'dnf', '-v'] failed with exit code 1. Stderr: Host and machine ids are equal (03b541e558e14fa9b3690b7fc8366e58): refusing to link journals warning: Generating 18 missing index(es), please wait... Fatal glibc error: CPU does not support x86-64-v2 Error in PREIN scriptlet in rpm package libutempter Error in POSTIN scriptlet in rpm package p11-kit-trust Error in PREIN scriptlet in rpm package ca-certificates Error in POSTIN scriptlet in rpm package libblkid Error in POSTIN scriptlet in rpm package systemd-libs Error in POSTIN scriptlet in rpm package util-linux-core Error in PREIN scriptlet in rpm package systemd Error in POSTIN scriptlet in rpm package dbus-common Error in PREIN scriptlet in rpm package dbus-broker Error in POSTIN scriptlet in rpm package elfutils-default-yama-scope Error unpacking rpm package gnupg2-2.3.3-2.el9_0.x86_64 Error in PREIN scriptlet in rpm package tpm2-tss Error in POSTIN scriptlet in rpm package dnf Error in POSTTRANS scriptlet in rpm package filesystem Error in POSTTRANS scriptlet in rpm package rpm Error in <unknown> scriptlet in rpm package glibc-common Error in <unknown> scriptlet in rpm package glib2 Error: Transaction failed Expected results: An inhibitor explaining the issue would be easier to apprehend than a glibc error during the DNF transaction. Additional info: * Workaround is to change the CPU model from the hypervisor side. * https://developers.redhat.com/blog/2021/01/05/building-red-hat-enterprise-linux-9-for-the-x86-64-v2-microarchitecture-level#background_of_the_x86_64_microarchitecture_levels * as per https://gitlab.com/x86-psABIs/x86-64-ABI (to be checked) x86-64 baseline: cmov cx8 fpu fxsr mmx osfxsr sce sse sse2 x86-64-v2: lm pni cx16 lahf_lm popcnt sse3 sse4_1 sse4_2 ssse3 * this KCS is related: RHEL 9 guest panic's during boot with following error "Fatal glibc error: CPU does not support x86-64-v2" https://access.redhat.com/solutions/6833751 * lscpu from customer Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 4 On-line CPU(s) list: 0-3 Thread(s) per core: 1 Core(s) per socket: 2 Socket(s): 2 NUMA node(s): 1 Vendor ID: GenuineIntel BIOS Vendor ID: GenuineIntel CPU family: 6 Model: 15 Model name: Intel(R) Xeon(R) Gold 6240R CPU @ 2.40GHz BIOS Model name: Intel(R) Xeon(R) Gold 6240R CPU @ 2.40GHz Stepping: 1 CPU MHz: 2394.374 BogoMIPS: 4788.74 Hypervisor vendor: VMware Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 1024K L3 cache: 36608K NUMA node0 CPU(s): 0-3 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc arch_perfmon nopl tsc_reliable nonstop_tsc cpuid pni ssse3 cx16 tsc_deadline_timer hypervisor lahf_lm pti tsc_adjust arat