RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2075424 - virt-customize fails with "error: Fatal glibc error: CPU does not support x86-64-v2" with TCG mode
Summary: virt-customize fails with "error: Fatal glibc error: CPU does not support x86...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: libguestfs
Version: 8.6
Hardware: x86_64
OS: Unspecified
unspecified
low
Target Milestone: rc
: ---
Assignee: Richard W.M. Jones
QA Contact: YongkuiGuo
URL:
Whiteboard:
Depends On: 2084566
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-04-14 08:07 UTC by YongkuiGuo
Modified: 2022-11-08 09:44 UTC (History)
6 users (show)

Fixed In Version: libguestfs-1.44.0-6.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-11-08 09:19:55 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
the whole log of virt-customize (85.05 KB, text/plain)
2022-04-14 08:07 UTC, YongkuiGuo 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-118944 0 None None None 2022-04-14 08:14:31 UTC
Red Hat Product Errata RHSA-2022:7472 0 None None None 2022-11-08 09:20:34 UTC

Description YongkuiGuo 2022-04-14 08:07:18 UTC 
Created attachment 1872402 [details]
the whole log of virt-customize

Description of problem:
When trying to modify rhel9 guest image with virt-customize on rhel8.6 host under TCG mode, the command fails with "guestfsd: error: Fatal glibc error: CPU does not support x86-64-v2".

Version-Release number of selected component (if applicable):
libguestfs-1.44.0-5.module+el8.6.0+13732+b2b9b31d.x86_64
qemu-kvm-core-6.2.0-6.module+el8.6.0+14165+5e5e76ac.x86_64
kernel-4.18.0-365.el8.x86_64


How reproducible:
100%


Steps:

1. On rhel8.6 host
# LIBGUESTFS_BACKEND_SETTINGS=force_tcg virt-customize -a rhel-guest-image-9.1-20220411.0.x86_64.qcow2 --run-command 'rpm -e cloud-init --nodeps' -v -x
...
rpm -e cloud-init --nodeps
"
Fatal glibc error: CPU does not support x86-64-v2
commandrvf: stdout=n stderr=n flags=0x0
commandrvf: umount /sysroot/sys
commandrvf: stdout=n stderr=n flags=0x0
commandrvf: umount /sysroot/proc
commandrvf: stdout=n stderr=n flags=0x0
commandrvf: umount /sysroot/dev/pts
commandrvf: stdout=n stderr=n flags=0x0
commandrvf: umount /sysroot/dev
guestfsd: error: Fatal glibc error: CPU does not support x86-64-v2
libguestfs: trace: sh = NULL (error)
libguestfs: trace: download "/tmp/builder.log" "/dev/stderr"
...

Actual results:
As above

Expected results:
I know trying to modify RHEL n+1 image on RHEL n is not supported officially. My colleague huzhao hit this issue, perhaps we can look at it.

Additional info:
1. No problem with libvirt mode and direct mode
2. No problem on rhel9 host to modify rhel9 image with TCG mode
Comment 2 YongkuiGuo 2022-04-14 08:26:08 UTC 
Hi rjones,

I got this doc from my coworker:https://developers.redhat.com/blog/2021/01/05/building-red-hat-enterprise-linux-9-for-the-x86-64-v2-microarchitecture-level#background_of_the_x86_64_microarchitecture_levels
RHEL9 only supports CPU x86-64-v2.
Comment 3 Richard W.M. Jones 2022-04-14 09:24:27 UTC 
(In reply to YongkuiGuo from comment #2)
> Hi rjones,
> 
> I got this doc from my
> coworker:https://developers.redhat.com/blog/2021/01/05/building-red-hat-
> enterprise-linux-9-for-the-x86-64-v2-microarchitecture-
> level#background_of_the_x86_64_microarchitecture_levels
> RHEL9 only supports CPU x86-64-v2.

RHEL 8.6 should be able to emulate this CPU model, but probably libguestfs
is not using the correct -cpu flags.  I'm looking into it.
Comment 4 YongkuiGuo 2022-04-14 09:55:37 UTC 
(In reply to Richard W.M. Jones from comment #3)
> (In reply to YongkuiGuo from comment #2)
> > Hi rjones,
> > 
> > I got this doc from my
> > coworker:https://developers.redhat.com/blog/2021/01/05/building-red-hat-
> > enterprise-linux-9-for-the-x86-64-v2-microarchitecture-
> > level#background_of_the_x86_64_microarchitecture_levels
> > RHEL9 only supports CPU x86-64-v2.
> 
> RHEL 8.6 should be able to emulate this CPU model, but probably libguestfs
> is not using the correct -cpu flags.  I'm looking into it.

Ok, thanks.
Comment 5 Richard W.M. Jones 2022-04-14 11:06:27 UTC 
I can reproduce this on RHEL AV.  A simple reproducer is:

$ LIBGUESTFS_BACKEND_SETTINGS=force_tcg virt-customize -a rhel9.qcow2 --run-command ls -v -x

Notes:

* You need a RHEL 9 or C9S disk image.

* You need -v -x to see the real error from inside the appliance, which is:
  guestfsd: error: Fatal glibc error: CPU does not support x86-64-v2

* As Yongkui noted, it doesn't happen without forcing TCG.
Comment 6 Richard W.M. Jones 2022-04-14 11:18:57 UTC 
libguestfs in RHEL 8.6 uses -cpu qemu64 in TCG mode which is not capable of
emulating all the features needed for RHEL 9 glibc.

libguestfs upstream uses -cpu max which works fine.  Also qemu/libvirt
in RHEL 8.6 now support -cpu max (which they did not in earlier versions).

To make it work in RHEL 8.6 it was sufficient to backport the following
upstream commits:

https://github.com/libguestfs/libguestfs/commit/30f74f38bd6e42e783ba80895f4d6826abddd417
https://github.com/libguestfs/libguestfs/commit/13ceb6a87b2869909a6a0e3c8caa962b72e4cb0e

Yongkui, I'm doing a scratch build now which I'll send you for testing.

I'm dev-acking this bug because I think this is important to get fixed
in RHEL 8.
Comment 7 YongkuiGuo 2022-04-14 11:46:10 UTC 
(In reply to Richard W.M. Jones from comment #6)
> libguestfs in RHEL 8.6 uses -cpu qemu64 in TCG mode which is not capable of
> emulating all the features needed for RHEL 9 glibc.
> 
> libguestfs upstream uses -cpu max which works fine.  Also qemu/libvirt
> in RHEL 8.6 now support -cpu max (which they did not in earlier versions).
> 
> To make it work in RHEL 8.6 it was sufficient to backport the following
> upstream commits:
> 
> https://github.com/libguestfs/libguestfs/commit/
> 30f74f38bd6e42e783ba80895f4d6826abddd417
> https://github.com/libguestfs/libguestfs/commit/
> 13ceb6a87b2869909a6a0e3c8caa962b72e4cb0e
> 
> Yongkui, I'm doing a scratch build now which I'll send you for testing.
> 
> I'm dev-acking this bug because I think this is important to get fixed
> in RHEL 8.

Thanks, I will try the scratch build once you finished to build it.
Comment 9 YongkuiGuo 2022-04-14 12:32:20 UTC 
Hi rjones,

The scratch build works well.

Steps:

# rpm -q libguestfs
libguestfs-1.44.0-6.el8.x86_64

# LIBGUESTFS_BACKEND_SETTINGS=force_tcg virt-customize -a rhel-guest-image-9.1-20220411.0.x86_64.qcow2 --run-command ls
[   0.0] Examining the guest ...
[  69.5] Setting a random seed
[  69.8] Running: ls
[  71.7] Finishing off

# cat /var/log/libvirt/qemu/guestfs-f2wjk9323eacsl4n.log
...
/usr/libexec/qemu-kvm \
-name guest=guestfs-f2wjk9323eacsl4n,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-8-guestfs-f2wjk9323eac/master-key.aes"}' \
-machine pc-i440fx-rhel7.6.0,usb=off,dump-guest-core=off,memory-backend=pc.ram,graphics=off \
-accel tcg \
-cpu max \
...
Comment 12 John Boero 2022-05-16 13:33:53 UTC 
Thanks I noticed this also affects CentOS 9 Streams as the default CPU type causes kernel panics upon boot.  The force_tcg workaround does not help here.  Is there any parameter or env var to customize the QEMU args for cpu type in virt-customize? 

https://www.reddit.com/r/CentOS/comments/srlq39/centos_stream_9_latest_image_broken/

Let me know if this warrants a separate issue being filed.
Thanks
Comment 13 Richard W.M. Jones 2022-05-16 13:42:22 UTC 
If you mean libguestfs, then this is already fixed in CentOS 9 Stream.
Make sure you're using the latest libguestfs package (1:1.48.2-2.el9),
and if it still fails then follow the instructions here to debug:
https://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs

However the reddit issue you linked above is nothing to do with
libguestfs, and everything to do with using the wrong CPU type
in the guest metadata.  Use one that supports x86_64-v2 and they'll
be able to start the C9S guest fine.
Comment 14 YongkuiGuo 2022-07-04 05:33:12 UTC 
Verified with package:
libguestfs-1.44.0-8.module+el8.7.0+15574+3df6e453.x86_64


Steps:

1. On rhel8.7 host
$ LIBGUESTFS_BACKEND_SETTINGS=force_tcg virt-customize -a rhel-guest-image-9.1-20220701.0.x86_64.qcow2 --run-command ls
[   0.0] Examining the guest ...
[ 101.8] Setting a random seed
[ 102.6] Setting the machine ID in /etc/machine-id
[ 102.6] Running: ls
[ 106.6] Finishing off
Comment 16 errata-xmlrpc 2022-11-08 09:19:55 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Low: virt:rhel and virt-devel:rhel security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7472
Note You need to log in before you can comment on or make changes to this bug.