Bug 2075424

Hi rjones,

I got this doc from my coworker:https://developers.redhat.com/blog/2021/01/05/building-red-hat-enterprise-linux-9-for-the-x86-64-v2-microarchitecture-level#background_of_the_x86_64_microarchitecture_levels
RHEL9 only supports CPU x86-64-v2.

(In reply to YongkuiGuo from comment #2)
> Hi rjones,
> 
> I got this doc from my
> coworker:https://developers.redhat.com/blog/2021/01/05/building-red-hat-
> enterprise-linux-9-for-the-x86-64-v2-microarchitecture-
> level#background_of_the_x86_64_microarchitecture_levels
> RHEL9 only supports CPU x86-64-v2.

RHEL 8.6 should be able to emulate this CPU model, but probably libguestfs
is not using the correct -cpu flags.  I'm looking into it.

(In reply to Richard W.M. Jones from comment #3)
> (In reply to YongkuiGuo from comment #2)
> > Hi rjones,
> > 
> > I got this doc from my
> > coworker:https://developers.redhat.com/blog/2021/01/05/building-red-hat-
> > enterprise-linux-9-for-the-x86-64-v2-microarchitecture-
> > level#background_of_the_x86_64_microarchitecture_levels
> > RHEL9 only supports CPU x86-64-v2.
> 
> RHEL 8.6 should be able to emulate this CPU model, but probably libguestfs
> is not using the correct -cpu flags.  I'm looking into it.

Ok, thanks.

I can reproduce this on RHEL AV.  A simple reproducer is:

$ LIBGUESTFS_BACKEND_SETTINGS=force_tcg virt-customize -a rhel9.qcow2 --run-command ls -v -x

Notes:

* You need a RHEL 9 or C9S disk image.

* You need -v -x to see the real error from inside the appliance, which is:
  guestfsd: error: Fatal glibc error: CPU does not support x86-64-v2

* As Yongkui noted, it doesn't happen without forcing TCG.

libguestfs in RHEL 8.6 uses -cpu qemu64 in TCG mode which is not capable of
emulating all the features needed for RHEL 9 glibc.

libguestfs upstream uses -cpu max which works fine.  Also qemu/libvirt
in RHEL 8.6 now support -cpu max (which they did not in earlier versions).

To make it work in RHEL 8.6 it was sufficient to backport the following
upstream commits:

https://github.com/libguestfs/libguestfs/commit/30f74f38bd6e42e783ba80895f4d6826abddd417
https://github.com/libguestfs/libguestfs/commit/13ceb6a87b2869909a6a0e3c8caa962b72e4cb0e

Yongkui, I'm doing a scratch build now which I'll send you for testing.

I'm dev-acking this bug because I think this is important to get fixed
in RHEL 8.

(In reply to Richard W.M. Jones from comment #6)
> libguestfs in RHEL 8.6 uses -cpu qemu64 in TCG mode which is not capable of
> emulating all the features needed for RHEL 9 glibc.
> 
> libguestfs upstream uses -cpu max which works fine.  Also qemu/libvirt
> in RHEL 8.6 now support -cpu max (which they did not in earlier versions).
> 
> To make it work in RHEL 8.6 it was sufficient to backport the following
> upstream commits:
> 
> https://github.com/libguestfs/libguestfs/commit/
> 30f74f38bd6e42e783ba80895f4d6826abddd417
> https://github.com/libguestfs/libguestfs/commit/
> 13ceb6a87b2869909a6a0e3c8caa962b72e4cb0e
> 
> Yongkui, I'm doing a scratch build now which I'll send you for testing.
> 
> I'm dev-acking this bug because I think this is important to get fixed
> in RHEL 8.

Thanks, I will try the scratch build once you finished to build it.

Hi rjones,

The scratch build works well.

Steps:

# rpm -q libguestfs
libguestfs-1.44.0-6.el8.x86_64

# LIBGUESTFS_BACKEND_SETTINGS=force_tcg virt-customize -a rhel-guest-image-9.1-20220411.0.x86_64.qcow2 --run-command ls
[   0.0] Examining the guest ...
[  69.5] Setting a random seed
[  69.8] Running: ls
[  71.7] Finishing off

# cat /var/log/libvirt/qemu/guestfs-f2wjk9323eacsl4n.log
...
/usr/libexec/qemu-kvm \
-name guest=guestfs-f2wjk9323eacsl4n,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-8-guestfs-f2wjk9323eac/master-key.aes"}' \
-machine pc-i440fx-rhel7.6.0,usb=off,dump-guest-core=off,memory-backend=pc.ram,graphics=off \
-accel tcg \
-cpu max \
...

Thanks I noticed this also affects CentOS 9 Streams as the default CPU type causes kernel panics upon boot.  The force_tcg workaround does not help here.  Is there any parameter or env var to customize the QEMU args for cpu type in virt-customize? 

https://www.reddit.com/r/CentOS/comments/srlq39/centos_stream_9_latest_image_broken/

Let me know if this warrants a separate issue being filed.
Thanks

If you mean libguestfs, then this is already fixed in CentOS 9 Stream.
Make sure you're using the latest libguestfs package (1:1.48.2-2.el9),
and if it still fails then follow the instructions here to debug:
https://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs

However the reddit issue you linked above is nothing to do with
libguestfs, and everything to do with using the wrong CPU type
in the guest metadata.  Use one that supports x86_64-v2 and they'll
be able to start the C9S guest fine.

Verified with package:
libguestfs-1.44.0-8.module+el8.7.0+15574+3df6e453.x86_64


Steps:

1. On rhel8.7 host
$ LIBGUESTFS_BACKEND_SETTINGS=force_tcg virt-customize -a rhel-guest-image-9.1-20220701.0.x86_64.qcow2 --run-command ls
[   0.0] Examining the guest ...
[ 101.8] Setting a random seed
[ 102.6] Setting the machine ID in /etc/machine-id
[ 102.6] Running: ls
[ 106.6] Finishing off

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Low: virt:rhel and virt-devel:rhel security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7472