The glibc message is intentional:
https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/s390/s390-64/dl-hwcap-check.h;h=87e18be6bd0c512bfffe0421fe243524193abdcf;hb=HEAD#l35

The question is whether TCG is able to or will be able to emulate this
feature now or in future.

It's generally useful for us to be able to run libguestfs nested w/o
nested KVM support, although it's not something that we would support
specifically on s390x (because we don't ship virt-v2v there).

(In reply to Richard W.M. Jones from comment #1)
> The glibc message is intentional:
> https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/s390/s390-64/dl-
> hwcap-check.h;h=87e18be6bd0c512bfffe0421fe243524193abdcf;hb=HEAD#l35
> 
> The question is whether TCG is able to or will be able to emulate this
> feature now or in future.

TCG already supports most z14 instructions. A handful of Vector FPU instructions are missing. With that, we can enable VXE and switch to a z14.

z14 support is on its way upstream: https://lore.kernel.org/qemu-devel/20210517142739.38597-1-david@redhat.com/

The latest state lives at: https://github.com/davidhildenbrand/qemu/tree/z14

With that, latest/greatest rhel9 should boot just fine.

Could reproduce the bug with this guest command:
/usr/libexec/qemu-kvm \
    -global virtio-blk-ccw.scsi=off \
    -no-user-config \
    -nodefaults \
    -display none \
    -machine accel=tcg \
    -cpu max \
    -m 1280 \
    -no-reboot \
    -rtc driftfix=slew \
    -kernel /var/tmp/.guestfs-0/appliance.d/kernel \
    -initrd /var/tmp/.guestfs-0/appliance.d/initrd \
    -object rng-random,filename=/dev/urandom,id=rng0 \
    -device virtio-rng-ccw,rng=rng0 \
    -device virtio-scsi-ccw,id=scsi \
    -drive file=/home/bfu/RHEL-8.5.0-20210506.n.0-s390x.qcow2,cache=unsafe,format=raw,id=hd0,if=none \
    -device scsi-hd,drive=hd0 \
    -drive file=/var/tmp/.guestfs-0/appliance.d/root,snapshot=on,id=appliance,cache=unsafe,if=none,format=raw \
    -device scsi-hd,drive=appliance \
    -device virtio-serial-ccw \
    -chardev stdio,id=charconsole0 \
    -device sclpconsole,chardev=charconsole0 \
    -append "panic=1 console=ttysclp0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=UUID=9e192f01-def5-40cc-aab2-743d668be445 selinux=0 guestfs_verbose=1 TERM=xterm-256color"

(In reply to bfu from comment #3)
> Could reproduce the bug with this guest command:
> /usr/libexec/qemu-kvm \
>     -global virtio-blk-ccw.scsi=off \
>     -no-user-config \
>     -nodefaults \
>     -display none \
>     -machine accel=tcg \
>     -cpu max \
>     -m 1280 \
>     -no-reboot \
>     -rtc driftfix=slew \
>     -kernel /var/tmp/.guestfs-0/appliance.d/kernel \
>     -initrd /var/tmp/.guestfs-0/appliance.d/initrd \
>     -object rng-random,filename=/dev/urandom,id=rng0 \
>     -device virtio-rng-ccw,rng=rng0 \
>     -device virtio-scsi-ccw,id=scsi \
>     -drive
> file=/home/bfu/RHEL-8.5.0-20210506.n.0-s390x.qcow2,cache=unsafe,format=raw,
> id=hd0,if=none \
>     -device scsi-hd,drive=hd0 \
>     -drive
> file=/var/tmp/.guestfs-0/appliance.d/root,snapshot=on,id=appliance,
> cache=unsafe,if=none,format=raw \
>     -device scsi-hd,drive=appliance \
>     -device virtio-serial-ccw \
>     -chardev stdio,id=charconsole0 \
>     -device sclpconsole,chardev=charconsole0 \
>     -append "panic=1 console=ttysclp0 edd=off udevtimeout=6000
> udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory
> usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1
> root=UUID=9e192f01-def5-40cc-aab2-743d668be445 selinux=0 guestfs_verbose=1
> TERM=xterm-256color"

qemu version:
qemu-kvm-6.0.0-5.el9.s390x
host kernel: 5.13.0-0.rc2.19.el9.s390x
guest kernel: 5.13.0-0.rc2.19.el9.s390x
libguestfs: libguestfs-1.45.6-6.el9.s390x

VXE support has been merged into upstream QEMU; it should flow into el9 qemu-kvm naturally via the next rebase.

Can you guys retry with upstream QEMU? Make sure not use a recent rhel9 kernel in the guest, because there was a VXE mis-detection BUG that I just recently fixed upstream.

I will close this as RAWHIDE if nobody objects.

Maybe better to leave it open, reassigned to libguestfs if necessary,
so that we ensure that it is tested before RHEL 9 is released.

I don't have the time nor the s390 machine to test it right now.

(In reply to Richard W.M. Jones from comment #6)
> Maybe better to leave it open, reassigned to libguestfs if necessary,
> so that we ensure that it is tested before RHEL 9 is released.

Sure, I can reassign then.

> 
> I don't have the time nor the s390 machine to test it right now.

You don't need a s390x machine, because we're talking about TCG ... but you'll most certainly need time. :)

(In reply to Richard W.M. Jones from comment #6)
> Maybe better to leave it open, reassigned to libguestfs if necessary,
> so that we ensure that it is tested before RHEL 9 is released.

Ok, then let's also set ITR 9.0.0 to make sure that it does not fall through the cracks.

This issue is fixed after qemu is rebased to 6.1.

Test with packages:
qemu-kvm-6.1.0-1.el9
libguestfs-1.45.6-13.el9
kernel-5.14.0-1.el9


Steps:

1. On s390x-kvm-020.lab.eng.rdu2.redhat.com env with RHEL-9.0.0-20210914.1 compose

# LIBGUESTFS_BACKEND_SETTINGS=force_tcg libguestfs-test-tool
...
LIBGUESTFS_BACKEND_SETTINGS=force_tcg
PATH=/root/.local/bin:/root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
XDG_RUNTIME_DIR=/run/user/0
SELinux: Enforcing
guestfs_get_append: (null)
guestfs_get_autosync: 1
guestfs_get_backend: libvirt
guestfs_get_backend_settings: [force_tcg]
guestfs_get_cachedir: /var/tmp
guestfs_get_hv: /usr/libexec/qemu-kvm
guestfs_get_memsize: 1280
guestfs_get_network: 0
guestfs_get_path: /usr/lib64/guestfs
guestfs_get_pgroup: 0
guestfs_get_program: libguestfs-test-tool
guestfs_get_recovery_proc: 1
guestfs_get_smp: 1
guestfs_get_sockdir: /tmp
guestfs_get_tmpdir: /tmp
guestfs_get_trace: 0
guestfs_get_verbose: 1
host_cpu: s390x
Launching appliance, timeout set to 600 seconds.
libguestfs: launch: program=libguestfs-test-tool
libguestfs: launch: version=1.45.6rhel=9,release=13.el9,libvirt
libguestfs: launch: backend registered: direct
libguestfs: launch: backend registered: libvirt
libguestfs: launch: backend registered: uml
libguestfs: launch: backend registered: unix
libguestfs: launch: backend=libvirt
libguestfs: launch: tmpdir=/tmp/libguestfssxKkmT
libguestfs: launch: umask=0022
libguestfs: launch: euid=0
libguestfs: libvirt version = 7006000 (7.6.0)
libguestfs: guest random name = guestfs-a7r7v4h0vn92q25w
libguestfs: connect to libvirt
libguestfs: opening libvirt handle: URI = qemu:///system, auth = default+wrapper, flags = 0
libguestfs: successfully opened libvirt handle: conn = 0x2aa3822b030
libguestfs: qemu version (reported by libvirt) = 6001000 (6.1.0)
libguestfs: get libvirt capabilities
libguestfs: parsing capabilities XML
libguestfs: parsing domcapabilities XML
libguestfs: build appliance
libguestfs: begin building supermin appliance
libguestfs: run supermin
libguestfs: command: run: /usr/bin/supermin
libguestfs: command: run: \ --build
libguestfs: command: run: \ --verbose
libguestfs: command: run: \ --if-newer
libguestfs: command: run: \ --lock /var/tmp/.guestfs-0/lock
libguestfs: command: run: \ --copy-kernel
libguestfs: command: run: \ -f ext2
libguestfs: command: run: \ --host-cpu s390x
libguestfs: command: run: \ /usr/lib64/guestfs/supermin.d
libguestfs: command: run: \ -o /var/tmp/.guestfs-0/appliance.d
supermin: version: 5.2.1
supermin: rpm: detected RPM version 4.16
supermin: rpm: detected RPM architecture s390x
supermin: package handler: fedora/rpm
supermin: acquiring lock on /var/tmp/.guestfs-0/lock
supermin: if-newer: output does not need rebuilding
libguestfs: finished building supermin appliance
libguestfs: command: run: qemu-img
libguestfs: command: run: \ create
libguestfs: command: run: \ -f qcow2
libguestfs: command: run: \ -o backing_file=/var/tmp/.guestfs-0/appliance.d/root,backing_fmt=raw
libguestfs: command: run: \ /tmp/libguestfssxKkmT/overlay2.qcow2
Formatting '/tmp/libguestfssxKkmT/overlay2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=4294967296 backing_file=/var/tmp/.guestfs-0/appliance.d/root backing_fmt=raw lazy_refcounts=off refcount_bits=16
libguestfs: create libvirt XML
libguestfs: command: run: dmesg | grep -Eoh 'lpj=[[:digit:]]+'
libguestfs: read_lpj_from_dmesg: external command exited with error status 1
libguestfs: read_lpj_from_files: no boot messages files are readable
libguestfs: libvirt XML:\n<?xml version="1.0"?>\n<domain type="qemu" xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0">\n  <name>guestfs-a7r7v4h0vn92q25w</name>\n  <memory unit="MiB">1280</memory>\n  <currentMemory unit="MiB">1280</currentMemory>\n  <cpu mode="maximum"/>\n  <vcpu>1</vcpu>\n  <clock offset="utc">\n    <timer name="rtc" tickpolicy="catchup"/>\n    <timer name="pit" tickpolicy="delay"/>\n  </clock>\n  <os>\n    <type>hvm</type>\n    <kernel>/var/tmp/.guestfs-0/appliance.d/kernel</kernel>\n    <initrd>/var/tmp/.guestfs-0/appliance.d/initrd</initrd>\n    <cmdline>panic=1 console=ttysclp0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=UUID=55a88236-55ec-4c6a-818d-b60363aecd62 selinux=0 guestfs_verbose=1 TERM=xterm-256color</cmdline>\n  </os>\n  <on_reboot>destroy</on_reboot>\n  <devices>\n    <rng model="virtio">\n      <backend model="random">/dev/urandom</backend>\n    </rng>\n    <controller type="scsi" index="0" model="virtio-scsi"/>\n    <disk device="disk" type="file">\n      <source file="/tmp/libguestfssxKkmT/scratch1.img"/>\n      <target dev="sda" bus="scsi"/>\n      <driver name="qemu" type="raw" cache="unsafe"/>\n      <address type="drive" controller="0" bus="0" target="0" unit="0"/>\n    </disk>\n    <disk type="file" device="disk">\n      <source file="/tmp/libguestfssxKkmT/overlay2.qcow2"/>\n      <target dev="sdb" bus="scsi"/>\n      <driver name="qemu" type="qcow2" cache="unsafe"/>\n      <address type="drive" controller="0" bus="0" target="1" unit="0"/>\n    </disk>\n    <console type="unix">\n      <source mode="connect" path="/tmp/libguestfsqZUVjX/console.sock"/>\n      <target type="sclp" port="0"/>\n    </console>\n    <channel type="unix">\n      <source mode="connect" path="/tmp/libguestfsqZUVjX/guestfsd.sock"/>\n      <target type="virtio" name="org.libguestfs.channel.0"/>\n    </channel>\n    <controller type="usb" model="none"/>\n    <memballoon model="none"/>\n  </devices>\n  <qemu:commandline>\n    <qemu:env name="TMPDIR" value="/var/tmp"/>\n  </qemu:commandline>\n</domain>\n
...
libguestfs: launch libvirt guest
KASLR disabled: CPU has no PRNG
[    0.344020] Linux version 5.14.0-1.el9.s390x (mockbuild.eng.bos.redhat.com) (gcc (GCC) 11.2.1 20210728 (Red Hat 11.2.1-2), GNU ld version 2.35.2-9.el9) #1 SMP Mon Aug 30 21:22:55 EDT 2021
[    0.346324] setup: Linux is running under KVM in 64-bit mode
[    0.367787] setup: The maximum memory size is 1280MB
[    0.369687] cpu: 1 configured CPUs, 0 standby CPUs
[    0.402303] Write protected kernel read-only data: 14728k
[    0.439695] Zone ranges:
[    0.441673]   DMA      [mem 0x0000000000000000-0x000000007fffffff]
[    0.442415]   Normal   empty
[    0.442458] Movable zone start for each node
[    0.442494] Early memory node ranges
[    0.442527]   node   0: [mem 0x0000000000000000-0x000000004fffffff]
[    0.442936] Initmem setup node 0 [mem 0x0000000000000000-0x000000004fffffff]
[    0.490216] percpu: Embedded 36 pages/cpu s107264 r8192 d32000 u147456
[    0.492564] Built 1 zonelists, mobility grouping on.  Total pages: 322560
[    0.492612] Policy zone: DMA
[    0.492770] Kernel command line: panic=1 console=ttysclp0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=UUID=55a88236-55ec-4c6a-818d-b60363aecd62 selinux=0 guestfs_verbose=1 TERM=xterm-256color
[    0.494836] cgroup: Disabling memory control group subsystem
[    0.495585] Unknown command line parameters: no_timer_check edd=off udevtimeout=6000 tsc=reliable guestfs_verbose=1
[    0.496695] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    0.497188] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[    0.498174] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.518721] Memory: 1250336K/1310720K available (8864K kernel code, 2284K rwdata, 5864K rodata, 3844K init, 864K bss, 60384K reserved, 0K cma-reserved)
[    0.519133] random: get_random_u64 called from kmem_cache_open+0x42/0x410 with crng_init=0
[    0.525249] SLUB: HWalign=256, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.525886] ftrace: allocating 29654 entries in 116 pages
[    0.681307] ftrace: allocated 116 pages with 4 groups
[    0.687981] rcu: Hierarchical RCU implementation.
[    0.688028] rcu: \tRCU restricting CPUs from NR_CPUS=512 to nr_cpu_ids=1.
[    0.688125] \tTrampoline variant of Tasks RCU enabled.
[    0.688145] \tRude variant of Tasks RCU enabled.
[    0.688163] \tTracing variant of Tasks RCU enabled.
[    0.688262] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[    0.688310] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
[    0.728059] NR_IRQS: 3, nr_irqs: 3, preallocated irqs: 3
[    0.732277] clocksource: tod: mask: 0xffffffffffffffff max_cycles: 0x3b0a9be803b0a9, max_idle_ns: 1805497147909793 ns
[    0.735546] Console: colour dummy device 80x25
[    0.739925] printk: console [ttysclp0] enabled
[    0.741448] pid_max: default: 32768 minimum: 301
[    0.742435] LSM: Security Framework initializing
[    0.745486] Yama: becoming mindful.
[    0.746239] LSM support for eBPF active
[    0.746716] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)
[    0.746771] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)
[    0.780408] rcu: Hierarchical SRCU implementation.
[    0.787258] smp: Bringing up secondary CPUs ...
[    0.787377] smp: Brought up 1 node, 1 CPU
[    0.798979] devtmpfs: initialized
[    0.806716] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.806886] futex hash table entries: 256 (order: 4, 65536 bytes, linear)
[    0.816959] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.819721] audit: initializing netlink subsys (disabled)
[    0.824003] Spectre V2 mitigation: execute trampolines
[    0.827485] audit: type=2000 audit(1631954023.759:1): state=initialized audit_enabled=0 res=1
[    0.844941] alg: self-tests disabled
[    0.856393] cryptd: max_cpu_qlen set to 1000
[    0.876454] iommu: Default domain type: Translated 
[    0.878294] SCSI subsystem initialized
[    0.878541] pps_core: LinuxPPS API ver. 1 registered
[    0.878565] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti>
[    0.878644] PTP clock support registered
[    0.887637] NetLabel: Initializing
[    0.887680] NetLabel:  domain hash size = 128
[    0.887703] NetLabel:  protocols = UNLABELED CIPSOv4 CALIPSO
[    0.888419] NetLabel:  unlabeled traffic allowed by default
[    1.209207] VFS: Disk quotas dquot_6.6.0
[    1.209577] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    1.211105] hugetlbfs: disabling because there are no supported hugepage sizes
[    1.255944] NET: Registered PF_INET protocol family
[    1.257208] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)
[    1.262060] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)
[    1.262195] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)
[    1.262422] TCP bind hash table entries: 16384 (order: 6, 262144 bytes, linear)
[    1.262639] TCP: Hash tables configured (established 16384 bind 16384)
[    1.264118] MPTCP token hash table entries: 2048 (order: 3, 49152 bytes, linear)
[    1.264376] UDP hash table entries: 1024 (order: 3, 32768 bytes, linear)
[    1.264598] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes, linear)
[    1.266070] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    1.266306] NET: Registered PF_XDP protocol family
[    1.275314] Trying to unpack rootfs image as initramfs...
[    1.297656] Freeing initrd memory: 3460K
[    1.299012] hypfs: The hardware system does not support hypfs
[    1.300242] hypfs: Initialization of hypfs failed with rc=-61
[    1.303446] Initialise system trusted keyrings
[    1.304742] workingset: timestamp_bits=45 max_order=19 bucket_order=0
[    1.372211] NET: Registered PF_ALG protocol family
[    1.372453] Key type asymmetric registered
[    1.372518] Asymmetric key parser 'x509' registered
[    1.372743] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)
[    1.373495] io scheduler mq-deadline registered
[    1.373572] io scheduler kyber registered
[    1.374080] io scheduler bfq registered
[    1.383055] atomic64_test: passed
[    1.384979] hvc_iucv: The z/VM IUCV HVC device driver cannot be used without z/VM
[    1.387344] rdac: device handler registered
[    1.387820] hp_sw: device handler registered
[    1.387859] emc: device handler registered
[    1.388275] alua: device handler registered
[    1.389282] drop_monitor: Initializing network drop monitor service
[    1.390657] Initializing XFRM netlink socket
[    1.392068] NET: Registered PF_INET6 protocol family
[    1.399210] Segment Routing with IPv6
[    1.399630] NET: Registered PF_PACKET protocol family
[    1.400416] mpls_gso: MPLS GSO support
[    1.401174] cio: Channel measurement facility initialized using format extended (mode autodetected)
[    1.402804] sclp_sd: Store Data request failed (eq=2, di=3, response=0x40f0, flags=0x00, status=0, rc=-5)
[    1.403653] ap: The hardware system does not support AP instructions
[    1.411112] registered taskstats version 1
[    1.412396] Loading compiled-in X.509 certificates
[    1.417823] random: fast init done
[    1.418501] random: crng init done
[    1.426958] Loaded X.509 cert 'Red Hat Enterprise Linux kernel signing key: f07d75deeaf2fe9065bb49167dbb055267c43ba1'
[    1.431025] Loaded X.509 cert 'Red Hat Enterprise Linux Driver Update Program (key 3): bf57f3e87362bc7229d9f465321773dfd1f77a80'
[    1.433394] Loaded X.509 cert 'Red Hat Enterprise Linux kpatch signing key: 4d38fd864ebe18c5f0b72e3852e2014c3a676fc8'
[    1.434640] Loaded X.509 cert 'Red Hat Secure Boot CA 2 (beta): 81f8ee475c3eedfbcea584bed7aedbd37d64b32a'
[    1.434771] page_owner is disabled
[    1.435738] Key type big_key registered
[    1.445301] Key type encrypted registered
[    1.445744] ima: No TPM chip found, activating TPM-bypass!
[    1.445840] Loading compiled-in module X.509 certificates
[    1.447916] Loaded X.509 cert 'Red Hat Enterprise Linux kernel signing key: f07d75deeaf2fe9065bb49167dbb055267c43ba1'
[    1.447994] ima: Allocated hash algorithm: sha256
[    1.449437] ima: No architecture policies found
[    1.449826] evm: Initialising EVM extended attributes:
[    1.449853] evm: security.selinux
[    1.449895] evm: security.SMACK64 (disabled)
[    1.449914] evm: security.SMACK64EXEC (disabled)
[    1.449933] evm: security.SMACK64TRANSMUTE (disabled)
[    1.449953] evm: security.SMACK64MMAP (disabled)
[    1.449971] evm: security.apparmor (disabled)
[    1.449990] evm: security.ima
[    1.450008] evm: security.capability
[    1.450033] evm: HMAC attrs: 0x1
[    1.468094] Freeing unused kernel image (initmem) memory: 3844K
[    1.469277] Write protected read-only-after-init data: 108k
[    1.631620] Checked W+X mappings: passed, no unexpected W+X pages found
[    1.632013] Run /init as init process
supermin: mounting /proc
supermin: ext2 mini initrd starting up: 5.2.1 glibc
...
===== TEST FINISHED OK =====

(In reply to YongkuiGuo from comment #9)
> This issue is fixed after qemu is rebased to 6.1.

Great, thanks for testing! ... do we have to move this BZ now manually to VERIFIED (since it's a TestOnly BZ)?

Yes it sounds like this bug has been verified.

Okay, change the status to VERIFIED per comment 9.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: libguestfs), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2317