Are you still having a problem?

What versions of the following packages are installed?

kernel
qemu-kvm
seabios-bin
libvirt
libguestfs

Yes, I still meet this issue. Here is the package version:

kernel: 4.18.0-193.71.1.rt13.121.el8_2.x86_64
qemu-kvm: qemu-kvm-4.2.0-29.module+el8.2.1+7297+a825794d.x86_64
seabios-bin: seabios-bin-1.13.0-2.module+el8.2.1+7284+aa32a2c4.noarch
libguestfs: libguestfs-1.40.2-24.module+el8.2.1+7154+47ffd890.x86_64

libvirt:
libvirt-daemon-driver-storage-iscsi-direct-6.0.0-25.module+el8.2.1+7154+47ffd890.x86_64
libvirt-daemon-driver-nwfilter-6.0.0-25.module+el8.2.1+7154+47ffd890.x86_64
libvirt-daemon-driver-storage-disk-6.0.0-25.module+el8.2.1+7154+47ffd890.x86_64
libvirt-daemon-driver-storage-6.0.0-25.module+el8.2.1+7154+47ffd890.x86_64
libvirt-daemon-driver-storage-mpath-6.0.0-25.module+el8.2.1+7154+47ffd890.x86_64
libvirt-daemon-6.0.0-25.module+el8.2.1+7154+47ffd890.x86_64
libvirt-daemon-driver-interface-6.0.0-25.module+el8.2.1+7154+47ffd890.x86_64
libvirt-daemon-driver-secret-6.0.0-25.module+el8.2.1+7154+47ffd890.x86_64
libvirt-daemon-driver-network-6.0.0-25.module+el8.2.1+7154+47ffd890.x86_64
libvirt-daemon-kvm-6.0.0-25.module+el8.2.1+7154+47ffd890.x86_64
libvirt-client-6.0.0-25.module+el8.2.1+7154+47ffd890.x86_64
libvirt-daemon-driver-storage-iscsi-6.0.0-25.module+el8.2.1+7154+47ffd890.x86_64
libvirt-libs-6.0.0-25.module+el8.2.1+7154+47ffd890.x86_64
libvirt-daemon-driver-storage-gluster-6.0.0-25.module+el8.2.1+7154+47ffd890.x86_64
libvirt-daemon-driver-storage-rbd-6.0.0-25.module+el8.2.1+7154+47ffd890.x86_64
libvirt-daemon-driver-nodedev-6.0.0-25.module+el8.2.1+7154+47ffd890.x86_64
libvirt-bash-completion-6.0.0-25.module+el8.2.1+7154+47ffd890.x86_64
libvirt-daemon-config-network-6.0.0-25.module+el8.2.1+7154+47ffd890.x86_64
libvirt-daemon-driver-storage-scsi-6.0.0-25.module+el8.2.1+7154+47ffd890.x86_64
libvirt-daemon-driver-storage-logical-6.0.0-25.module+el8.2.1+7154+47ffd890.x86_64
libvirt-daemon-driver-qemu-6.0.0-25.module+el8.2.1+7154+47ffd890.x86_64
libvirt-daemon-driver-storage-core-6.0.0-25.module+el8.2.1+7154+47ffd890.x86_64

> kernel: 4.18.0-193.71.1.rt13.121.el8_2.x86_64

It's probably going to be a problem with the real time kernel.  I would
guess if you use the non-RT kernel it'll be OK.

As for exactly why is going to be a bit of a mystery, but the RT kernels
aren't designed for virtualization use AIUI.

Hello Richard

You are right. Actually, If I use the corresponding stock kernel, this issue was disappeared. From a testing standpoint, this issue belongs to a released version. If there is no customer-related bug involved in this issue, I don't think this issue will affect my test. Or whether to allow RT-related development to assess the extent of the issue's impact？

I'm not sure to what extent we support libguestfs on the RT kernel.  I *think*
we've had this issue come up in the past, but my bugzilla skills are not allowing
me to find any bugs at the moment.

We might have Virt customers using the RT kernel on 8.2, so it might be a good idea to debug this.

@Minxi, are you using CPU isolation? That is, can you reproduce this simply by installing the RT kernel and advanced Virt? Or do you install tuned and configure the nfv-guest/host profiles as well?

@Minxi or @Richard, do we know what the error message is? What is actually failing?

Hello Luiz

This issue is very easy to reproduce. Don't do any CPU isolation. Just install RT kernel and advanced Virt, then run virt-customize can reproduce this issue.

when I want to change a configuration on a guest image(such as change login password, copying a file from host to a guest image, or run some command on a guest image, not in a running VM), virt-customize is a good choice. On this issue, virt-customize can't use as well on RT kernel, so I can't modify some configuration on the guest image.

(In reply to Richard W.M. Jones from comment #6)
> I'm not sure to what extent we support libguestfs on the RT kernel.  I
> *think*
> we've had this issue come up in the past, but my bugzilla skills are not
> allowing
> me to find any bugs at the moment.

This one: https://bugzilla.redhat.com/show_bug.cgi?id=1795973


I can 'not' reproduce this issue. 

Steps:

1. Reserve a bare metal from beaker with distro 'RHEL-8.2.1-updates-20200901.0'

2. Install libguestfs related packages from the repo: http://download.eng.pek2.redhat.com/rhel-8/rel-eng/ADVANCED-VIRT-8/latest-ADVANCED-VIRT-8.2.1-RHEL-8/compose/Advanced-virt/x86_64/os

3. Install the latest kernel-rt and kernel-rt-debug packages from the reppo: http://download.eng.pek2.redhat.com/rhel-8/nightly/updates/RHEL-8/RHEL-8.2.1-20220207.n.0/compose/RT/x86_64/os

4. 
# rpm -q libguestfs qemu-kvm libvirt-daemon kernel-rt kernel-rt-debug seabios-bin
libguestfs-1.40.2-24.module+el8.2.1+7154+47ffd890.x86_64
qemu-kvm-4.2.0-29.module+el8.2.1+7297+a825794d.x86_64
libvirt-daemon-6.0.0-25.module+el8.2.1+7154+47ffd890.x86_64
kernel-rt-4.18.0-193.71.1.rt13.121.el8_2.x86_64
kernel-rt-debug-4.18.0-193.71.1.rt13.121.el8_2.x86_64
seabios-bin-1.13.0-2.module+el8.2.1+7284+aa32a2c4.noarch

5. libguestfs-test-tool works well
# libguestfs-test-tool
     ************************************************************
     *                    IMPORTANT NOTICE
     *
     * When reporting bugs, include the COMPLETE, UNEDITED
     * output below in your bug report.
     *
     ************************************************************
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
XDG_RUNTIME_DIR=/run/user/0
SELinux: Enforcing
guestfs_get_append: (null)
guestfs_get_autosync: 1
guestfs_get_backend: libvirt
guestfs_get_backend_settings: []
guestfs_get_cachedir: /var/tmp
guestfs_get_hv: /usr/libexec/qemu-kvm
guestfs_get_memsize: 768
guestfs_get_network: 0
guestfs_get_path: /usr/lib64/guestfs
guestfs_get_pgroup: 0
guestfs_get_program: libguestfs-test-tool
guestfs_get_recovery_proc: 1
guestfs_get_smp: 1
guestfs_get_sockdir: /tmp
guestfs_get_tmpdir: /tmp
guestfs_get_trace: 0
guestfs_get_verbose: 1
host_cpu: x86_64
Launching appliance, timeout set to 600 seconds.
libguestfs: launch: program=libguestfs-test-tool
libguestfs: launch: version=1.40.2rhel=8,release=24.module+el8.2.1+7154+47ffd890,libvirt
libguestfs: launch: backend registered: unix
libguestfs: launch: backend registered: uml
libguestfs: launch: backend registered: libvirt
libguestfs: launch: backend registered: direct
libguestfs: launch: backend=libvirt
libguestfs: launch: tmpdir=/tmp/libguestfswVcPkr
libguestfs: launch: umask=0022
libguestfs: launch: euid=0
libguestfs: libvirt version = 6000000 (6.0.0)
libguestfs: guest random name = guestfs-ixhmtokg4zekbgnn
libguestfs: connect to libvirt
libguestfs: opening libvirt handle: URI = qemu:///system, auth = default+wrapper, flags = 0
libguestfs: successfully opened libvirt handle: conn = 0x5621e50817d0
libguestfs: qemu version (reported by libvirt) = 4002000 (4.2.0)
libguestfs: get libvirt capabilities
libguestfs: parsing capabilities XML
libguestfs: build appliance
libguestfs: begin building supermin appliance
libguestfs: run supermin
libguestfs: command: run: /usr/bin/supermin
libguestfs: command: run: \ --build
libguestfs: command: run: \ --verbose
libguestfs: command: run: \ --if-newer
libguestfs: command: run: \ --lock /var/tmp/.guestfs-0/lock
libguestfs: command: run: \ --copy-kernel
libguestfs: command: run: \ -f ext2
libguestfs: command: run: \ --host-cpu x86_64
libguestfs: command: run: \ /usr/lib64/guestfs/supermin.d
libguestfs: command: run: \ -o /var/tmp/.guestfs-0/appliance.d
supermin: version: 5.1.19
supermin: rpm: detected RPM version 4.14
supermin: package handler: fedora/rpm
supermin: acquiring lock on /var/tmp/.guestfs-0/lock
supermin: build: /usr/lib64/guestfs/supermin.d
supermin: reading the supermin appliance
supermin: build: visiting /usr/lib64/guestfs/supermin.d/base.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/daemon.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/excludefiles type uncompressed excludefiles
supermin: build: visiting /usr/lib64/guestfs/supermin.d/hostfiles type uncompressed hostfiles
supermin: build: visiting /usr/lib64/guestfs/supermin.d/init.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/packages type uncompressed packages
supermin: build: visiting /usr/lib64/guestfs/supermin.d/udev-rules.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/zz-packages-xfs type uncompressed packages
supermin: mapping package names to installed packages
supermin: resolving full list of package dependencies
supermin: build: 184 packages, including dependencies
supermin: build: 36481 files
supermin: build: 10211 files, after matching excludefiles
supermin: build: 10225 files, after adding hostfiles
supermin: build: 10211 files, after removing unreadable files
supermin: build: 10246 files, after munging
supermin: kernel: looking for kernel using environment variables ...
supermin: kernel: looking for kernels in /lib/modules/*/vmlinuz ...
supermin: kernel: picked vmlinuz /lib/modules/4.18.0-193.71.1.rt13.121.el8_2.x86_64+debug/vmlinuz
supermin: kernel: kernel_version 4.18.0-193.71.1.rt13.121.el8_2.x86_64+debug
supermin: kernel: modpath /lib/modules/4.18.0-193.71.1.rt13.121.el8_2.x86_64+debug
supermin: ext2: creating empty ext2 filesystem '/var/tmp/.guestfs-0/appliance.d.andl9r3n/root'
supermin: ext2: populating from base image
supermin: ext2: copying files from host filesystem
supermin: ext2: copying kernel modules
supermin: ext2: creating minimal initrd '/var/tmp/.guestfs-0/appliance.d.andl9r3n/initrd'
supermin: ext2: wrote 28 modules to minimal initrd
supermin: renaming /var/tmp/.guestfs-0/appliance.d.andl9r3n to /var/tmp/.guestfs-0/appliance.d
libguestfs: finished building supermin appliance
libguestfs: command: run: qemu-img
libguestfs: command: run: \ create
libguestfs: command: run: \ -f qcow2
libguestfs: command: run: \ -o backing_file=/var/tmp/.guestfs-0/appliance.d/root,backing_fmt=raw
libguestfs: command: run: \ /tmp/libguestfswVcPkr/overlay2.qcow2
Formatting '/tmp/libguestfswVcPkr/overlay2.qcow2', fmt=qcow2 size=4294967296 backing_file=/var/tmp/.guestfs-0/appliance.d/root backing_fmt=raw cluster_size=65536 lazy_refcounts=off refcount_bits=16
libguestfs: create libvirt XML
libguestfs: libvirt XML:\n<?xml version="1.0"?>\n<domain type="kvm" xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0">\n  <name>guestfs-ixhmtokg4zekbgnn</name>\n  <memory unit="MiB">768</memory>\n  <currentMemory unit="MiB">768</currentMemory>\n  <cpu mode="host-passthrough">\n    <model fallback="allow"/>\n  </cpu>\n  <vcpu>1</vcpu>\n  <clock offset="utc">\n    <timer name="rtc" tickpolicy="catchup"/>\n    <timer name="pit" tickpolicy="delay"/>\n    <timer name="hpet" present="no"/>\n  </clock>\n  <os>\n    <type>hvm</type>\n    <kernel>/var/tmp/.guestfs-0/appliance.d/kernel</kernel>\n    <initrd>/var/tmp/.guestfs-0/appliance.d/initrd</initrd>\n    <cmdline>panic=1 console=ttyS0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=/dev/sdb selinux=0 guestfs_verbose=1 TERM=xterm-256color</cmdline>\n    <bios useserial="yes"/>\n  </os>\n  <on_reboot>destroy</on_reboot>\n  <devices>\n    <rng model="virtio">\n      <backend model="random">/dev/urandom</backend>\n    </rng>\n    <controller type="scsi" index="0" model="virtio-scsi"/>\n    <disk device="disk" type="file">\n      <source file="/tmp/libguestfswVcPkr/scratch1.img"/>\n      <target dev="sda" bus="scsi"/>\n      <driver name="qemu" type="raw" cache="unsafe"/>\n      <address type="drive" controller="0" bus="0" target="0" unit="0"/>\n    </disk>\n    <disk type="file" device="disk">\n      <source file="/tmp/libguestfswVcPkr/overlay2.qcow2"/>\n      <target dev="sdb" bus="scsi"/>\n      <driver name="qemu" type="qcow2" cache="unsafe"/>\n      <address type="drive" controller="0" bus="0" target="1" unit="0"/>\n    </disk>\n    <serial type="unix">\n      <source mode="connect" path="/tmp/libguestfsyC9CEZ/console.sock"/>\n      <target port="0"/>\n    </serial>\n    <channel type="unix">\n      <source mode="connect" path="/tmp/libguestfsyC9CEZ/guestfsd.sock"/>\n      <target type="virtio" name="org.libguestfs.channel.0"/>\n    </channel>\n    <controller type="usb" model="none"/>\n    <memballoon model="none"/>\n  </devices>\n  <qemu:commandline>\n    <qemu:env name="TMPDIR" value="/var/tmp"/>\n  </qemu:commandline>\n</domain>\n
libguestfs: command: run: ls
libguestfs: command: run: \ -a
libguestfs: command: run: \ -l
libguestfs: command: run: \ -R
libguestfs: command: run: \ -Z /var/tmp/.guestfs-0
libguestfs: /var/tmp/.guestfs-0:
libguestfs: total 0
libguestfs: drwxr-xr-x. 3 root root unconfined_u:object_r:user_tmp_t:s0  37 Feb  8 02:00 .
libguestfs: drwxrwxrwt. 6 root root system_u:object_r:tmp_t:s0          200 Feb  8 02:00 ..
libguestfs: drwxr-xr-x. 2 root root unconfined_u:object_r:user_tmp_t:s0  46 Feb  8 02:00 appliance.d
libguestfs: -rw-r--r--. 1 root root unconfined_u:object_r:user_tmp_t:s0   0 Feb  8 02:00 lock
libguestfs: 
libguestfs: /var/tmp/.guestfs-0/appliance.d:
libguestfs: total 390148
libguestfs: drwxr-xr-x. 2 root root unconfined_u:object_r:user_tmp_t:s0         46 Feb  8 02:00 .
libguestfs: drwxr-xr-x. 3 root root unconfined_u:object_r:user_tmp_t:s0         37 Feb  8 02:00 ..
libguestfs: -rw-r--r--. 1 root root unconfined_u:object_r:user_tmp_t:s0   10731520 Feb  8 02:00 initrd
libguestfs: -rwxr-xr-x. 1 root root unconfined_u:object_r:user_tmp_t:s0   20286832 Feb  8 02:00 kernel
libguestfs: -rw-r--r--. 1 root root unconfined_u:object_r:user_tmp_t:s0 4294967296 Feb  8 02:00 root
libguestfs: command: run: ls
libguestfs: command: run: \ -a
libguestfs: command: run: \ -l
libguestfs: command: run: \ -Z /tmp/libguestfsyC9CEZ
libguestfs: total 4
libguestfs: drwxr-xr-x.  2 root root unconfined_u:object_r:user_tmp_t:s0   47 Feb  8 02:00 .
libguestfs: drwxrwxrwt. 11 root root system_u:object_r:tmp_t:s0          4096 Feb  8 02:00 ..
libguestfs: srw-rw----.  1 root qemu unconfined_u:object_r:user_tmp_t:s0    0 Feb  8 02:00 console.sock
libguestfs: srw-rw----.  1 root qemu unconfined_u:object_r:user_tmp_t:s0    0 Feb  8 02:00 guestfsd.sock
libguestfs: launch libvirt guest
libguestfs: responding to serial console Device Status Report
\x1b[1;256r\x1b[256;256H\x1b[6n
Google, Inc.
Serial Graphics Adapter 11/20/19
SGABIOS $Id$ (mockbuild@) Wed Nov 20 15:03:34 UTC 2019
Term: 80x24
4 0
SeaBIOS (version 1.13.0-2.module+el8.2.1+7284+aa32a2c4)
Machine UUID c3fb41ef-8b18-456f-8eb6-d0af15668c6f
Booting from ROM...
\x1b[2J[    0.000000] Linux version 4.18.0-193.71.1.rt13.121.el8_2.x86_64+debug (mockbuild.eng.bos.redhat.com) (gcc version 8.3.1 20191121 (Red Hat 8.3.1-5) (GCC)) #1 SMP PREEMPT RT Tue Dec 7 12:54:12 EST 2021
[    0.000000] Command line: panic=1 console=ttyS0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=/dev/sdb selinux=0 guestfs_verbose=1 TERM=xterm-256color
[    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
[    0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
[    0.000000] BIOS-provided physical RAM map:
...
===== TEST FINISHED OK =====

6. virt-customize also works
# virt-customize -v -x -a rhel-guest-image-8.2-501.x86_64.qcow2 --root-password password:"redhat"
[   0.0] Examining the guest ...
libguestfs: trace: set_verbose true
libguestfs: trace: set_verbose = 0
libguestfs: trace: set_network true
libguestfs: trace: set_network = 0
libguestfs: trace: add_drive "rhel-guest-image-8.2-501.x86_64.qcow2" "readonly:false" "protocol:file" "discard:besteffort"
libguestfs: trace: add_drive = 0
libguestfs: trace: launch
libguestfs: trace: max_disks
libguestfs: trace: max_disks = 255
libguestfs: trace: get_tmpdir
libguestfs: trace: get_tmpdir = "/tmp"
libguestfs: trace: version
libguestfs: trace: version = <struct guestfs_version = major: 1, minor: 40, release: 2, extra: rhel=8,release=24.module+el8.2.1+7154+47ffd890,libvirt, >
libguestfs: trace: get_backend
libguestfs: trace: get_backend = "libvirt"
libguestfs: launch: program=virt-customize
libguestfs: launch: version=1.40.2rhel=8,release=24.module+el8.2.1+7154+47ffd890,libvirt
libguestfs: launch: backend registered: unix
libguestfs: launch: backend registered: uml
libguestfs: launch: backend registered: libvirt
libguestfs: launch: backend registered: direct
libguestfs: launch: backend=libvirt
libguestfs: launch: tmpdir=/tmp/libguestfsN3VRtB
libguestfs: launch: umask=0022
libguestfs: launch: euid=0
libguestfs: libvirt version = 6000000 (6.0.0)
libguestfs: guest random name = guestfs-p2siu85ez955d50s
libguestfs: connect to libvirt
libguestfs: opening libvirt handle: URI = qemu:///system, auth = default+wrapper, flags = 0
libguestfs: successfully opened libvirt handle: conn = 0x560c3d4c3cc0
libguestfs: qemu version (reported by libvirt) = 4002000 (4.2.0)
libguestfs: get libvirt capabilities
libguestfs: parsing capabilities XML
libguestfs: trace: get_backend_setting "force_tcg"
libguestfs: trace: get_backend_setting = NULL (error)
libguestfs: trace: get_backend_setting "internal_libvirt_label"
libguestfs: trace: get_backend_setting = NULL (error)
libguestfs: trace: get_backend_setting "internal_libvirt_imagelabel"
libguestfs: trace: get_backend_setting = NULL (error)
libguestfs: trace: get_backend_setting "internal_libvirt_norelabel_disks"
libguestfs: trace: get_backend_setting = NULL (error)
libguestfs: build appliance
libguestfs: trace: get_cachedir
libguestfs: trace: get_cachedir = "/var/tmp"
libguestfs: begin building supermin appliance
libguestfs: run supermin
libguestfs: command: run: /usr/bin/supermin
libguestfs: command: run: \ --build
libguestfs: command: run: \ --verbose
libguestfs: command: run: \ --if-newer
libguestfs: command: run: \ --lock /var/tmp/.guestfs-0/lock
libguestfs: command: run: \ --copy-kernel
libguestfs: command: run: \ -f ext2
libguestfs: command: run: \ --host-cpu x86_64
libguestfs: command: run: \ /usr/lib64/guestfs/supermin.d
libguestfs: command: run: \ -o /var/tmp/.guestfs-0/appliance.d
supermin: version: 5.1.19
supermin: rpm: detected RPM version 4.14
supermin: package handler: fedora/rpm
supermin: acquiring lock on /var/tmp/.guestfs-0/lock
supermin: build: /usr/lib64/guestfs/supermin.d
supermin: reading the supermin appliance
supermin: build: visiting /usr/lib64/guestfs/supermin.d/base.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/daemon.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/excludefiles type uncompressed excludefiles
supermin: build: visiting /usr/lib64/guestfs/supermin.d/hostfiles type uncompressed hostfiles
supermin: build: visiting /usr/lib64/guestfs/supermin.d/init.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/packages type uncompressed packages
supermin: build: visiting /usr/lib64/guestfs/supermin.d/udev-rules.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/zz-packages-xfs type uncompressed packages
supermin: mapping package names to installed packages
supermin: resolving full list of package dependencies
supermin: build: 184 packages, including dependencies
supermin: build: 36481 files
supermin: build: 10211 files, after matching excludefiles
supermin: build: 10225 files, after adding hostfiles
supermin: build: 10211 files, after removing unreadable files
supermin: build: 10246 files, after munging
supermin: kernel: looking for kernel using environment variables ...
supermin: kernel: looking for kernels in /lib/modules/*/vmlinuz ...
supermin: kernel: picked vmlinuz /lib/modules/4.18.0-193.71.1.rt13.121.el8_2.x86_64+debug/vmlinuz
supermin: kernel: kernel_version 4.18.0-193.71.1.rt13.121.el8_2.x86_64+debug
supermin: kernel: modpath /lib/modules/4.18.0-193.71.1.rt13.121.el8_2.x86_64+debug
supermin: ext2: creating empty ext2 filesystem '/var/tmp/.guestfs-0/appliance.d.fstt5zko/root'
supermin: ext2: populating from base image
supermin: ext2: copying files from host filesystem
supermin: ext2: copying kernel modules
supermin: ext2: creating minimal initrd '/var/tmp/.guestfs-0/appliance.d.fstt5zko/initrd'
supermin: ext2: wrote 28 modules to minimal initrd
supermin: renaming /var/tmp/.guestfs-0/appliance.d.fstt5zko to /var/tmp/.guestfs-0/appliance.d
libguestfs: finished building supermin appliance
libguestfs: trace: disk_create "/tmp/libguestfsN3VRtB/overlay1.qcow2" "qcow2" -1 "backingfile:/var/tmp/.guestfs-0/appliance.d/root" "backingformat:raw"
libguestfs: command: run: qemu-img
libguestfs: command: run: \ create
libguestfs: command: run: \ -f qcow2
libguestfs: command: run: \ -o backing_file=/var/tmp/.guestfs-0/appliance.d/root,backing_fmt=raw
libguestfs: command: run: \ /tmp/libguestfsN3VRtB/overlay1.qcow2
Formatting '/tmp/libguestfsN3VRtB/overlay1.qcow2', fmt=qcow2 size=4294967296 backing_file=/var/tmp/.guestfs-0/appliance.d/root backing_fmt=raw cluster_size=65536 lazy_refcounts=off refcount_bits=16
libguestfs: trace: disk_create = 0
libguestfs: trace: get_sockdir
libguestfs: trace: get_sockdir = "/tmp"
libguestfs: create libvirt XML
libguestfs: trace: disk_format "rhel-guest-image-8.2-501.x86_64.qcow2"
libguestfs: command: run: qemu-img --help | grep -sqE -- '\binfo\b.*-U\b'
libguestfs: command: run: qemu-img
libguestfs: command: run: \ info
libguestfs: command: run: \ -U
libguestfs: command: run: \ --output json
libguestfs: command: run: \ ./rhel-guest-image-8.2-501.x86_64.qcow2
libguestfs: parse_json: qemu-img info JSON output:\n{\n    "virtual-size": 10737418240,\n    "filename": "./rhel-guest-image-8.2-501.x86_64.qcow2",\n    "cluster-size": 65536,\n    "format": "qcow2",\n    "actual-size": 1199112192,\n    "format-specific": {\n        "type": "qcow2",\n        "data": {\n            "compat": "0.10",\n            "refcount-bits": 16\n        }\n    },\n    "dirty-flag": false\n}\n\n
libguestfs: trace: disk_format = "qcow2"
libguestfs: trace: get_cachedir
libguestfs: trace: get_cachedir = "/var/tmp"
libguestfs: libvirt XML:\n<?xml version="1.0"?>\n<domain type="kvm" xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0">\n  <name>guestfs-p2siu85ez955d50s</name>\n  <memory unit="MiB">768</memory>\n  <currentMemory unit="MiB">768</currentMemory>\n  <cpu mode="host-passthrough">\n    <model fallback="allow"/>\n  </cpu>\n  <vcpu>1</vcpu>\n  <clock offset="utc">\n    <timer name="rtc" tickpolicy="catchup"/>\n    <timer name="pit" tickpolicy="delay"/>\n    <timer name="hpet" present="no"/>\n  </clock>\n  <os>\n    <type>hvm</type>\n    <kernel>/var/tmp/.guestfs-0/appliance.d/kernel</kernel>\n    <initrd>/var/tmp/.guestfs-0/appliance.d/initrd</initrd>\n    <cmdline>panic=1 console=ttyS0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=/dev/sdb selinux=0 guestfs_verbose=1 guestfs_network=1 TERM=xterm-256color</cmdline>\n    <bios useserial="yes"/>\n  </os>\n  <on_reboot>destroy</on_reboot>\n  <devices>\n    <rng model="virtio">\n      <backend model="random">/dev/urandom</backend>\n    </rng>\n    <controller type="scsi" index="0" model="virtio-scsi"/>\n    <disk device="disk" type="file">\n      <source file="/home/rhel-guest-image-8.2-501.x86_64.qcow2"/>\n      <target dev="sda" bus="scsi"/>\n      <driver name="qemu" type="qcow2" cache="writeback" discard="unmap"/>\n      <address type="drive" controller="0" bus="0" target="0" unit="0"/>\n    </disk>\n    <disk type="file" device="disk">\n      <source file="/tmp/libguestfsN3VRtB/overlay1.qcow2"/>\n      <target dev="sdb" bus="scsi"/>\n      <driver name="qemu" type="qcow2" cache="unsafe"/>\n      <address type="drive" controller="0" bus="0" target="1" unit="0"/>\n    </disk>\n    <serial type="unix">\n      <source mode="connect" path="/tmp/libguestfs3WAdBH/console.sock"/>\n      <target port="0"/>\n    </serial>\n    <channel type="unix">\n      <source mode="connect" path="/tmp/libguestfs3WAdBH/guestfsd.sock"/>\n      <target type="virtio" name="org.libguestfs.channel.0"/>\n    </channel>\n    <controller type="usb" model="none"/>\n    <memballoon model="none"/>\n  </devices>\n  <qemu:commandline>\n    <qemu:env name="TMPDIR" value="/var/tmp"/>\n    <qemu:arg value="-netdev"/>\n    <qemu:arg value="user,id=usernet,net=169.254.0.0/16"/>\n    <qemu:arg value="-device"/>\n    <qemu:arg value="virtio-net-pci,netdev=usernet"/>\n  </qemu:commandline>\n</domain>\n
libguestfs: trace: get_cachedir
libguestfs: trace: get_cachedir = "/var/tmp"
libguestfs: command: run: ls
libguestfs: command: run: \ -a
libguestfs: command: run: \ -l
libguestfs: command: run: \ -R
libguestfs: command: run: \ -Z /var/tmp/.guestfs-0
libguestfs: /var/tmp/.guestfs-0:
libguestfs: total 0
libguestfs: drwxr-xr-x. 3 root root unconfined_u:object_r:user_tmp_t:s0  37 Feb  8 02:04 .
libguestfs: drwxrwxrwt. 6 root root system_u:object_r:tmp_t:s0          200 Feb  8 02:04 ..
libguestfs: drwxr-xr-x. 2 root root unconfined_u:object_r:user_tmp_t:s0  46 Feb  8 02:04 appliance.d
libguestfs: -rw-r--r--. 1 root root unconfined_u:object_r:user_tmp_t:s0   0 Feb  8 02:04 lock
libguestfs: 
libguestfs: /var/tmp/.guestfs-0/appliance.d:
libguestfs: total 390148
libguestfs: drwxr-xr-x. 2 root root unconfined_u:object_r:user_tmp_t:s0         46 Feb  8 02:04 .
libguestfs: drwxr-xr-x. 3 root root unconfined_u:object_r:user_tmp_t:s0         37 Feb  8 02:04 ..
libguestfs: -rw-r--r--. 1 root root unconfined_u:object_r:user_tmp_t:s0   10731520 Feb  8 02:04 initrd
libguestfs: -rwxr-xr-x. 1 root root unconfined_u:object_r:user_tmp_t:s0   20286832 Feb  8 02:04 kernel
libguestfs: -rw-r--r--. 1 root root unconfined_u:object_r:user_tmp_t:s0 4294967296 Feb  8 02:04 root
libguestfs: command: run: ls
libguestfs: command: run: \ -a
libguestfs: command: run: \ -l
libguestfs: command: run: \ -Z /tmp/libguestfs3WAdBH
libguestfs: total 4
libguestfs: drwxr-xr-x.  2 root root unconfined_u:object_r:user_tmp_t:s0   47 Feb  8 02:04 .
libguestfs: drwxrwxrwt. 11 root root system_u:object_r:tmp_t:s0          4096 Feb  8 02:04 ..
libguestfs: srw-rw----.  1 root qemu unconfined_u:object_r:user_tmp_t:s0    0 Feb  8 02:04 console.sock
libguestfs: srw-rw----.  1 root qemu unconfined_u:object_r:user_tmp_t:s0    0 Feb  8 02:04 guestfsd.sock
libguestfs: launch libvirt guest
libguestfs: responding to serial console Device Status Report
\x1b[1;256r\x1b[256;256H\x1b[6n
Google, Inc.
Serial Graphics Adapter 11/20/19
SGABIOS $Id$ (mockbuild@) Wed Nov 20 15:03:34 UTC 2019
Term: 80x24
4 0
SeaBIOS (version 1.13.0-2.module+el8.2.1+7284+aa32a2c4)
Machine UUID 1af5efaa-c0ce-41dc-b8e8-eaabfa8597ef


iPXE (http://ipxe.org) 00:05.0 C100 PCI2.10 PnP PMM+2FF90BD0+2FED0BD0 C100
                                                                               


Booting from ROM...
\x1b[2J[    0.000000] Linux version 4.18.0-193.71.1.rt13.121.el8_2.x86_64+debug (mockbuild.eng.bos.redhat.com) (gcc version 8.3.1 20191121 (Red Hat 8.3.1-5) (GCC)) #1 SMP PREEMPT RT Tue Dec 7 12:54:12 EST 2021
[    0.000000] Command line: panic=1 console=ttyS0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=/dev/sdb selinux=0 guestfs_verbose=1 guestfs_network=1 TERM=xterm-256color
[    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
...
[  25.4] Finishing off
libguestfs: trace: shutdown
libguestfs: trace: internal_autosync
guestfsd: => umount_all (0x2f) took 1.37 secs
guestfsd: <= internal_autosync (0x11a) request length 40 bytes
umount-all: /proc/mounts: fsname=/dev/root dir=/ type=ext2 opts=rw,noatime freq=0 passno=0
umount-all: /proc/mounts: fsname=/proc dir=/proc type=proc opts=rw,relatime freq=0 passno=0
umount-all: /proc/mounts: fsname=/sys dir=/sys type=sysfs opts=rw,relatime freq=0 passno=0
umount-all: /proc/mounts: fsname=tmpfs dir=/run type=tmpfs opts=rw,nosuid,relatime,size=68200k,mode=755 freq=0 passno=0
umount-all: /proc/mounts: fsname=/dev dir=/dev type=devtmpfs opts=rw,relatime,size=158968k,nr_inodes=39742,mode=755 freq=0 passno=0
umount-all: /proc/mounts: fsname=/dev/pts dir=/dev/pts type=devpts opts=rw,relatime,mode=600,ptmxmode=000 freq=0 passno=0
umount-all: /proc/mounts: fsname=shmfs dir=/dev/shm type=tmpfs opts=rw,relatime freq=0 passno=0
commandrvf: stdout=n stderr=y flags=0x0
commandrvf: udevadm --debug settle -E /dev/sdb
calling: settle
commandrvf: stdout=n stderr=y flags=0x0
commandrvf: udevadm --debug settle -E /dev/sda
calling: settle
fsync /dev/sda
libguestfs: trace: internal_autosync = 0
libguestfs: calling virDomainDestroy flags=VIR_DOMAIN_DESTROY_GRACEFUL
libguestfs: trace: shutdown = 0
libguestfs: trace: close
libguestfs: closing guestfs handle 0x560c3d4c20d0 (state 0)
libguestfs: command: run: rm
libguestfs: command: run: \ -rf /tmp/libguestfsN3VRtB
libguestfs: command: run: rm
libguestfs: command: run: \ -rf /tmp/libguestfs3WAdBH

7. virt-customize works with direct mode
# LIBGUESTFS_BACKEND=direct virt-customize -v -x -a rhel-guest-image-8.2-501.x86_64.qcow2 --root-password password:"redhat"
[   0.0] Examining the guest ...
libguestfs: trace: set_verbose true
libguestfs: trace: set_verbose = 0
libguestfs: trace: set_network true
libguestfs: trace: set_network = 0
libguestfs: trace: add_drive "rhel-guest-image-8.2-501.x86_64.qcow2" "readonly:false" "protocol:file" "discard:besteffort"
libguestfs: trace: add_drive = 0
libguestfs: trace: launch
libguestfs: trace: max_disks
libguestfs: trace: max_disks = 255
libguestfs: trace: get_tmpdir
libguestfs: trace: get_tmpdir = "/tmp"
libguestfs: trace: version
libguestfs: trace: version = <struct guestfs_version = major: 1, minor: 40, release: 2, extra: rhel=8,release=24.module+el8.2.1+7154+47ffd890,libvirt, >
libguestfs: trace: get_backend
libguestfs: trace: get_backend = "direct"
libguestfs: launch: program=virt-customize
libguestfs: launch: version=1.40.2rhel=8,release=24.module+el8.2.1+7154+47ffd890,libvirt
libguestfs: launch: backend registered: unix
libguestfs: launch: backend registered: uml
libguestfs: launch: backend registered: libvirt
libguestfs: launch: backend registered: direct
libguestfs: launch: backend=direct
libguestfs: launch: tmpdir=/tmp/libguestfst9Aqcf
libguestfs: launch: umask=0022
libguestfs: launch: euid=0
libguestfs: trace: get_cachedir
libguestfs: trace: get_cachedir = "/var/tmp"
libguestfs: begin building supermin appliance
libguestfs: run supermin
libguestfs: command: run: /usr/bin/supermin
libguestfs: command: run: \ --build
libguestfs: command: run: \ --verbose
libguestfs: command: run: \ --if-newer
libguestfs: command: run: \ --lock /var/tmp/.guestfs-0/lock
libguestfs: command: run: \ --copy-kernel
libguestfs: command: run: \ -f ext2
libguestfs: command: run: \ --host-cpu x86_64
libguestfs: command: run: \ /usr/lib64/guestfs/supermin.d
libguestfs: command: run: \ -o /var/tmp/.guestfs-0/appliance.d
supermin: version: 5.1.19
supermin: rpm: detected RPM version 4.14
supermin: package handler: fedora/rpm
supermin: acquiring lock on /var/tmp/.guestfs-0/lock
supermin: build: /usr/lib64/guestfs/supermin.d
supermin: reading the supermin appliance
supermin: build: visiting /usr/lib64/guestfs/supermin.d/base.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/daemon.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/excludefiles type uncompressed excludefiles
supermin: build: visiting /usr/lib64/guestfs/supermin.d/hostfiles type uncompressed hostfiles
supermin: build: visiting /usr/lib64/guestfs/supermin.d/init.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/packages type uncompressed packages
supermin: build: visiting /usr/lib64/guestfs/supermin.d/udev-rules.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/zz-packages-xfs type uncompressed packages
supermin: mapping package names to installed packages
supermin: resolving full list of package dependencies
supermin: build: 184 packages, including dependencies
supermin: build: 36481 files
supermin: build: 10211 files, after matching excludefiles
supermin: build: 10225 files, after adding hostfiles
supermin: build: 10211 files, after removing unreadable files
supermin: build: 10246 files, after munging
supermin: kernel: looking for kernel using environment variables ...
supermin: kernel: looking for kernels in /lib/modules/*/vmlinuz ...
supermin: kernel: picked vmlinuz /lib/modules/4.18.0-193.71.1.rt13.121.el8_2.x86_64+debug/vmlinuz
supermin: kernel: kernel_version 4.18.0-193.71.1.rt13.121.el8_2.x86_64+debug
supermin: kernel: modpath /lib/modules/4.18.0-193.71.1.rt13.121.el8_2.x86_64+debug
supermin: ext2: creating empty ext2 filesystem '/var/tmp/.guestfs-0/appliance.d.qelxddaf/root'
supermin: ext2: populating from base image
supermin: ext2: copying files from host filesystem
supermin: ext2: copying kernel modules
supermin: ext2: creating minimal initrd '/var/tmp/.guestfs-0/appliance.d.qelxddaf/initrd'
supermin: ext2: wrote 28 modules to minimal initrd
supermin: renaming /var/tmp/.guestfs-0/appliance.d.qelxddaf to /var/tmp/.guestfs-0/appliance.d
libguestfs: finished building supermin appliance
libguestfs: begin testing qemu features
libguestfs: trace: get_cachedir
libguestfs: trace: get_cachedir = "/var/tmp"
libguestfs: checking for previously cached test results of /usr/libexec/qemu-kvm, in /var/tmp/.guestfs-0
libguestfs: command: run: /usr/libexec/qemu-kvm
libguestfs: command: run: \ -display none
libguestfs: command: run: \ -help
libguestfs: command: run: /usr/libexec/qemu-kvm
libguestfs: command: run: \ -display none
libguestfs: command: run: \ -machine accel=kvm:tcg
libguestfs: command: run: \ -device ?
libguestfs: command: run: echo '{ "execute": "qmp_capabilities" }' '{ "execute": "query-qmp-schema" }' '{ "execute": "quit" }'  | "/usr/libexec/qemu-kvm" -display none -machine "accel=kvm:tcg" -qmp stdio
libguestfs: command: run: echo '{ "execute": "qmp_capabilities" }' '{ "execute": "query-kvm" }' '{ "execute": "quit" }'  | "/usr/libexec/qemu-kvm" -display none -machine "accel=kvm:tcg" -qmp stdio
libguestfs: saving test results
libguestfs: qemu version: 4.2
libguestfs: qemu mandatory locking: yes
libguestfs: qemu KVM: enabled
libguestfs: trace: get_backend_setting "force_tcg"
libguestfs: trace: get_backend_setting = NULL (error)
libguestfs: trace: get_sockdir
libguestfs: trace: get_sockdir = "/tmp"
libguestfs: finished testing qemu features
libguestfs: trace: get_backend_setting "gdb"
libguestfs: trace: get_backend_setting = NULL (error)
/usr/libexec/qemu-kvm \
    -global virtio-blk-pci.scsi=off \
    -no-user-config \
    -enable-fips \
    -nodefaults \
    -display none \
    -machine accel=kvm:tcg \
    -cpu host \
    -m 768 \
    -no-reboot \
    -rtc driftfix=slew \
    -no-hpet \
    -global kvm-pit.lost_tick_policy=discard \
    -kernel /var/tmp/.guestfs-0/appliance.d/kernel \
    -initrd /var/tmp/.guestfs-0/appliance.d/initrd \
    -object rng-random,filename=/dev/urandom,id=rng0 \
    -device virtio-rng-pci,rng=rng0 \
    -device virtio-scsi-pci,id=scsi \
    -drive file=/home/rhel-guest-image-8.2-501.x86_64.qcow2,cache=writeback,discard=unmap,id=hd0,if=none \
    -device scsi-hd,drive=hd0 \
    -drive file=/var/tmp/.guestfs-0/appliance.d/root,snapshot=on,id=appliance,cache=unsafe,if=none,format=raw \
    -device scsi-hd,drive=appliance \
    -device virtio-serial-pci \
    -serial stdio \
    -device sga \
    -chardev socket,path=/tmp/libguestfsc9I26f/guestfsd.sock,id=channel0 \
    -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 \
    -netdev user,id=usernet,net=169.254.0.0/16 \
    -device virtio-net-pci,netdev=usernet \
    -append "panic=1 console=ttyS0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=/dev/sdb selinux=0 guestfs_verbose=1 guestfs_network=1 TERM=xterm-256color"
libguestfs: responding to serial console Device Status Report
\x1b[1;256r\x1b[256;256H\x1b[6n
Google, Inc.
Serial Graphics Adapter 11/20/19
SGABIOS $Id$ (mockbuild@) Wed Nov 20 15:03:34 UTC 2019
Term: 80x24
4 0
SeaBIOS (version 1.13.0-2.module+el8.2.1+7284+aa32a2c4)


iPXE (http://ipxe.org) 00:05.0 C100 PCI2.10 PnP PMM+2FF907A0+2FED07A0 C100
                                                                               


Booting from ROM...
\x1b[2J[    0.000000] Linux version 4.18.0-193.71.1.rt13.121.el8_2.x86_64+debug (mockbuild.eng.bos.redhat.com) (gcc version 8.3.1 20191121 (Red Hat 8.3.1-5) (GCC)) #1 SMP PREEMPT RT Tue Dec 7 12:54:12 EST 2021
[    0.000000] Command line: panic=1 console=ttyS0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=/dev/sdb selinux=0 guestfs_verbose=1 guestfs_network=1 TERM=xterm-256color
[    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
[    0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
[    0.000000] BIOS-provided physical RAM map:
...
[  32.9] Finishing off

> @Minxi or @Richard, do we know what the error message is? What is actually failing?

In the original bug report there was no output after the usual
SeaBIOS messages, so either we didn't get into the kernel at all,
or we did get into the kernel but earlyprintk didn't work (most
likely the former).  Unfortunately it's impossible to tell what
really happened without a local reproducer and a ton of debugging.

(In reply to Richard W.M. Jones from comment #10)
> > @Minxi or @Richard, do we know what the error message is? What is actually failing?
> 
> In the original bug report there was no output after the usual
> SeaBIOS messages, so either we didn't get into the kernel at all,
> or we did get into the kernel but earlyprintk didn't work (most
> likely the former).  Unfortunately it's impossible to tell what
> really happened without a local reproducer and a ton of debugging.

Richard, can you offer an advice if/how we should move forward?

If I understood the report correctly, this failure only affects 8.2 (ie. it works with more recent RT kernels). We'll have Virt customers on 8.2.z for a possibly a long time. Also, if this is a kernel regression of some kind that could potentially affect other RT kernel customers, then it makes it more important to debug it.

The Virt-RT engineers could have a look, but we'd need a reproducer (Prasad, could you have a look?).

I can only offer suggestions I'm afraid.  If we cannot reproduce it locally
then we cannot debug it.

It should be possible to reproduce it by:

 - Install RHEL 8.2 on baremetal

 - Install RT kernel

 - Install libguestfs

 - Run: libguestfs-test-tool

If the bug is reproduced, you'll see output like this:

  Booting from ROM...
  \x1b[2Jlibguestfs: child_cleanup: 0x562783851870: child process died

where the "Booting from ROM" message and \x1b[2J control codes come
from seabios, there is no kernel output, then about 600 seconds
later libguestfs times out waiting for the kernel and prints the debug message.

(Yongkui tried that above and couldn't reproduce it, so maybe it's hardware dependent)

I still can not reproduce this issue. Probably this problem has something with specific hardware.

If needed I can provide my test environment to dev to see this issue. or any other help I can provide.

(In reply to Luiz Capitulino from comment #13)
> The Virt-RT engineers could have a look, but we'd need a reproducer (Prasad,
> could you have a look?).

* Yes, will check.

# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.2 (Ootpa)

#
# rpm -q libguestfs libvirt-daemon qemu-kvm seabios-bin kernel-rt
libguestfs-1.40.2-24.module+el8.2.1+7154+47ffd890.x86_64
libvirt-daemon-6.0.0-25.module+el8.2.1+7154+47ffd890.x86_64
qemu-kvm-4.2.0-29.module+el8.2.1+7297+a825794d.x86_64
seabios-bin-1.13.0-2.module+el8.2.1+7284+aa32a2c4.noarch
kernel-rt-4.18.0-193.71.1.rt13.121.el8_2.x86_64
#
# uname -r
4.18.0-193.el8.x86_64
#
#
# LIBGUESTFS_MEMSIZE=2048 LIBGUESTFS_BACKEND=direct virt-customize -a /var/lib/libvirt/images/r84vm.qcow2 --root-password password:"redhat"
[   0.0] Examining the guest ...
[  11.9] Setting a random seed
[  12.0] Setting passwords
[  14.9] Finishing off
#
#
# libguestfs-test-tool
...
     * When reporting bugs, include the COMPLETE, UNEDITED
     * output below in your bug report.
     *
     ************************************************************
...
===== TEST FINISHED OK =====


# uname -r
4.18.0-193.71.1.rt13.121.el8_2.x86_64

#
# LIBGUESTFS_MEMSIZE=2048 LIBGUESTFS_BACKEND=direct virt-customize -a /var/lib/libvirt/images/f35vm.qcow2 --root-password password:"redhat"
[   0.0] Examining the guest ...
[   7.9] Setting a random seed
[   7.9] Setting passwords
[  10.4] Finishing off
#
#
# LIBGUESTFS_MEMSIZE=2048 libguestfs-test-tool
     * When reporting bugs, include the COMPLETE, UNEDITED
     * output below in your bug report.
     *
     ************************************************************
...
===== TEST FINISHED OK =====


This issue of libguestfs throwing an error about child process running the guest VM dying is not reproducing.
The issue may have something to do with a guest image I guess, not sure. It does not seem to show up now.

@yoguo: @mhou: ...wdyt?

Hello Prasad

Sorry for delay responed your message. I can reproduce this issue on my environment. I will share my envionment through e-mail thread. Here is my test step:

# rpm -q libguestfs libvirt-daemon qemu-kvm seabios-bin kernel-rt
libguestfs-1.40.2-24.module+el8.2.1+7154+47ffd890.x86_64
libvirt-daemon-6.0.0-25.module+el8.2.1+7154+47ffd890.x86_64
qemu-kvm-4.2.0-29.module+el8.2.1+7297+a825794d.x86_64
seabios-bin-1.13.0-2.module+el8.2.1+7284+aa32a2c4.noarch
kernel-rt-4.18.0-193.70.1.rt13.120.el8_2.x86_64


# LIBGUESTFS_MEMSIZE=2048 LIBGUESTFS_BACKEND=direct virt-customize -v -x -a /root/rhel7.9.qcow2 --root-password password:"redhat"
[   0.0] Examining the guest ...
libguestfs: trace: set_verbose true
libguestfs: trace: set_verbose = 0
libguestfs: trace: set_network true
libguestfs: trace: set_network = 0
libguestfs: trace: add_drive "/root/rhel7.9.qcow2" "readonly:false" "protocol:file" "discard:besteffort"
libguestfs: trace: add_drive = 0
libguestfs: trace: launch
libguestfs: trace: max_disks
libguestfs: trace: max_disks = 255
libguestfs: trace: get_tmpdir
libguestfs: trace: get_tmpdir = "/tmp"
libguestfs: trace: version
libguestfs: trace: version = <struct guestfs_version = major: 1, minor: 40, release: 2, extra: rhel=8,release=24.module+el8.2.1+7154+47ffd890,libvirt, >
libguestfs: trace: get_backend
libguestfs: trace: get_backend = "direct"
libguestfs: launch: program=virt-customize
libguestfs: launch: version=1.40.2rhel=8,release=24.module+el8.2.1+7154+47ffd890,libvirt
libguestfs: launch: backend registered: unix
libguestfs: launch: backend registered: uml
libguestfs: launch: backend registered: libvirt
libguestfs: launch: backend registered: direct
libguestfs: launch: backend=direct
libguestfs: launch: tmpdir=/tmp/libguestfsXXkMH4
libguestfs: launch: umask=0022
libguestfs: launch: euid=0
libguestfs: trace: get_cachedir
libguestfs: trace: get_cachedir = "/var/tmp"
libguestfs: begin building supermin appliance
libguestfs: run supermin
libguestfs: command: run: /usr/bin/supermin
libguestfs: command: run: \ --build
libguestfs: command: run: \ --verbose
libguestfs: command: run: \ --if-newer
libguestfs: command: run: \ --lock /var/tmp/.guestfs-0/lock
libguestfs: command: run: \ --copy-kernel
libguestfs: command: run: \ -f ext2
libguestfs: command: run: \ --host-cpu x86_64
libguestfs: command: run: \ /usr/lib64/guestfs/supermin.d
libguestfs: command: run: \ -o /var/tmp/.guestfs-0/appliance.d
supermin: version: 5.1.19
supermin: rpm: detected RPM version 4.14
supermin: package handler: fedora/rpm
supermin: acquiring lock on /var/tmp/.guestfs-0/lock
supermin: if-newer: output does not need rebuilding
libguestfs: finished building supermin appliance
libguestfs: begin testing qemu features
libguestfs: trace: get_cachedir
libguestfs: trace: get_cachedir = "/var/tmp"
libguestfs: checking for previously cached test results of /usr/libexec/qemu-kvm, in /var/tmp/.guestfs-0
libguestfs: loading previously cached test results
libguestfs: qemu version: 4.2
libguestfs: qemu mandatory locking: yes
libguestfs: qemu KVM: enabled
libguestfs: trace: get_backend_setting "force_tcg"
libguestfs: trace: get_backend_setting = NULL (error)
libguestfs: trace: get_sockdir
libguestfs: trace: get_sockdir = "/tmp"
libguestfs: finished testing qemu features
libguestfs: trace: get_backend_setting "gdb"
libguestfs: trace: get_backend_setting = NULL (error)
/usr/libexec/qemu-kvm \
    -global virtio-blk-pci.scsi=off \
    -no-user-config \
    -enable-fips \
    -nodefaults \
    -display none \
    -machine accel=kvm:tcg \
    -cpu host \
    -m 2048 \
    -no-reboot \
    -rtc driftfix=slew \
    -no-hpet \
    -global kvm-pit.lost_tick_policy=discard \
    -kernel /var/tmp/.guestfs-0/appliance.d/kernel \
    -initrd /var/tmp/.guestfs-0/appliance.d/initrd \
    -object rng-random,filename=/dev/urandom,id=rng0 \
    -device virtio-rng-pci,rng=rng0 \
    -device virtio-scsi-pci,id=scsi \
    -drive file=/root/rhel7.9.qcow2,cache=writeback,discard=unmap,id=hd0,if=none \
    -device scsi-hd,drive=hd0 \
    -drive file=/var/tmp/.guestfs-0/appliance.d/root,snapshot=on,id=appliance,cache=unsafe,if=none,format=raw \
    -device scsi-hd,drive=appliance \
    -device virtio-serial-pci \
    -serial stdio \
    -device sga \
    -chardev socket,path=/tmp/libguestfs9bDbSw/guestfsd.sock,id=channel0 \
    -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 \
    -netdev user,id=usernet,net=169.254.0.0/16 \
    -device virtio-net-pci,netdev=usernet \
    -append "panic=1 console=ttyS0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=/dev/sdb selinux=0 guestfs_verbose=1 guestfs_network=1 TERM=xterm"
libguestfs: responding to serial console Device Status Report
\x1b[1;256r\x1b[256;256H\x1b[6n
Google, Inc.
Serial Graphics Adapter 11/20/19
SGABIOS $Id$ (mockbuild@) Wed Nov 20 15:03:34 UTC 2019
Term: 80x24
4 0
SeaBIOS (version 1.13.0-2.module+el8.2.1+7284+aa32a2c4)


iPXE (http://ipxe.org) 00:05.0 C100 PCI2.10 PnP PMM+7FF907A0+7FED07A0 C100
                                                                               


Booting from ROM...
\x1b[2Jlibguestfs: child_cleanup: 0x56348dff4870: child process died
libguestfs: sending SIGTERM to process 25488
libguestfs: qemu maxrss 259236K
libguestfs: trace: launch = -1 (error)
virt-customize: error: libguestfs error: guestfs_launch failed, see earlier 
error messages
libguestfs: trace: close
libguestfs: closing guestfs handle 0x56348dff4870 (state 0)
libguestfs: command: run: rm
libguestfs: command: run: \ -rf /tmp/libguestfsXXkMH4
libguestfs: command: run: rm
libguestfs: command: run: \ -rf /tmp/libguestfs9bDbSw

Hello Minxi,

* Thank you for sharing the output. Going through the logs this issue does not seem related to the realtime kernel. I'm able to reproduce it with a non-realtime kernel too.
 
# rpm -q kernel
kernel-4.18.0-193.el8.x86_64

* It is related to the console unix socket, which is getting deleted for some reason. It causes the guest process to terminate, which leads to the error

    libguestfs: trace: launch = -1 (error)
    virt-customize: error: libguestfs error: guestfs_launch failed

* libvirtd(8) logs show

    Domain id=2 name='guestfs-gf0paodqqinxr0p7' uuid=7505f366-6029-406f-9d53-d398596a4ea2 is tainted: custom-argv
    Domain id=2 name='guestfs-gf0paodqqinxr0p7' uuid=7505f366-6029-406f-9d53-d398596a4ea2 is tainted: host-cpu
   internal error: End of file from qemu monitor

   info : libvirt version: 6.0.0, package: 25.6.module+el8.2.1+12457+868e9540 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2021-08-31-09:13:22, )
   info : hostname: dell-per750-03.rhts.eng.pek2.redhat.com
   warning : virSecurityDACTransactionRun:292 : Ignoring failed restore attempt on /tmp/libguestfs0lZYeW/overlay2.qcow2
   warning : virSecurityDACTransactionRun:292 : Ignoring failed restore attempt on /var/lib/libvirt/images/f34vm.qcow2
   warning : virSecurityDACTransactionRun:292 : Ignoring failed restore attempt on /tmp/libguestfs0lZYeW/overlay1.qcow2
 
   internal error: child reported (status=125): unable to stat: /tmp/libguestfs0dYTfY/console.sock: No such file or directory
   Unable to run security manager transaction


* I set-up the [Advanced-virt] repository on another machine and run the test, but all seems to work fine there.
  /me wondering what's going on...?

@rjones: ...wdyt?

I don't think the console socket is being deleted.  What happens
if we don't detect the kernel and userspace inside the appliance
coming up within (by default) 600 seconds, then libguestfs kills
qemu and deletes all the temporary files.  The libvirtd messages
are consistent with this, and it's normal behaviour (on this error path).

The problem is that the kernel is not starting, or maybe it is
starting but hanging/crashing very early before it is able to
print any messages.

These sorts of bugs are very hard to diagnose in my experience.
They usually involve me recompiling the kernel and inserting
more printk's very early on to try to find out exactly where it
is hanging.

If I was going to guess I'd say that the RT kernel isn't compatible
with TCG + -cpu max.

Do we have a test system I can log into (eg in Beaker) where
this problem is reproducible?  I'll need to compile and install
some stuff, so a throwaway system there would be ideal.

Hello Rjones,

> I don't think the console socket is being deleted.
> ...then libguestfs kills qemu and deletes all the temporary files.
>

* Yes, running strace(1) against libguestfs-test-tool(1) confirms that socket files get deleted after qemu terminates.
===
...
accept4(7, NULL, NULL, SOCK_CLOEXEC|SOCK_NONBLOCK) = 8   <= console_socket
close(7)
...
write(2, "libguestfs: error: error reading"..., 95libguestfs: error: error reading console messages from the appliance: Connection reset by peer) = 95
...
unlink("/tmp/libguestfs1lN6WV/guestfsd.sock") = 0
unlink("/tmp/libguestfs1lN6WV/console.sock") = 0
...
====

* I compiled libguestfs-1.40.2 from
  -> https://download.libguestfs.org/1.40-stable/

And going through the trace, it seems to follow following path

guestfs_launch
 -> guestfs_impl_launch
  -> launch_libvirt
   -> guestfs_int_recv_from_daemon
    -> recv_from_daemon
     -> read_data
      -> handle_log_message
       -> perrorf (g, _("error reading console messages from the appliance")): Connection reset by peer

By this stage, Qemu process is already gone.


* Guest boot-up seems to hang after printing

  Serial Graphics Adapter 11/20/19                                                                                                                                                               
  SGABIOS $Id$ (mockbuild@) Wed Nov 20 15:03:34 UTC 2019
  Booting from ROM...
  ...

  Likely guest kernel boot has not initiated yet.

* I wonder why is it only happening on the this one machine and not others.
  I tried to replicate the same 'Advanced-virt' qemu-kvm/libvirt/libguestfs versions
  on another machines, issue does not show up there.

Thank you.

@mhou: would it be possible to share the machine with Rjones?

Thank you.

I'm afraid this has just missed the non-blocker deadline for 8.6, so
it's more likely to go into 8.7.

Upstream fix:
https://github.com/libguestfs/supermin/commit/c44d685e662e001f5fe70f0a98d0964cb561e1ec

(In reply to Prasad J Pandit from comment #25)
> * I wonder why is it only happening on the this one machine and not others.
>   I tried to replicate the same 'Advanced-virt' qemu-kvm/libvirt/libguestfs
> versions
>   on another machines, issue does not show up there.

It's because the kernel-rt-debug kernel is installed on this machine.
Libguestfs chooses this kernel to boot from, and it doesn't work.  A
simple reproducer is below.

My fix was to make libguestfs/supermin pick a non-debug kernel instead.

----- reproducer -----

exec /usr/libexec/qemu-kvm \
    -no-user-config \
    -enable-fips \
    -nodefaults \
    -display none \
    -machine accel=kvm:tcg \
    -cpu host \
    -m 2048 \
    -no-reboot \
    -rtc driftfix=slew \
    -no-hpet \
    -global kvm-pit.lost_tick_policy=discard \
    -kernel /boot/vmlinuz-4.18.0-193.70.1.rt13.120.el8_2.x86_64+debug \
    -initrd /var/tmp/.guestfs-0/appliance.d/initrd \
    -object rng-random,filename=/dev/urandom,id=rng0 \
    -device virtio-rng-pci,rng=rng0 \
    -device virtio-scsi-pci,id=scsi \
    -drive file=/var/tmp/.guestfs-0/appliance.d/root,snapshot=on,id=appliance,cache=unsafe,if=none,format=raw \
    -device scsi-hd,drive=appliance \
    -device virtio-serial-pci \
    -serial stdio \
    -device sga \
    -append "console=ttyS0 printk.time=1 root=/dev/sda selinux=0 guestfs_verbose=1 TERM=xterm-256color ignore_loglevel"

-------

(In reply to mhou from comment #31)
> Hello Richard
> 
> Do you still need this machine to troubleshoot more about the bug? If yes, I
> will keep reserved for you, otherwise, I need to take this machine to do
> more tests.

I've finished with it now, thanks.

Nice debugging work, Prasad and Rich!

Quick question: can you speculate if kernel-debug RT could have a booting issue that's unrelated to Virt/TCG and would happen on bare-metal as well? If yes, I might get the RT kernel team attention to this issue.

That machine is running 4.18.0-193.el8.x86_64 (non-debug) on the
host, and I didn't want to reboot it to see if the debug kernel would
have booted.  The only thing I tested was the debug kernel as a guest
under KVM (reproducer in comment 32).  I don't know if the RT kernels
at all, or the debug kernels in particular, are meant to work under KVM.

I'd like to say that it's difficult to reproduce this issue on other machines even though the debug RT kernel packages are installed. I tried but failed to reproduce it(using debug RT kernel, see comment 9). Anyway, the fix (supermin ignores the debug kernel) is ok.

(In reply to YongkuiGuo from comment #35)
> I'd like to say that it's difficult to reproduce this issue on other
> machines even though the debug RT kernel packages are installed. I tried but
> failed to reproduce it(using debug RT kernel, see comment 9). 

I think to reproduce it, the kernel-rt-debug kernel must be a higher
version than any other kernel/kernel-rt installed on the system.
supermin will choose the higher version number first, and only if
that is a +debug kernel will it fail.

If kernel-rt-debug is the highest version and it doesn't fail, then
send me the output of

libguestfs-test-tool
ls -l /lib/modules/*/vmlinuz

(In reply to Richard W.M. Jones from comment #36)
> I think to reproduce it, the kernel-rt-debug kernel must be a higher
> version than any other kernel/kernel-rt installed on the system.
> supermin will choose the higher version number first, and only if
> that is a +debug kernel will it fail.
> 
> If kernel-rt-debug is the highest version and it doesn't fail, then
> send me the output of
> 
> libguestfs-test-tool
> ls -l /lib/modules/*/vmlinuz

Still fails to reproduce on rhel8.6 host

# uname -r
4.18.0-367.el8.x86_64

# rpm -q kernel kernel-rt
kernel-4.18.0-367.el8.x86_64
kernel-rt-4.18.0-367.rt7.152.el8.x86_64

# rpm -q kernel-rt-debug
kernel-rt-debug-4.18.0-369.rt7.154.el8.x86_64

# ls -l /lib/modules/*/vmlinuz
-rwxr-xr-x. 1 root root 10460352 Feb 10 15:17 /lib/modules/4.18.0-367.el8.x86_64/vmlinuz
-rwxr-xr-x. 1 root root 10337472 Feb 14 10:29 /lib/modules/4.18.0-367.rt7.152.el8.x86_64/vmlinuz
-rwxr-xr-x. 1 root root 11365568 Feb 23 15:53 /lib/modules/4.18.0-369.rt7.154.el8.x86_64+debug/vmlinuz

# libguestfs-test-tool
     ************************************************************
     *                    IMPORTANT NOTICE
     *
     * When reporting bugs, include the COMPLETE, UNEDITED
     * output below in your bug report.
     *
     ************************************************************
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
XDG_RUNTIME_DIR=/run/user/0
SELinux: Enforcing
guestfs_get_append: (null)
guestfs_get_autosync: 1
guestfs_get_backend: libvirt
guestfs_get_backend_settings: []
guestfs_get_cachedir: /var/tmp
guestfs_get_hv: /usr/libexec/qemu-kvm
guestfs_get_memsize: 1280
guestfs_get_network: 0
guestfs_get_path: /usr/lib64/guestfs
guestfs_get_pgroup: 0
guestfs_get_program: libguestfs-test-tool
guestfs_get_recovery_proc: 1
guestfs_get_smp: 1
guestfs_get_sockdir: /tmp
guestfs_get_tmpdir: /tmp
guestfs_get_trace: 0
guestfs_get_verbose: 1
host_cpu: x86_64
Launching appliance, timeout set to 600 seconds.
libguestfs: launch: program=libguestfs-test-tool
libguestfs: launch: version=1.44.0rhel=8,release=5.module+el8.6.0+13732+b2b9b31d,libvirt
libguestfs: launch: backend registered: unix
libguestfs: launch: backend registered: uml
libguestfs: launch: backend registered: libvirt
libguestfs: launch: backend registered: direct
libguestfs: launch: backend=libvirt
libguestfs: launch: tmpdir=/tmp/libguestfsmQjQMw
libguestfs: launch: umask=0022
libguestfs: launch: euid=0
libguestfs: libvirt version = 8000000 (8.0.0)
libguestfs: guest random name = guestfs-51iu167zji5srfc6
libguestfs: connect to libvirt
libguestfs: opening libvirt handle: URI = qemu:///system, auth = default+wrapper, flags = 0
libguestfs: successfully opened libvirt handle: conn = 0x55ad08784030
libguestfs: qemu version (reported by libvirt) = 6002000 (6.2.0)
libguestfs: get libvirt capabilities
libguestfs: parsing capabilities XML
libguestfs: parsing domcapabilities XML
libguestfs: build appliance
libguestfs: begin building supermin appliance
libguestfs: run supermin
libguestfs: command: run: /usr/bin/supermin
libguestfs: command: run: \ --build
libguestfs: command: run: \ --verbose
libguestfs: command: run: \ --if-newer
libguestfs: command: run: \ --lock /var/tmp/.guestfs-0/lock
libguestfs: command: run: \ --copy-kernel
libguestfs: command: run: \ -f ext2
libguestfs: command: run: \ --host-cpu x86_64
libguestfs: command: run: \ /usr/lib64/guestfs/supermin.d
libguestfs: command: run: \ -o /var/tmp/.guestfs-0/appliance.d
supermin: version: 5.2.1
supermin: rpm: detected RPM version 4.14
supermin: rpm: detected RPM architecture x86_64
supermin: package handler: fedora/rpm
supermin: acquiring lock on /var/tmp/.guestfs-0/lock
supermin: build: /usr/lib64/guestfs/supermin.d
supermin: reading the supermin appliance
supermin: build: visiting /usr/lib64/guestfs/supermin.d/base.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/daemon.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/excludefiles type uncompressed excludefiles
supermin: build: visiting /usr/lib64/guestfs/supermin.d/hostfiles type uncompressed hostfiles
supermin: build: visiting /usr/lib64/guestfs/supermin.d/init.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/packages type uncompressed packages
supermin: build: visiting /usr/lib64/guestfs/supermin.d/udev-rules.tar.gz type gzip base image (tar)
supermin: mapping package names to installed packages
supermin: resolving full list of package dependencies
supermin: build: 175 packages, including dependencies
supermin: build: 33809 files
supermin: build: 6679 files, after matching excludefiles
supermin: build: 6693 files, after adding hostfiles
supermin: build: 6680 files, after removing unreadable files
supermin: build: 6715 files, after munging
supermin: kernel: looking for kernel using environment variables ...
supermin: kernel: looking for kernels in /lib/modules/*/vmlinuz ...
supermin: kernel: picked vmlinuz /lib/modules/4.18.0-369.rt7.154.el8.x86_64+debug/vmlinuz
supermin: kernel: kernel_version 4.18.0-369.rt7.154.el8.x86_64+debug
supermin: kernel: modpath /lib/modules/4.18.0-369.rt7.154.el8.x86_64+debug
supermin: ext2: creating empty ext2 filesystem '/var/tmp/.guestfs-0/appliance.d.jkjf6b5u/root'
supermin: ext2: populating from base image
supermin: ext2: copying files from host filesystem
supermin: ext2: copying kernel modules
supermin: ext2: creating minimal initrd '/var/tmp/.guestfs-0/appliance.d.jkjf6b5u/initrd'
supermin: ext2: wrote 36 modules to minimal initrd
supermin: renaming /var/tmp/.guestfs-0/appliance.d.jkjf6b5u to /var/tmp/.guestfs-0/appliance.d
libguestfs: finished building supermin appliance
libguestfs: command: run: qemu-img
libguestfs: command: run: \ create
libguestfs: command: run: \ -f qcow2
libguestfs: command: run: \ -o backing_file=/var/tmp/.guestfs-0/appliance.d/root,backing_fmt=raw
libguestfs: command: run: \ /tmp/libguestfsmQjQMw/overlay2.qcow2
Formatting '/tmp/libguestfsmQjQMw/overlay2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=4294967296 backing_file=/var/tmp/.guestfs-0/appliance.d/root backing_fmt=raw lazy_refcounts=off refcount_bits=16
libguestfs: create libvirt XML
libguestfs: command: run: file
libguestfs: command: run: \ -- /var/tmp/.guestfs-0/appliance.d/root
libguestfs: libvirt XML:\n<?xml version="1.0"?>\n<domain type="kvm" xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0">\n  <name>guestfs-51iu167zji5srfc6</name>\n  <memory unit="MiB">1280</memory>\n  <currentMemory unit="MiB">1280</currentMemory>\n  <cpu mode="host-passthrough">\n    <model fallback="allow"/>\n  </cpu>\n  <vcpu>1</vcpu>\n  <clock offset="utc">\n    <timer name="rtc" tickpolicy="catchup"/>\n    <timer name="pit" tickpolicy="delay"/>\n    <timer name="hpet" present="no"/>\n  </clock>\n  <os>\n    <type>hvm</type>\n    <kernel>/var/tmp/.guestfs-0/appliance.d/kernel</kernel>\n    <initrd>/var/tmp/.guestfs-0/appliance.d/initrd</initrd>\n    <cmdline>panic=1 console=ttyS0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=UUID=743e66cb-0a0b-4232-ab17-f37fcc5ffc1c selinux=0 guestfs_verbose=1 TERM=xterm-256color</cmdline>\n    <bios useserial="yes"/>\n  </os>\n  <on_reboot>destroy</on_reboot>\n  <devices>\n    <rng model="virtio">\n      <backend model="random">/dev/urandom</backend>\n    </rng>\n    <controller type="scsi" index="0" model="virtio-scsi"/>\n    <disk device="disk" type="file">\n      <source file="/tmp/libguestfsmQjQMw/scratch1.img"/>\n      <target dev="sda" bus="scsi"/>\n      <driver name="qemu" type="raw" cache="unsafe"/>\n      <address type="drive" controller="0" bus="0" target="0" unit="0"/>\n    </disk>\n    <disk type="file" device="disk">\n      <source file="/tmp/libguestfsmQjQMw/overlay2.qcow2"/>\n      <target dev="sdb" bus="scsi"/>\n      <driver name="qemu" type="qcow2" cache="unsafe"/>\n      <address type="drive" controller="0" bus="0" target="1" unit="0"/>\n    </disk>\n    <serial type="unix">\n      <source mode="connect" path="/tmp/libguestfsBMgK5u/console.sock"/>\n      <target port="0"/>\n    </serial>\n    <channel type="unix">\n      <source mode="connect" path="/tmp/libguestfsBMgK5u/guestfsd.sock"/>\n      <target type="virtio" name="org.libguestfs.channel.0"/>\n    </channel>\n    <controller type="usb" model="none"/>\n    <memballoon model="none"/>\n  </devices>\n  <qemu:commandline>\n    <qemu:env name="TMPDIR" value="/var/tmp"/>\n  </qemu:commandline>\n</domain>\n
libguestfs: command: run: ls
libguestfs: command: run: \ -a
libguestfs: command: run: \ -l
libguestfs: command: run: \ -R
libguestfs: command: run: \ -Z /var/tmp/.guestfs-0
libguestfs: /var/tmp/.guestfs-0:
libguestfs: total 0
libguestfs: drwxr-xr-x. 3 root root unconfined_u:object_r:user_tmp_t:s0  37 Mar  4 04:03 .
libguestfs: drwxrwxrwt. 6 root root system_u:object_r:tmp_t:s0          200 Mar  4 04:03 ..
libguestfs: drwxr-xr-x. 2 root root unconfined_u:object_r:user_tmp_t:s0  46 Mar  4 04:03 appliance.d
libguestfs: -rw-r--r--. 1 root root unconfined_u:object_r:user_tmp_t:s0   0 Mar  4 04:03 lock
libguestfs: 
libguestfs: /var/tmp/.guestfs-0/appliance.d:
libguestfs: total 274620
libguestfs: drwxr-xr-x. 2 root root unconfined_u:object_r:user_tmp_t:s0         46 Mar  4 04:03 .
libguestfs: drwxr-xr-x. 3 root root unconfined_u:object_r:user_tmp_t:s0         37 Mar  4 04:03 ..
libguestfs: -rw-r--r--. 1 root root unconfined_u:object_r:user_tmp_t:s0    5726208 Mar  4 04:03 initrd
libguestfs: -rwxr-xr-x. 1 root root unconfined_u:object_r:user_tmp_t:s0   11365568 Mar  4 04:03 kernel
libguestfs: -rw-r--r--. 1 root root unconfined_u:object_r:user_tmp_t:s0 4294967296 Mar  4 04:03 root
libguestfs: command: run: ls
libguestfs: command: run: \ -a
libguestfs: command: run: \ -l
libguestfs: command: run: \ -Z /tmp/libguestfsBMgK5u
libguestfs: total 4
libguestfs: drwxr-xr-x.  2 root root unconfined_u:object_r:user_tmp_t:s0   47 Mar  4 04:03 .
libguestfs: drwxrwxrwt. 11 root root system_u:object_r:tmp_t:s0          4096 Mar  4 04:03 ..
libguestfs: srw-rw----.  1 root qemu unconfined_u:object_r:user_tmp_t:s0    0 Mar  4 04:03 console.sock
libguestfs: srw-rw----.  1 root qemu unconfined_u:object_r:user_tmp_t:s0    0 Mar  4 04:03 guestfsd.sock
libguestfs: launch libvirt guest
\x1bc\x1b[?7l\x1b[2J\x1b[0mSeaBIOS (version 1.15.0-1.module+el8.6.0+13725+61ae1949)
Machine UUID 985ebafc-f1ec-469b-a530-fcee2db45753
Booting from ROM..\x1bc\x1b[?7l\x1b[2J[    0.000000] Linux version 4.18.0-369.rt7.154.el8.x86_64+debug (mockbuild.eng.bos.redhat.com) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-10) (GCC)) #1 SMP PREEMPT_RT Wed Feb 23 15:25:34 EST 2022
[    0.000000] Command line: panic=1 console=ttyS0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=UUID=743e66cb-0a0b-4232-ab17-f37fcc5ffc1c selinux=0 guestfs_verbose=1 TERM=xterm-256color
[    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
[    0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
[    0.000000] signal: max sigframe size: 1776
[    0.000000] BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000004fffcfff] usable
[    0.000000] BIOS-e820: [mem 0x000000004fffd000-0x000000004fffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 2.8 present.
[    0.000000] DMI: Red Hat KVM, BIOS 1.15.0-1.module+el8.6.0+13725+61ae1949 04/01/2014
[    0.000000] Hypervisor detected: KVM
...
===== TEST FINISHED OK =====

Sorry for uploading the same log of libguestfs-test-tool twice.

Interestingly we picked the debug kernel and booted it:

> supermin: kernel: picked vmlinuz /lib/modules/4.18.0-369.rt7.154.el8.x86_64+debug/vmlinuz
> supermin: kernel: kernel_version 4.18.0-369.rt7.154.el8.x86_64+debug
> supermin: kernel: modpath /lib/modules/4.18.0-369.rt7.154.el8.x86_64+debug

It seems as if the other debug kernel which failed (4.18.0-193.71.1.rt13.121.el8_2)
probably has a bug that stops it from booting on qemu (or qemu has a bug
that breaks emulation).

Anyway, choosing non-debug kernels is better because the debug ones run
slower and use more RAM, assuming they're anything like the Fedora debug
kernels.  So the change to supermin is still valid.

(In reply to Richard W.M. Jones from comment #42)
> Anyway, choosing non-debug kernels is better because the debug ones run
> slower and use more RAM, assuming they're anything like the Fedora debug
> kernels.  So the change to supermin is still valid.

Yes. Ignoring the debug RT kernel is the most direct way to avoid this kind of issue. BTW, could you help change the 'Product' from RHEL AV to RHEL?

Change component to RHEL since RHEL AV & RHEL have converged in >= 8.6

I don't think this is a blocker bug, so probably we will not be able
to fix it in RHEL 8.6 and it'll have to wait til 8.7.

I had to remove the TestOnly keyword because:

remote: *** Resolves:
remote: ***   Unapproved:
remote: ***     rhbz#2051332 (qe_test_coverage?, qa_ack+, internal_target_release=8.7.0, devel_ack+, mirror+, release+, stale-, TestOnly, Triaged, ZStream)
remote: ***   This is unapproved because at least one of the BZs contains the TestOnly keyword. TestOnly keyword must be removed in order to pass approval.

On rhel8.7 host:

1.
# rpm -q kernel kernel-rt kernel-rt-debug
kernel-4.18.0-369.el8.x86_64
kernel-rt-4.18.0-369.rt7.154.el8.x86_64
kernel-rt-debug-4.18.0-369.rt7.154.el8.x86_64


2.Tested with previous supermin package: supermin-5.2.1-1.module+el8.6.0+12861+13975d62.x86_64

# libguestfs-test-tool
     ************************************************************
     *                    IMPORTANT NOTICE
     *
     * When reporting bugs, include the COMPLETE, UNEDITED
     * output below in your bug report.
     *
     ************************************************************
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
XDG_RUNTIME_DIR=/run/user/0
SELinux: Enforcing
guestfs_get_append: (null)
guestfs_get_autosync: 1
guestfs_get_backend: libvirt
guestfs_get_backend_settings: []
guestfs_get_cachedir: /var/tmp
guestfs_get_hv: /usr/libexec/qemu-kvm
guestfs_get_memsize: 1280
guestfs_get_network: 0
guestfs_get_path: /usr/lib64/guestfs
guestfs_get_pgroup: 0
guestfs_get_program: libguestfs-test-tool
guestfs_get_recovery_proc: 1
guestfs_get_smp: 1
guestfs_get_sockdir: /tmp
guestfs_get_tmpdir: /tmp
guestfs_get_trace: 0
guestfs_get_verbose: 1
host_cpu: x86_64
Launching appliance, timeout set to 600 seconds.
libguestfs: launch: program=libguestfs-test-tool
libguestfs: launch: version=1.44.0rhel=8,release=5.module+el8.6.0+13732+b2b9b31d,libvirt
libguestfs: launch: backend registered: unix
libguestfs: launch: backend registered: uml
libguestfs: launch: backend registered: libvirt
libguestfs: launch: backend registered: direct
libguestfs: launch: backend=libvirt
libguestfs: launch: tmpdir=/tmp/libguestfsHvvTdy
libguestfs: launch: umask=0022
libguestfs: launch: euid=0
libguestfs: libvirt version = 8000000 (8.0.0)
libguestfs: guest random name = guestfs-yrkb1h2bigcrsmtg
libguestfs: connect to libvirt
libguestfs: opening libvirt handle: URI = qemu:///system, auth = default+wrapper, flags = 0
libguestfs: successfully opened libvirt handle: conn = 0x55ae8ce7e030
libguestfs: qemu version (reported by libvirt) = 6002000 (6.2.0)
libguestfs: get libvirt capabilities
libguestfs: parsing capabilities XML
libguestfs: parsing domcapabilities XML
libguestfs: build appliance
libguestfs: begin building supermin appliance
libguestfs: run supermin
libguestfs: command: run: /usr/bin/supermin
libguestfs: command: run: \ --build
libguestfs: command: run: \ --verbose
libguestfs: command: run: \ --if-newer
libguestfs: command: run: \ --lock /var/tmp/.guestfs-0/lock
libguestfs: command: run: \ --copy-kernel
libguestfs: command: run: \ -f ext2
libguestfs: command: run: \ --host-cpu x86_64
libguestfs: command: run: \ /usr/lib64/guestfs/supermin.d
libguestfs: command: run: \ -o /var/tmp/.guestfs-0/appliance.d
supermin: version: 5.2.1
supermin: rpm: detected RPM version 4.14
supermin: rpm: detected RPM architecture x86_64
supermin: package handler: fedora/rpm
supermin: acquiring lock on /var/tmp/.guestfs-0/lock
supermin: build: /usr/lib64/guestfs/supermin.d
supermin: reading the supermin appliance
supermin: build: visiting /usr/lib64/guestfs/supermin.d/base.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/daemon.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/excludefiles type uncompressed excludefiles
supermin: build: visiting /usr/lib64/guestfs/supermin.d/hostfiles type uncompressed hostfiles
supermin: build: visiting /usr/lib64/guestfs/supermin.d/init.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/packages type uncompressed packages
supermin: build: visiting /usr/lib64/guestfs/supermin.d/udev-rules.tar.gz type gzip base image (tar)
supermin: mapping package names to installed packages
supermin: resolving full list of package dependencies
supermin: build: 175 packages, including dependencies
supermin: build: 33813 files
supermin: build: 6683 files, after matching excludefiles
supermin: build: 6697 files, after adding hostfiles
supermin: build: 6685 files, after removing unreadable files
supermin: build: 6720 files, after munging
supermin: kernel: looking for kernel using environment variables ...
supermin: kernel: looking for kernels in /lib/modules/*/vmlinuz ...
supermin: kernel: picked vmlinuz /lib/modules/4.18.0-369.rt7.154.el8.x86_64+debug/vmlinuz
supermin: kernel: kernel_version 4.18.0-369.rt7.154.el8.x86_64+debug
supermin: kernel: modpath /lib/modules/4.18.0-369.rt7.154.el8.x86_64+debug
supermin: ext2: creating empty ext2 filesystem '/var/tmp/.guestfs-0/appliance.d.1wijz8eh/root'
supermin: ext2: populating from base image
supermin: ext2: copying files from host filesystem
supermin: ext2: copying kernel modules
supermin: ext2: creating minimal initrd '/var/tmp/.guestfs-0/appliance.d.1wijz8eh/initrd'
supermin: ext2: wrote 36 modules to minimal initrd
supermin: renaming /var/tmp/.guestfs-0/appliance.d.1wijz8eh to /var/tmp/.guestfs-0/appliance.d
libguestfs: finished building supermin appliance
libguestfs: command: run: qemu-img
libguestfs: command: run: \ create
libguestfs: command: run: \ -f qcow2
libguestfs: command: run: \ -o backing_file=/var/tmp/.guestfs-0/appliance.d/root,backing_fmt=raw
libguestfs: command: run: \ /tmp/libguestfsHvvTdy/overlay2.qcow2
Formatting '/tmp/libguestfsHvvTdy/overlay2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=4294967296 backing_file=/var/tmp/.guestfs-0/appliance.d/root backing_fmt=raw lazy_refcounts=off refcount_bits=16
libguestfs: create libvirt XML
libguestfs: command: run: file
libguestfs: command: run: \ -- /var/tmp/.guestfs-0/appliance.d/root
libguestfs: libvirt XML:\n<?xml version="1.0"?>\n<domain type="kvm" xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0">\n  <name>guestfs-yrkb1h2bigcrsmtg</name>\n  <memory unit="MiB">1280</memory>\n  <currentMemory unit="MiB">1280</currentMemory>\n  <cpu mode="host-passthrough">\n    <model fallback="allow"/>\n  </cpu>\n  <vcpu>1</vcpu>\n  <clock offset="utc">\n    <timer name="rtc" tickpolicy="catchup"/>\n    <timer name="pit" tickpolicy="delay"/>\n    <timer name="hpet" present="no"/>\n  </clock>\n  <os>\n    <type>hvm</type>\n    <kernel>/var/tmp/.guestfs-0/appliance.d/kernel</kernel>\n    <initrd>/var/tmp/.guestfs-0/appliance.d/initrd</initrd>\n    <cmdline>panic=1 console=ttyS0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=UUID=8495c444-92af-4290-ac6c-e416be6622cd selinux=0 guestfs_verbose=1 TERM=xterm-256color</cmdline>\n    <bios useserial="yes"/>\n  </os>\n  <on_reboot>destroy</on_reboot>\n  <devices>\n    <rng model="virtio">\n      <backend model="random">/dev/urandom</backend>\n    </rng>\n    <controller type="scsi" index="0" model="virtio-scsi"/>\n    <disk device="disk" type="file">\n      <source file="/tmp/libguestfsHvvTdy/scratch1.img"/>\n      <target dev="sda" bus="scsi"/>\n      <driver name="qemu" type="raw" cache="unsafe"/>\n      <address type="drive" controller="0" bus="0" target="0" unit="0"/>\n    </disk>\n    <disk type="file" device="disk">\n      <source file="/tmp/libguestfsHvvTdy/overlay2.qcow2"/>\n      <target dev="sdb" bus="scsi"/>\n      <driver name="qemu" type="qcow2" cache="unsafe"/>\n      <address type="drive" controller="0" bus="0" target="1" unit="0"/>\n    </disk>\n    <serial type="unix">\n      <source mode="connect" path="/tmp/libguestfsGQ1qqz/console.sock"/>\n      <target port="0"/>\n    </serial>\n    <channel type="unix">\n      <source mode="connect" path="/tmp/libguestfsGQ1qqz/guestfsd.sock"/>\n      <target type="virtio" name="org.libguestfs.channel.0"/>\n    </channel>\n    <controller type="usb" model="none"/>\n    <memballoon model="none"/>\n  </devices>\n  <qemu:commandline>\n    <qemu:env name="TMPDIR" value="/var/tmp"/>\n  </qemu:commandline>\n</domain>\n
libguestfs: command: run: ls
libguestfs: command: run: \ -a
libguestfs: command: run: \ -l
libguestfs: command: run: \ -R
libguestfs: command: run: \ -Z /var/tmp/.guestfs-0
libguestfs: /var/tmp/.guestfs-0:
libguestfs: total 0
libguestfs: drwxr-xr-x. 3 root root unconfined_u:object_r:user_tmp_t:s0  37 Mar  8 04:09 .
libguestfs: drwxrwxrwt. 5 root root system_u:object_r:tmp_t:s0          121 Mar  8 04:09 ..
libguestfs: drwxr-xr-x. 2 root root unconfined_u:object_r:user_tmp_t:s0  46 Mar  8 04:09 appliance.d
libguestfs: -rw-r--r--. 1 root root unconfined_u:object_r:user_tmp_t:s0   0 Mar  8 04:09 lock
libguestfs: 
libguestfs: /var/tmp/.guestfs-0/appliance.d:
libguestfs: total 274636
libguestfs: drwxr-xr-x. 2 root root unconfined_u:object_r:user_tmp_t:s0         46 Mar  8 04:09 .
libguestfs: drwxr-xr-x. 3 root root unconfined_u:object_r:user_tmp_t:s0         37 Mar  8 04:09 ..
libguestfs: -rw-r--r--. 1 root root unconfined_u:object_r:user_tmp_t:s0    5726208 Mar  8 04:09 initrd
libguestfs: -rwxr-xr-x. 1 root root unconfined_u:object_r:user_tmp_t:s0   11365568 Mar  8 04:09 kernel
libguestfs: -rw-r--r--. 1 root root unconfined_u:object_r:user_tmp_t:s0 4294967296 Mar  8 04:09 root
libguestfs: command: run: ls
libguestfs: command: run: \ -a
libguestfs: command: run: \ -l
libguestfs: command: run: \ -Z /tmp/libguestfsGQ1qqz
libguestfs: total 4
libguestfs: drwxr-xr-x.  2 root root unconfined_u:object_r:user_tmp_t:s0   47 Mar  8 04:09 .
libguestfs: drwxrwxrwt. 11 root root system_u:object_r:tmp_t:s0          4096 Mar  8 04:09 ..
libguestfs: srw-rw----.  1 root qemu unconfined_u:object_r:user_tmp_t:s0    0 Mar  8 04:09 console.sock
libguestfs: srw-rw----.  1 root qemu unconfined_u:object_r:user_tmp_t:s0    0 Mar  8 04:09 guestfsd.sock
libguestfs: launch libvirt guest
\x1bc\x1b[?7l\x1b[2J\x1b[0mSeaBIOS (version 1.15.0-1.module+el8.6.0+13725+61ae1949)
Machine UUID 8581bc39-2091-40d3-8495-56468a2c4720
Booting from ROM..\x1bc\x1b[?7l\x1b[2J[    0.000000] Linux version 4.18.0-369.rt7.154.el8.x86_64+debug (mockbuild.eng.bos.redhat.com) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-10) (GCC)) #1 SMP PREEMPT_RT Wed Feb 23 15:25:34 EST 2022
[    0.000000] Command line: panic=1 console=ttyS0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=UUID=8495c444-92af-4290-ac6c-e416be6622cd selinux=0 guestfs_verbose=1 TERM=xterm-256color
...
===== TEST FINISHED OK =====

The debug RT kernel was picked by supermin.


3.Tested with the fixed supermin package: supermin-5.2.1-2.module+el8.7.0+14396+56bc2570.x86_64

# libguestfs-test-tool
     ************************************************************
     *                    IMPORTANT NOTICE
     *
     * When reporting bugs, include the COMPLETE, UNEDITED
     * output below in your bug report.
     *
     ************************************************************
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
XDG_RUNTIME_DIR=/run/user/0
SELinux: Enforcing
guestfs_get_append: (null)
guestfs_get_autosync: 1
guestfs_get_backend: libvirt
guestfs_get_backend_settings: []
guestfs_get_cachedir: /var/tmp
guestfs_get_hv: /usr/libexec/qemu-kvm
guestfs_get_memsize: 1280
guestfs_get_network: 0
guestfs_get_path: /usr/lib64/guestfs
guestfs_get_pgroup: 0
guestfs_get_program: libguestfs-test-tool
guestfs_get_recovery_proc: 1
guestfs_get_smp: 1
guestfs_get_sockdir: /tmp
guestfs_get_tmpdir: /tmp
guestfs_get_trace: 0
guestfs_get_verbose: 1
host_cpu: x86_64
Launching appliance, timeout set to 600 seconds.
libguestfs: launch: program=libguestfs-test-tool
libguestfs: launch: version=1.44.0rhel=8,release=5.module+el8.6.0+13732+b2b9b31d,libvirt
libguestfs: launch: backend registered: unix
libguestfs: launch: backend registered: uml
libguestfs: launch: backend registered: libvirt
libguestfs: launch: backend registered: direct
libguestfs: launch: backend=libvirt
libguestfs: launch: tmpdir=/tmp/libguestfs2EG86U
libguestfs: launch: umask=0022
libguestfs: launch: euid=0
libguestfs: libvirt version = 8000000 (8.0.0)
libguestfs: guest random name = guestfs-aezowj4a4x8sv4kc
libguestfs: connect to libvirt
libguestfs: opening libvirt handle: URI = qemu:///system, auth = default+wrapper, flags = 0
libguestfs: successfully opened libvirt handle: conn = 0x5575c3282030
libguestfs: qemu version (reported by libvirt) = 6002000 (6.2.0)
libguestfs: get libvirt capabilities
libguestfs: parsing capabilities XML
libguestfs: parsing domcapabilities XML
libguestfs: build appliance
libguestfs: begin building supermin appliance
libguestfs: run supermin
libguestfs: command: run: /usr/bin/supermin
libguestfs: command: run: \ --build
libguestfs: command: run: \ --verbose
libguestfs: command: run: \ --if-newer
libguestfs: command: run: \ --lock /var/tmp/.guestfs-0/lock
libguestfs: command: run: \ --copy-kernel
libguestfs: command: run: \ -f ext2
libguestfs: command: run: \ --host-cpu x86_64
libguestfs: command: run: \ /usr/lib64/guestfs/supermin.d
libguestfs: command: run: \ -o /var/tmp/.guestfs-0/appliance.d
supermin: version: 5.2.1
supermin: rpm: detected RPM version 4.14
supermin: rpm: detected RPM architecture x86_64
supermin: package handler: fedora/rpm
supermin: acquiring lock on /var/tmp/.guestfs-0/lock
supermin: build: /usr/lib64/guestfs/supermin.d
supermin: reading the supermin appliance
supermin: build: visiting /usr/lib64/guestfs/supermin.d/base.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/daemon.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/excludefiles type uncompressed excludefiles
supermin: build: visiting /usr/lib64/guestfs/supermin.d/hostfiles type uncompressed hostfiles
supermin: build: visiting /usr/lib64/guestfs/supermin.d/init.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/packages type uncompressed packages
supermin: build: visiting /usr/lib64/guestfs/supermin.d/udev-rules.tar.gz type gzip base image (tar)
supermin: mapping package names to installed packages
supermin: resolving full list of package dependencies
supermin: build: 175 packages, including dependencies
supermin: build: 33813 files
supermin: build: 6683 files, after matching excludefiles
supermin: build: 6697 files, after adding hostfiles
supermin: build: 6685 files, after removing unreadable files
supermin: build: 6720 files, after munging
supermin: kernel: looking for kernel using environment variables ...
supermin: kernel: looking for kernels in /lib/modules/*/vmlinuz ...
supermin: kernel: picked vmlinuz /lib/modules/4.18.0-369.rt7.154.el8.x86_64/vmlinuz
supermin: kernel: kernel_version 4.18.0-369.rt7.154.el8.x86_64
supermin: kernel: modpath /lib/modules/4.18.0-369.rt7.154.el8.x86_64
supermin: ext2: creating empty ext2 filesystem '/var/tmp/.guestfs-0/appliance.d.6fmn3krb/root'
supermin: ext2: populating from base image
supermin: ext2: copying files from host filesystem
supermin: ext2: copying kernel modules
supermin: ext2: creating minimal initrd '/var/tmp/.guestfs-0/appliance.d.6fmn3krb/initrd'
supermin: ext2: wrote 36 modules to minimal initrd
supermin: renaming /var/tmp/.guestfs-0/appliance.d.6fmn3krb to /var/tmp/.guestfs-0/appliance.d
libguestfs: finished building supermin appliance
libguestfs: command: run: qemu-img
libguestfs: command: run: \ create
libguestfs: command: run: \ -f qcow2
libguestfs: command: run: \ -o backing_file=/var/tmp/.guestfs-0/appliance.d/root,backing_fmt=raw
libguestfs: command: run: \ /tmp/libguestfs2EG86U/overlay2.qcow2
Formatting '/tmp/libguestfs2EG86U/overlay2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=4294967296 backing_file=/var/tmp/.guestfs-0/appliance.d/root backing_fmt=raw lazy_refcounts=off refcount_bits=16
libguestfs: create libvirt XML
libguestfs: command: run: file
libguestfs: command: run: \ -- /var/tmp/.guestfs-0/appliance.d/root
libguestfs: libvirt XML:\n<?xml version="1.0"?>\n<domain type="kvm" xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0">\n  <name>guestfs-aezowj4a4x8sv4kc</name>\n  <memory unit="MiB">1280</memory>\n  <currentMemory unit="MiB">1280</currentMemory>\n  <cpu mode="host-passthrough">\n    <model fallback="allow"/>\n  </cpu>\n  <vcpu>1</vcpu>\n  <clock offset="utc">\n    <timer name="rtc" tickpolicy="catchup"/>\n    <timer name="pit" tickpolicy="delay"/>\n    <timer name="hpet" present="no"/>\n  </clock>\n  <os>\n    <type>hvm</type>\n    <kernel>/var/tmp/.guestfs-0/appliance.d/kernel</kernel>\n    <initrd>/var/tmp/.guestfs-0/appliance.d/initrd</initrd>\n    <cmdline>panic=1 console=ttyS0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=UUID=3c1e4f94-b6ba-40f9-8f70-e7b0b5e65f4d selinux=0 guestfs_verbose=1 TERM=xterm-256color</cmdline>\n    <bios useserial="yes"/>\n  </os>\n  <on_reboot>destroy</on_reboot>\n  <devices>\n    <rng model="virtio">\n      <backend model="random">/dev/urandom</backend>\n    </rng>\n    <controller type="scsi" index="0" model="virtio-scsi"/>\n    <disk device="disk" type="file">\n      <source file="/tmp/libguestfs2EG86U/scratch1.img"/>\n      <target dev="sda" bus="scsi"/>\n      <driver name="qemu" type="raw" cache="unsafe"/>\n      <address type="drive" controller="0" bus="0" target="0" unit="0"/>\n    </disk>\n    <disk type="file" device="disk">\n      <source file="/tmp/libguestfs2EG86U/overlay2.qcow2"/>\n      <target dev="sdb" bus="scsi"/>\n      <driver name="qemu" type="qcow2" cache="unsafe"/>\n      <address type="drive" controller="0" bus="0" target="1" unit="0"/>\n    </disk>\n    <serial type="unix">\n      <source mode="connect" path="/tmp/libguestfsBKI0cV/console.sock"/>\n      <target port="0"/>\n    </serial>\n    <channel type="unix">\n      <source mode="connect" path="/tmp/libguestfsBKI0cV/guestfsd.sock"/>\n      <target type="virtio" name="org.libguestfs.channel.0"/>\n    </channel>\n    <controller type="usb" model="none"/>\n    <memballoon model="none"/>\n  </devices>\n  <qemu:commandline>\n    <qemu:env name="TMPDIR" value="/var/tmp"/>\n  </qemu:commandline>\n</domain>\n
libguestfs: command: run: ls
libguestfs: command: run: \ -a
libguestfs: command: run: \ -l
libguestfs: command: run: \ -R
libguestfs: command: run: \ -Z /var/tmp/.guestfs-0
libguestfs: /var/tmp/.guestfs-0:
libguestfs: total 0
libguestfs: drwxr-xr-x. 3 root root unconfined_u:object_r:user_tmp_t:s0  37 Mar  8 04:03 .
libguestfs: drwxrwxrwt. 5 root root system_u:object_r:tmp_t:s0          121 Mar  8 04:03 ..
libguestfs: drwxr-xr-x. 2 root root unconfined_u:object_r:user_tmp_t:s0  46 Mar  8 04:03 appliance.d
libguestfs: -rw-r--r--. 1 root root unconfined_u:object_r:user_tmp_t:s0   0 Mar  8 04:02 lock
libguestfs: 
libguestfs: /var/tmp/.guestfs-0/appliance.d:
libguestfs: total 270108
libguestfs: drwxr-xr-x. 2 root root unconfined_u:object_r:user_tmp_t:s0         46 Mar  8 04:03 .
libguestfs: drwxr-xr-x. 3 root root unconfined_u:object_r:user_tmp_t:s0         37 Mar  8 04:03 ..
libguestfs: -rw-r--r--. 1 root root unconfined_u:object_r:user_tmp_t:s0    5346816 Mar  8 04:03 initrd
libguestfs: -rwxr-xr-x. 1 root root unconfined_u:object_r:user_tmp_t:s0   10333376 Mar  8 04:03 kernel
libguestfs: -rw-r--r--. 1 root root unconfined_u:object_r:user_tmp_t:s0 4294967296 Mar  8 04:03 root
libguestfs: command: run: ls
libguestfs: command: run: \ -a
libguestfs: command: run: \ -l
libguestfs: command: run: \ -Z /tmp/libguestfsBKI0cV
libguestfs: total 4
libguestfs: drwxr-xr-x.  2 root root unconfined_u:object_r:user_tmp_t:s0   47 Mar  8 04:03 .
libguestfs: drwxrwxrwt. 11 root root system_u:object_r:tmp_t:s0          4096 Mar  8 04:03 ..
libguestfs: srw-rw----.  1 root qemu unconfined_u:object_r:user_tmp_t:s0    0 Mar  8 04:03 console.sock
libguestfs: srw-rw----.  1 root qemu unconfined_u:object_r:user_tmp_t:s0    0 Mar  8 04:03 guestfsd.sock
libguestfs: launch libvirt guest
\x1bc\x1b[?7l\x1b[2J\x1b[0mSeaBIOS (version 1.15.0-1.module+el8.6.0+13725+61ae1949)
Machine UUID d987f608-c2f0-44be-bcde-d4ada4d71010
Booting from ROM..\x1bc\x1b[?7l\x1b[2J[    0.000000] Linux version 4.18.0-369.rt7.154.el8.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-10) (GCC)) #1 SMP PREEMPT_RT Wed Feb 23 15:53:30 EST 2022
[    0.000000] Command line: panic=1 console=ttyS0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=UUID=3c1e4f94-b6ba-40f9-8f70-e7b0b5e65f4d selinux=0 guestfs_verbose=1 TERM=xterm-256color
...
===== TEST FINISHED OK =====

Supermin ignored the debug RT kernel and use RT kernel instead. The behavior is as expected.

Verified this bug since the test case for this bug has been automated and passed in the latest nightly compose test.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Low: virt:rhel and virt-devel:rhel security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7472