RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2027375 - Can't use libguestfs tools when zfcpdump kernel installed
Summary: Can't use libguestfs tools when zfcpdump kernel installed
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: supermin
Version: 9.0
Hardware: s390x
OS: Unspecified
unspecified
high
Target Milestone: rc
: ---
Assignee: Virtualization Maintenance
QA Contact: YongkuiGuo
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-11-29 13:52 UTC by smitterl
Modified: 2022-05-17 13:32 UTC (History)
6 users (show)

Fixed In Version: supermin-5.2.1-7.el9
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-05-17 13:18:16 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-104158 0 None None None 2021-11-29 13:53:30 UTC
Red Hat Product Errata RHBA-2022:2488 0 None None None 2022-05-17 13:18:20 UTC

Description smitterl 2021-11-29 13:52:16 UTC 
Description of problem:
IIUC, supermin uses the latest kernel from /boot [1]. On a system with kernel-zfcpdump.s390x, it fails to find virtio drivers in there. Therefore, several tools fail with supermin reporting

"supermin: waiting another 1024000000 ns for root UUID to appear
This usually means your kernel doesn't support virtio, or supermin was unable
to load some kernel modules (see module loading messages above)."

although the currently loaded kernel does contain the virtio modules.



Version-Release number of selected component (if applicable):
supermin-5.2.1-4.el9.s390x

How reproducible:
100%


Steps to Reproduce:
1. yum install -y kernel-zfcpdump.s390x
2. yum install -y libguestfs-tools
3. libguestfs-test-tool

Actual results:
Tests fail:
...
supermin: mounting /sys
supermin: waiting another 1024000000 ns for root UUID to appear
This usually means your kernel doesn't support virtio, or supermin was unable
to load some kernel modules (see module loading messages above).
supermin: waiting another 2048000000 ns for root UUID to appear
supermin: waiting another 4096000000 ns for root UUID to appear
supermin: waiting another 8192000000 ns for root UUID to appear
supermin: waiting another 16384000000 ns for root UUID to appear
supermin: waiting another 32768000000 ns for root UUID to appear
supermin: waiting another 65536000000 ns for root UUID to appear
libguestfs: error: appliance closed the connection unexpectedly, see earlier error messages
libguestfs: child_cleanup: 0x2aa1f657ea0: child process died
libguestfs: error: guestfs_launch failed, see earlier error messages


Expected results:
Tests pass.


Additional info:
1. As by kerneldoc[1] it looks like this kernel can be ignored by supermin.
2. kernel-zfcpdump is not installed per default. It was installed via usage of wildcard to get the full set of a specific kernel build for a test round.
3. Setting severity 'high' because once a customer hits the issue they might not easily solve this issue.
4. The issue can be solved by setting the SUPERMIN_KERNEL envvar, e.g.:

# SUPERMIN_KERNEL=/boot/vmlinuz-5.14.0-15.mr128_211112_1256.el9.s390x SUPERMIN_MODULES=/lib/modules/5.14.0-15.mr128_211112_1256.el9.s390x libguestfs-test-tool

[1]https://github.com/libguestfs/supermin/blob/9c2ca5f2803122fdeb405a8daedc5c6d8cf66a19/src/format_ext2_kernel.ml#L116
[2]https://www.kernel.org/doc/html/latest/s390/zfcpdump.html
Comment 1 Richard W.M. Jones 2021-11-29 14:38:38 UTC 
14:06 < cohuck> zfcpdump is a stripped-down kernel that is only used to be able to dump to scsi disks; they should be excluded from normal testing, I guess
Comment 2 Richard W.M. Jones 2021-11-29 14:48:29 UTC 
Upstream fix:
https://github.com/libguestfs/supermin/commit/9fbe476d4df0b01568d3668e6121cae7c779c8c7

Yongkui, could you look at this for QE ACK?  It requires an s390x system
to reproduce the problem unfortunately.  The reproducer is in comment 0.
Comment 3 YongkuiGuo 2021-11-30 04:22:16 UTC 
(In reply to Richard W.M. Jones from comment #2)
> Upstream fix:
> https://github.com/libguestfs/supermin/commit/
> 9fbe476d4df0b01568d3668e6121cae7c779c8c7
> 
> Yongkui, could you look at this for QE ACK?  It requires an s390x system
> to reproduce the problem unfortunately.  The reproducer is in comment 0.

Ack+, I can reproduce this issue on s390x env.
Comment 4 YongkuiGuo 2021-12-01 02:26:44 UTC 
Test with the fixed supermin:

# rpm -q libguestfs supermin
libguestfs-1.46.0-5.el9.s390x
supermin-5.2.1-6.el9.s390x

# rpm -qa | grep kernel
kernel-core-5.14.0-21.el9.s390x
kernel-modules-5.14.0-21.el9.s390x
kernel-5.14.0-21.el9.s390x
kernel-tools-5.14.0-21.el9.s390x
kernel-zfcpdump-core-5.14.0-21.el9.s390x
kernel-zfcpdump-modules-5.14.0-21.el9.s390x
kernel-zfcpdump-5.14.0-21.el9.s390x


# libguestfs-test-tool
PATH=/root/.local/bin:/root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
XDG_RUNTIME_DIR=/run/user/0
SELinux: Enforcing
guestfs_get_append: (null)
guestfs_get_autosync: 1
guestfs_get_backend: libvirt
guestfs_get_backend_settings: []
guestfs_get_cachedir: /var/tmp
guestfs_get_hv: /usr/libexec/qemu-kvm
guestfs_get_memsize: 1280
guestfs_get_network: 0
guestfs_get_path: /usr/lib64/guestfs
guestfs_get_pgroup: 0
guestfs_get_program: libguestfs-test-tool
guestfs_get_recovery_proc: 1
guestfs_get_smp: 1
guestfs_get_sockdir: /tmp
guestfs_get_tmpdir: /tmp
guestfs_get_trace: 0
guestfs_get_verbose: 1
host_cpu: s390x
Launching appliance, timeout set to 600 seconds.
libguestfs: launch: program=libguestfs-test-tool
libguestfs: launch: version=1.46.0rhel=9,release=5.el9,libvirt
libguestfs: launch: backend registered: direct
libguestfs: launch: backend registered: libvirt
libguestfs: launch: backend registered: uml
libguestfs: launch: backend registered: unix
libguestfs: launch: backend=libvirt
libguestfs: launch: tmpdir=/tmp/libguestfsnvRb09
libguestfs: launch: umask=0022
libguestfs: launch: euid=0
libguestfs: libvirt version = 7009000 (7.9.0)
libguestfs: guest random name = guestfs-8t51mpa0m49x67dy
libguestfs: connect to libvirt
libguestfs: opening libvirt handle: URI = qemu:///system, auth = default+wrapper, flags = 0
libguestfs: successfully opened libvirt handle: conn = 0x2aa1e36b030
libguestfs: qemu version (reported by libvirt) = 6001000 (6.1.0)
libguestfs: get libvirt capabilities
libguestfs: parsing capabilities XML
libguestfs: parsing domcapabilities XML
libguestfs: build appliance
libguestfs: begin building supermin appliance
libguestfs: run supermin
libguestfs: command: run: /usr/bin/supermin
libguestfs: command: run: \ --build
libguestfs: command: run: \ --verbose
libguestfs: command: run: \ --if-newer
libguestfs: command: run: \ --lock /var/tmp/.guestfs-0/lock
libguestfs: command: run: \ --copy-kernel
libguestfs: command: run: \ -f ext2
libguestfs: command: run: \ --host-cpu s390x
libguestfs: command: run: \ /usr/lib64/guestfs/supermin.d
libguestfs: command: run: \ -o /var/tmp/.guestfs-0/appliance.d
supermin: version: 5.2.1
supermin: rpm: detected RPM version 4.16
supermin: rpm: detected RPM architecture s390x
supermin: package handler: fedora/rpm
supermin: acquiring lock on /var/tmp/.guestfs-0/lock
supermin: build: /usr/lib64/guestfs/supermin.d
supermin: reading the supermin appliance
supermin: build: visiting /usr/lib64/guestfs/supermin.d/base.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/daemon.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/excludefiles type uncompressed excludefiles
supermin: build: visiting /usr/lib64/guestfs/supermin.d/hostfiles type uncompressed hostfiles
supermin: build: visiting /usr/lib64/guestfs/supermin.d/init.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/packages type uncompressed packages
supermin: build: visiting /usr/lib64/guestfs/supermin.d/udev-rules.tar.gz type gzip base image (tar)
supermin: mapping package names to installed packages
supermin: resolving full list of package dependencies
supermin: build: 171 packages, including dependencies
supermin: build: 32567 files
supermin: build: 7905 files, after matching excludefiles
supermin: build: 7916 files, after adding hostfiles
supermin: build: 7904 files, after removing unreadable files
supermin: build: 7930 files, after munging
supermin: kernel: looking for kernel using environment variables ...
supermin: kernel: looking for kernels in /lib/modules/*/vmlinuz ...
supermin: kernel: picked vmlinuz /lib/modules/5.14.0-21.el9.s390x+zfcpdump/vmlinuz
supermin: kernel: kernel_version 5.14.0-21.el9.s390x+zfcpdump
supermin: kernel: modpath /lib/modules/5.14.0-21.el9.s390x+zfcpdump
supermin: ext2: creating empty ext2 filesystem '/var/tmp/.guestfs-0/appliance.d.d3kya1z6/root'
supermin: ext2: populating from base image
supermin: ext2: copying files from host filesystem
supermin: ext2: copying kernel modules
supermin: ext2: creating minimal initrd '/var/tmp/.guestfs-0/appliance.d.d3kya1z6/initrd'
supermin: ext2: wrote 0 modules to minimal initrd
supermin: renaming /var/tmp/.guestfs-0/appliance.d.d3kya1z6 to /var/tmp/.guestfs-0/appliance.d
libguestfs: finished building supermin appliance
libguestfs: command: run: qemu-img --help | grep -sqE -- '\binfo\b.*-U\b'
libguestfs: command: run: qemu-img
libguestfs: command: run: \ info
libguestfs: command: run: \ -U
libguestfs: command: run: \ --output json
libguestfs: command: run: \ /var/tmp/.guestfs-0/appliance.d/root
libguestfs: parse_json: qemu-img info JSON output:\n{\n    "virtual-size": 4294967296,\n    "filename": "/var/tmp/.guestfs-0/appliance.d/root",\n    "format": "raw",\n    "actual-size": 230006784,\n    "dirty-flag": false\n}\n\n
libguestfs: command: run: qemu-img
libguestfs: command: run: \ create
libguestfs: command: run: \ -f qcow2
libguestfs: command: run: \ -o backing_file=/var/tmp/.guestfs-0/appliance.d/root,backing_fmt=raw
libguestfs: command: run: \ /tmp/libguestfsnvRb09/overlay2.qcow2
Formatting '/tmp/libguestfsnvRb09/overlay2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=4294967296 backing_file=/var/tmp/.guestfs-0/appliance.d/root backing_fmt=raw lazy_refcounts=off refcount_bits=16
libguestfs: create libvirt XML
libguestfs: libvirt XML:\n<?xml version="1.0"?>\n<domain type="kvm" xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0">\n  <name>guestfs-8t51mpa0m49x67dy</name>\n  <memory unit="MiB">1280</memory>\n  <currentMemory unit="MiB">1280</currentMemory>\n  <cpu mode="maximum"/>\n  <vcpu>1</vcpu>\n  <clock offset="utc">\n    <timer name="rtc" tickpolicy="catchup"/>\n    <timer name="pit" tickpolicy="delay"/>\n  </clock>\n  <os>\n    <type>hvm</type>\n    <kernel>/var/tmp/.guestfs-0/appliance.d/kernel</kernel>\n    <initrd>/var/tmp/.guestfs-0/appliance.d/initrd</initrd>\n    <cmdline>panic=1 console=ttysclp0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=UUID=3858c78d-b199-4941-a1cb-5a6a4324b3ba selinux=0 guestfs_verbose=1 TERM=xterm-256color</cmdline>\n  </os>\n  <on_reboot>destroy</on_reboot>\n  <devices>\n    <rng model="virtio">\n      <backend model="random">/dev/urandom</backend>\n    </rng>\n    <controller type="scsi" index="0" model="virtio-scsi"/>\n    <disk device="disk" type="file">\n      <source file="/tmp/libguestfsnvRb09/scratch1.img"/>\n      <target dev="sda" bus="scsi"/>\n      <driver name="qemu" type="raw" cache="unsafe"/>\n      <address type="drive" controller="0" bus="0" target="0" unit="0"/>\n    </disk>\n    <disk type="file" device="disk">\n      <source file="/tmp/libguestfsnvRb09/overlay2.qcow2"/>\n      <target dev="sdb" bus="scsi"/>\n      <driver name="qemu" type="qcow2" cache="unsafe"/>\n      <address type="drive" controller="0" bus="0" target="1" unit="0"/>\n    </disk>\n    <console type="unix">\n      <source mode="connect" path="/tmp/libguestfsHxiHUA/console.sock"/>\n      <target type="sclp" port="0"/>\n    </console>\n    <channel type="unix">\n      <source mode="connect" path="/tmp/libguestfsHxiHUA/guestfsd.sock"/>\n      <target type="virtio" name="org.libguestfs.channel.0"/>\n    </channel>\n    <controller type="usb" model="none"/>\n    <memballoon model="none"/>\n  </devices>\n  <qemu:commandline>\n    <qemu:env name="TMPDIR" value="/var/tmp"/>\n  </qemu:commandline>\n</domain>\n
libguestfs: command: run: ls
libguestfs: command: run: \ -a
libguestfs: command: run: \ -l
libguestfs: command: run: \ -R
libguestfs: command: run: \ -Z /var/tmp/.guestfs-0
libguestfs: /var/tmp/.guestfs-0:
libguestfs: total 4
libguestfs: drwxr-xr-x. 3 root root unconfined_u:object_r:user_tmp_t:s0   37 Nov 30 21:18 .
libguestfs: drwxrwxrwt. 8 root root system_u:object_r:tmp_t:s0          4096 Nov 30 21:18 ..
libguestfs: drwxr-xr-x. 2 root root unconfined_u:object_r:user_tmp_t:s0   46 Nov 30 21:18 appliance.d
libguestfs: -rw-r--r--. 1 root root unconfined_u:object_r:user_tmp_t:s0    0 Nov 30 21:18 lock
libguestfs: 
libguestfs: /var/tmp/.guestfs-0/appliance.d:
libguestfs: total 228744
libguestfs: drwxr-xr-x. 2 root root unconfined_u:object_r:user_tmp_t:s0         46 Nov 30 21:18 .
libguestfs: drwxr-xr-x. 3 root root unconfined_u:object_r:user_tmp_t:s0         37 Nov 30 21:18 ..
libguestfs: -rw-r--r--. 1 root root unconfined_u:object_r:user_tmp_t:s0     623104 Nov 30 21:18 initrd
libguestfs: -rwxr-xr-x. 1 root root unconfined_u:object_r:user_tmp_t:s0    3600384 Nov 30 21:18 kernel
libguestfs: -rw-r--r--. 1 root root unconfined_u:object_r:user_tmp_t:s0 4294967296 Nov 30 21:18 root
libguestfs: command: run: ls
libguestfs: command: run: \ -a
libguestfs: command: run: \ -l
libguestfs: command: run: \ -Z /tmp/libguestfsHxiHUA
libguestfs: total 4
libguestfs: drwxr-xr-x.  2 root root unconfined_u:object_r:user_tmp_t:s0   47 Nov 30 21:18 .
libguestfs: drwxrwxrwt. 21 root root system_u:object_r:tmp_t:s0          4096 Nov 30 21:18 ..
libguestfs: srw-rw----.  1 root qemu unconfined_u:object_r:user_tmp_t:s0    0 Nov 30 21:18 console.sock
libguestfs: srw-rw----.  1 root qemu unconfined_u:object_r:user_tmp_t:s0    0 Nov 30 21:18 guestfsd.sock
libguestfs: launch libvirt guest
[    0.044776] Linux version 5.14.0-21.el9.s390x+zfcpdump (mockbuild.eng.bos.redhat.com) (gcc (GCC) 11.2.1 20211019 (Red Hat 11.2.1-6), GNU ld version 2.35.2-10.el9) #1 SMP Thu Nov 25 21:48:13 EST 2021
[    0.044803] setup: Linux is running under KVM in 64-bit mode
[    0.045514] setup: The maximum memory size is 1280MB
[    0.045760] cpu: 1 configured CPUs, 0 standby CPUs
[    0.045874] Write protected kernel read-only data: 6620k
[    0.045953] Zone ranges:
[    0.045957]   DMA      [mem 0x0000000000000000-0x000000007fffffff]
[    0.045962]   Normal   empty
[    0.045965] Movable zone start for each node
[    0.045966] Early memory node ranges
[    0.045968]   node   0: [mem 0x0000000000000000-0x000000004fffffff]
[    0.045974] Initmem setup node 0 [mem 0x0000000000000000-0x000000004fffffff]
[    0.095622] percpu: Embedded 19 pages/cpu s45568 r0 d32256 u77824
[    0.095687] Built 1 zonelists, mobility grouping on.  Total pages: 322560
[    0.095698] Kernel command line: panic=1 console=ttysclp0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=UUID=3858c78d-b199-4941-a1cb-5a6a4324b3ba selinux=0 guestfs_verbose=1 TERM=xterm-256color
[    0.095782] Unknown command line parameters: no_timer_check edd=off udevtimeout=6000 cgroup_disable=memory tsc=reliable selinux=0 guestfs_verbose=1
[    0.099935] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    0.101996] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[    0.102027] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.109956] Memory: 1277080K/1310720K available (3740K kernel code, 848K rwdata, 2880K rodata, 972K init, 264K bss, 33640K reserved, 0K cma-reserved)
[    0.109967] random: get_random_u64 called from kmem_cache_open+0x3c/0x230 with crng_init=0
[    0.110082] SLUB: HWalign=256, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.110191] rcu: Hierarchical RCU implementation.
[    0.110194] rcu: \tRCU restricting CPUs from NR_CPUS=32 to nr_cpu_ids=1.
[    0.110197] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[    0.110198] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
[    0.110204] NR_IRQS: 3, nr_irqs: 3, preallocated irqs: 3
[    0.117525] clocksource: tod: mask: 0xffffffffffffffff max_cycles: 0x3b0a9be803b0a9, max_idle_ns: 1805497147909793 ns
[    0.117655] Console: colour dummy device 80x25
[    0.117840] printk: console [ttysclp0] enabled
[    0.117908] pid_max: default: 32768 minimum: 301
[    0.117971] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)
[    0.117985] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)
[    0.118562] rcu: Hierarchical SRCU implementation.
[    0.118682] smp: Bringing up secondary CPUs ...
[    0.118684] smp: Brought up 1 node, 1 CPU
[    0.119444] devtmpfs: initialized
[    0.119609] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.119617] futex hash table entries: 256 (order: 4, 65536 bytes, linear)
[    0.119772] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.119900] Spectre V2 mitigation: etokens
[    0.121101] alg: self-tests disabled
[    0.121387] cryptd: max_cpu_qlen set to 1000
[    0.122137] SCSI subsystem initialized
[    0.122151] pps_core: LinuxPPS API ver. 1 registered
[    0.122152] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti>
[    0.122155] PTP clock support registered
[    0.187003] Unpacking initramfs...
[    0.187767] Freeing initrd memory: 608K
[    0.188584] workingset: timestamp_bits=62 max_order=19 bucket_order=0
[    0.198322] NET: Registered PF_ALG protocol family
[    0.198339] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
[    0.198343] io scheduler mq-deadline registered
[    0.198345] io scheduler kyber registered
[    0.198387] io scheduler bfq registered
[    0.198452] atomic64_test: passed
[    0.199032] zram: Added device: zram0
[    0.200776] random: fast init done
[    0.201901] random: crng init done
[    0.203319] null_blk: module loaded
[    0.203361] rdac: device handler registered
[    0.203390] hp_sw: device handler registered
[    0.203391] emc: device handler registered
[    0.203411] alua: device handler registered
[    0.203888] Rounding down aligned max_sectors from 4294967295 to 4294967288
[    0.203954] db_root: cannot open: /etc/target
[    0.204027] pps_ldisc: PPS line discipline registered
[    0.204045] mpls_gso: MPLS GSO support
[    0.204138] cio: Channel measurement facility initialized using format extended (mode autodetected)
[    0.204567] sclp_sd: Store Data request failed (eq=2, di=3, response=0x40f0, flags=0x00, status=0, rc=-5)
[    0.204608] vmur: The z/VM virtual unit record device driver cannot be loaded without z/VM
[    0.205223] Freeing unused kernel image (initmem) memory: 972K
[    0.217615] Write protected read-only-after-init data: 24k
[    0.217703] Checked W+X mappings: passed, no unexpected W+X pages found
[    0.217711] Run /init as init process
supermin: mounting /proc
supermin: ext2 mini initrd starting up: 5.2.1 glibc
supermin: cmdline: panic=1 console=ttysclp0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=UUID=3858c78d-b199-4941-a1cb-5a6a4324b3ba selinux=0 guestfs_verbose=1 TERM=xterm-256color
supermin: uptime: 0.21 0.01
supermin: mounting /sys
supermin: waiting another 1024000000 ns for root UUID to appear
This usually means your kernel doesn't support virtio, or supermin was unable
to load some kernel modules (see module loading messages above).
supermin: waiting another 2048000000 ns for root UUID to appear
supermin: waiting another 4096000000 ns for root UUID to appear
supermin: waiting another 8192000000 ns for root UUID to appear
supermin: waiting another 16384000000 ns for root UUID to appear
supermin: waiting another 32768000000 ns for root UUID to appear
supermin: waiting another 65536000000 ns for root UUID to appear
supermin: waiting another 131072000000 ns for root UUID to appear
... hang


The supermin-5.2.1-6 can not resolve this issue. The kernel version 5.14.0-21.el9.s390x+zfcpdump is still be selected.
Comment 5 Richard W.M. Jones 2021-12-01 09:57:46 UTC 
> supermin: kernel: picked vmlinuz /lib/modules/5.14.0-21.el9.s390x+zfcpdump/vmlinuz

It picked the wrong kernel for some reason, so this isn't fixed ...
Comment 6 Richard W.M. Jones 2021-12-01 10:38:27 UTC 
The previous patch was wrong.  This commit should fix it:

https://github.com/libguestfs/supermin/commit/f53868ce875fc17527696a85b48c67fefa3176e7
Comment 7 Richard W.M. Jones 2021-12-01 13:12:11 UTC 
FWIW I tested this on s390x and it now appears to work:
supermin: kernel: picked vmlinuz /lib/modules/5.14.0-15.mr128_211112_1256.el9.s390x/vmlinuz
Comment 8 YongkuiGuo 2021-12-01 13:24:21 UTC 
Test with the latest fixed supermin:

# rpm -q libguestfs supermin
libguestfs-1.46.0-5.el9.s390x
supermin-5.2.1-7.el9.s390x

# rpm -qa | grep  kernel
kernel-core-5.14.0-21.el9.s390x
kernel-modules-5.14.0-21.el9.s390x
kernel-5.14.0-21.el9.s390x
kernel-tools-5.14.0-21.el9.s390x
kernel-zfcpdump-core-5.14.0-21.el9.s390x
kernel-zfcpdump-modules-5.14.0-21.el9.s390x
kernel-zfcpdump-5.14.0-21.el9.s390x

# hostname
ibm-z-523.lab.eng.rdu2.redhat.com


# libguestfs-test-tool
PATH=/root/.local/bin:/root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
XDG_RUNTIME_DIR=/run/user/0
SELinux: Enforcing
guestfs_get_append: (null)
guestfs_get_autosync: 1
guestfs_get_backend: libvirt
guestfs_get_backend_settings: []
guestfs_get_cachedir: /var/tmp
guestfs_get_hv: /usr/libexec/qemu-kvm
guestfs_get_memsize: 1280
guestfs_get_network: 0
guestfs_get_path: /usr/lib64/guestfs
guestfs_get_pgroup: 0
guestfs_get_program: libguestfs-test-tool
guestfs_get_recovery_proc: 1
guestfs_get_smp: 1
guestfs_get_sockdir: /tmp
guestfs_get_tmpdir: /tmp
guestfs_get_trace: 0
guestfs_get_verbose: 1
host_cpu: s390x
Launching appliance, timeout set to 600 seconds.
libguestfs: launch: program=libguestfs-test-tool
libguestfs: launch: version=1.46.0rhel=9,release=5.el9,libvirt
libguestfs: launch: backend registered: direct
libguestfs: launch: backend registered: libvirt
libguestfs: launch: backend registered: uml
libguestfs: launch: backend registered: unix
libguestfs: launch: backend=libvirt
libguestfs: launch: tmpdir=/tmp/libguestfsjZaZDq
libguestfs: launch: umask=0022
libguestfs: launch: euid=0
libguestfs: libvirt version = 7009000 (7.9.0)
libguestfs: guest random name = guestfs-o7o7utgvyhz3qz4d
libguestfs: connect to libvirt
libguestfs: opening libvirt handle: URI = qemu:///system, auth = default+wrapper, flags = 0
libguestfs: successfully opened libvirt handle: conn = 0x2aa0807b030
libguestfs: qemu version (reported by libvirt) = 6001000 (6.1.0)
libguestfs: get libvirt capabilities
libguestfs: parsing capabilities XML
libguestfs: parsing domcapabilities XML
libguestfs: build appliance
libguestfs: begin building supermin appliance
libguestfs: run supermin
libguestfs: command: run: /usr/bin/supermin
libguestfs: command: run: \ --build
libguestfs: command: run: \ --verbose
libguestfs: command: run: \ --if-newer
libguestfs: command: run: \ --lock /var/tmp/.guestfs-0/lock
libguestfs: command: run: \ --copy-kernel
libguestfs: command: run: \ -f ext2
libguestfs: command: run: \ --host-cpu s390x
libguestfs: command: run: \ /usr/lib64/guestfs/supermin.d
libguestfs: command: run: \ -o /var/tmp/.guestfs-0/appliance.d
supermin: version: 5.2.1
supermin: rpm: detected RPM version 4.16
supermin: rpm: detected RPM architecture s390x
supermin: package handler: fedora/rpm
supermin: acquiring lock on /var/tmp/.guestfs-0/lock
supermin: build: /usr/lib64/guestfs/supermin.d
supermin: reading the supermin appliance
supermin: build: visiting /usr/lib64/guestfs/supermin.d/base.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/daemon.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/excludefiles type uncompressed excludefiles
supermin: build: visiting /usr/lib64/guestfs/supermin.d/hostfiles type uncompressed hostfiles
supermin: build: visiting /usr/lib64/guestfs/supermin.d/init.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/packages type uncompressed packages
supermin: build: visiting /usr/lib64/guestfs/supermin.d/udev-rules.tar.gz type gzip base image (tar)
supermin: mapping package names to installed packages
supermin: resolving full list of package dependencies
supermin: build: 171 packages, including dependencies
supermin: build: 32567 files
supermin: build: 7905 files, after matching excludefiles
supermin: build: 7916 files, after adding hostfiles
supermin: build: 7904 files, after removing unreadable files
supermin: build: 7930 files, after munging
supermin: kernel: looking for kernel using environment variables ...
supermin: kernel: looking for kernels in /lib/modules/*/vmlinuz ...
supermin: kernel: picked vmlinuz /lib/modules/5.14.0-21.el9.s390x/vmlinuz
supermin: kernel: kernel_version 5.14.0-21.el9.s390x
supermin: kernel: modpath /lib/modules/5.14.0-21.el9.s390x
...
===== TEST FINISHED OK =====

It works.
Comment 11 YongkuiGuo 2021-12-08 06:17:19 UTC 
Verified this bug as there is no issue on the latest RHEL9 compose which includes the fixed supermin package.
Comment 15 errata-xmlrpc 2022-05-17 13:18:16 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: supermin), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2488
Note You need to log in before you can comment on or make changes to this bug.