Bug 1795973

Created attachment 1656213 [details]
libguestfs-test-tool output

Created attachment 1656214 [details]
Complete list of installed packages

This isn't rt related; I dropped back to Linux amd-daytona-04.khw1.lab.eng.bos.redhat.com 4.18.0-171.el8.x86_64+debug #1 SMP Fri Jan 17 15:00:12 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

and it's still broken in the same way.

David, could you check if it's actually using the non-RT kernel for libguestfs?
The output of libguestfs-test-tool can be used to check this.  Libguestfs will
pick the highest numbered installed (not necessarily running) kernel.

step 1: It works with non-RT kernel but also the rhel-av module; I need to test with non-RT and non-AV

Yeh you're right, it's fine without rt even with the non-AV qemu

I can reproduce this issue only when the following kernel-rt-debug* packages also have been installed.
-----
kernel-rt-debug-modules-extra-4.18.0-172.rt13.29.el8.x86_64
kernel-rt-debug-core-4.18.0-172.rt13.29.el8.x86_64
kernel-rt-debug-modules-4.18.0-172.rt13.29.el8.x86_64
kernel-rt-debug-4.18.0-172.rt13.29.el8.x86_64
kernel-rt-debug-devel-4.18.0-172.rt13.29.el8.x86_64
-----
As long as I removed the above kernel-rt-debug* packages, the libguestfs-test-tool works fine.

One rather crude way to fix this might be to exclude RT kernels.  We
previously excluded Xen kernels as you can see from the adjacent
test in the patch below.  I don't really like this approach however.
We exclude Xen kernels for a good reason - they cannot boot on qemu -
but there should be no similar problem for RT kernels as far as I
know.  Although RT kernels would normally be used on baremetal, they
should at least work on qemu even if it's not optimal.

commit 5680e912a43092bff0e5e5f62cf6a7886ca564b7 (HEAD -> master)
Author: Richard W.M. Jones <rjones>
Date:   Tue Feb 4 10:21:29 2020 +0000

    kernel: Exclude realtime kernels from consideration (RHBZ#1795973).

diff --git a/src/format_ext2_kernel.ml b/src/format_ext2_kernel.ml
index 98bff3a..d7f3054 100644
--- a/src/format_ext2_kernel.ml
+++ b/src/format_ext2_kernel.ml
@@ -156,6 +156,8 @@ and kernel_filter patterns is_arm all_files =
       ) all_files in
   let files =
     List.filter (fun filename -> find filename "xen" = -1) files in
+  let files =
+    List.filter (fun filename -> find filename "-rt-" = -1) files in
   let files =
     if not is_arm then files
     else (

(In reply to YongkuiGuo from comment #7)
> I can reproduce this issue only when the following kernel-rt-debug* packages
> also have been installed.
> -----
> kernel-rt-debug-modules-extra-4.18.0-172.rt13.29.el8.x86_64
> kernel-rt-debug-core-4.18.0-172.rt13.29.el8.x86_64
> kernel-rt-debug-modules-4.18.0-172.rt13.29.el8.x86_64
> kernel-rt-debug-4.18.0-172.rt13.29.el8.x86_64
> kernel-rt-debug-devel-4.18.0-172.rt13.29.el8.x86_64
> -----
> As long as I removed the above kernel-rt-debug* packages, the
> libguestfs-test-tool works fine.

Could you attach the output with libguestfs-test-tool when the RT debug
kernels are installed?

Also it would be really useful if you could try this:

  $ virt-rescue --scratch
  ><rescue> ls /

to see if the ls command hangs or segfaults.

Created attachment 1657525 [details]
libguestfs-test-tool log with kernel-rt-debug* installed

$ virt-rescue --scratch
Formatting '/tmp/libguestfs6QMI2v/overlay2.qcow2', fmt=qcow2 size=4294967296 backing_file=/var/tmp/.guestfs-0/appliance.d/root backing_fmt=raw cluster_size=65536 lazy_refcounts=off refcount_bits=16
[    0.000000] ACPI BIOS Error (bug): A valid RSDP was not found (20190703/tbxfroot-210)
[    1.708688] usbserial: usb_serial_init - registering generic driver failed
[    1.712624] usbserial: usb_serial_init - returning with error -19
supermin: mounting /proc
supermin: ext2 mini initrd starting up: 5.1.19 glibc
insmod: init_module: nfit.ko: No such device
Starting /init script ...
[/usr/lib/tmpfiles.d/mdadm.conf:1] Line references path below legacy directory /var/run/, updating /var/run/mdadm → /run/mdadm; please update the tmpfiles.d/ drop-in file accordingly.
[/usr/lib/tmpfiles.d/systemd.conf:11] Unknown group 'utmp'.
[/usr/lib/tmpfiles.d/systemd.conf:22] Unknown group 'systemd-journal'.
[/usr/lib/tmpfiles.d/systemd.conf:23] Unknown group 'systemd-journal'.
starting version 239
/init: line 116: echo: write error: Invalid argument
/init: line 116: echo: write error: Invalid argument
mdadm: No arrays found in config file or automatically
/init: line 142: lvmetad: command not found
mdadm: No arrays found in config file or automatically
/init: line 152: ldmtool: command not found
/init: line 255:   282 Segmentation fault      $cmd

------------------------------------------------------------

Welcome to virt-rescue, the libguestfs rescue shell.

Note: The contents of / (root) are the rescue appliance.
You have to mount the guest’s partitions under /sysroot
before you can examine them.

><rescue> ls /     --hang

(In reply to YongkuiGuo from comment #11)
> Created attachment 1657525 [details]
> libguestfs-test-tool log with kernel-rt-debug* installed

Thanks - that is the same as what Dave Gilbert observed - ie. guestfsd
segfaults, ...

> ><rescue> ls /     --hang

... and this is what we both observed too.

kernel-rt-4.18.0-172.rt13.29.el8.x86_64 => works
kernel-rt-debug-4.18.0-172.rt13.29.el8.x86_64 => broken

Setting LIBGUESTFS_MEMSIZE=2048 causes both test cases to succeed.  It could
be that the debug kernel uses more memory.

I suspect the fix for this is simply:

commit ba82fb023de8ff0d271b4b82e3ef12a8d169728e
Author: Richard W.M. Jones <rjones>
Date:   Wed May 16 13:40:27 2018 +0100

    lib: Increase default memory assigned to the appliance.
    
    With recent Linux kernels, adding and partitioning 255 disks causes
    the appliance to run out of memory.  This causes a test failure in
    tests/disks/test-255-disks.sh.  This change gives the appliance enough
    memory to complete the test.

This is not included in libguestfs from RHEL non-AV 8, but it is
included in RHEL AV already.

The default memory size is 768M in RHEL-AV(non-aarch64/powerpc).  The command 'LIBGUESTFS_MEMSIZE=768 libguestfs-test-tool' works both kernel-rt and kernel-rt-debug.

Hi rjones,
Is there any requirements to cover the RT kernel for libguestfs from QE side? I'd like to know the reason why you use the RT kernel. Thanks.

(In reply to YongkuiGuo from comment #18)
> Hi rjones,
> Is there any requirements to cover the RT kernel for libguestfs from QE
> side? I'd like to know the reason why you use the RT kernel. Thanks.

While I believe this is not actually a bug related to the RT kernel,
I do think it may be worth including a simple test where we run
libguestfs-test-tool with only an RT kernel installed.  Assuming
that isn't going to be too much work.

I don't know whether or not it's a *requirement* that we test RT kernels.
Maybe we can ask Martin Tessun about that.  But it'd be nice to test
that it doesn't break.

(In reply to Richard W.M. Jones from comment #19)
> While I believe this is not actually a bug related to the RT kernel,
> I do think it may be worth including a simple test where we run
> libguestfs-test-tool with only an RT kernel installed.  Assuming
> that isn't going to be too much work.
> 
> I don't know whether or not it's a *requirement* that we test RT kernels.
> Maybe we can ask Martin Tessun about that.  But it'd be nice to test
> that it doesn't break.

Ok, i get it. Thanks.

Just to update things:

 - Bug is NOT caused by the RT kernel.

 - Bug is caused by the appliance running out of memory.  Using LIBGUESTFS_MEMSIZE=2048
   (for example) will fix the problem.

 - There is an upstream patch which provides more memory to the appliance, but
   it is only applied to RHEL AV branch, but this but was found in RHEL non-AV.

 - We will soon copy the RHEL AV package to RHEL (I think in RHEL 8.3) so
   we don't really need to apply the patch specially to RHEL right now.

I suspect we ought to close this bug.

rjones, I tried on RHEL8.3 and there was no problem. We can close this bug.