Bug 1945835

Jarda - sending your way as it seems this component used to be picked up by Pino... I suspect another for the backlog, but will let you decide...

Klaus - seems this was lost in Jarda's backlog. It's a virt-tools related component - I'll let you and Rich decide how to proceed.

(In reply to John Ferlan from comment #4)
> Klaus - seems this was lost in Jarda's backlog. It's a virt-tools related
> component - I'll let you and Rich decide how to proceed.

Thanks. Will park it in the Backlog until Rich is back and we can discuss, as this seems relatively important.

Let's bring this back. Assigning for Laszlo as he has been working on updating our P2V lately.

Since our current P2V iso is RHEL 7 based, I believe NVMe wasn't really a 'thing' back then, so hopefully this will all 'just work' once we have a newer P2V version, hopefully based on RHEL9?

Still, Laszlo, you might make this tentatively dependent on the RHEL9 P2V RFE.

Thanks,

 -Klaus

Preparatory patch:

[p2v PATCH] build: remove gnulib
Message-Id: <20220321151618.22282-1-lersek>
https://listman.redhat.com/archives/libguestfs/2022-March/028444.html

(In reply to Laszlo Ersek from comment #17)
> Preparatory patch:
> 
> [p2v PATCH] build: remove gnulib
> Message-Id: <20220321151618.22282-1-lersek>
> https://listman.redhat.com/archives/libguestfs/2022-March/028444.html

Commit 16c50a347801.

Moving to POST because there's a fix posted or upstream.

Note for readers: we don't yet have a way to deliver this fix to
customers, but we're working on that separately (bug 2038105).

Apologies, I wasn't informative enough in my comment 18 -- there I had left this BZ in ASSIGNED status on purpose. Commit 16c50a347801 is a prep patch (gnulib removal); the actual work for recognizing NVMe block devices is still to be done, per comment 7 and comment 12.

My intent was to post two separate patches for this BZ (rather than a series), given that the gnulib removal could have required several iterations and a separate discussion, from the NVMe pattern addition. So I'm moving this back to ASSIGNED now; there's more upstream work for me to do here. Again I'm sorry for being unclear in comment 18!

Some findings related to testing:

- libvirt does not support emulated NVMe devices, only assigned (= passthrough) ones, so the only option to test the nvme pattern is to run QEMU directly.

- in the QEMU tree, "docs/system/devices/nvme.rst" is helpful, regarding the command line params

- because my physical host is Fedora 35, the "virt-p2v.img" target attempts to build a fedora-35 based bootable image, with virt-builder. Unfortunately, the image is unbootable. The grub menu comes up, but once it loads the kernel / initrd, the guest is dead -- no logs (not even on serial), nothing

- attempting to check "virt-p2v.img" with guestfish, I even get:

><fs> list-filesystems 
/dev/sda1: unknown
/dev/sda2: xfs
/dev/sda3: xfs

which is inexplicable.

- inspired by commit 5ead90798eac ("tests: bump Fedora image to 32", 2020-07-03), which changed the "physical machine" image to F32, I modified Makefile.am to build a fedora-32 based bootable image as well:

@@ -296,7 +296,7 @@ virt-p2v.img: \
            p2v.service \
            virt-p2v \
            virt-p2v-make-disk
-       $(top_builddir)/run virt-p2v-make-disk -o $@
+       $(top_builddir)/run virt-p2v-make-disk -o $@ -v fedora-32
 
 # Tests.
 
Unfortunately, with this in place, the image is not even built, as it hangs in:

> [  17.7] Updating packages
> ...
> dnf --verbose -y --best upgrade

Same issue with "fedora-34".

(In a separate F34 VM, "dnf update" works OK.)

So ATM p2v is untestable to me.

(In reply to Laszlo Ersek from comment #21)
> - because my physical host is Fedora 35, the "virt-p2v.img" target attempts
> to build a fedora-35 based bootable image, with virt-builder. Unfortunately,
> the image is unbootable. The grub menu comes up, but once it loads the
> kernel / initrd, the guest is dead -- no logs (not even on serial), nothing

Mine actually tried to build a fedora-36 image because of my
half-Rawhide install!  Anyway I added a similar patch to you to
force it to use fedora-35 instead.  (Note I don't think fedora-32
will work because the Fedora mirrors won't still have the packages
needed to do the update).

The build does *appear* to hang at the dnf command, but actually
it's not hanging, it's just not telling you what it's doing.  The
reason is that the libguestfs protocol doesn't transmit the
messages from dnf until the command completes, which sucks but
there's not much we can do about it.  (The only continuous messages
you see from the libguestfs appliance are console messages, since
those are sent over a separate channel.  Since dnf doesn't print
any messages to the console, this doesn't apply).

Anyway, leaving it alone for about 10 minutes does complete the
fedora-35 image.

However it doesn't boot correctly.  It seems as if it's trying to
boot the guest instead of virt-p2v.  This used to work fine so I
wonder if something changed in qemu/firmware/command line?  I'll
keep plugging away to see if I can make it work.

This is my patch, but if you're on Fedora 35 already you shouldn't need this.
Although I suppose it can be useful to have a different version of the
guest vs virt-p2v, just so it's easy to tell which one is booting.

diff --git a/Makefile.am b/Makefile.am
index a5742b0..08357e1 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -296,7 +296,7 @@ virt-p2v.img: \
            p2v.service \
            virt-p2v \
            virt-p2v-make-disk
-       $(top_builddir)/run virt-p2v-make-disk -o $@
+       $(top_builddir)/run virt-p2v-make-disk -o $@ -v fedora-35

Oh wait, the physical machine was F32 and virt-p2v was F35, so
it was trying to boot p2v.  Doesn't explain yet why it doesn't
boot ...

So several things are going on.  virt-builder (or more likely, dracut)
appears to have a bug where if you update the kernel in the image then
it will have a broken command line.  To observe this:

$ virt-builder fedora-35 --update 
$ qemu-system-x86_64 -machine accel=kvm:tcg -cpu host -m 2048 -drive file=fedora-35.img,format=raw,if=virtio

Interrupt grub, and go and look at the kernel command line of the
top (5.16) kernel and you'll see it's got stuff from the libguestfs
appliance, including a root= parameter that is definitely wrong.

The older kernel (part of the base image) is fine though, so you could
still boot by selecting that entry.

Anyhow back to virt-p2v - if you interrupt grub there and select the
older, working kernel then it boots up to the point where it tries to
mount the root filesystem off USB.  I added "ignore_loglevel
rd.plymouth=0 plymouth.enable=0" to try to print a bit more information about
what it's doing, but then things got really strange and I started to see actual
disk errors.  I wonder if the qemu USB mass storage emulation has bitrotted
because no one uses it?  It's a bit random, sometimes there are no
errors.

I'm working on the assumption that qemu USB boot has stopped working, or
dracut is missing some drivers.

https://listman.redhat.com/archives/libguestfs/2022-March/028468.html
https://listman.redhat.com/archives/libguestfs/2022-March/028469.html
https://listman.redhat.com/archives/libguestfs/2022-March/028470.html

With these patches, "make run-virt-p2v-in-a-vm" works fine and I was
able to run virt-p2v and complete a conversion successfully.

(Note these patches are required by but unrelated to fixing the current bug,
hence status should stay as ASSIGNED.)

So the one-liner fix seems to work.

I modified the "run-virt-p2v-in-a-vm" makefile target in a separate patch, to test a "physical" machine with a system disk (as before) plus two NVMe disks (well, two namespaces of one NVMe device) that both map the blank disk image ("blank-part.img").

I've also set up a dedicated RHEL9 nightly VM for the "conversion server" role.

I'm going to attach some screenshots from the client side (where p2v ran), the conversion log (compressed) from the server side (the RHEL9 VM), and the output XML (also from the server side, from /var/tmp). The md5sums of the nvme-origianted disks on the destination match the blank image's md5sum on the source.

Created attachment 1868129 [details]
virt-p2v screen listing the NVMe disks

Created attachment 1868130 [details]
virt-p2v screen showing conversion completion

Created attachment 1868131 [details]
conversion log & other artifacts from the RHEL9 conversion server

Created attachment 1868132 [details]
conversion output domain XML

Rich, assuming I have an "-o local" conversion output (p2v.xml and all
three disk images), how do I most easily import those on an existent
libvirtd host? I figure I should test booting the converted domain, and
see how the nvme disks (now exposed as virtio-block disks, vd*) look
"from the inside". Thanks!

(In reply to Laszlo Ersek from comment #31)
> Rich, assuming I have an "-o local" conversion output (p2v.xml and all
> three disk images), how do I most easily import those on an existent
> libvirtd host? I figure I should test booting the converted domain, and
> see how the nvme disks (now exposed as virtio-block disks, vd*) look
> "from the inside". Thanks!

-o local, like the one I did via p2v yesterday:

-rw-r--r--. 1 rjones rjones   6442450944 Mar 23 18:25  p2v-sda
-rw-r--r--. 1 rjones rjones         1650 Mar 23 18:25  p2v.xml

creates a disk image and libvirt XML.  You can just import that straight
into libvirt using "virsh create p2v.xml".  Running that either as non-root
or root will work, but note they will import the VM into separate
namespaces, and I think non-root may have some limitations (no network?)

Also note you can add --console to immediately start a virsh console
session with the new domain.  (Use virt-viewer to see the GUI)

Thanks, I've read up on "virsh create" now -- I've never used it before. (Not surprisingly, as I've had no use for either transient domains, or one-off configurations of persistent domains.)

I should have mentioned that virsh define + virsh start is another way to do this.
I cannot remember the difference though - does one create a permanent guest and
the other create a transient guest?  Or maybe they're the same, but with more
steps.

Also virsh undefine removes the guest definition (permanently).

Created attachment 1868348 [details]
conversion artifacts with updated test case

I've udpated the test case: not only does it add two NVMe namespaces (under an NVMe device), but it also exposes the "physical machine" disk image as another (separate) NVMe device. See the artifacts (logs, screenshots etc) attached.

Additionally, after the conversion, I imported the output domain into libvirtd, and booted it successfully. The blank disks have md5sums identical to the original blank disk.

Upstream patch set:

[p2v PATCH 0/2] recognize NVMe devices
Message-Id: <20220325170514.14431-1-lersek>
https://listman.redhat.com/archives/libguestfs/2022-March/028485.html

(In reply to Laszlo Ersek from comment #35)

> Upstream patch set:
> 
> [p2v PATCH 0/2] recognize NVMe devices
> Message-Id: <20220325170514.14431-1-lersek>
> https://listman.redhat.com/archives/libguestfs/2022-March/028485.html

Commit range 307a346cebc8..c1a4bd83a8bb.

Moving to POST (refer to comment 19).

Rich rebased virt-p2v for bug 1990052; with the same effort, we inherit the upstream fix for this issue too.

Tested with:
livecd-p2v-202211091434.iso
virt-v2v-2.0.7-6/7.el9.x86_64

Use virt-p2v to convert RHEL8/RHEL9/Win 2022 os installed in physical host with nvme disk, the nvme disk can be detected by virt-p2v and the conversion can be finished successfully.

Based on the comments, I close the bug.