Bug 2151752

(In reply to mxie from comment #2)
> Created attachment 1930956 [details]
> virt-v2v-convert-win2022-guest.log

Here's the conflict, between the actual driver "fwcfg" and the stub driver "qemufwcfg":

windows: copying guest tools bits: '/usr/share/virtio-win/virtio-win.iso:fwcfg/2k22/amd64/fwcfg.cat' -> '/Windows/Drivers/VirtIO/fwcfg.cat'
windows: copying guest tools bits: '/usr/share/virtio-win/virtio-win.iso:fwcfg/2k22/amd64/fwcfg.inf' -> '/Windows/Drivers/VirtIO/fwcfg.inf'
windows: copying guest tools bits: '/usr/share/virtio-win/virtio-win.iso:fwcfg/2k22/amd64/fwcfg.pdb' -> '/Windows/Drivers/VirtIO/fwcfg.pdb'
windows: copying guest tools bits: '/usr/share/virtio-win/virtio-win.iso:fwcfg/2k22/amd64/fwcfg.sys' -> '/Windows/Drivers/VirtIO/fwcfg.sys'
windows: copying guest tools bits: '/usr/share/virtio-win/virtio-win.iso:qemufwcfg/2k22/amd64/qemufwcfg.cat' -> '/Windows/Drivers/VirtIO/qemufwcfg.cat'
windows: copying guest tools bits: '/usr/share/virtio-win/virtio-win.iso:qemufwcfg/2k22/amd64/qemufwcfg.inf' -> '/Windows/Drivers/VirtIO/qemufwcfg.inf'

Thanks for the data. Comments:

(1) I can't find the screenshots
"win11-copied-drivers-after-virt-v2v-2.0.7-7.bz2151752.png" and the
screenshot for Win2022 (I think you mispasted the name of that file).
Can you please attach both files?

(2) The logs are useful, thanks. In the win11 case, I see:

> windows: copying guest tools bits (via libosinfo): 'host:/usr/share/virtio-win/drivers/by-os/amd64/w11/qemufwcfg.inf' -> '/Windows/Drivers/VirtIO/qemufwcfg.inf'
> windows: copying guest tools bits (via libosinfo): 'host:/usr/share/virtio-win/drivers/by-os/amd64/w11/qemufwcfg.cat' -> '/Windows/Drivers/VirtIO/qemufwcfg.cat'

This means the following:

- virt-v2v copies the stub driver from the w11 subdirectory of the
  installed virtio-win RPM

- the windows version seems to be a match, as far as I can tell.

- *libosinfo* only knows about the stub driver, so there is no conflict
  between both drivers; the patch does not need to remove any drivers.

- However, this may be a problem in itself.
  virtio-win-1.9.30-0.el9_1.noarch RPM certainly provides both drivers,
  as host side files:

  /usr/share/virtio-win/drivers/by-os/amd64/w11/fwcfg.cat
  /usr/share/virtio-win/drivers/by-os/amd64/w11/fwcfg.inf
  /usr/share/virtio-win/drivers/by-os/amd64/w11/fwcfg.sys
  /usr/share/virtio-win/drivers/by-os/amd64/w11/qemufwcfg.cat
  /usr/share/virtio-win/drivers/by-os/amd64/w11/qemufwcfg.inf

  so this is an omission in libosinfo. I've checked, and even upstream
  osinfo-db (@ 72c69622e6db) does not know about the full driver
  "fwcfg", only the stub driver "qemufwcfg".

- Either way, from comment#10, the result is (in the converted guest):
  "qemufwcfg device has no driver". This seems to imply that the
  qemufwcfg stub driver fails to bind the device for some reason.

(3) In the win2022 case, the log says:

> windows: copying guest tools bits: '/usr/share/virtio-win/virtio-win.iso:fwcfg/2k22/amd64/fwcfg.cat' -> '/Windows/Drivers/VirtIO/fwcfg.cat'
> windows: copying guest tools bits: '/usr/share/virtio-win/virtio-win.iso:fwcfg/2k22/amd64/fwcfg.inf' -> '/Windows/Drivers/VirtIO/fwcfg.inf'
> windows: copying guest tools bits: '/usr/share/virtio-win/virtio-win.iso:fwcfg/2k22/amd64/fwcfg.pdb' -> '/Windows/Drivers/VirtIO/fwcfg.pdb'
> windows: copying guest tools bits: '/usr/share/virtio-win/virtio-win.iso:fwcfg/2k22/amd64/fwcfg.sys' -> '/Windows/Drivers/VirtIO/fwcfg.sys'
> windows: copying guest tools bits: '/usr/share/virtio-win/virtio-win.iso:qemufwcfg/2k22/amd64/qemufwcfg.cat' -> '/Windows/Drivers/VirtIO/qemufwcfg.cat'
> windows: copying guest tools bits: '/usr/share/virtio-win/virtio-win.iso:qemufwcfg/2k22/amd64/qemufwcfg.inf' -> '/Windows/Drivers/VirtIO/qemufwcfg.inf'

and then it also says:

> windows: skipping qemufwcfg stub driver in favor of fwcfg driver

Meaning:

- Virt-v2v copies both the stub driver and the actual driver from the
  2k22 directories of the (loop-mounted) virtio-win ISO.

- The windows version is a match, AFAICT

- libosinfo is not used for selecting the drivers to copy, as libosinfo
  does not know anything about guest drivers for win2022:

> libosinfo: loaded OS: http://microsoft.com/win/2k22
> libosinfo drivers before filtering:
> [nothing]
> libosinfo drivers after filtering:
> [nothing]

- Because both drivers are copied, the patch takes effect and it removes
  the stub driver. The complete driver (fwcfg) remains in place.

- The result is, per comment#10: "qemufwcfg device cannot start because
  of bad data". This implies that the complete driver fails to bind the
  device for some reason.

So, my verdict thus far seems to be that, with the patch applied, we
*never* create a situation where the stub and the complete drivers could
conflict with each other; yet *either* driver fails to bind the device.

This leads me to believe that this is a genuine virtio-win problem.

Here's what I'm going to do:

- check if I can reproduce the symptom with either driver *without*
  virt-v2v in the picture; that is, after loading these drivers from
  virtio-win into a Win2019 guest that's installed originally under
  libvirt

- try to reproduce the symptom via conversion from ESXi, again using
  Win2019. Last time I couldn't do this, but now I know that I need to
  display the "hidden devices" in Device Manager.

(In reply to Laszlo Ersek from comment #14)

> Here's what I'm going to do:
>
> - check if I can reproduce the symptom with either driver *without*
>   virt-v2v in the picture; that is, after loading these drivers from
>   virtio-win into a Win2019 guest that's installed originally under
>   libvirt

After installing the stub driver like this, three things happen:

- the "QEMU FWCfg Device" entry appears in the System devices subtree of
  Device Manager

- there are no warning or error marks on the icon

- the Properties dialog says, "No drivers are installed for this
  device".

  This matches comment#1 precisely, and matching comment 10 too ("Win11:
  qemufwcfg device has no driver").

After uninstalling the "QEMU FWCfg Device" together with its *stub*
driver from Device Manager, and then installing the *complete* driver,
three things happen:

- the "QEMU FwCfg Device" entry appears in the System devices subtree of
  Device Manager (note the different spelling: lower-case "w" in "FwCfg"
  as opposed to "FWCfg" from above)

- there is a yellow triangle with an exclamation mark in it, over the
  icon

- the properties dialog says, "This device cannot start. (Code 10) Bad
  data."

  This matches the screenshot in comment#0 precisely, and matches
  comment 10 as well ("Win2022: qemufwcfg device cannot start because of
  bad data")

Alright, so my analysis is complete; here's the verdict:

- There is no difference between Win2022, Win2019, and Win11. The only
  difference in behavior is due to installing the stub driver versus the
  complete driver. The symptom only *appears* related to the Windows
  version because libosinfo has "uneven" support for these Windows
  releases, and some Windows versions cause the stub driver to be
  installed, and some other Windows versions cause the complete driver
  to be installed.

- My patch is correct in any case -- regardless of actual driver
  behavior --, as both drivers are never intended to be installed at the
  same time.

- The stub driver fails with virt-v2v entirely out of the picture, the
  same way as it does after virt-v2v conversion.

- The complete driver fails with virt-v2v entirely out of the picture,
  the same way as it does after virt-v2v conversion. It's true that this
  failure mode differs from how the stub driver fails, but that's
  totally orthogonal to virt-v2v.

Summary:

(1) The *particular symptoms reported* are NOTABUG in virt-v2v.

(2) The problem should be reported for virtio-win. (I can imagine that
    the stub driver actually behaves "by design" -- "No drivers are
    installed for this device" might just be the expected message for a
    *stub* driver. The same argument probably does not apply to the
    complete fwcfg driver, which does produce an exclamation mark and
    Code 10 in Device Manager.)

(3) We might want to report an issue for osinfo-db, so that it learn
    about Win2022, and that it learn about the complete fwcfg driver.

(4) We should still upstream my patch (because both drivers should never
    be installed together). If necessary, we can reuse this BZ for that
    purpose.

(In reply to Laszlo Ersek from comment #15)

> (1) The *particular symptoms reported* are NOTABUG in virt-v2v.
> 
> (2) The problem should be reported for virtio-win. (I can imagine that
>     the stub driver actually behaves "by design" -- "No drivers are
>     installed for this device" might just be the expected message for a
>     *stub* driver. The same argument probably does not apply to the
>     complete fwcfg driver, which does produce an exclamation mark and
>     Code 10 in Device Manager.)

https://bugzilla.redhat.com/show_bug.cgi?id=2153745

> (3) We might want to report an issue for osinfo-db, so that it learn
>     about Win2022, and that it learn about the complete fwcfg driver.

https://bugzilla.redhat.com/show_bug.cgi?id=2153741

> (4) We should still upstream my patch (because both drivers should never
>     be installed together). If necessary, we can reuse this BZ for that
>     purpose.

[v2v PATCH] windows_virtio: favor "fwcfg" over "qemufwcfg"
Message-Id: <20221215111544.18511-1-lersek>
https://listman.redhat.com/archives/libguestfs/2022-December/030395.html

(In reply to Laszlo Ersek from comment #18)

> [v2v PATCH] windows_virtio: favor "fwcfg" over "qemufwcfg"
> Message-Id: <20221215111544.18511-1-lersek>
> https://listman.redhat.com/archives/libguestfs/2022-December/030395.html

Merged upstream as commit 0596edf29aa4.

Tried with the versions:

libguestfs-1.48.4-4.el9.x86_64
osinfo-db-20221130-1.el9.noarch
libosinfo-1.10.0-1.el9.x86_64
virt-v2v-2.2.0-1.el9.x86_64


Steps:
1. Convert win2022/win11/win2019 guests from VMware by v2v

# virt-v2v -ic vpx://root.212.149/data/10.73.212.36/?no_verify=1 -o rhv-upload -of raw -os nfs_data -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /v2v-ops/rhvpasswd  -oo rhv-cafile=/v2v-ops/ca22.pem -oo rhv-cluster=Default esx8.0-win11-x86_64 -it vddk -io vddk-libdir=/home/vddk8.0.0 -io vddk-thumbprint=D1:03:96:7E:11:3D:7C:4C:B6:50:28:1B:63:74:B5:40:5F:9D:9F:94 -ip /v2v-ops/esxpw -on esx8.0-win11-x86_64-rhv-upload -v -x |& tee > /2152465/win11-rhv-upload.log


2.  Start guests after finishing conversion. Check "Device Manager" -> "System devices" -> Show hidden devices:

Win2022: QEMU Fwfg Device(a yellow triangle with an exclamation mark over the icon): This device cannot start (Code 10) Bad data.   
Win2019: QEMU Fwfg Device(a yellow triangle with an exclamation mark over the icon): This device cannot start (Code 10) Bad data.
bz2153741

Win11: QEMU FWCfg Device: No drivers are installed for this device.

3. Check qemufwcfg info in v2v debug log, v2v installs qemufwcfg drivers for windows guests during conversion

# grep 'windows: skipping qemufwcfg stub driver in favor of fwcfg driver' win2022-rhv-upload.log
windows: skipping qemufwcfg stub driver in favor of fwcfg driver
 
# grep 'windows: skipping qemufwcfg stub driver in favor of fwcfg driver' win2019-rhv-upload.log
windows: skipping qemufwcfg stub driver in favor of fwcfg driver

# grep 'windows: skipping qemufwcfg stub driver in favor of fwcfg driver' win11-rhv-upload.log
#

Marking as Verified:Tested.

Tried with the versions:

libguestfs-1.48.4-4.el9.x86_64
osinfo-db-20221130-1.el9.noarch
libosinfo-1.10.0-1.el9.x86_64
virt-v2v-2.2.0-4.el9.x86_64
virtio-win-1.9.32-0.el9_1.noarch

Steps:
1. Convert win2022/win11/win2019 guests from VMware by v2v

# virt-v2v -ic vpx://root.213.207/data/10.73.212.38/?no_verify=1 -o rhv-upload -of raw -os nfs_data -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /v2v-ops/rhvpasswd  -oo rhv-cafile=/v2v-ops/ca22.pem -oo rhv-cluster=Default esx7.0-win2019-x86_64 -it vddk -io vddk-libdir=/home/vddk7.0.3 -io vddk-thumbprint=09:9E:54:CF:C3:36:11:9D:7D:B6:45:E0:85:74:4D:DB:CE:24:7B:46 -ip /v2v-ops/esxpw -on esx7.0-win2019-x86_64-rhv-upload -v -x |& tee >  ./win2019.log

2.  Start guests after finishing the conversion. Check "Device Manager" -> "System devices" -> Show hidden devices:

Win2022: QEMU Fwfg Device(a yellow triangle with an exclamation mark over the icon): This device cannot start (Code 10) Bad data.   
Win2019: QEMU Fwfg Device(a yellow triangle with an exclamation mark over the icon): This device cannot start (Code 10) Bad data.
Win11: QEMU FWCfg Device: No drivers are installed for this device.

3. Check qemufwcfg info in v2v debug log, v2v installs qemufwcfg drivers for windows guests during conversion

# grep 'windows: skipping qemufwcfg stub driver in favor of fwcfg driver' win11.log 
# 
# grep 'windows: skipping qemufwcfg stub driver in favor of fwcfg driver' win2019.log 
windows: skipping qemufwcfg stub driver in favor of fwcfg driver
# grep 'windows: skipping qemufwcfg stub driver in favor of fwcfg driver' win2022.log 
windows: skipping qemufwcfg stub driver in favor of fwcfg driver

Marking as Verified.

Tried with the versions:

libguestfs-1.48.4-4.el9.x86_64
osinfo-db-20221130-1.el9.noarch
libosinfo-1.10.0-1.el9.x86_64
virt-v2v-2.2.0-4.el9.x86_64
virtio-win-1.9.32-0.el9_1.noarch

Steps:
1. Convert win2022/win11/win2019 guests from VMware by v2v

# virt-v2v -ic vpx://root.213.207/data/10.73.212.38/?no_verify=1 -o rhv-upload -of raw -os nfs_data -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /v2v-ops/rhvpasswd  -oo rhv-cafile=/v2v-ops/ca22.pem -oo rhv-cluster=Default esx7.0-win2019-x86_64 -it vddk -io vddk-libdir=/home/vddk7.0.3 -io vddk-thumbprint=09:9E:54:CF:C3:36:11:9D:7D:B6:45:E0:85:74:4D:DB:CE:24:7B:46 -ip /v2v-ops/esxpw -on esx7.0-win2019-x86_64-rhv-upload -v -x |& tee >  ./win2019.log

2.  Start guests after finishing the conversion. Check "Device Manager" -> "System devices" -> Show hidden devices:

Win2022: QEMU Fwfg Device(a yellow triangle with an exclamation mark over the icon): This device cannot start (Code 10) Bad data.   
Win2019: QEMU Fwfg Device(a yellow triangle with an exclamation mark over the icon): This device cannot start (Code 10) Bad data.
Win11: QEMU FWCfg Device: No drivers are installed for this device.

3. Check qemufwcfg info in v2v debug log, v2v installs qemufwcfg drivers for windows guests during conversion

# grep 'windows: skipping qemufwcfg stub driver in favor of fwcfg driver' win11.log 
# 
# grep 'windows: skipping qemufwcfg stub driver in favor of fwcfg driver' win2019.log 
windows: skipping qemufwcfg stub driver in favor of fwcfg driver
# grep 'windows: skipping qemufwcfg stub driver in favor of fwcfg driver' win2022.log 
windows: skipping qemufwcfg stub driver in favor of fwcfg driver

Marking as Verified.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (virt-v2v bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:2313