Bug 2112801

Tingting, can you try starting it with a serial connection.  I think this
command should work:

# virsh start dell-per740-74-test --console

There may be additional information in the serial output which is not
printed on the virtual display.

If that doesn't help, then there should be even more information available
if you interrupt the boot at GRUB, and edit the boot command line:

-> Remove: rhgb quiet

-> Add: ignore_loglevel

Then hit Ctrl-x key to boot.

(In reply to Richard W.M. Jones from comment #3)
> Tingting, can you try starting it with a serial connection.  I think this
> command should work:
> 
> # virsh start dell-per740-74-test --console
> 
> There may be additional information in the serial output which is not
> printed on the virtual display.
> 
> If that doesn't help, then there should be even more information available
> if you interrupt the boot at GRUB, and edit the boot command line:
> 
> -> Remove: rhgb quiet
> 
> -> Add: ignore_loglevel
> 
> Then hit Ctrl-x key to boot.

Please also add:

  earlyprintk=ttyS0,115200n8 console=ttyS0,115200n8

In other words, the kernel command line should not include either "rhgb" or "quiet", and should include all of "ignore_loglevel earlyprintk=ttyS0,115200n8 console=ttyS0,115200n8".

Furthermore, please start a "screen" session at first, enable output capture with "Control-A Shift-H", then launch the domain with

  virsh start --console --paused dell-per740-74-test

and once you have a message that the console has been connected, *then* resume the domain with a "virsh resume" command from a different terminal.

This makes sure that zero log lines will be lost.

Now, to the actual problem. I've checked the conv log, and the code we put in place for bug 2076013 appears active. In the log and the output metadata, we have:

  gcaps_default_cpu = false

and

  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='16' threads='2'/>
  </cpu>

which look correct for a RHEL-9.1 guest (note "host-passthrough").

Hopefully the kernel log of the converted domain will tell us more.

(In reply to Laszlo Ersek from comment #4)
> (In reply to Richard W.M. Jones from comment #3)
> > Tingting, can you try starting it with a serial connection.  I think this
> > command should work:
> > 
> > # virsh start dell-per740-74-test --console
> > 
> > There may be additional information in the serial output which is not
> > printed on the virtual display.
> > 
> > If that doesn't help, then there should be even more information available
> > if you interrupt the boot at GRUB, and edit the boot command line:
> > 
> > -> Remove: rhgb quiet
> > 
> > -> Add: ignore_loglevel
> > 
> > Then hit Ctrl-x key to boot.
> 
> Please also add:
> 
>   earlyprintk=ttyS0,115200n8 console=ttyS0,115200n8
> 
> In other words, the kernel command line should not include either "rhgb" or
> "quiet", and should include all of "ignore_loglevel
> earlyprintk=ttyS0,115200n8 console=ttyS0,115200n8".

Attached the logs, pls refer to comment 6.

> 
> Furthermore, please start a "screen" session at first, enable output capture
> with "Control-A Shift-H", then launch the domain with

Tried this one, "Control-A Shift-H" didn't work.

> 
>   virsh start --console --paused dell-per740-74-test
> 
> and once you have a message that the console has been connected, *then*
> resume the domain with a "virsh resume" command from a different terminal.

Hi Tingting,

please ignore the "screen" problem; you managed to capture the guest output anyway, so "screen" is not relevant any longer.

Regarding the hang itself. From "physical.xml" in comment 2, I see:

    <disk type="network" device="disk">
      <driver name="qemu" type="raw"/>
      <source protocol="nbd">
        <host name="localhost" port="33505"/>
      </source>
      <target dev="sda"/>
    </disk>

that is, there is only one hard disk to export from the physical machine.

Then, from comment 6, we have:

[ TIME ] Timed out waiting for device [0;1;…er/rhel_dell--per740--74-home.
[DEPEND] Dependency failed for /home.
[DEPEND] Dependency failed for Local File Systems.
[DEPEND] Dependency failed for Mark… need to relabel after reboot.

In other words, the "home" logical volume cannot be mounted for some reason, and that breaks (via systemd dependencies) the rest of system boot.

This is quite strange, as the "swap" and "root" logical volumes can be mounted successfully!

From the conversion log in comment 2, I see:

+ lvm lvs
  LV   VG                  Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home rhel_dell-per740-74 -wi-a----- <1.02t                                                    
  root rhel_dell-per740-74 -wi-a----- 70.00g                                                    
  swap rhel_dell-per740-74 -wi-a-----  4.00g                                                    

The home local volume is like 1TB in size. The copying finishes in ~2 minutes however:

[ 647.1] Copying disk 1/1
[ 764.9] Creating output metadata

so the disk is either very sparse, or the network is super fast. Either way, this should not cause any problems.

One thing that surprises me is the following kernel command line option:

  resume=/dev/mapper/rhel_dell--per740--74-swap

The resume attempt is unsuccessful in the converted guest:

         Starting Resume from hiber…/rhel_dell--per740--74-swap...
[    3.249854] PM: Image not found (code -22)
[  OK  ] Finished Resume from hiber…er/rhel_dell--per740--74-swap.

which I think is expected, and normal.

so... I have no idea what could be wrong; can you perhaps provide more information about the filesystems on the source (physical) machine?

The thing we should *actually* analyze is why the systemd dependency times out; unfortunately, that's very difficult. I've always found systemd dependency failures virtually impossible to analyze. There never is an indication of what is waiting for what. Any ideas?

In the converted guest boot log, we see

[  OK  ] Found device /dev/mapper/rhel_dell--per740--74-root.
[  OK  ] Found device /dev/mapper/rhel_dell--per740--74-swap.

but not a similar message for "home".

The "home" LV seems to be found and even properly trimmed in the conversion log :/

If you start "guestfish" separately on the guest disk (after conversion), can you see the "home" logical volume, and the filesystem on it? Thanks.

No answers, just some random thoughts ...

(In reply to Laszlo Ersek from comment #9)
> [ TIME ] Timed out waiting for device [0;1;…er/rhel_dell--per740--74-home.
> [DEPEND] Dependency failed for /home.
> [DEPEND] Dependency failed for Local File Systems.
> [DEPEND] Dependency failed for Mark… need to relabel after reboot.
> 
> In other words, the "home" logical volume cannot be mounted for some reason,
> and that breaks (via systemd dependencies) the rest of system boot.

Don't discount some kind of disk corruption.  The path we use for
copying the disks in virt-v2v >= 2.0 is quite different from the one
in older virt-v2v and it's entirely possible it is broken in some way.

> One thing that surprises me is the following kernel command line option:
> 
>   resume=/dev/mapper/rhel_dell--per740--74-swap

We attempt to rewrite resume=... options from guests during conversion:
https://github.com/libguestfs/virt-v2v/blob/b13bb26341ec1daba2a06fedabec6d32de7a24bf/convert/convert_linux.ml#L1200

However the presence of this option in the first place does worry me
slightly.  Could it be that the guest was suspended?  In that case
could the filesystem on disk be not fully shutdown consistent, could
virt-v2v have replayed the filesystem log when mounting the disks
during conversion, and then could that have messed up the filesystem
completely when the guest was resumed after conversion?  It's a horrible
thought.  For Windows guests we require the guest to be fully shut
down before conversion (see information about "fast startup" in
the manual).

> so... I have no idea what could be wrong; can you perhaps provide more
> information about the filesystems on the source (physical) machine?

This information should be present in the conversion log.

(In reply to Richard W.M. Jones from comment #11)
> No answers, just some random thoughts ...
> 
> (In reply to Laszlo Ersek from comment #9)
> > [ TIME ] Timed out waiting for device [0;1;…er/rhel_dell--per740--74-home.
> > [DEPEND] Dependency failed for /home.
> > [DEPEND] Dependency failed for Local File Systems.
> > [DEPEND] Dependency failed for Mark… need to relabel after reboot.
> > 
> > In other words, the "home" logical volume cannot be mounted for some reason,
> > and that breaks (via systemd dependencies) the rest of system boot.
> 
> Don't discount some kind of disk corruption.  The path we use for
> copying the disks in virt-v2v >= 2.0 is quite different from the one
> in older virt-v2v and it's entirely possible it is broken in some way.
> 
> > One thing that surprises me is the following kernel command line option:
> > 
> >   resume=/dev/mapper/rhel_dell--per740--74-swap
> 
> We attempt to rewrite resume=... options from guests during conversion:
> https://github.com/libguestfs/virt-v2v/blob/
> b13bb26341ec1daba2a06fedabec6d32de7a24bf/convert/convert_linux.ml#L1200

Huh, I modified the neighborhood of that code just recently -- I hadn't realized what the "resume" regexp stood for.

> However the presence of this option in the first place does worry me
> slightly.  Could it be that the guest was suspended?  In that case
> could the filesystem on disk be not fully shutdown consistent, could
> virt-v2v have replayed the filesystem log when mounting the disks
> during conversion, and then could that have messed up the filesystem
> completely when the guest was resumed after conversion?  It's a horrible
> thought.  For Windows guests we require the guest to be fully shut
> down before conversion (see information about "fast startup" in
> the manual).

Indeed, I remember it!

> > so... I have no idea what could be wrong; can you perhaps provide more
> > information about the filesystems on the source (physical) machine?
> 
> This information should be present in the conversion log.

I've found:

libguestfs: trace: v2v: list_filesystems = ["/dev/sda1", "xfs", "/dev/rhel_dell-per740-74/home", "xfs", "/dev/rhel_dell-per740-74/root", "xfs", "/dev/rhel_dell-per740-74/swap", "swap"]

Seems pretty standard. I'll try to poke around in the disk image (from comment 12).

Thanks.

More thinking aloud:  I wondered if perhaps old virt-v2v removed
the resume=XXX entry on conversion (instead of rewriting the device
name) and that I'd made a mistake during transscription.
However I checked back over the old code:

https://github.com/rwmjones/old-virt-v2v/blob/da8b1187d3acda2aa9be2d4fdd07b5a78ce16d72/lib/Sys/VirtConvert/Converter/Linux.pm#L2700

and as far as I can tell it's doing the same as what we are doing now.
Maybe we *should* be removing resume=XXX if we find it in the grub
configuration.

I think I know what's going on.

After logging in to the converted guest, the root and swap LVs are active, but the home LV is nowhere to be seen. Additionally, when I run commands such as "pvs" and "lvs", I get complaints of the form

  Devices file sys_wwid naa.6b07b250f0ebff0029f1a9b404bcfffa PVID HSIfTMJIOu8SdKzfONSC7fjOWYA9AasV last seen on /dev/sda2 not found.

So, yes, this is another consequence of recent LVM using "/etc/lvm/devices/system.devices" as a list of devices that need (and suffice!) to be scanned for PV headers. In this guest, it contains:

> # LVM uses devices listed in this file.
> # Created by LVM command lvmdevices pid 3435 at Thu Jul 14 03:08:21 2022
> VERSION=1.1.2
> IDTYPE=sys_wwid IDNAME=naa.6b07b250f0ebff0029f1a9b404bcfffa DEVNAME=/dev/sda2 PVID=HSIfTMJIOu8SdKzfONSC7fjOWYA9AasV PART=2

where "naa.6b07b250f0ebff0029f1a9b404bcfffa" is the WWID <https://en.wikipedia.org/wiki/WWID> of the *physical* disk of the system that virt-p2v converted. Obviously this WWID is absent from the converted-to domain.

Note that, if I run lvs and pvs with the devices file disabled, I get totally sensible output:

# pvs --devicesfile='' --verbose
  PV         VG                  Fmt  Attr PSize PFree DevSize PV UUID                               
  /dev/vda2  rhel_dell-per740-74 lvm2 a--  1.09t    0    1.09t HSIfTM-JIOu-8SdK-zfON-SC7f-jOWY-A9AasV

# lvs --devicesfile='' --verbose
  LV   VG                  #Seg Attr       LSize  Maj Min KMaj KMin Pool Origin Data%  Meta%  Move Cpy%Sync Log Convert LV UUID                                LProfile
  home rhel_dell-per740-74    1 -wi------- <1.02t  -1  -1   -1   -1                                                     xSlK88-2IU3-rJeD-Dg0H-fmrf-rgBX-X8KBEH         
  root rhel_dell-per740-74    1 -wi-ao---- 70.00g  -1  -1  253    0                                                     SbExQF-m0AZ-SNXI-8OtR-lv13-zxXk-mJ1Khz         
  swap rhel_dell-per740-74    1 -wi-ao----  4.00g  -1  -1  253    1                                                     VejrfO-5yIa-OytB-bUhp-Q588-lCIp-2nEALs         

As you can see, in the "pvs" output, the PV UUID "HSIfTM-JIOu-8SdK-zfON-SC7f-jOWY-A9AasV" *does* match "PVID" in "/etc/lvm/devices/system.devices". Meaning that our conversion makes sense, it's just that we carry over "/etc/lvm/devices/system.devices", which -- incorrectly, for the conversion -- binds the PV to the physical disk "naa.6b07b250f0ebff0029f1a9b404bcfffa".

Now, we have in fact *earlier logic* in virt-v2v to prevent this from happening; refer to "convert/convert_linux.ml":

    (* Delete blkid caches if they exist, since they will refer to the old
     * device names.  blkid will rebuild these on demand.
     *
     * Delete the LVM cache since it will contain references to the
     * old devices (RHBZ#1164853).
     *)
    List.iter g#rm_f [
      "/etc/blkid/blkid.tab"; "/etc/blkid.tab";
      "/etc/lvm/cache/.cache"
    ];

This is what we now need to extend with the removal of "/etc/lvm/devices/system.devices".

(In reply to Laszlo Ersek from comment #15)
> I think I know what's going on.
> 
> After logging in to the converted guest,

by which I meant the *emergency shell* of the converted guest (obviously I can't log in normally, as the boot process never completes!)

Notably, when booting the guest with "debug", the journal contains entries such as:

systemd-udevd[958]: vda2: /usr/lib/udev/rules.d/69-dm-lvm.rules:19 LINK 'disk/by-id/lvm-pv-uuid-HSIfTM-JIOu-8SdK-zfON-SC7f-jOWY-A9AasV'
systemd-udevd[958]: vda2: /usr/lib/udev/rules.d/69-dm-lvm.rules:82 Importing properties from results of '/usr/sbin/lvm pvscan --cache --listvg --checkcomplete --vgonline --autoactivation event --udevoutput --journal=output /dev/vda2'
systemd-udevd[958]: vda2: Starting '/usr/sbin/lvm pvscan --cache --listvg --checkcomplete --vgonline --autoactivation event --udevoutput --journal=output /dev/vda2'
lvm[967]: /dev/vda2 excluded: device is not in devices file.
lvm[967]: WARNING: no autoactivation for /dev/vda2: not found in system.devices.

Two things of note are here:

- Udev invokes pvscan with "--cache", therefore the *intent* is to discover new devices. That's what libguestfs used to do as well as the *sole* action to disable device caching in lvm. However, this is no longer sufficient; "udev", just like libguestfs, should also pass "--devicesfile=''", so that the new devices file be ignored as well.

- lvm clearly reports that /dev/vda2 is excluded from auto-activation. :/

guestfish works alright on the disk image BTW, as we'd taught libguestfs to ignore the devices file.

(In reply to Laszlo Ersek from comment #15)
> Now, we have in fact *earlier logic* in virt-v2v to prevent this from
> happening; refer to "convert/convert_linux.ml":
> 
>     (* Delete blkid caches if they exist, since they will refer to the old
>      * device names.  blkid will rebuild these on demand.
>      *
>      * Delete the LVM cache since it will contain references to the
>      * old devices (RHBZ#1164853).
>      *)
>     List.iter g#rm_f [
>       "/etc/blkid/blkid.tab"; "/etc/blkid.tab";
>       "/etc/lvm/cache/.cache"
>     ];
> 
> This is what we now need to extend with the removal of
> "/etc/lvm/devices/system.devices".

Good one, yes I agree.  LVM will rebuild this file if it's missing?

Another note: this symptom seems specific to P2V (rather than pure V2V) because:

https://man7.org/linux/man-pages/man8/lvmdevices.8.html

       • sys_wwid uses the wwid reported by sysfs.  This is the first
         choice for non-virtual devices.

       • sys_serial uses the serial number reported by sysfs.  This is
         the second choice for non-virtual devices.

       • [...]

       • devname the device name is used if no other type applies.

In other words, the reason we've (apparently) not seen this thus far with pure virt-v2v is that the LVM devices file is not populated, either at all, or at least with a sys_wwid, when the *source* installation is on a virtual disk of some sort.

Here's the plan: I intend to build a patched virt-v2v package for QE, so they can retest the P2V conversion (using the same P2V ISO). If the conversion works, I'll submit the patch upstream.

(In reply to Richard W.M. Jones from comment #19)
> (In reply to Laszlo Ersek from comment #15)
> > Now, we have in fact *earlier logic* in virt-v2v to prevent this from
> > happening; refer to "convert/convert_linux.ml":
> > 
> >     (* Delete blkid caches if they exist, since they will refer to the old
> >      * device names.  blkid will rebuild these on demand.
> >      *
> >      * Delete the LVM cache since it will contain references to the
> >      * old devices (RHBZ#1164853).
> >      *)
> >     List.iter g#rm_f [
> >       "/etc/blkid/blkid.tab"; "/etc/blkid.tab";
> >       "/etc/lvm/cache/.cache"
> >     ];
> > 
> > This is what we now need to extend with the removal of
> > "/etc/lvm/devices/system.devices".
> 
> Good one, yes I agree.  LVM will rebuild this file if it's missing?

The answer to that seems nuanced <https://man7.org/linux/man-pages/man8/lvmdevices.8.html> (the whole thing is worth reading; for the purpose of this question, I'll quote):

"With no devices file, lvm will use any device on the system [...] If the system devices file does not yet exist, the pvcreate or vgcreate commands will create it if they see no existing VGs on the system.  lvmdevices --addev and vgimportdevices will always create a new devices file if it does not yet exist."

[v2v PATCH] convert_linux: remove LVM2 "devices file"
Message-Id: <20220805084426.9200-1-lersek>
https://listman.redhat.com/archives/libguestfs/2022-August/029577.html

(In reply to Laszlo Ersek from comment #24)
> [v2v PATCH] convert_linux: remove LVM2 "devices file"
> Message-Id: <20220805084426.9200-1-lersek>
> https://listman.redhat.com/archives/libguestfs/2022-August/029577.html

Upstream commit 8e4b732e8b43.

I'll build a package later.

After several rounds of testing, I find the root cause, summarize as below.

Tested with latest build:
virt-v2v-2.0.7-5.el9

1. If I didn't change vCPU number in virt-p2v window, eg:the orignial CPU is 32, then the default vCPU is set as 32. After conversion ,there is cpu related config in guest xml as below:

  <vcpu placement='static'>32</vcpu>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' cores='16' threads='2'/>
  </cpu>

The guest can be booted successfully with latest build, but the guest will hang with previous virt-v2v build, so I think the bug has been fixed.

2.If I change vCPU number in virt-p2v window, eg:the orignial CPU is 32, I set the vCPU as 8.I will get warning during conversion as below, pls refer Bug 1590721.
virt-v2v: warning: could not define libvirt domain: unsupported configuration: CPU topology doesn't match maximum vcpu count.
The libvirt XML is still available in ‘/tmp/v2vlibvirtc38452.xml’.  Try running ‘virsh -c qemu:///system define /tmp/v2vlibvirtc38452.xml’ yourself instead.

After conversion, I check the xml, the cpu config looks like:
  <vcpu>8</vcpu>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='16' threads='2'/>
  </cpu>

So I removed the cpu mode part and boot the guest, then the cpu config in xml looks like:
  <vcpu placement='static'>8</vcpu>
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>qemu64</model>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='lahf_lm'/>
    <feature policy='disable' name='svm'/>
  </cpu>

Then the guest will hang as comment 29.

3. If I change the cpu config in scenario 2 as below, the guest will booted successfully.
  <vcpu placement='static'>8</vcpu>
  <cpu mode='host-passthrough' check='none' migratable='on'/>


Refer to the above comments, the bug has been fixed and I move the bug to VERIFIED.

For Scenario 2, I am not sure whether it's a bug. I am not familiar with cpu feature and will contact cpu feature owner to check it, comments are welcomed if you know the answer.

> For Scenario 2, I am not sure whether it's a bug. I am not familiar with cpu feature and will contact cpu feature owner to check it, comments are welcomed if you know the answer.

It does sound like it's a bug, yes.  Basically we should never create
libvirt XML configuration that is inconsistent like that, and you should
never have to edit the XML.

(In reply to tingting zheng from comment #31)
> After several rounds of testing, I find the root cause, summarize as below.
> 
> Tested with latest build:
> virt-v2v-2.0.7-5.el9

……
> 
> 2.If I change vCPU number in virt-p2v window, eg:the orignial CPU is 32, I
> set the vCPU as 8.I will get warning during conversion as below, pls refer
> Bug 1590721.
> virt-v2v: warning: could not define libvirt domain: unsupported
> configuration: CPU topology doesn't match maximum vcpu count.
> The libvirt XML is still available in ‘/tmp/v2vlibvirtc38452.xml’.  Try
> running ‘virsh -c qemu:///system define /tmp/v2vlibvirtc38452.xml’ yourself
> instead.
> 
> After conversion, I check the xml, the cpu config looks like:
>   <vcpu>8</vcpu>
>   <cpu mode='host-passthrough'>
>     <topology sockets='1' cores='16' threads='2'/>
>   </cpu>
> 
> So I removed the cpu mode part and boot the guest, then the cpu config in
> xml looks like:
>   <vcpu placement='static'>8</vcpu>
>   <cpu mode='custom' match='exact' check='full'>
>     <model fallback='forbid'>qemu64</model>
>     <feature policy='require' name='x2apic'/>
>     <feature policy='require' name='hypervisor'/>
>     <feature policy='require' name='lahf_lm'/>
>     <feature policy='disable' name='svm'/>
>   </cpu>
> 
> Then the guest will hang as comment 29.
> 
> 3. If I change the cpu config in scenario 2 as below, the guest will booted
> successfully.
>   <vcpu placement='static'>8</vcpu>
>   <cpu mode='host-passthrough' check='none' migratable='on'/>
> 
> 
> Refer to the above comments, the bug has been fixed and I move the bug to
> VERIFIED.
> 
> For Scenario 2, I am not sure whether it's a bug. I am not familiar with cpu
> feature and will contact cpu feature owner to check it, comments are
> welcomed if you know the answer.

Confirm with cpu feature owner, qemu64 will be the default cpu model for guest if user doesn't supply a <cpu/> element,details pls refer to bug 1598162. But qemu64 is not official supported on RHEL8/RHEL9 guest, so the guest will hang or kernel panic. 

If I change the cpu model to host-passthrough or host-model, the guest can be booted successfully.

So for scenario 2, it's not related to virt-v2v, it's a libvirt known issue and no more bug is needed.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Low: virt-v2v security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7968