Created attachment 958293 [details]
boot log

Hi James, I need the virt-v2v -v -x output too.

Created attachment 958294 [details]
virt-v2v -v -x

I guess we must be screwing up /etc/fstab after conversion.  I'm
not quite sure how exactly.  Is it convenient to run this command
on the guest *after* conversion:

  virt-cat -a LPLCLDWA273-sda -a LPLCLDWA273-sdb -a LPLCLDWA273-sdc -a LPLCLDWA273-sdd /etc/fstab

[root@LPLCLDVTV01 LPLCLDWA273]# virt-cat -a LPLCLDWA273-sda -a LPLCLDWA273-sdb -a LPLCLDWA273-sdc -a LPLCLDWA273-sdd /etc/fstab

#
# /etc/fstab
# Created by anaconda on Fri May 27 23:01:08 2011
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/VG00-lv_root /                       ext4    defaults        1 1
/dev/mapper/VG00-lv_amex /amex                   ext4    defaults        1 2
UUID=902cf782-97ae-4576-b97d-1834b57727ff /boot                   ext4    defaults        1 2
/dev/mapper/VG00-lv_home /home                   ext4    defaults        1 2
/dev/mapper/VG00-lv_logs /logs                   ext4    defaults        1 2
/dev/mapper/VG00-lv_opt /opt                    ext4    defaults        1 2
/dev/mapper/VG00-lv_tmp /tmp                    ext4    defaults        1 2
/dev/mapper/VG00-lv_var /var                    ext4    defaults        1 2
/dev/mapper/VG00-lv_swap swap                    swap    defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0

Did it definitely boot before conversion?

I think I'm going to need you to send me this guest unfortunately.

 - - -

However if you don't want to do that, you could try 'virt-diff'-ing
the guest before and after to see what files changed (below I will
give a summary of what files *I* think virt-v2v changed).  To virt-diff
you'll have to unpack the OVA:

  $ tar xf LPLCLDWA273.ova
  $ gunzip *.vmdk.gz

and diff the before and after guests by doing:

  $ virt-diff -a LPLCLDWA273-disk1.vmdk -a LPLCLDWA273-disk2.vmdk \
              -a LPLCLDWA273-disk3.vmdk -a LPLCLDWA273-disk4.vmdk \
              -A LPLCLDWA273-sda -A LPLCLDWA273-sdb \
              -A LPLCLDWA273-sdc -A LPLCLDWA273-sdd | less

That will give you a diff-like output summarizing all the changes
made.

 - - -

I think that this virt-v2v run should only have changed the following
files:

 - Removed /var/lib/rpm/__db.00*
 - Edited /etc/fstab but probably it's identical (or whitespace only changes)
 - Edited /boot/grub/menu.lst but probably identical
 - Edited /boot/grub/device.map but probably identical
 - Edited /etc/sysconfig/grub but probably identical
 - Removed /etc/blkid/blkid.tab & /etc/blkid.tab
 - Added scsi_hostadapter entry to /etc/modprobe.d/virt-v2v-added.conf
 - Reran dracut

Out of those, I suspect only rerunning dracut would kill boot like
this (be nice to see full boot messages btw).  However dracut didn't
print any errors or warnings when it ran.

Working on uploading the ova. Attaching the virt-diff in the mean time.

Created attachment 958359 [details]
virt-diff of guest

I don't really have a strong theory about why this doesn't convert,
but I do have a couple of observations:

(1) It doesn't boot before conversion either, with the same error.

(2) The guest wasn't cleanly shutdown before it was exported to the
OVA.  Perhaps the guest had a sudden poweroff event, or maybe it was
running?  (Although I expect that VMware would prevent you from exporting
a guest which was running).

If the guest was not cleanly shutdown, then I would expect conversion
to work, but a clean shutdown would be preferable.  If the guest was
running, then the OVA could be inconsistent and all bets are off.

Could you go back and check that the guest really worked on the
origin?

If it works on the origin, I suggest ensuring it is cleanly shut
down, exporting it again to OVA, and converting it again, and
then checking that the conversion boots by hand, eg:

$ virt-v2v -i ova new.ova -o local -os /var/tmp -of raw
$ /usr/libexec/qemu-kvm -m 2048 -drive file=/var/tmp/LPLCLDWA273-sda,if=virtio -drive file=/var/tmp/LPLCLDWA273-sdb,if=virtio -drive file=/var/tmp/LPLCLDWA273-sdc,if=virtio -drive file=/var/tmp/LPLCLDWA273-sdd,if=virtio

Also: Is this a one-off problem, or do we have a class of guests
all exhibiting the same thing?

At the moment this is a one off thing. I've only been given 3 RHEL guests though, so 1/3 have this problem.

Rebooting got the errors back to what I had seen before. Still unable to find volume groups though. I'll attach the full boot log and I uploaded the new ova as 1164853-LPLCLDWA273_reboot.ova.

Created attachment 958790 [details]
full boot log

(In reply to jmighion from comment #13)
> Rebooting got the errors back to what I had seen before. Still unable to
> find volume groups though.

I'm not clear what you mean by this.  In comment 11 I was concerned
because the guest had not been shut down correctly before the OVA
was exported, as evidenced by the messages in the log (comment 4).
What does "Rebooting got the errors back to what I had seen before" mean?

I had seen the errors in the boot log previously but was no longer seeing them. It was probably because the guest was brought down incorrectly sometime. The root cause appears to be the same, no volume groups can be found.

Thanks to Bryn, I think this guest is hitting
https://bugzilla.redhat.com/show_bug.cgi?id=1165596

There's supposed to be a z-stream fix coming, and they should try
installing that fix before conversion.

(In reply to Richard W.M. Jones from comment #17)
> Thanks to Bryn, I think this guest is hitting
> https://bugzilla.redhat.com/show_bug.cgi?id=1165596
> 
> There's supposed to be a z-stream fix coming, and they should try
> installing that fix before conversion.

(The bug #1165596 is only a problem if lvmetad is used - global/use_lvmetad=1 in /etc/lvm/lvm.conf and lvm2-lvmetad service running. Please check if this is the case - lvmetad is not used by default in RHEL6 so the 1165596 and the solution is applicable only if configuration was changed to use the lvmetad.)

No, use_lvmetad is not set.

However there's some kind of LVM weirdness going on with this
guest (which is also the case before conversion).  I'm still
unclear on whether we have tested that this guest boots on
the source hypervisor.

It does boot on the source hypervisor.

Created attachment 959294 [details]
/etc/lvm/lvm.conf from the guest

Finally I was able to find a solution.  It makes no sense, but it works.

AFTER doing the conversion, you have to fix /etc/lvm/lvm.conf
'filter', like this:

$ guestfish -a LPLCLDWA273-sda \
            -a LPLCLDWA273-sdb \
            -a LPLCLDWA273-sdc \
            -a LPLCLDWA273-sdd -i
><fs> emacs /etc/lvm/lvm.conf

Locate the commented out line:

    # filter = [ "a/.*/" ]                                                      

UNCOMMENT this line.

Save the file, exit guestfish, and the guest will now boot.

It's my understanding that the default LVM filter should be "all
devices", and so my change to the configuration file should have
no effect.

Attached is the lvm.conf file before I made this edit.

(In reply to Richard W.M. Jones from comment #21)
> Created attachment 959294 [details]
> /etc/lvm/lvm.conf from the guest
> 
> Finally I was able to find a solution.  It makes no sense, but it works.

Thanks to Bryn we have a resolution: It's not uncommenting
the filter line which fixes the problem, it's the act of
touching lvm.conf.  This causes the file to be newer than
/etc/lvm/cache/.cache which causes LVM to ignore/rebuild its
cache.

The correct solution therefore is to delete /etc/lvm/cache/.cache
during conversion:

https://github.com/libguestfs/libguestfs/commit/63d67ac8ac6ae609514c995704a4dbd6f5ff5647

This fix will be in 1.28.1-1.13.el7

Tingting, can you QA ack this.  The fix is already included
in libguestfs-1.28.1-1.13.el7.

Tested on our end and their working. Thanks!

Hi jmighion,

I try to reproduce this issue with packages, but failed, please help me check the steps, thanks:

virt-v2v-1.28.1-1.12.el7.x86_64
libguestfs-1.28.1-1.12.el7.x86_64

Steps:
1. Use virt-v2v to convert RHEL image to libvirt image
# ls /home/test/
esx5.5-rhel6.6-x86_64.ova

# /usr/bin/virt-v2v  -i ova esx-rhel6-ova.tar  -o local  -of raw -os /var/tmp -on test1
[   0.0] Opening the source -i ova esx-rhel6-ova.tar
[   1.0] Creating an overlay to protect the source from being modified
[   2.0] Opening the overlay
[   5.0] Initializing the target -o local -os /var/tmp
[   5.0] Inspecting the overlay
[  15.0] Checking for sufficient free disk space in the guest
[  15.0] Estimating space required on target for each disk
[  15.0] Converting Red Hat Enterprise Linux Server release 6.5 (Santiago) to run on KVM
virt-v2v: warning: /files/boot/grub/device.map/hd0 references unknown 
device "sda".  You may have to fix this entry manually after conversion.
virt-v2v: warning: /files/etc/sysconfig/grub/boot references unknown device 
"sda".  You may have to fix this entry manually after conversion.
virt-v2v: This guest has virtio drivers installed.
[  38.0] Mapping filesystem data to avoid copying unused and blank areas
[  40.0] Closing the overlay
[  40.0] Copying disk 1/1 to /var/tmp/test1-sda (raw)
    (100.00/100%)
[  88.0] Creating output metadata
[  88.0] Finishing off

2. Then convert this converted disk to openstack:
~(keystone_admin)]# virt-v2v -i disk /var/tmp/test1-sda  -o glance 
[   0.0] Opening the source -i disk /var/tmp/test1-sda
[   0.0] Creating an overlay to protect the source from being modified
[   1.0] Opening the overlay
[   3.0] Initializing the target -o glance
[   4.0] Inspecting the overlay
[  12.0] Checking for sufficient free disk space in the guest
[  12.0] Estimating space required on target for each disk
[  12.0] Converting Red Hat Enterprise Linux Server release 6.5 (Santiago) to run on KVM
virt-v2v: warning: /files/boot/grub/device.map/hd0 references unknown 
device "sda".  You may have to fix this entry manually after conversion.
virt-v2v: warning: /files/etc/sysconfig/grub/boot references unknown device 
"sda".  You may have to fix this entry manually after conversion.
virt-v2v: This guest has virtio drivers installed.
[  37.0] Mapping filesystem data to avoid copying unused and blank areas
[  37.0] Closing the overlay
[  37.0] Copying disk 1/1 to /var/tmp/glance.RpNPDU/sda (raw)
    (100.00/100%)
[ 102.0] Creating output metadata
+------------------+--------------------------------------+
| Property         | Value                                |
+------------------+--------------------------------------+
| checksum         | 0bf19cf36369a7f4f324328b2d04b676     |
| container_format | bare                                 |
| created_at       | 2014-11-21T05:59:47                  |
| deleted          | False                                |
| deleted_at       | None                                 |
| disk_format      | raw                                  |
| id               | b227fd42-8941-4426-9ea4-8a44e8957818 |
| is_public        | False                                |
| min_disk         | 0                                    |
| min_ram          | 0                                    |
| name             | test1-sda                            |
| owner            | c3a9c07d56aa401b84da161f28cfc06f     |
| protected        | False                                |
| size             | 8589934592                           |
| status           | active                               |
| updated_at       | 2014-11-21T06:01:33                  |
| virtual_size     | None                                 |
+------------------+--------------------------------------+
+----------------------------+--------------------------------------+
| Property                   | Value                                |
+----------------------------+--------------------------------------+
| Property 'architecture'    | x86_64                               |
| Property 'hw_disk_bus'     | virtio                               |
| Property 'hw_vif_model'    | virtio                               |
| Property 'hypervisor_type' | kvm                                  |
| Property 'os_distro'       | rhel                                 |
| Property 'os_type'         | linux                                |
| Property 'os_version'      | 6.5                                  |
| Property 'vm_mode'         | hvm                                  |
| checksum                   | 0bf19cf36369a7f4f324328b2d04b676     |
| container_format           | bare                                 |
| created_at                 | 2014-11-21T05:59:47                  |
| deleted                    | False                                |
| deleted_at                 | None                                 |
| disk_format                | raw                                  |
| id                         | b227fd42-8941-4426-9ea4-8a44e8957818 |
| is_public                  | False                                |
| min_disk                   | 0                                    |
| min_ram                    | 2048                                 |
| name                       | test1-sda                            |
| owner                      | c3a9c07d56aa401b84da161f28cfc06f     |
| protected                  | False                                |
| size                       | 8589934592                           |
| status                     | active                               |
| updated_at                 | 2014-11-21T06:01:34                  |
| virtual_size               | None                                 |
+----------------------------+--------------------------------------+
[ 211.0] Finishing off

3. Boot up guest:
My result: guest can boot up correctly.

# ps -ef |grep qemu-kvm
qemu      5777     1 68 04:50 ?        00:00:37 /usr/libexec/qemu-kvm -name instance-00000010 -S -machine pc-i440fx-rhel7.1.0,accel=kvm,usb=off -cpu SandyBridge,+pdpe1gb,+osxsave,+dca,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -m 2048 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid 961ae588-c47e-4d38-bce7-b4b193d58668 -smbios type=1,manufacturer=Red Hat,product=OpenStack Compute,version=2014.1.3-4.el7ost,serial=36363131-3930-3643-5533-3431335a4e36,uuid=961ae588-c47e-4d38-bce7-b4b193d58668 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000010.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/nova/instances/961ae588-c47e-4d38-bce7-b4b193d58668/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=29 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:d6:4e:62,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/961ae588-c47e-4d38-bce7-b4b193d58668/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 0.0.0.0:0 -k en-us -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on



My question is:
Q1: my test guest just has one disk, does it matter to reproduce this issue, i see bug description has 4 disk totally.

Q2: what's the method you used to copy this converted disk to openstack server?
I just use
~(keystone_admin)]# virt-v2v -i disk /var/tmp/test1-sda  -o glance

And if something missed please tell me, thanks.

I'll offer a suggestion here.  You're only going to hit this bug if:

(a) the guest uses logical volumes

(b) the guest has LVs *other* than root & swap, ie, you need to add
extra LVs to initial install (they don't need to store anything,
they just have to exist and have a filesystem in them and be mentioned
in /etc/fstab)

(c) the timestamp on /etc/lvm/cache/.cache is newer than the
timestamp on /etc/lvm/lvm.conf.

Then when you do the conversion, LVM will first mount root and swap
successfully (it doesn't use the LVM cache for this).  But later it
will try to fsck your extra LV, but it will now be using the LVM
cache, which will contain a reference to the old disk (/dev/sdX),
and that's when it fails.

zhoujunqin it looks like Richard Jones already described what was missing from that test.
To answer your questions though, I don't think the guest has to have multiple disks. It just needs multiple LVs.
The way we're booting these guests on OpenStack is by creating a volume with cinder, which essentially leaves a shell volume-uuid on disk. We use virt-v2v to create a raw file (virt-v2v -v -x -i ova LPLCLDWA273.ova -o local -of raw). We then copy the raw file over the shell volume that cinder created and boot from that using nova boot --flavor uuid --block-device id=uuid,source=volume,dest=volume,bootindex=0 --block-device etc guest_name.

Hi jmighion and rjones,
thanks for your kind help, and with the attached ova file, i can reproduce this issue.
libguestfs-1.28.1-1.12.el7.x86_64
virt-v2v-1.28.1-1.12.el7.x86_64

Steps:
1. Use virt-v2v to convert RHEL image to libvirt image

# virt-v2v  -i ova 1164853-LPLCLDWA273_reboot.ova  -o local -os /var/tmp -of raw 
[   0.0] Opening the source -i ova 1164853-LPLCLDWA273_reboot.ova
[ 220.0] Creating an overlay to protect the source from being modified
[ 237.0] Opening the overlay
[ 243.0] Initializing the target -o local -os /var/tmp
[ 243.0] Inspecting the overlay
[ 249.0] Checking for sufficient free disk space in the guest
[ 249.0] Estimating space required on target for each disk
[ 249.0] Converting Red Hat Enterprise Linux Server release 6.6 (Santiago) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 278.0] Mapping filesystem data to avoid copying unused and blank areas
[ 281.0] Closing the overlay
[ 281.0] Copying disk 1/4 to /var/tmp/LPLCLDWA273-sda (raw)
    (100.00/100%)
[ 396.0] Copying disk 2/4 to /var/tmp/LPLCLDWA273-sdb (raw)
    (100.00/100%)
[ 406.0] Copying disk 3/4 to /var/tmp/LPLCLDWA273-sdc (raw)
    (100.00/100%)
[ 407.0] Copying disk 4/4 to /var/tmp/LPLCLDWA273-sdd (raw)
    (100.00/100%)
[ 407.0] Creating output metadata
[ 407.0] Finishing off

2. Use qemu-kvm command boot up guest directly:

#/usr/libexec/qemu-kvm -m 2048 -drive file=/var/tmp/LPLCLDWA273-sda,if=virtio -drive file=/var/tmp/LPLCLDWA273-sdb,if=virtio -drive file=/var/tmp/LPLCLDWA273-sdc,if=virtio -drive file=/var/tmp/LPLCLDWA273-sdd,if=virtio

3. The guest cannot boot because no volume group is found and the filesystem checks fail.

Then try to verify this bug with new build:
libguestfs-1.28.1-1.13.el7.x86_64
virt-v2v-1.28.1-1.13.el7.x86_64

1.
# virt-v2v -i ova 1164853-LPLCLDWA273_reboot.ova  -o local -os /var/tmp/ -of raw 
[   0.0] Opening the source -i ova 1164853-LPLCLDWA273_reboot.ova
[ 212.0] Creating an overlay to protect the source from being modified
[ 215.0] Opening the overlay
[ 228.0] Initializing the target -o local -os /var/tmp/
[ 228.0] Inspecting the overlay
[ 234.0] Checking for sufficient free disk space in the guest
[ 234.0] Estimating space required on target for each disk
[ 234.0] Converting Red Hat Enterprise Linux Server release 6.6 (Santiago) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 265.0] Mapping filesystem data to avoid copying unused and blank areas
[ 268.0] Closing the overlay
[ 268.0] Copying disk 1/4 to /var/tmp/LPLCLDWA273-sda (raw)
    (100.00/100%)
[ 379.0] Copying disk 2/4 to /var/tmp/LPLCLDWA273-sdb (raw)
    (100.00/100%)
[ 390.0] Copying disk 3/4 to /var/tmp/LPLCLDWA273-sdc (raw)
    (100.00/100%)
[ 390.0] Copying disk 4/4 to /var/tmp/LPLCLDWA273-sdd (raw)
    (100.00/100%)
[ 390.0] Creating output metadata
[ 391.0] Finishing off

2. After conversion, use qemu-kvm cmd boot up guest directly:

#/usr/libexec/qemu-kvm -m 2048 -drive file=/var/tmp/LPLCLDWA273-sda,if=virtio -drive file=/var/tmp/LPLCLDWA273-sdb,if=virtio -drive file=/var/tmp/LPLCLDWA273-sdc,if=virtio -drive file=/var/tmp/LPLCLDWA273-sdd,if=virtio

Result: Guest boot up with no error, we can use connect to it in another terminal via
# vncviewer localhost 

So move this bug from ON_QA to VERIFIED.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-0303.html