Bug 1138182

Summary: xen guest will be kernel panic after converted by virt-v2v
Product: Red Hat Enterprise Linux 7 Reporter: tingting zheng <tzheng>
Component: libguestfsAssignee: Richard W.M. Jones <rjones>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 7.1CC: codong, juzhou, mbooth, mzhan, ptoscano, rjones, tzheng
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: V2V
Fixed In Version: libguestfs-1.27.41-1.1.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-03-05 13:44:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
screenshot of xen guest kernel panic after converted by virt-v2v
none
Log file of converting xen guest
none
Log file of converting xen guest -new none

Description tingting zheng 2014-09-04 08:24:26 UTC
Created attachment 934325 [details]
screenshot of xen guest kernel panic after converted by virt-v2v

Description
xen guest will be kernel panic after converted by virt-v2v

Version:
libguestfs-tools-c-1.27.32-1.1.el7.x86_64
libguestfs-1.27.32-1.1.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Copy xen guest image and xml file from xen server to v2v server:

2.Run virt-v2v to connect a xen pv/hvm guest either by -i disk or -i libvirtxml.
# export LIBGUESTFS_BACKEND=direct
# virt-v2v -i disk -o local -os /var/tmp rhel6.6-pv-x64-test.img
[   0.0] Opening the source -i disk rhel6.6-pv-x64-test.img
[   0.0] Creating an overlay to protect the source from being modified
[   0.0] Opening the overlay
[   6.0] Initializing the target -o local -os /var/tmp
[   6.0] Inspecting the overlay
[  12.0] Converting Red Hat Enterprise Linux Server release 6.6 Beta (Santiago) to run on KVM
virt-v2v: warning: /files/boot/grub/device.map/hd0 references unknown
device "xvda".  You may have to fix this entry manually after
conversion.
[  35.0] Mapping filesystem data to avoid copying unused and blank areas
[  36.0] Closing the overlay
[  36.0] Copying disk 1/1 to /var/tmp/rhel6.6-pv-x64-test-sda (raw)
    (100.00/100%)
[ 170.0] Creating output metadata
[ 170.0] Finishing off

# virt-v2v -i libvirtxml  -os default xen-hvm-rhel6.4-x86_64.xml  -of qcow2
[   0.0] Opening the source -i libvirtxml xen-hvm-rhel6.4-x86_64.xml
[   0.0] Creating an overlay to protect the source from being modified
[   0.0] Opening the overlay
[   8.0] Initializing the target -o libvirt -os default
[   8.0] Inspecting the overlay
[  13.0] Converting Red Hat Enterprise Linux Server release 6.4 (Santiago) to run on KVM
virt-v2v: warning: /files/boot/grub/device.map/hd0 references unknown
device "xvda".  You may have to fix this entry manually after
conversion.
[  35.0] Mapping filesystem data to avoid copying unused and blank areas
[  35.0] Closing the overlay
[  36.0] Copying disk 1/1 to /var/lib/libvirt/images/xen-hvm-rhel6.4-x86_64-sda (qcow2)
    (100.00/100%)
[ 124.0] Creating output metadata
Pool default refreshed

Domain xen-hvm-rhel6.4-x86_64 defined from /tmp/v2vlibvirt7a7162.xml

[ 125.0] Finishing off

3.After conversion,try to boot the guest,it fails to boot with kernel panic:could not find /lib/modules/2.6.*/modules.dep:No such file or directory.

Actual results:
As description.

Expected results:
Guest can be booted successfully after converion by virt-v2v.

Comment 2 Richard W.M. Jones 2014-09-05 13:41:43 UTC
I'm not sure what the problem is here, but a couple of things
would help me to diagnose it further.

(0) Try virt-v2v 1.27.38 -- which hopefully I'll be uploading
later today.

(1) In virt-v2v 1.27.38 it should be possible to import directly
from the remote Xen hypervisor, ie:

  virt-v2v -ic xen+ssh://root@xen [...]

Read the section in the virt-v2v manual about "INPUT FROM RHEL 5 XEN"
first however.

(2) If it's still failing, can you add the -v -x options and
then give the complete output.  It will be very large, so
redirect it into a file like this:

  virt-v2v -v -x [.....] |& tee /tmp/log

Comment 3 tingting zheng 2014-09-09 06:55:48 UTC
(In reply to Richard W.M. Jones from comment #2)
> I'm not sure what the problem is here, but a couple of things
> would help me to diagnose it further.
> 
> (0) Try virt-v2v 1.27.38 -- which hopefully I'll be uploading
> later today.

I tried the below version,kernal panic again.
virt-v2v-1.27.39-1.1.el7.x86_64
libguestfs-1.27.39-1.1.el7.x86_64
libguestfs-tools-c-1.27.39-1.1.el7.x86_64

> (1) In virt-v2v 1.27.38 it should be possible to import directly
> from the remote Xen hypervisor, ie:
> 
>   virt-v2v -ic xen+ssh://root@xen [...]
> 
> Read the section in the virt-v2v manual about "INPUT FROM RHEL 5 XEN"
> first however.

Tried to use -ic xen+ssh://root@xen [...]
# sshkey-gen
# ssh-copy-id -i /root/.ssh/id_rsa.pub root@$xenserver
# eval `ssh-agent`
# ssh-add
# virt-v2v -v -x -ic xen+ssh://10.66.106.64 -os default --network default rhel6.6-pv-x64-test |& tee /tmp/v2v-xen.log

Conversion is successful,but guest still fails to boot,attach the log file for reference.

Comment 4 tingting zheng 2014-09-09 06:57:47 UTC
Created attachment 935557 [details]
Log file of converting xen guest

Comment 5 tingting zheng 2014-09-09 08:07:40 UTC
Created attachment 935572 [details]
Log file of converting xen guest -new

Pls ignore comment 4,I've uploaded the new log file.

Comment 6 Richard W.M. Jones 2014-09-09 14:25:26 UTC
The bug seems to be caused because we call dracut with the wrong
kernel version parameter.  Dracut itself prints an error:

/sbin/dracut --add-drivers "virtio virtio_ring virtio_blk virtio_net virtio_pci" \
  /boot/initramfs-2.6.32-496.el6.x86_64.img 2.6.32-496.el6
find: `/lib/modules/2.6.32-496.el6/': No such file or directory
find: `/lib/modules/2.6.32-496.el6/': No such file or directory
find: `/lib/modules/2.6.32-496.el6/': No such file or directory
find: `/lib/modules/2.6.32-496.el6/': No such file or directory
find: `/lib/modules/2.6.32-496.el6/': No such file or directory
find: `/lib/modules/2.6.32-496.el6/': No such file or directory

but doesn't actually exit with a detectable status code, and
therefore we continue, but the initramfs of the guest has been
destroyed, and hence the VM won't boot.

I was able to reproduce this problem easily simply by running virt-v2v
on a RHEL 6.5 guest image -- no Xen is needed.

The incorrect dracut parameter is fixed in this commit:

https://github.com/libguestfs/libguestfs/commit/59136518f13cb489cc17a411267d54444d7c8ac6

Unfortunately the second problem of dracut not exiting with
a useful status code is harder to fix.  What happens is that
it creates a corrupted initramfs (if it created nothing at
all, we could at least detect that).  Of course even fixing
dracut won't help since we cannot go back in time to fix
old guests with broken dracut/mkinitrd.

Comment 8 zhoujunqin 2014-09-10 11:08:19 UTC
I can reproduce this issue as bug description.

And try to verify it with new version:
virt-v2v-1.27.41-1.1.el7.x86_64
libguestfs-1.27.41-1.1.el7.x86_64

Tried to use -ic xen+ssh://root@xen [...]
# sshkey-gen
# ssh-copy-id -i /root/.ssh/id_rsa.pub root@$xenserver
# eval `ssh-agent`
# ssh-add
# virt-v2v -v -x -ic xen+ssh://10.66.106.64 -os default --network default rhel6.6-pv-x64-test 

Result: Conversion is successful, and guest successfully to boot up.
Move to VERIFIED.

Comment 10 errata-xmlrpc 2015-03-05 13:44:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-0303.html