RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1141145 - virt-v2v fails to convert xen pv guests.
Summary: virt-v2v fails to convert xen pv guests.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libguestfs
Version: 7.1
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Richard W.M. Jones
QA Contact: Virtualization Bugs
URL:
Whiteboard: V2V
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-09-12 10:33 UTC by tingting zheng
Modified: 2015-03-05 13:44 UTC (History)
8 users (show)

Fixed In Version: libguestfs-1.27.51-1.1.el7
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1141663 (view as bug list)
Environment:
Last Closed: 2015-03-05 13:44:36 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Detailed log file (414.13 KB, text/plain)
2014-09-16 07:27 UTC, tingting zheng
no flags Details
Debug info with xenblk error (416.32 KB, text/plain)
2014-09-17 03:02 UTC, tingting zheng
no flags Details
sceenshot of xen guest hang during boot (2.79 KB, image/png)
2014-09-18 06:04 UTC, tingting zheng
no flags Details
Log file of converesiong by virt-v2v (450.52 KB, text/plain)
2014-09-22 02:49 UTC, tingting zheng
no flags Details
test-broken-grub.pl (3.50 KB, text/plain)
2014-09-24 14:03 UTC, Richard W.M. Jones
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:0303 0 normal SHIPPED_LIVE libguestfs bug fix and enhancement update 2015-03-05 17:34:44 UTC

Description tingting zheng 2014-09-12 10:33:15 UTC
Description
virt-v2v fails to convert xen pv guests.

Version:
libguestfs-1.27.43-1.1.el7.x86_64
virt-v2v-1.27.43-1.1.el7.x86_64
libvirt-1.2.8-2.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Prepare a xen pv guest.
As virt-v2v manual said:Older versions of virt-v2v could turn a Xen paravirtualized (PV) guest into a KVM guest by installing anew kernel.  This version of virt-v2v does not attempt to install any new kernels.  Instead it will give you an error if there are only Xen PV kernels available.

So install kernel.
# rpm -qa |grep kernel
kernel-xen-2.6.18-308.el5
kernel-2.6.18-308.el5

2.Use virt-v2v to convert guest from xen.
# virt-v2v -ic xen+ssh://10.66.106.64 -os default  xen-pv-rhel5.8-x86_64
[   0.0] Opening the source -i libvirt -ic xen+ssh://10.66.106.64 xen-pv-rhel5.8-x86_64
libvirt: Remote Driver error : unknown procedure: 212
[  16.0] Creating an overlay to protect the source from being modified
[  30.0] Opening the overlay
[  52.0] Initializing the target -o libvirt -os default
[  52.0] Inspecting the overlay
[  86.0] Checking for sufficient free disk space in the guest
[  86.0] Converting Red Hat Enterprise Linux Server release 5.8 (Tikanga) to run on KVM
virt-v2v: error: multiple files in /boot could be the initramfs matching 
kernel 2.6.18-308.el5.  This could be a bug in virt-v2v.

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

3.Check xen guest:
# ll /boot/
total 15037
-rw-r--r-- 1 root root   67541 Jan 27  2012 config-2.6.18-308.el5
-rw-r--r-- 1 root root   67182 Jan 27  2012 config-2.6.18-308.el5xen
drwxr-xr-x 2 root root    1024 May 11 23:29 grub
-rw------- 1 root root 3327788 May 11 23:29 initrd-2.6.18-308.el5.img
-rw------- 1 root root 3335566 Feb  2  2012 initrd-2.6.18-308.el5xen.img
drwx------ 2 root root   12288 Feb  2  2012 lost+found
-rw-r--r-- 1 root root  116635 Jan 27  2012 symvers-2.6.18-308.el5.gz
-rw-r--r-- 1 root root  116295 Jan 27  2012 symvers-2.6.18-308.el5xen.gz
-rw-r--r-- 1 root root 1275921 Jan 27  2012 System.map-2.6.18-308.el5
-rw-r--r-- 1 root root 1237606 Jan 27  2012 System.map-2.6.18-308.el5xen
-rw-r--r-- 1 root root 2115644 Jan 27  2012 vmlinuz-2.6.18-308.el5
-rw-r--r-- 1 root root 2209057 Jan 27  2012 vmlinuz-2.6.18-308.el5xen
-rw-r--r-- 1 root root  427345 Jan 27  2012 xen.gz-2.6.18-308.el5
-rwxr-xr-x 1 root root  994384 Jan 27  2012 xen-syms-2.6.18-308.el5


Actual results:
As described.

Expected results:
virt-v2v can convert xen pv guest successfully.

Additional info:

Comment 2 Richard W.M. Jones 2014-09-12 15:04:37 UTC
I added this patch which should fix this:

https://github.com/libguestfs/libguestfs/commit/e1ed66e2b1c6b94bdfe10616667fdbc299ecb800

It'll be in 1.27.44.

If for some reason it doesn't fix it, can you add the -v -x options
and include the log file in your next report.

Comment 3 Richard W.M. Jones 2014-09-13 06:41:43 UTC
This bug needs QA ack.

Comment 4 tingting zheng 2014-09-15 02:34:09 UTC
(In reply to Richard W.M. Jones from comment #3)
> This bug needs QA ack.

Done.

Comment 5 tingting zheng 2014-09-15 03:26:02 UTC
Tested with:
virt-v2v-1.27.45-1.1.el7.x86_64
libguestfs-1.27.45-1.1.el7.x86_64

# virt-v2v -ic xen+ssh://10.66.106.64 -os default  xen-pv-rhel5.8-x86_64 -v -x 
[   0.0] Opening the source -i libvirt -ic xen+ssh://10.66.106.64 xen-pv-rhel5.8-x86_64
libvirt: Remote Driver error : unknown procedure: 212
virt-v2v: error: in the libvirt XML metadata, <os><type arch='...'> is 
missing or empty

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]
libvirt xml is:
<domain type='xen' id='5'>
  <name>xen-pv-rhel5.8-x86_64</name>
  <uuid>a25c918c-d838-6323-1944-6ae214992d5d</uuid>
  <memory>524288</memory>
  <currentMemory>524288</currentMemory>
  <vcpu>1</vcpu>
  <bootloader>/usr/bin/pygrub</bootloader>
  <os>
    <type>linux</type>
    <kernel>/var/lib/xen/boot_kernel.r5G2qO</kernel>
    <initrd>/var/lib/xen/boot_ramdisk.w1Y-8E</initrd>
    <cmdline>ro root=/dev/VolGroup00/LogVol00 rhgb quiet</cmdline>
  </os>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <disk type='file' device='disk'>
      <driver name='tap' type='aio'/>
      <source file='/var/lib/xen/images/xen/xen-pv/xen-pv-rhel5.8-x86_64.img'/>
      <target dev='xvda' bus='xen'/>
    </disk>
    <interface type='bridge'>
      <mac address='00:16:3e:f0:6c:2e'/>
      <source bridge='xenbr0'/>
      <script path='vif-bridge'/>
      <target dev='vif5.0'/>
    </interface>
    <console type='pty' tty='/dev/pts/1'>
      <source path='/dev/pts/1'/>
      <target port='0'/>
    </console>
    <input type='mouse' bus='xen'/>
    <graphics type='vnc' port='5900' autoport='yes' keymap='en-us'/>
  </devices>
</domain>

Comment 6 tingting zheng 2014-09-15 05:21:54 UTC
In comment 5,the guest is running,after I shutdown the guest,another error will show as below:
# virt-v2v -ic xen+ssh://10.66.106.64 -os default  xen-pv-rhel5.8-x86_64 -v -x |& tee /tmp/v2v-xenpv.log
[   0.0] Opening the source -i libvirt -ic xen+ssh://10.66.106.64 xen-pv-rhel5.8-x86_64
libvirt: Remote Driver error : unknown procedure: 212
virt-v2v: error: access: No such file or directory: 
/var/lib/xen/images/xen/xen-pv/xen-pv-rhel5.8-x86_64.img

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]
libvirt xml is:
<domain type='xen'>
  <name>xen-pv-rhel5.8-x86_64</name>
  <uuid>a25c918c-d838-6323-1944-6ae214992d5d</uuid>
  <memory>524288</memory>
  <currentMemory>524288</currentMemory>
  <vcpu>1</vcpu>
  <bootloader>/usr/bin/pygrub</bootloader>
  <os>
    <type arch='x86_64' machine='xenpv'>linux</type>
  </os>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <disk type='file' device='disk'>
      <driver name='tap' type='aio'/>
      <source file='/var/lib/xen/images/xen/xen-pv/xen-pv-rhel5.8-x86_64.img'/>
      <target dev='xvda' bus='xen'/>
    </disk>
    <interface type='bridge'>
      <mac address='00:16:3e:f0:6c:2e'/>
      <source bridge='xenbr0'/>
      <script path='vif-bridge'/>
    </interface>
    <console type='pty'>
      <target port='0'/>
    </console>
    <input type='mouse' bus='xen'/>
    <graphics type='vnc' port='-1' autoport='yes' keymap='en-us'/>
  </devices>
</domain>

Comment 7 tingting zheng 2014-09-15 07:36:02 UTC
Bug 1141654 records issue in comment 6.

Comment 9 tingting zheng 2014-09-16 07:25:51 UTC
Tested with:
libguestfs-1.27.46-1.1.el7.x86_64
virt-v2v-1.27.46-1.1.el7.x86_64

There is still error info shows:
virt-v2v: error: libguestfs error: command: grubby: bad argument
--set-kernel: unknown option

I will attach detailed debug info.

Comment 10 tingting zheng 2014-09-16 07:27:05 UTC
Created attachment 937871 [details]
Detailed log file

Refer to the above comments,move the bug back to ASSIGNED.

Comment 11 Richard W.M. Jones 2014-09-16 14:58:04 UTC
This should fix it:
https://github.com/libguestfs/libguestfs/commit/d86016da8e5d9afd01c75067198aecad15d402c9
Will be included in libguestfs >= 1.27.47.

Comment 13 tingting zheng 2014-09-17 03:00:43 UTC
Tested with:
libguestfs-1.27.47-1.1.el7.x86_64
virt-v2v-1.27.47-1.1.el7.x86_64

virt-v2v still fails with the below error,detailed debug info will be attached.
# virt-v2v -ic xen+ssh://10.66.106.64 -os default  xen-pv-rhel5.8-x86_64
[   0.0] Opening the source -i libvirt -ic xen+ssh://10.66.106.64 xen-pv-rhel5.8-x86_64
[  16.0] Creating an overlay to protect the source from being modified
[  30.0] Opening the overlay
[ 231.0] Initializing the target -o libvirt -os default
[ 232.0] Inspecting the overlay
[ 244.0] Checking for sufficient free disk space in the guest
[ 244.0] Estimating space required on target for each disk
[ 244.0] Converting Red Hat Enterprise Linux Server release 5.8 (Tikanga) to run on KVM
virt-v2v: error: libguestfs error: command: No module xenblk found for 
kernel 2.6.18-308.el5, aborting.

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Comment 14 tingting zheng 2014-09-17 03:02:05 UTC
Created attachment 938311 [details]
Debug info with xenblk error

Comment 17 tingting zheng 2014-09-18 06:03:30 UTC
Tested with:
# rpm -qa libguestfs virt-v2v
libguestfs-1.27.48-1.1.el7.x86_64
virt-v2v-1.27.48-1.1.el7.x86_64

xen pv guest with regular kernel installed can be converted successfully.
However,after conversion,the guest can not be booted successfully,refer to the screenshot.

Comment 18 tingting zheng 2014-09-18 06:04:10 UTC
Created attachment 938755 [details]
sceenshot of xen guest hang during boot

Comment 19 Richard W.M. Jones 2014-09-18 16:50:54 UTC
I tried to reproduce this and I'm afraid I could not.

What I did was:

(1) Installed RHEL 5.8 PV x86-64 guest on my RHEL 5 Xen host.

(2) I logged into the guest, registered it with RHN, and installed the
ordinary non-Xen kernel, ie:

  rhn_register
  yum install kernel

(3) Shut down the guest.

(4) Converted it with virt-v2v:

$ LIBGUESTFS_BACKEND=direct ./run virt-v2v -ic xen+ssh://root@v2v-rhel5xen rhel58-x86_64-pv -o local -os /var/tmp
[   0.0] Opening the source -i libvirt -ic xen+ssh://root@v2v-rhel5xen rhel58-x86_64-pv
[   1.0] Creating an overlay to protect the source from being modified
[   2.0] Opening the overlay
[  25.0] Initializing the target -o local -os /var/tmp
[  25.0] Inspecting the overlay
[  73.0] Checking for sufficient free disk space in the guest
[  73.0] Estimating space required on target for each disk
[  73.0] Converting Red Hat Enterprise Linux Server release 5.8 (Tikanga) to run on KVM
[ 244.0] Mapping filesystem data to avoid copying unused and blank areas
[ 250.0] Closing the overlay
[ 250.0] Copying disk 1/1 to /var/tmp/rhel58-x86_64-pv-sda (raw)
    (100.00/100%)
[2532.0] Creating output metadata
[2532.0] Finishing off

(5) I booted it using qemu-kvm.  NOTE I'm using virtio-blk drivers:

$ qemu-kvm -drive file=/var/tmp/rhel58-x86_64-pv-sda,if=virtio -m 1024 

It booted perfectly.

-----

How did you boot the guest?  Did you use qemu-kvm directly, or did you
use libvirt + 'virsh define' etc?

If you used qemu-kvm, what was the command line that you used?

Did you perform the same steps (1)-(5) as I did above, or did you
install or convert the guest in a different way?

Comment 20 tingting zheng 2014-09-19 03:21:53 UTC
(In reply to Richard W.M. Jones from comment #19)
> I tried to reproduce this and I'm afraid I could not.
> 
> What I did was:
> 
> (1) Installed RHEL 5.8 PV x86-64 guest on my RHEL 5 Xen host.
> 
> (2) I logged into the guest, registered it with RHN, and installed the
> ordinary non-Xen kernel, ie:
> 
>   rhn_register
>   yum install kernel

I downloaded kernel by myself:kernel-2.6.18-308.el5.x86_64.rpm,and then use rpm -ivh kernel-2.6.18-308.el5.x86_64.rpm to install kernel.


> 
> (3) Shut down the guest.
> 
> (4) Converted it with virt-v2v:
> 
> $ LIBGUESTFS_BACKEND=direct ./run virt-v2v -ic xen+ssh://root@v2v-rhel5xen
> rhel58-x86_64-pv -o local -os /var/tmp
> [   0.0] Opening the source -i libvirt -ic xen+ssh://root@v2v-rhel5xen
> rhel58-x86_64-pv
> [   1.0] Creating an overlay to protect the source from being modified
> [   2.0] Opening the overlay
> [  25.0] Initializing the target -o local -os /var/tmp
> [  25.0] Inspecting the overlay
> [  73.0] Checking for sufficient free disk space in the guest
> [  73.0] Estimating space required on target for each disk
> [  73.0] Converting Red Hat Enterprise Linux Server release 5.8 (Tikanga) to
> run on KVM
> [ 244.0] Mapping filesystem data to avoid copying unused and blank areas
> [ 250.0] Closing the overlay
> [ 250.0] Copying disk 1/1 to /var/tmp/rhel58-x86_64-pv-sda (raw)
>     (100.00/100%)
> [2532.0] Creating output metadata
> [2532.0] Finishing off

Almost the same way:
# virt-v2v -ic xen+ssh://10.66.106.64 -os default  xen-pv-rhel5.8-x86_64
[   0.0] Opening the source -i libvirt -ic xen+ssh://10.66.106.64 xen-pv-rhel5.8-x86_64
[  15.0] Creating an overlay to protect the source from being modified
[  32.0] Opening the overlay
[  54.0] Initializing the target -o libvirt -os default
[  54.0] Inspecting the overlay
[  67.0] Checking for sufficient free disk space in the guest
[  67.0] Estimating space required on target for each disk
[  67.0] Converting Red Hat Enterprise Linux Server release 5.8 (Tikanga) to run on KVM
[  85.0] Mapping filesystem data to avoid copying unused and blank areas
[  87.0] Closing the overlay
[  87.0] Copying disk 1/1 to /var/lib/libvirt/images/xen-pv-rhel5.8-x86_64-sda (raw)
    (100.00/100%)
[ 240.0] Creating output metadata
Pool default refreshed

Domain xen-pv-rhel5.8-x86_64 defined from /tmp/v2vlibvirtff6f58.xml

[ 240.0] Finishing off



> (5) I booted it using qemu-kvm.  NOTE I'm using virtio-blk drivers:
> 
> $ qemu-kvm -drive file=/var/tmp/rhel58-x86_64-pv-sda,if=virtio -m 1024 
> 
> It booted perfectly.
> 
> -----
> 
> How did you boot the guest?  Did you use qemu-kvm directly, or did you
> use libvirt + 'virsh define' etc?
> 
> If you used qemu-kvm, what was the command line that you used?
> 
> Did you perform the same steps (1)-(5) as I did above, or did you
> install or convert the guest in a different way?

I boot it from virsh,as the guest has been defined by virt-v2v,I just use virsh start to boot it.

The qemu command line shows as below:
qemu     21548     1 99 23:11 ?        00:04:48 /usr/libexec/qemu-kvm -name xen-pv-rhel5.8-x86_64 -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off -m 512 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid b618552f-eb42-4304-a67c-3af7d4e3ae59 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/xen-pv-rhel5.8-x86_64.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -no-acpi -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/libvirt/images/xen-pv-rhel5.8-x86_64-sda,if=none,id=drive-virtio-disk0,format=raw,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -vnc 127.0.0.1:1 -k en-us -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on
root     21625 18501  0 23:16 pts/1    00:00:00 grep --color=auto 27

Btw,I use the same way to install kernel/conversion for guest xen-pv-rhel5.8-i386,after conversion,the guest can be booted successfully.

Comment 21 Richard W.M. Jones 2014-09-19 11:43:16 UTC
I'm missing the virt-v2v -v -x output from the guest.  Can you
send me that first.  It looks as if I will (later) need the
whole disk image (from before conversion) too.

Comment 22 tingting zheng 2014-09-22 02:49:38 UTC
Created attachment 939863 [details]
Log file of converesiong by virt-v2v

Comment 23 Richard W.M. Jones 2014-09-22 17:31:51 UTC
Let's try this fix:

https://github.com/libguestfs/libguestfs/commit/cee9a4993d4435b57aa233b7b678a9b7cc5c1d53

I have tested a RHEL 5 guest conversion with this change, and it works
for me, but it also worked before.

Fix will be in virt-v2v >= 1.27.51.

Comment 24 tingting zheng 2014-09-23 07:30:28 UTC
Tested with:
libguestfs-1.27.52-1.1.el7.x86_64
virt-v2v-1.27.52-1.1.el7.x86_64

Convert the same xen pv guest to libvirt:
# virt-v2v -ic xen+ssh://10.66.106.64 -os default  xen-pv-rhel5.8-x86_64
[   0.0] Opening the source -i libvirt -ic xen+ssh://10.66.106.64 xen-pv-rhel5.8-x86_64
[  16.0] Creating an overlay to protect the source from being modified
[  30.0] Opening the overlay
[ 202.0] Initializing the target -o libvirt -os default
[ 202.0] Inspecting the overlay
[ 221.0] Checking for sufficient free disk space in the guest
[ 221.0] Estimating space required on target for each disk
[ 221.0] Converting Red Hat Enterprise Linux Server release 5.8 (Tikanga) to run on KVM
This guest has virtio drivers installed.
[ 238.0] Mapping filesystem data to avoid copying unused and blank areas
[ 239.0] Closing the overlay
[ 239.0] Copying disk 1/1 to /var/lib/libvirt/images/xen-pv-rhel5.8-x86_64-sda (raw)
    (100.00/100%)
[ 407.0] Creating output metadata
Pool default refreshed

Domain xen-pv-rhel5.8-x86_64 defined from /tmp/v2vlibvirt555b44.xml

[ 408.0] Finishing off

Guest can be converted successfully but it still fails to boot.

I checked my original guest on xen server,it can be booted successfully.

Btw,I converted another rhel5.8 guest on xen server,it can be booted successfully.

# virt-v2v -ic xen+ssh://10.66.106.64 -os default  pv-rhel58-nolvm
[   0.0] Opening the source -i libvirt -ic xen+ssh://10.66.106.64 pv-rhel58-nolvm
[  16.0] Creating an overlay to protect the source from being modified
[  30.0] Opening the overlay
[  52.0] Initializing the target -o libvirt -os default
[  52.0] Inspecting the overlay
[  76.0] Checking for sufficient free disk space in the guest
[  76.0] Estimating space required on target for each disk
[  76.0] Converting Red Hat Enterprise Linux Server release 5.8 (Tikanga) to run on KVM
This guest has virtio drivers installed.
[  90.0] Mapping filesystem data to avoid copying unused and blank areas
[  91.0] Closing the overlay
[  91.0] Copying disk 1/1 to /var/lib/libvirt/images/pv-rhel58-nolvm-sda (raw)
    (100.00/100%)
[ 216.0] Creating output metadata
Pool default refreshed

Domain pv-rhel58-nolvm defined from /tmp/v2vlibvirtfa2d3e.xml

I didn't say any other difference between these 2 rhel5.8 x86_64 guests except the failed one has no swap partition.

I think it's better you check the orininal image on xen server,I will send you a mail about the user/password of my xen server.

Comment 26 Richard W.M. Jones 2014-09-24 09:54:54 UTC
After painfully debugging the grub-legacy boot process, I have
identified the problem, although not yet why it happens.

Grub-legacy encodes the sector offset of the stage1.5 file
into the master boot record:

00000000  eb 48 90 10 8e d0 bc 00  b0 b8 00 00 8e d8 8e c0  |.H..............|
00000010  fb be 00 7c bf 00 06 b9  00 02 f3 a4 ea 21 06 00  |...|.........!..|
00000020  00 be be 07 38 04 75 0b  83 c6 10 81 fe fe 07 75  |....8.u........u|
00000030  f3 eb 16 b4 02 b0 01 bb  00 7c b2 80 8a 74 03 02  |.........|...t..|
00000040  80 00 00 80 41 94 01 00  00 08 fa 90 90 f6 c2 80  |....A...........|
                      |---------|
                      stage2_sector = 0x19441 (sectors)
                                    = 0x3288200 (bytes)
                                            |
    +---------- points to disk offset ------+
    |
    V
03288200  52 56 be 03 81 e8 28 01  5e bf f8 81 66 8b 2d 83  |RV....(.^...f.-.|
03288210  7d 04 00 0f 84 ca 00 80  7c ff 00 74 3e 66 8b 1d  |}.......|..t>f..|
03288220  66 31 c0 b0 7f 39 45 04  7f 03 8b 45 04 29 45 04  |f1...9E....E.)E.|
03288230  66 01 05 c7 04 10 00 89  44 02 66 89 5c 08 c7 44  |f.......D.f.\..D|
03288240  06 00 70 50 66 31 c0 89  44 04 66 89 44 0c b4 42  |..pPf1..D.f.D..B|
03288250  cd 13 0f 82 9f 00 bb 00  70 eb 56 66 8b 05 66 31  |........p.Vf..f1|
03288260  d2 66 f7 34 88 54 0a 66  31 d2 66 f7 74 04 88 54  |.f.4.T.f1.f.t..T|
03288270  0b 89 44 0c 3b 44 08 7d  74 8b 04 2a 44 0a 39 45  |..D.;D.}t..*D.9E|
03288280  04 7f 03 8b 45 04 29 45  04 66 01 05 8a 54 0d c0  |....E.)E.f...T..|
03288290  e2 06 8a 4c 0a fe c1 08  d1 8a 6c 0c 5a 52 8a 74  |...L......l.ZR.t|
032882a0  0b 50 bb 00 70 8e c3 31  db b4 02 cd 13 72 46 8c  |.P..p..1.....rF.|
032882b0  c3 8e 45 06 58 c1 e0 05  01 45 06 60 1e c1 e0 04  |..E.X....E.`....|
032882c0  89 c1 31 ff 31 f6 8e db  fc f3 a4 1f be 12 81 e8  |..1.1...........|
032882d0  5e 00 61 83 7d 04 00 0f  85 3c ff 83 ef 08 e9 2e  |^.a.}....<......|
032882e0  ff be 14 81 e8 49 00 5a  ea 00 82 00 00 be 17 81  |.....I.Z........|
032882f0  e8 3d 00 eb 06 be 1c 81  e8 35 00 be 21 81 e8 2f  |.=.......5..!../|
03288300  00 eb fe 4c 6f 61 64 69  6e 67 20 73 74 61 67 65  |...Loading stage|
03288310  32 00 2e 00 0d 0a 00 47  65 6f 6d 00 52 65 61 64  |2......Geom.Read|
03288320  00 20 45 72 72 6f 72 00  bb 01 00 b4 0e cd 10 46  |. Error........F|
03288330  8a 04 3c 00 75 f2 c3 00  00 00 00 00 00 00 00 00  |..<.u...........|
03288340  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

The above is taken from tzheng's original disk image.

After conversion using virt-v2v, the stage2_sector value in the MBR
is identical.  However the code is not at location 0x3288200 any
longer.  That disk offset seems to contain zeroes, and the file has
moved up a bit to offset 0x32a2200.

Essentially something must move the file (or perhaps the whole
partition...?) around.  I'm not sure what that is.

Comment 27 Richard W.M. Jones 2014-09-24 10:09:14 UTC
In fact the original disk image (before conversion) is broken, as you
can prove by the following simple test:

(1) Create a protective overlay over the original disk image:

qemu-img create -f qcow2 -b xen-pv-rhel5.8-x86_64.img overlay.qcow2

(2) Demonstrate that the disk image gets as far as grub:

qemu-kvm -m 1024 -drive file=overlay.qcow2,format=qcow2,if=ide

(Close qemu as soon as it gets to the red RHEL 5 grub screen)

(3) Zero out all free space in /boot:

guestfish -a overlay.qcow2 -i zero-free-space /boot

(4) Repeat step (2).  It will hang at the "GRUB _" prompt.

This demonstrates that the grub-legacy MBR contains a stage2_sector
offset that points to a deleted file (probably the file is /boot/grub/
e2fs_stage1_5).  It only happened to work by accident because although
the file is deleted, its contents remain on the disk.

However virt-v2v aggressively trims deleted files to reduce the amount
of data that we copy.  This is normally a safe thing to do.  In this
case though it breaks the boot which was accidentally working before.

Tingting: It would be interesting if you could reproduce how this
guest was originally created, so we can see if there is a root cause
bug in RHEL 5.

Comment 28 Richard W.M. Jones 2014-09-24 14:03:00 UTC
Created attachment 940786 [details]
test-broken-grub.pl

Attached is a program which you can run against a grub-legacy
disk image, and it will tell you if it is broken in the way
described in comment 26 and comment 27 above.

Typical usage (non-broken disk image):

$ /tmp/test-broken-grub.pl rhel58-x86_64-pv.img 
Product name: Red Hat Enterprise Linux Server release 5.8 (Tikanga)
Type/distro:  linux/rhel
Boot dev:     /dev/sda1
Boot fs:      ext3
Grub:         grub-legacy
OK: grub configuration in this guest is fine

Output when the grub-legacy is broken:

$ /tmp/test-broken-grub.pl /tmp/xen-pv-rhel5.8-x86_64.img
Product name: Red Hat Enterprise Linux Server release 5.8 (Tikanga)
Type/distro:  linux/rhel
Boot dev:     /dev/sda1
Boot fs:      ext3
Grub:         grub-legacy
BROKEN GUEST: grub configuration points to a deleted stage2 file!

I ran this on a bunch of random guests, and didn't find any broken ones.

Comment 29 tingting zheng 2014-09-25 06:48:56 UTC
(In reply to Richard W.M. Jones from comment #28)
> Created attachment 940786 [details]
> test-broken-grub.pl
> 
> Attached is a program which you can run against a grub-legacy
> disk image, and it will tell you if it is broken in the way
> described in comment 26 and comment 27 above.
> 
> Typical usage (non-broken disk image):
> 
> $ /tmp/test-broken-grub.pl rhel58-x86_64-pv.img 
> Product name: Red Hat Enterprise Linux Server release 5.8 (Tikanga)
> Type/distro:  linux/rhel
> Boot dev:     /dev/sda1
> Boot fs:      ext3
> Grub:         grub-legacy
> OK: grub configuration in this guest is fine
> 
> Output when the grub-legacy is broken:
> 
> $ /tmp/test-broken-grub.pl /tmp/xen-pv-rhel5.8-x86_64.img
> Product name: Red Hat Enterprise Linux Server release 5.8 (Tikanga)
> Type/distro:  linux/rhel
> Boot dev:     /dev/sda1
> Boot fs:      ext3
> Grub:         grub-legacy
> BROKEN GUEST: grub configuration points to a deleted stage2 file!
> 
> I ran this on a bunch of random guests, and didn't find any broken ones.


I downloaded the script and run to test the original xen guest,it shows as a broken guest.
original guest:
 ./test-broken-grub.pl /root/xen-pv-rhel5.8-x86_64.img 
Product name: Red Hat Enterprise Linux Server release 5.8 (Tikanga)
Type/distro:  linux/rhel
Boot dev:     /dev/sda1
Boot fs:      ext3
Grub:         grub-legacy
BROKEN GUEST: grub configuration points to a deleted stage2 file!

Comment 30 tingting zheng 2015-01-04 08:23:08 UTC
Tested with:
libguestfs-1.28.1-1.18.el7.x86_64
virt-v2v-1.28.1-1.18.el7.x86_64

(In reply to Richard W.M. Jones from comment #27)
> In fact the original disk image (before conversion) is broken, as you
> can prove by the following simple test:
> 
> (1) Create a protective overlay over the original disk image:
> 
> qemu-img create -f qcow2 -b xen-pv-rhel5.8-x86_64.img overlay.qcow2
> 
> (2) Demonstrate that the disk image gets as far as grub:
> 
> qemu-kvm -m 1024 -drive file=overlay.qcow2,format=qcow2,if=ide
> 
> (Close qemu as soon as it gets to the red RHEL 5 grub screen)
> 
> (3) Zero out all free space in /boot:
> 
> guestfish -a overlay.qcow2 -i zero-free-space /boot
> 
> (4) Repeat step (2).  It will hang at the "GRUB _" prompt.
> 
> This demonstrates that the grub-legacy MBR contains a stage2_sector
> offset that points to a deleted file (probably the file is /boot/grub/
> e2fs_stage1_5).  It only happened to work by accident because although
> the file is deleted, its contents remain on the disk.
> 
> However virt-v2v aggressively trims deleted files to reduce the amount
> of data that we copy.  This is normally a safe thing to do.  In this
> case though it breaks the boot which was accidentally working before.
> 
> Tingting: It would be interesting if you could reproduce how this
> guest was originally created, so we can see if there is a root cause
> bug in RHEL 5.

I tried to install a new rhel5.8 x86_64 pv guest and can not reproduce this broken grub one.

For the new guest,the grub configuration is fine.
# ./test-broken-grub.pl  rhel58.img 
Product name: Red Hat Enterprise Linux Server release 5.8 (Tikanga)
Type/distro:  linux/rhel
Boot dev:     /dev/sda1
Boot fs:      ext3
Grub:         grub-legacy
OK: grub configuration in this guest is fine

Use latest virt-v2v to convert the guest,guest can be converted and booted successfully.

Refer to comment 27,28,29,comment 17 is caused by broken grub.For guest with fine grub,virt-v2v can convert and boot guest successfully,so I think the issue described in this bug has been fixed,I will move the bug to VERIFIED,if related issues can be reproduced in the near future,I will create a new bug to track it.

Comment 32 errata-xmlrpc 2015-03-05 13:44:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-0303.html


Note You need to log in before you can comment on or make changes to this bug.