Bug 1141145 - virt-v2v fails to convert xen pv guests.
Summary: virt-v2v fails to convert xen pv guests.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libguestfs
Version: 7.1
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Richard W.M. Jones
QA Contact: Virtualization Bugs
URL:
Whiteboard: V2V
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-09-12 10:33 UTC by tingting zheng
Modified: 2015-03-05 13:44 UTC (History)
8 users (show)

Fixed In Version: libguestfs-1.27.51-1.1.el7
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1141663 (view as bug list)
Environment:
Last Closed: 2015-03-05 13:44:36 UTC
Target Upstream Version:


Attachments (Terms of Use)
Detailed log file (414.13 KB, text/plain)
2014-09-16 07:27 UTC, tingting zheng
no flags Details
Debug info with xenblk error (416.32 KB, text/plain)
2014-09-17 03:02 UTC, tingting zheng
no flags Details
sceenshot of xen guest hang during boot (2.79 KB, image/png)
2014-09-18 06:04 UTC, tingting zheng
no flags Details
Log file of converesiong by virt-v2v (450.52 KB, text/plain)
2014-09-22 02:49 UTC, tingting zheng
no flags Details
test-broken-grub.pl (3.50 KB, text/plain)
2014-09-24 14:03 UTC, Richard W.M. Jones
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:0303 0 normal SHIPPED_LIVE libguestfs bug fix and enhancement update 2015-03-05 17:34:44 UTC

Description tingting zheng 2014-09-12 10:33:15 UTC
Description
virt-v2v fails to convert xen pv guests.

Version:
libguestfs-1.27.43-1.1.el7.x86_64
virt-v2v-1.27.43-1.1.el7.x86_64
libvirt-1.2.8-2.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Prepare a xen pv guest.
As virt-v2v manual said:Older versions of virt-v2v could turn a Xen paravirtualized (PV) guest into a KVM guest by installing anew kernel.  This version of virt-v2v does not attempt to install any new kernels.  Instead it will give you an error if there are only Xen PV kernels available.

So install kernel.
# rpm -qa |grep kernel
kernel-xen-2.6.18-308.el5
kernel-2.6.18-308.el5

2.Use virt-v2v to convert guest from xen.
# virt-v2v -ic xen+ssh://10.66.106.64 -os default  xen-pv-rhel5.8-x86_64
[   0.0] Opening the source -i libvirt -ic xen+ssh://10.66.106.64 xen-pv-rhel5.8-x86_64
libvirt: Remote Driver error : unknown procedure: 212
[  16.0] Creating an overlay to protect the source from being modified
[  30.0] Opening the overlay
[  52.0] Initializing the target -o libvirt -os default
[  52.0] Inspecting the overlay
[  86.0] Checking for sufficient free disk space in the guest
[  86.0] Converting Red Hat Enterprise Linux Server release 5.8 (Tikanga) to run on KVM
virt-v2v: error: multiple files in /boot could be the initramfs matching 
kernel 2.6.18-308.el5.  This could be a bug in virt-v2v.

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

3.Check xen guest:
# ll /boot/
total 15037
-rw-r--r-- 1 root root   67541 Jan 27  2012 config-2.6.18-308.el5
-rw-r--r-- 1 root root   67182 Jan 27  2012 config-2.6.18-308.el5xen
drwxr-xr-x 2 root root    1024 May 11 23:29 grub
-rw------- 1 root root 3327788 May 11 23:29 initrd-2.6.18-308.el5.img
-rw------- 1 root root 3335566 Feb  2  2012 initrd-2.6.18-308.el5xen.img
drwx------ 2 root root   12288 Feb  2  2012 lost+found
-rw-r--r-- 1 root root  116635 Jan 27  2012 symvers-2.6.18-308.el5.gz
-rw-r--r-- 1 root root  116295 Jan 27  2012 symvers-2.6.18-308.el5xen.gz
-rw-r--r-- 1 root root 1275921 Jan 27  2012 System.map-2.6.18-308.el5
-rw-r--r-- 1 root root 1237606 Jan 27  2012 System.map-2.6.18-308.el5xen
-rw-r--r-- 1 root root 2115644 Jan 27  2012 vmlinuz-2.6.18-308.el5
-rw-r--r-- 1 root root 2209057 Jan 27  2012 vmlinuz-2.6.18-308.el5xen
-rw-r--r-- 1 root root  427345 Jan 27  2012 xen.gz-2.6.18-308.el5
-rwxr-xr-x 1 root root  994384 Jan 27  2012 xen-syms-2.6.18-308.el5


Actual results:
As described.

Expected results:
virt-v2v can convert xen pv guest successfully.

Additional info:

Comment 2 Richard W.M. Jones 2014-09-12 15:04:37 UTC
I added this patch which should fix this:

https://github.com/libguestfs/libguestfs/commit/e1ed66e2b1c6b94bdfe10616667fdbc299ecb800

It'll be in 1.27.44.

If for some reason it doesn't fix it, can you add the -v -x options
and include the log file in your next report.

Comment 3 Richard W.M. Jones 2014-09-13 06:41:43 UTC
This bug needs QA ack.

Comment 4 tingting zheng 2014-09-15 02:34:09 UTC
(In reply to Richard W.M. Jones from comment #3)
> This bug needs QA ack.

Done.

Comment 5 tingting zheng 2014-09-15 03:26:02 UTC
Tested with:
virt-v2v-1.27.45-1.1.el7.x86_64
libguestfs-1.27.45-1.1.el7.x86_64

# virt-v2v -ic xen+ssh://10.66.106.64 -os default  xen-pv-rhel5.8-x86_64 -v -x 
[   0.0] Opening the source -i libvirt -ic xen+ssh://10.66.106.64 xen-pv-rhel5.8-x86_64
libvirt: Remote Driver error : unknown procedure: 212
virt-v2v: error: in the libvirt XML metadata, <os><type arch='...'> is 
missing or empty

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]
libvirt xml is:
<domain type='xen' id='5'>
  <name>xen-pv-rhel5.8-x86_64</name>
  <uuid>a25c918c-d838-6323-1944-6ae214992d5d</uuid>
  <memory>524288</memory>
  <currentMemory>524288</currentMemory>
  <vcpu>1</vcpu>
  <bootloader>/usr/bin/pygrub</bootloader>
  <os>
    <type>linux</type>
    <kernel>/var/lib/xen/boot_kernel.r5G2qO</kernel>
    <initrd>/var/lib/xen/boot_ramdisk.w1Y-8E</initrd>
    <cmdline>ro root=/dev/VolGroup00/LogVol00 rhgb quiet</cmdline>
  </os>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <disk type='file' device='disk'>
      <driver name='tap' type='aio'/>
      <source file='/var/lib/xen/images/xen/xen-pv/xen-pv-rhel5.8-x86_64.img'/>
      <target dev='xvda' bus='xen'/>
    </disk>
    <interface type='bridge'>
      <mac address='00:16:3e:f0:6c:2e'/>
      <source bridge='xenbr0'/>
      <script path='vif-bridge'/>
      <target dev='vif5.0'/>
    </interface>
    <console type='pty' tty='/dev/pts/1'>
      <source path='/dev/pts/1'/>
      <target port='0'/>
    </console>
    <input type='mouse' bus='xen'/>
    <graphics type='vnc' port='5900' autoport='yes' keymap='en-us'/>
  </devices>
</domain>

Comment 6 tingting zheng 2014-09-15 05:21:54 UTC
In comment 5,the guest is running,after I shutdown the guest,another error will show as below:
# virt-v2v -ic xen+ssh://10.66.106.64 -os default  xen-pv-rhel5.8-x86_64 -v -x |& tee /tmp/v2v-xenpv.log
[   0.0] Opening the source -i libvirt -ic xen+ssh://10.66.106.64 xen-pv-rhel5.8-x86_64
libvirt: Remote Driver error : unknown procedure: 212
virt-v2v: error: access: No such file or directory: 
/var/lib/xen/images/xen/xen-pv/xen-pv-rhel5.8-x86_64.img

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]
libvirt xml is:
<domain type='xen'>
  <name>xen-pv-rhel5.8-x86_64</name>
  <uuid>a25c918c-d838-6323-1944-6ae214992d5d</uuid>
  <memory>524288</memory>
  <currentMemory>524288</currentMemory>
  <vcpu>1</vcpu>
  <bootloader>/usr/bin/pygrub</bootloader>
  <os>
    <type arch='x86_64' machine='xenpv'>linux</type>
  </os>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <disk type='file' device='disk'>
      <driver name='tap' type='aio'/>
      <source file='/var/lib/xen/images/xen/xen-pv/xen-pv-rhel5.8-x86_64.img'/>
      <target dev='xvda' bus='xen'/>
    </disk>
    <interface type='bridge'>
      <mac address='00:16:3e:f0:6c:2e'/>
      <source bridge='xenbr0'/>
      <script path='vif-bridge'/>
    </interface>
    <console type='pty'>
      <target port='0'/>
    </console>
    <input type='mouse' bus='xen'/>
    <graphics type='vnc' port='-1' autoport='yes' keymap='en-us'/>
  </devices>
</domain>

Comment 7 tingting zheng 2014-09-15 07:36:02 UTC
Bug 1141654 records issue in comment 6.

Comment 9 tingting zheng 2014-09-16 07:25:51 UTC
Tested with:
libguestfs-1.27.46-1.1.el7.x86_64
virt-v2v-1.27.46-1.1.el7.x86_64

There is still error info shows:
virt-v2v: error: libguestfs error: command: grubby: bad argument
--set-kernel: unknown option

I will attach detailed debug info.

Comment 10 tingting zheng 2014-09-16 07:27:05 UTC
Created attachment 937871 [details]
Detailed log file

Refer to the above comments,move the bug back to ASSIGNED.

Comment 11 Richard W.M. Jones 2014-09-16 14:58:04 UTC
This should fix it:
https://github.com/libguestfs/libguestfs/commit/d86016da8e5d9afd01c75067198aecad15d402c9
Will be included in libguestfs >= 1.27.47.

Comment 13 tingting zheng 2014-09-17 03:00:43 UTC
Tested with:
libguestfs-1.27.47-1.1.el7.x86_64
virt-v2v-1.27.47-1.1.el7.x86_64

virt-v2v still fails with the below error,detailed debug info will be attached.
# virt-v2v -ic xen+ssh://10.66.106.64 -os default  xen-pv-rhel5.8-x86_64
[   0.0] Opening the source -i libvirt -ic xen+ssh://10.66.106.64 xen-pv-rhel5.8-x86_64
[  16.0] Creating an overlay to protect the source from being modified
[  30.0] Opening the overlay
[ 231.0] Initializing the target -o libvirt -os default
[ 232.0] Inspecting the overlay
[ 244.0] Checking for sufficient free disk space in the guest
[ 244.0] Estimating space required on target for each disk
[ 244.0] Converting Red Hat Enterprise Linux Server release 5.8 (Tikanga) to run on KVM
virt-v2v: error: libguestfs error: command: No module xenblk found for 
kernel 2.6.18-308.el5, aborting.

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Comment 14 tingting zheng 2014-09-17 03:02:05 UTC
Created attachment 938311 [details]
Debug info with xenblk error

Comment 17 tingting zheng 2014-09-18 06:03:30 UTC
Tested with:
# rpm -qa libguestfs virt-v2v
libguestfs-1.27.48-1.1.el7.x86_64
virt-v2v-1.27.48-1.1.el7.x86_64

xen pv guest with regular kernel installed can be converted successfully.
However,after conversion,the guest can not be booted successfully,refer to the screenshot.

Comment 18 tingting zheng 2014-09-18 06:04:10 UTC
Created attachment 938755 [details]
sceenshot of xen guest hang during boot

Comment 19 Richard W.M. Jones 2014-09-18 16:50:54 UTC
I tried to reproduce this and I'm afraid I could not.

What I did was:

(1) Installed RHEL 5.8 PV x86-64 guest on my RHEL 5 Xen host.

(2) I logged into the guest, registered it with RHN, and installed the
ordinary non-Xen kernel, ie:

  rhn_register
  yum install kernel

(3) Shut down the guest.

(4) Converted it with virt-v2v:

$ LIBGUESTFS_BACKEND=direct ./run virt-v2v -ic xen+ssh://root@v2v-rhel5xen rhel58-x86_64-pv -o local -os /var/tmp
[   0.0] Opening the source -i libvirt -ic xen+ssh://root@v2v-rhel5xen rhel58-x86_64-pv
[   1.0] Creating an overlay to protect the source from being modified
[   2.0] Opening the overlay
[  25.0] Initializing the target -o local -os /var/tmp
[  25.0] Inspecting the overlay
[  73.0] Checking for sufficient free disk space in the guest
[  73.0] Estimating space required on target for each disk
[  73.0] Converting Red Hat Enterprise Linux Server release 5.8 (Tikanga) to run on KVM
[ 244.0] Mapping filesystem data to avoid copying unused and blank areas
[ 250.0] Closing the overlay
[ 250.0] Copying disk 1/1 to /var/tmp/rhel58-x86_64-pv-sda (raw)
    (100.00/100%)
[2532.0] Creating output metadata
[2532.0] Finishing off

(5) I booted it using qemu-kvm.  NOTE I'm using virtio-blk drivers:

$ qemu-kvm -drive file=/var/tmp/rhel58-x86_64-pv-sda,if=virtio -m 1024 

It booted perfectly.

-----

How did you boot the guest?  Did you use qemu-kvm directly, or did you
use libvirt + 'virsh define' etc?

If you used qemu-kvm, what was the command line that you used?

Did you perform the same steps (1)-(5) as I did above, or did you
install or convert the guest in a different way?

Comment 20 tingting zheng 2014-09-19 03:21:53 UTC
(In reply to Richard W.M. Jones from comment #19)
> I tried to reproduce this and I'm afraid I could not.
> 
> What I did was:
> 
> (1) Installed RHEL 5.8 PV x86-64 guest on my RHEL 5 Xen host.
> 
> (2) I logged into the guest, registered it with RHN, and installed the
> ordinary non-Xen kernel, ie:
> 
>   rhn_register
>   yum install kernel

I downloaded kernel by myself:kernel-2.6.18-308.el5.x86_64.rpm,and then use rpm -ivh kernel-2.6.18-308.el5.x86_64.rpm to install kernel.


> 
> (3) Shut down the guest.
> 
> (4) Converted it with virt-v2v:
> 
> $ LIBGUESTFS_BACKEND=direct ./run virt-v2v -ic xen+ssh://root@v2v-rhel5xen
> rhel58-x86_64-pv -o local -os /var/tmp
> [   0.0] Opening the source -i libvirt -ic xen+ssh://root@v2v-rhel5xen
> rhel58-x86_64-pv
> [   1.0] Creating an overlay to protect the source from being modified
> [   2.0] Opening the overlay
> [  25.0] Initializing the target -o local -os /var/tmp
> [  25.0] Inspecting the overlay
> [  73.0] Checking for sufficient free disk space in the guest
> [  73.0] Estimating space required on target for each disk
> [  73.0] Converting Red Hat Enterprise Linux Server release 5.8 (Tikanga) to
> run on KVM
> [ 244.0] Mapping filesystem data to avoid copying unused and blank areas
> [ 250.0] Closing the overlay
> [ 250.0] Copying disk 1/1 to /var/tmp/rhel58-x86_64-pv-sda (raw)
>     (100.00/100%)
> [2532.0] Creating output metadata
> [2532.0] Finishing off

Almost the same way:
# virt-v2v -ic xen+ssh://10.66.106.64 -os default  xen-pv-rhel5.8-x86_64
[   0.0] Opening the source -i libvirt -ic xen+ssh://10.66.106.64 xen-pv-rhel5.8-x86_64
[  15.0] Creating an overlay to protect the source from being modified
[  32.0] Opening the overlay
[  54.0] Initializing the target -o libvirt -os default
[  54.0] Inspecting the overlay
[  67.0] Checking for sufficient free disk space in the guest
[  67.0] Estimating space required on target for each disk
[  67.0] Converting Red Hat Enterprise Linux Server release 5.8 (Tikanga) to run on KVM
[  85.0] Mapping filesystem data to avoid copying unused and blank areas
[  87.0] Closing the overlay
[  87.0] Copying disk 1/1 to /var/lib/libvirt/images/xen-pv-rhel5.8-x86_64-sda (raw)
    (100.00/100%)
[ 240.0] Creating output metadata
Pool default refreshed

Domain xen-pv-rhel5.8-x86_64 defined from /tmp/v2vlibvirtff6f58.xml

[ 240.0] Finishing off



> (5) I booted it using qemu-kvm.  NOTE I'm using virtio-blk drivers:
> 
> $ qemu-kvm -drive file=/var/tmp/rhel58-x86_64-pv-sda,if=virtio -m 1024 
> 
> It booted perfectly.
> 
> -----
> 
> How did you boot the guest?  Did you use qemu-kvm directly, or did you
> use libvirt + 'virsh define' etc?
> 
> If you used qemu-kvm, what was the command line that you used?
> 
> Did you perform the same steps (1)-(5) as I did above, or did you
> install or convert the guest in a different way?

I boot it from virsh,as the guest has been defined by virt-v2v,I just use virsh start to boot it.

The qemu command line shows as below:
qemu     21548     1 99 23:11 ?        00:04:48 /usr/libexec/qemu-kvm -name xen-pv-rhel5.8-x86_64 -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off -m 512 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid b618552f-eb42-4304-a67c-3af7d4e3ae59 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/xen-pv-rhel5.8-x86_64.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -no-acpi -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/libvirt/images/xen-pv-rhel5.8-x86_64-sda,if=none,id=drive-virtio-disk0,format=raw,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -vnc 127.0.0.1:1 -k en-us -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on
root     21625 18501  0 23:16 pts/1    00:00:00 grep --color=auto 27

Btw,I use the same way to install kernel/conversion for guest xen-pv-rhel5.8-i386,after conversion,the guest can be booted successfully.

Comment 21 Richard W.M. Jones 2014-09-19 11:43:16 UTC
I'm missing the virt-v2v -v -x output from the guest.  Can you
send me that first.  It looks as if I will (later) need the
whole disk image (from before conversion) too.

Comment 22 tingting zheng 2014-09-22 02:49:38 UTC
Created attachment 939863 [details]
Log file of converesiong by virt-v2v

Comment 23 Richard W.M. Jones 2014-09-22 17:31:51 UTC
Let's try this fix:

https://github.com/libguestfs/libguestfs/commit/cee9a4993d4435b57aa233b7b678a9b7cc5c1d53

I have tested a RHEL 5 guest conversion with this change, and it works
for me, but it also worked before.

Fix will be in virt-v2v >= 1.27.51.

Comment 24 tingting zheng 2014-09-23 07:30:28 UTC
Tested with:
libguestfs-1.27.52-1.1.el7.x86_64
virt-v2v-1.27.52-1.1.el7.x86_64

Convert the same xen pv guest to libvirt:
# virt-v2v -ic xen+ssh://10.66.106.64 -os default  xen-pv-rhel5.8-x86_64
[   0.0] Opening the source -i libvirt -ic xen+ssh://10.66.106.64 xen-pv-rhel5.8-x86_64
[  16.0] Creating an overlay to protect the source from being modified
[  30.0] Opening the overlay
[ 202.0] Initializing the target -o libvirt -os default
[ 202.0] Inspecting the overlay
[ 221.0] Checking for sufficient free disk space in the guest
[ 221.0] Estimating space required on target for each disk
[ 221.0] Converting Red Hat Enterprise Linux Server release 5.8 (Tikanga) to run on KVM
This guest has virtio drivers installed.
[ 238.0] Mapping filesystem data to avoid copying unused and blank areas
[ 239.0] Closing the overlay
[ 239.0] Copying disk 1/1 to /var/lib/libvirt/images/xen-pv-rhel5.8-x86_64-sda (raw)
    (100.00/100%)
[ 407.0] Creating output metadata
Pool default refreshed

Domain xen-pv-rhel5.8-x86_64 defined from /tmp/v2vlibvirt555b44.xml

[ 408.0] Finishing off

Guest can be converted successfully but it still fails to boot.

I checked my original guest on xen server,it can be booted successfully.

Btw,I converted another rhel5.8 guest on xen server,it can be booted successfully.

# virt-v2v -ic xen+ssh://10.66.106.64 -os default  pv-rhel58-nolvm
[   0.0] Opening the source -i libvirt -ic xen+ssh://10.66.106.64 pv-rhel58-nolvm
[  16.0] Creating an overlay to protect the source from being modified
[  30.0] Opening the overlay
[  52.0] Initializing the target -o libvirt -os default
[  52.0] Inspecting the overlay
[  76.0] Checking for sufficient free disk space in the guest
[  76.0] Estimating space required on target for each disk
[  76.0] Converting Red Hat Enterprise Linux Server release 5.8 (Tikanga) to run on KVM
This guest has virtio drivers installed.
[  90.0] Mapping filesystem data to avoid copying unused and blank areas
[  91.0] Closing the overlay
[  91.0] Copying disk 1/1 to /var/lib/libvirt/images/pv-rhel58-nolvm-sda (raw)
    (100.00/100%)
[ 216.0] Creating output metadata
Pool default refreshed

Domain pv-rhel58-nolvm defined from /tmp/v2vlibvirtfa2d3e.xml

I didn't say any other difference between these 2 rhel5.8 x86_64 guests except the failed one has no swap partition.

I think it's better you check the orininal image on xen server,I will send you a mail about the user/password of my xen server.

Comment 26 Richard W.M. Jones 2014-09-24 09:54:54 UTC
After painfully debugging the grub-legacy boot process, I have
identified the problem, although not yet why it happens.

Grub-legacy encodes the sector offset of the stage1.5 file
into the master boot record:

00000000  eb 48 90 10 8e d0 bc 00  b0 b8 00 00 8e d8 8e c0  |.H..............|
00000010  fb be 00 7c bf 00 06 b9  00 02 f3 a4 ea 21 06 00  |...|.........!..|
00000020  00 be be 07 38 04 75 0b  83 c6 10 81 fe fe 07 75  |....8.u........u|
00000030  f3 eb 16 b4 02 b0 01 bb  00 7c b2 80 8a 74 03 02  |.........|...t..|
00000040  80 00 00 80 41 94 01 00  00 08 fa 90 90 f6 c2 80  |....A...........|
                      |---------|
                      stage2_sector = 0x19441 (sectors)
                                    = 0x3288200 (bytes)
                                            |
    +---------- points to disk offset ------+
    |
    V
03288200  52 56 be 03 81 e8 28 01  5e bf f8 81 66 8b 2d 83  |RV....(.^...f.-.|
03288210  7d 04 00 0f 84 ca 00 80  7c ff 00 74 3e 66 8b 1d  |}.......|..t>f..|
03288220  66 31 c0 b0 7f 39 45 04  7f 03 8b 45 04 29 45 04  |f1...9E....E.)E.|
03288230  66 01 05 c7 04 10 00 89  44 02 66 89 5c 08 c7 44  |f.......D.f.\..D|
03288240  06 00 70 50 66 31 c0 89  44 04 66 89 44 0c b4 42  |..pPf1..D.f.D..B|
03288250  cd 13 0f 82 9f 00 bb 00  70 eb 56 66 8b 05 66 31  |........p.Vf..f1|
03288260  d2 66 f7 34 88 54 0a 66  31 d2 66 f7 74 04 88 54  |.f.4.T.f1.f.t..T|
03288270  0b 89 44 0c 3b 44 08 7d  74 8b 04 2a 44 0a 39 45  |..D.;D.}t..*D.9E|
03288280  04 7f 03 8b 45 04 29 45  04 66 01 05 8a 54 0d c0  |....E.)E.f...T..|
03288290  e2 06 8a 4c 0a fe c1 08  d1 8a 6c 0c 5a 52 8a 74  |...L......l.ZR.t|
032882a0  0b 50 bb 00 70 8e c3 31  db b4 02 cd 13 72 46 8c  |.P..p..1.....rF.|
032882b0  c3 8e 45 06 58 c1 e0 05  01 45 06 60 1e c1 e0 04  |..E.X....E.`....|
032882c0  89 c1 31 ff 31 f6 8e db  fc f3 a4 1f be 12 81 e8  |..1.1...........|
032882d0  5e 00 61 83 7d 04 00 0f  85 3c ff 83 ef 08 e9 2e  |^.a.}....<......|
032882e0  ff be 14 81 e8 49 00 5a  ea 00 82 00 00 be 17 81  |.....I.Z........|
032882f0  e8 3d 00 eb 06 be 1c 81  e8 35 00 be 21 81 e8 2f  |.=.......5..!../|
03288300  00 eb fe 4c 6f 61 64 69  6e 67 20 73 74 61 67 65  |...Loading stage|
03288310  32 00 2e 00 0d 0a 00 47  65 6f 6d 00 52 65 61 64  |2......Geom.Read|
03288320  00 20 45 72 72 6f 72 00  bb 01 00 b4 0e cd 10 46  |. Error........F|
03288330  8a 04 3c 00 75 f2 c3 00  00 00 00 00 00 00 00 00  |..<.u...........|
03288340  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

The above is taken from tzheng's original disk image.

After conversion using virt-v2v, the stage2_sector value in the MBR
is identical.  However the code is not at location 0x3288200 any
longer.  That disk offset seems to contain zeroes, and the file has
moved up a bit to offset 0x32a2200.

Essentially something must move the file (or perhaps the whole
partition...?) around.  I'm not sure what that is.

Comment 27 Richard W.M. Jones 2014-09-24 10:09:14 UTC
In fact the original disk image (before conversion) is broken, as you
can prove by the following simple test:

(1) Create a protective overlay over the original disk image:

qemu-img create -f qcow2 -b xen-pv-rhel5.8-x86_64.img overlay.qcow2

(2) Demonstrate that the disk image gets as far as grub:

qemu-kvm -m 1024 -drive file=overlay.qcow2,format=qcow2,if=ide

(Close qemu as soon as it gets to the red RHEL 5 grub screen)

(3) Zero out all free space in /boot:

guestfish -a overlay.qcow2 -i zero-free-space /boot

(4) Repeat step (2).  It will hang at the "GRUB _" prompt.

This demonstrates that the grub-legacy MBR contains a stage2_sector
offset that points to a deleted file (probably the file is /boot/grub/
e2fs_stage1_5).  It only happened to work by accident because although
the file is deleted, its contents remain on the disk.

However virt-v2v aggressively trims deleted files to reduce the amount
of data that we copy.  This is normally a safe thing to do.  In this
case though it breaks the boot which was accidentally working before.

Tingting: It would be interesting if you could reproduce how this
guest was originally created, so we can see if there is a root cause
bug in RHEL 5.

Comment 28 Richard W.M. Jones 2014-09-24 14:03:00 UTC
Created attachment 940786 [details]
test-broken-grub.pl

Attached is a program which you can run against a grub-legacy
disk image, and it will tell you if it is broken in the way
described in comment 26 and comment 27 above.

Typical usage (non-broken disk image):

$ /tmp/test-broken-grub.pl rhel58-x86_64-pv.img 
Product name: Red Hat Enterprise Linux Server release 5.8 (Tikanga)
Type/distro:  linux/rhel
Boot dev:     /dev/sda1
Boot fs:      ext3
Grub:         grub-legacy
OK: grub configuration in this guest is fine

Output when the grub-legacy is broken:

$ /tmp/test-broken-grub.pl /tmp/xen-pv-rhel5.8-x86_64.img
Product name: Red Hat Enterprise Linux Server release 5.8 (Tikanga)
Type/distro:  linux/rhel
Boot dev:     /dev/sda1
Boot fs:      ext3
Grub:         grub-legacy
BROKEN GUEST: grub configuration points to a deleted stage2 file!

I ran this on a bunch of random guests, and didn't find any broken ones.

Comment 29 tingting zheng 2014-09-25 06:48:56 UTC
(In reply to Richard W.M. Jones from comment #28)
> Created attachment 940786 [details]
> test-broken-grub.pl
> 
> Attached is a program which you can run against a grub-legacy
> disk image, and it will tell you if it is broken in the way
> described in comment 26 and comment 27 above.
> 
> Typical usage (non-broken disk image):
> 
> $ /tmp/test-broken-grub.pl rhel58-x86_64-pv.img 
> Product name: Red Hat Enterprise Linux Server release 5.8 (Tikanga)
> Type/distro:  linux/rhel
> Boot dev:     /dev/sda1
> Boot fs:      ext3
> Grub:         grub-legacy
> OK: grub configuration in this guest is fine
> 
> Output when the grub-legacy is broken:
> 
> $ /tmp/test-broken-grub.pl /tmp/xen-pv-rhel5.8-x86_64.img
> Product name: Red Hat Enterprise Linux Server release 5.8 (Tikanga)
> Type/distro:  linux/rhel
> Boot dev:     /dev/sda1
> Boot fs:      ext3
> Grub:         grub-legacy
> BROKEN GUEST: grub configuration points to a deleted stage2 file!
> 
> I ran this on a bunch of random guests, and didn't find any broken ones.


I downloaded the script and run to test the original xen guest,it shows as a broken guest.
original guest:
 ./test-broken-grub.pl /root/xen-pv-rhel5.8-x86_64.img 
Product name: Red Hat Enterprise Linux Server release 5.8 (Tikanga)
Type/distro:  linux/rhel
Boot dev:     /dev/sda1
Boot fs:      ext3
Grub:         grub-legacy
BROKEN GUEST: grub configuration points to a deleted stage2 file!

Comment 30 tingting zheng 2015-01-04 08:23:08 UTC
Tested with:
libguestfs-1.28.1-1.18.el7.x86_64
virt-v2v-1.28.1-1.18.el7.x86_64

(In reply to Richard W.M. Jones from comment #27)
> In fact the original disk image (before conversion) is broken, as you
> can prove by the following simple test:
> 
> (1) Create a protective overlay over the original disk image:
> 
> qemu-img create -f qcow2 -b xen-pv-rhel5.8-x86_64.img overlay.qcow2
> 
> (2) Demonstrate that the disk image gets as far as grub:
> 
> qemu-kvm -m 1024 -drive file=overlay.qcow2,format=qcow2,if=ide
> 
> (Close qemu as soon as it gets to the red RHEL 5 grub screen)
> 
> (3) Zero out all free space in /boot:
> 
> guestfish -a overlay.qcow2 -i zero-free-space /boot
> 
> (4) Repeat step (2).  It will hang at the "GRUB _" prompt.
> 
> This demonstrates that the grub-legacy MBR contains a stage2_sector
> offset that points to a deleted file (probably the file is /boot/grub/
> e2fs_stage1_5).  It only happened to work by accident because although
> the file is deleted, its contents remain on the disk.
> 
> However virt-v2v aggressively trims deleted files to reduce the amount
> of data that we copy.  This is normally a safe thing to do.  In this
> case though it breaks the boot which was accidentally working before.
> 
> Tingting: It would be interesting if you could reproduce how this
> guest was originally created, so we can see if there is a root cause
> bug in RHEL 5.

I tried to install a new rhel5.8 x86_64 pv guest and can not reproduce this broken grub one.

For the new guest,the grub configuration is fine.
# ./test-broken-grub.pl  rhel58.img 
Product name: Red Hat Enterprise Linux Server release 5.8 (Tikanga)
Type/distro:  linux/rhel
Boot dev:     /dev/sda1
Boot fs:      ext3
Grub:         grub-legacy
OK: grub configuration in this guest is fine

Use latest virt-v2v to convert the guest,guest can be converted and booted successfully.

Refer to comment 27,28,29,comment 17 is caused by broken grub.For guest with fine grub,virt-v2v can convert and boot guest successfully,so I think the issue described in this bug has been fixed,I will move the bug to VERIFIED,if related issues can be reproduced in the near future,I will create a new bug to track it.

Comment 32 errata-xmlrpc 2015-03-05 13:44:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-0303.html


Note You need to log in before you can comment on or make changes to this bug.