Bug 1550168

Summary: virt-v2v -o rhv generated OVF causes oVirt to crash with NullPointerException in LibvirtVmXmlBuilder.writeRng if SpecParams is missing
Product: Red Hat Enterprise Linux 7 Reporter: Richard W.M. Jones <rjones>
Component: libguestfsAssignee: Richard W.M. Jones <rjones>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.5CC: ahadas, lsurette, mavital, mxie, mzhan, ptoscano, Rhev-m-bugs, rjones, srevivo, tgolembi, tzheng, xchen, ykaul
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: V2V
Fixed In Version: libguestfs-1.38.1-1.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1550123 Environment:
Last Closed: 2018-10-30 07:45:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Virt RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1550123, 1551055    
Bug Blocks:    
Attachments:
Description Flags
v2v-memory-rng-7.5.z.log none

Description Richard W.M. Jones 2018-02-28 17:09:29 UTC 
+++ This bug was initially created as a clone of Bug #1550123 +++

Description of problem:

libguestfs creates OVF XML which looks like this for some guests:

      <Item>
        <rasd:Caption>RNG Device</rasd:Caption>
        <rasd:InstanceId>feeae0f9-ce77-4df7-8daf-0d128ef0e0e2</rasd:InstanceId>
        <rasd:ResourceType>0</rasd:ResourceType>
        <Type>rng</Type>
        <Device>virtio</Device>
      </Item>

      <Item>
        <rasd:Caption>Memory Ballooning Device</rasd:Caption>
        <rasd:InstanceId>b7015425-3e07-4f69-bac2-29649b2acea3</rasd:InstanceId>
        <rasd:ResourceType>0</rasd:ResourceType>
        <Type>balloon</Type>
        <Device>memballoon</Device>
      </Item>

Unfortunately this XML causes oVirt 4.2 to crash, see bug 1550123 for details.

Version-Release number of selected component (if applicable):

libguestfs <= 1.38.0
ovirt-engine-4.2.2.2-0.0.master.20180225172203.gitd7cf125.el7.centos.noarch

How reproducible:

100%

Steps to Reproduce:
1. Convert a Fedora or RHEL 7 guest to oVirt 4.2 using -o rhv.
2. Try to boot the guest on the target oVirt.
3. Boot with fail with a strange (and bogus) error about:

"There are no hosts to use. Check that the cluster contains at least one host in Up state."

The real error can be seen in engine.log - it's a Java NullPointerException in oVirt.

Additional information:

Upstream fix:
https://www.redhat.com/archives/libguestfs/2018-February/msg00187.html
Comment 2 Pino Toscano 2018-04-11 16:23:57 UTC 
Fixed upstream with
https://github.com/libguestfs/libguestfs/commit/d7b31559c291bc8af3419679facdf174e6e647c1
which is in libguestfs >= 1.39.1.
Comment 3 Pino Toscano 2018-05-02 11:21:39 UTC 
Fix backported in 1.38.1.
Comment 5 mxie@redhat.com 2018-05-07 10:23:14 UTC 
Try to reproduce the bug with below builds:
virt-v2v-1.36.10-6.el7_5.2.x86_64
libguestfs-1.36.10-6.el7_5.2.x86_64

Reproduce steps:
Scenario1:
1.1 Prepare a rhel7 guest which has RNG device and Memory Ballooning Device
# virsh dumpxml rhel7.5
.... 
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </memballoon>
    <rng model='virtio'>
      <backend model='random'>/dev/urandom</backend>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/>
    </rng>
....

1.2 Convert guest to rhv4.2 by virt-v2v using -o rhv
# virt-v2v rhel7.5 -o rhv -os 10.66.144.40:/home/nfs_export -on memory-rng-7.5.z
[   0.0] Opening the source -i libvirt rhel7.5
[   0.0] Creating an overlay to protect the source from being modified
[   0.5] Initializing the target -o rhv -os 10.66.144.40:/home/nfs_export
[   0.9] Opening the overlay
[  31.3] Inspecting the overlay
[ 140.2] Checking for sufficient free disk space in the guest
[ 140.2] Estimating space required on target for each disk
[ 140.2] Converting Red Hat Enterprise Linux Server 7.5 (Maipo) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 260.3] Mapping filesystem data to avoid copying unused and blank areas
[ 260.9] Closing the overlay
[ 262.0] Checking if the guest needs BIOS or UEFI to boot
[ 262.0] Assigning disks to buses
[ 262.0] Copying disk 1/1 to /tmp/v2v.29t5Rn/6f2055e4-b776-4ffd-94e6-f444e4931b76/images/bcf165d2-9b27-48ed-8e5c-7b84ebd154a3/fcb2ea87-15bb-42f1-b5b7-803293f88bb0 (qcow2)
    (100.00/100%)
[ 364.6] Creating output metadata
[ 364.7] Finishing off

1.3 Check converted guest's ovf after finishing conversion but can't find RNG deivce and memory Ballooning Device, meanwhile,
# cat 7fff3b26-5325-4116-b606-acaaf1f5ddcd/7fff3b26-5325-4116-b606-acaaf1f5ddcd.ovf 


Sceanario2:
2.1 Add RND device and memory Ballooning Device info according to comment0 into converted guest's ovf manually after step1.2
# vi 7fff3b26-5325-4116-b606-acaaf1f5ddcd/7fff3b26-5325-4116-b606-acaaf1f5ddcd.ovf
....
     <Item>
        <rasd:Caption>RNG Device</rasd:Caption>
        <rasd:InstanceId>feeae0f9-ce77-4df7-8daf-0d128ef0e0e2</rasd:InstanceId>
        <rasd:ResourceType>0</rasd:ResourceType>
        <Type>rng</Type>
        <Device>virtio</Device>
      </Item>
      <Item>
        <rasd:Caption>Memory Ballooning Device</rasd:Caption>
        <rasd:InstanceId>b7015425-3e07-4f69-bac2-29649b2acea3</rasd:InstanceId>
        <rasd:ResourceType>0</rasd:ResourceType>
        <Type>balloon</Type>
        <Device>memballoon</Device>
      </Item>
....

2.2 Import guest after editing ovf from export domain to data domain
2.3 Power on guest,but guest can't be ran due to same error with bug
#cat engine.log
....
2018-05-07 18:03:56,966+08 ERROR [org.ovirt.engine.core.vdsbroker.CreateVDSCommand] (EE-ManagedThreadFactory-engine-Thread-118856) [9501d505-a1d3-4427-a2bf-cbee5af9f26c] Failed to create VM: java.lang.NullPointerException
	at org.ovirt.engine.core.vdsbroker.builder.vminfo.LibvirtVmXmlBuilder.writeRng(LibvirtVmXmlBuilder.java:1431) [vdsbroker.jar:]
....




Verify the bug with below builds:
virt-v2v-1.38.1-1.el7.x86_64
libguestfs-1.38.1-1.el7.x86_64

Steps:
1.Convert a rhel7 guest which has RNG device and Memory Ballooning Device by virt-v2v using -o rhv
2.Check converted guest's ovf after finishing conversion and there are RNG device and memory Ballooning Device in ovf, and these devices have specified spec-params.
# cat 207a4c12-072f-4873-8ada-efc448187c99/207a4c12-072f-4873-8ada-efc448187c99.ovf 
 
     <Item>
        <rasd:Caption>RNG Device</rasd:Caption>
        <rasd:InstanceId>12631d09-c5d9-4b6d-8926-670e4fe252b7</rasd:InstanceId>
        <rasd:ResourceType>0</rasd:ResourceType>
        <Type>rng</Type>
        <Device>virtio</Device>
        <SpecParams>
          <source>urandom</source>
        </SpecParams>
      </Item>
      <Item>
        <rasd:Caption>Memory Ballooning Device</rasd:Caption>
        <rasd:InstanceId>118d2331-8623-4aa5-a445-664f198d328f</rasd:InstanceId>
        <rasd:ResourceType>0</rasd:ResourceType>
        <Type>balloon</Type>
        <Device>memballoon</Device>
        <SpecParams>
          <model>virtio</model>
        </SpecParams>
      </Item>

3.Guest could boot into os normally after importing from export domain to data domain


Hi rjones,

   V2V will delete RNG and Memory Ballooning Device from guest during conversion if libguestfs<=1.38.0, so I have to add RNG and Memory Ballooning to converted guest on rhv4.2 manually as scenario2 to reproduce the bug, do you think these reproduce steps of scenario2 are reasonable?

Thanks
Comment 6 Pino Toscano 2018-05-07 11:00:41 UTC 
Hi mxie,

virt-v2v outputs the RNG and memory balloon devices to the OVF if the kernel in the guest supports virtio-rng and virtio-balloon. The rng and balloon devices in the libvirt XML are ignored.
Can you please attach the log of virt-v2v at the step 1.2 with -v -x?
Comment 7 mxie@redhat.com 2018-05-08 03:31:25 UTC 
Created attachment 1432952 [details]
v2v-memory-rng-7.5.z.log
Comment 9 Pino Toscano 2018-05-09 13:12:22 UTC 
(In reply to mxie from comment #5)
>    V2V will delete RNG and Memory Ballooning Device from guest during
> conversion if libguestfs<=1.38.0, so I have to add RNG and Memory Ballooning
> to converted guest on rhv4.2 manually as scenario2 to reproduce the bug, do
> you think these reproduce steps of scenario2 are reasonable?

It is a bit stretched as test case, but could work.  Another option is to pick an older build from brew, for example libguestfs-1.38.0-1.el7, so it produces the OVF with memory balloon and random devices but without the fix.
Comment 11 mxie@redhat.com 2018-05-10 08:43:24 UTC 
I can reproduce the bug with below builds:
virt-v2v-1.38.0-1.el7.x86_64
libguestfs-1.38.0-1.el7.x86_64

Reproduce steps:
1. Convert a rhel7 guest to rhv4.2 by virt-v2v using -o rhv
# virt-v2v rhel7.5 -o rhv -os 10.66.144.40:/home/nfs_export -on no-rng-mem
[   0.0] Opening the source -i libvirt rhel7.5
[   0.0] Creating an overlay to protect the source from being modified
[   0.8] Initializing the target -o rhv -os 10.66.144.40:/home/nfs_export
[   1.1] Opening the overlay
[ 439.7] Inspecting the overlay
[ 588.1] Checking for sufficient free disk space in the guest
[ 588.1] Estimating space required on target for each disk
[ 588.1] Converting Red Hat Enterprise Linux Server 7.5 (Maipo) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 824.0] Mapping filesystem data to avoid copying unused and blank areas
[ 826.4] Closing the overlay
[ 828.2] Checking if the guest needs BIOS or UEFI to boot
[ 828.2] Assigning disks to buses
[ 828.2] Copying disk 1/1 to /tmp/v2v.7SSPvV/ea9cb06f-8bf9-4fc8-a247-478e754d898a/images/1bfbce40-c01b-4cea-b7ff-ed5f112069ee/6d4177bf-1bd6-4776-8e4b-edc0e2ce5127 (qcow2)
    (100.00/100%)
[ 979.7] Creating output metadata
[ 979.8] Finishing off


2. Check converted guest's ovf after finishing conversion and could find RNG deivce and memory Ballooning Device without specified spec-params
....
     <Item>
        <rasd:Caption>RNG Device</rasd:Caption>
        <rasd:InstanceId>109c8226-6081-4437-8db2-de729f76260c</rasd:InstanceId>
        <rasd:ResourceType>0</rasd:ResourceType>
        <Type>rng</Type>
        <Device>virtio</Device>
      </Item>
      <Item>
        <rasd:Caption>Memory Ballooning Device</rasd:Caption>
        <rasd:InstanceId>59b27b33-0a24-47bb-bca8-336bded25c28</rasd:InstanceId>
        <rasd:ResourceType>0</rasd:ResourceType>
        <Type>balloon</Type>
        <Device>memballoon</Device>
      </Item>
      <Item>
.....

3.Failed to ran guest and get same error with bug in engine.log 
2018-05-10 16:35:33,880+08 ERROR [org.ovirt.engine.core.vdsbroker.CreateVDSCommand] (EE-ManagedThreadFactory-engine-Thread-186792) [f3946278-e735-4650-8bbf-16408fb3a712] Failed to create VM: java.lang.NullPointerException
	at org.ovirt.engine.core.vdsbroker.builder.vminfo.LibvirtVmXmlBuilder.writeRng(LibvirtVmXmlBuilder.java:1431) [vdsbroker.jar:]
Comment 12 mxie@redhat.com 2018-06-05 09:16:07 UTC 
Verify the bug with below builds:
virt-v2v-1.38.2-3.el7.x86_64
libguestfs-1.38.2-3.el7.x86_64
libvirt-4.3.0-1.el7.x86_64
qemu-kvm-rhev-2.12.0-3.el7.x86_64

Steps:
1. Convert a rhel7 guest to rhv4.2 by virt-v2v 
#  virt-v2v -ic vpx://root.73.141/data/10.73.75.219?no_verify=1 esx6.7-rhel7.5-x86_64 -o rhv -os 10.66.144.40:/home/nfs_export -b ovirtmgmt --password-file /tmp/passwd
[   0.0] Opening the source -i libvirt -ic vpx://root.73.141/data/10.73.75.219?no_verify=1 esx6.7-rhel7.5-x86_64
[   2.0] Creating an overlay to protect the source from being modified
[   3.1] Initializing the target -o rhv -os 10.66.144.40:/home/nfs_export
[   3.4] Opening the overlay
[  26.6] Inspecting the overlay
[ 206.8] Checking for sufficient free disk space in the guest
[ 206.8] Estimating space required on target for each disk
[ 206.8] Converting Red Hat Enterprise Linux Server 7.5 (Maipo) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[1544.8] Mapping filesystem data to avoid copying unused and blank areas
[1551.4] Closing the overlay
[1554.8] Checking if the guest needs BIOS or UEFI to boot
[1554.8] Assigning disks to buses
[1554.8] Copying disk 1/1 to /tmp/v2v.7RYevN/ea9cb06f-8bf9-4fc8-a247-478e754d898a/images/934071de-bc9f-4f53-9eaa-57430651a46a/e5367eeb-1471-4335-a542-e816bc4cfb28 (raw)
    (100.00/100%)
[2332.5] Creating output metadata
[2332.6] Finishing off

2. Check converted guest's ovf at export domain after finishing conversion and could find RNG deivce and memory Ballooning Device with specified spec-params
#cat 355f261f-65f3-4ce1-9e46-4926c0e6dd11/355f261f-65f3-4ce1-9e46-4926c0e6dd11.ovf
....
    <Item>
        <rasd:Caption>RNG Device</rasd:Caption>
        <rasd:InstanceId>5022155a-3b1a-4a10-a6ff-ef80ba742f30</rasd:InstanceId>
        <rasd:ResourceType>0</rasd:ResourceType>
        <Type>rng</Type>
        <Device>virtio</Device>
        <SpecParams>
          <source>urandom</source>
        </SpecParams>
      </Item>
      <Item>
        <rasd:Caption>Memory Ballooning Device</rasd:Caption>
        <rasd:InstanceId>ec609589-b500-465b-8311-cd76e1bec452</rasd:InstanceId>
        <rasd:ResourceType>0</rasd:ResourceType>
        <Type>balloon</Type>
        <Device>memballoon</Device>
        <SpecParams>
          <model>virtio</model>
        </SpecParams>
      </Item>
....

3.Import guest from export domain to data domain, power on guest normally and checkpoints of guest are passed


Result:
   Virt-v2v could add RNG Device and Memory Ballooning Device with specified spec-params for guest during conversion, so move the bug from ON_QA to VERIFIED
Comment 14 errata-xmlrpc 2018-10-30 07:45:24 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:3021