1550168 – virt-v2v -o rhv generated OVF causes oVirt to crash with NullPointerException in LibvirtVmXmlBuilder.writeRng if SpecParams is missing

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1550168 - virt-v2v -o rhv generated OVF causes oVirt to crash with NullPointerException in LibvirtVmXmlBuilder.writeRng if SpecParams is missing

Summary: virt-v2v -o rhv generated OVF causes oVirt to crash with NullPointerException...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	libguestfs
Sub Component:
Version:	7.5
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	high
Target Milestone:	rc
Target Release:	---
Assignee:	Richard W.M. Jones
QA Contact:	Virtualization Bugs
Docs Contact:
URL:
Whiteboard:	V2V
Depends On:	1550123 1551055
Blocks:
TreeView+	depends on / blocked

Reported:	2018-02-28 17:09 UTC by Richard W.M. Jones
Modified:	2019-04-28 10:45 UTC (History)
CC List:	13 users (show)
Fixed In Version:	libguestfs-1.38.1-1.el7
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:	1550123
Environment:
Last Closed:	2018-10-30 07:45:24 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
v2v-memory-rng-7.5.z.log (1.35 MB, text/plain) 2018-05-08 03:31 UTC, mxie@redhat.com	no flags	Details
View All

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHEA-2018:3021	0	None	None	None	2018-10-30 07:47:20 UTC

Description Richard W.M. Jones 2018-02-28 17:09:29 UTC

+++ This bug was initially created as a clone of Bug #1550123 +++

Description of problem:

libguestfs creates OVF XML which looks like this for some guests:

      <Item>
        <rasd:Caption>RNG Device</rasd:Caption>
        <rasd:InstanceId>feeae0f9-ce77-4df7-8daf-0d128ef0e0e2</rasd:InstanceId>
        <rasd:ResourceType>0</rasd:ResourceType>
        <Type>rng</Type>
        <Device>virtio</Device>
      </Item>

      <Item>
        <rasd:Caption>Memory Ballooning Device</rasd:Caption>
        <rasd:InstanceId>b7015425-3e07-4f69-bac2-29649b2acea3</rasd:InstanceId>
        <rasd:ResourceType>0</rasd:ResourceType>
        <Type>balloon</Type>
        <Device>memballoon</Device>
      </Item>

Unfortunately this XML causes oVirt 4.2 to crash, see bug 1550123 for details.

Version-Release number of selected component (if applicable):

libguestfs <= 1.38.0
ovirt-engine-4.2.2.2-0.0.master.20180225172203.gitd7cf125.el7.centos.noarch

How reproducible:

100%

Steps to Reproduce:
1. Convert a Fedora or RHEL 7 guest to oVirt 4.2 using -o rhv.
2. Try to boot the guest on the target oVirt.
3. Boot with fail with a strange (and bogus) error about:

"There are no hosts to use. Check that the cluster contains at least one host in Up state."

The real error can be seen in engine.log - it's a Java NullPointerException in oVirt.

Additional information:

Upstream fix:
https://www.redhat.com/archives/libguestfs/2018-February/msg00187.html

Comment 2 Pino Toscano 2018-04-11 16:23:57 UTC

Fixed upstream with
https://github.com/libguestfs/libguestfs/commit/d7b31559c291bc8af3419679facdf174e6e647c1
which is in libguestfs >= 1.39.1.

Comment 3 Pino Toscano 2018-05-02 11:21:39 UTC

Fix backported in 1.38.1.

Comment 5 mxie@redhat.com 2018-05-07 10:23:14 UTC

Try to reproduce the bug with below builds:
virt-v2v-1.36.10-6.el7_5.2.x86_64
libguestfs-1.36.10-6.el7_5.2.x86_64

Reproduce steps:
Scenario1:
1.1 Prepare a rhel7 guest which has RNG device and Memory Ballooning Device
# virsh dumpxml rhel7.5
.... 
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </memballoon>
    <rng model='virtio'>
      <backend model='random'>/dev/urandom</backend>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/>
    </rng>
....

1.2 Convert guest to rhv4.2 by virt-v2v using -o rhv
# virt-v2v rhel7.5 -o rhv -os 10.66.144.40:/home/nfs_export -on memory-rng-7.5.z
[   0.0] Opening the source -i libvirt rhel7.5
[   0.0] Creating an overlay to protect the source from being modified
[   0.5] Initializing the target -o rhv -os 10.66.144.40:/home/nfs_export
[   0.9] Opening the overlay
[  31.3] Inspecting the overlay
[ 140.2] Checking for sufficient free disk space in the guest
[ 140.2] Estimating space required on target for each disk
[ 140.2] Converting Red Hat Enterprise Linux Server 7.5 (Maipo) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 260.3] Mapping filesystem data to avoid copying unused and blank areas
[ 260.9] Closing the overlay
[ 262.0] Checking if the guest needs BIOS or UEFI to boot
[ 262.0] Assigning disks to buses
[ 262.0] Copying disk 1/1 to /tmp/v2v.29t5Rn/6f2055e4-b776-4ffd-94e6-f444e4931b76/images/bcf165d2-9b27-48ed-8e5c-7b84ebd154a3/fcb2ea87-15bb-42f1-b5b7-803293f88bb0 (qcow2)
    (100.00/100%)
[ 364.6] Creating output metadata
[ 364.7] Finishing off

1.3 Check converted guest's ovf after finishing conversion but can't find RNG deivce and memory Ballooning Device, meanwhile,
# cat 7fff3b26-5325-4116-b606-acaaf1f5ddcd/7fff3b26-5325-4116-b606-acaaf1f5ddcd.ovf 


Sceanario2:
2.1 Add RND device and memory Ballooning Device info according to comment0 into converted guest's ovf manually after step1.2
# vi 7fff3b26-5325-4116-b606-acaaf1f5ddcd/7fff3b26-5325-4116-b606-acaaf1f5ddcd.ovf
....
     <Item>
        <rasd:Caption>RNG Device</rasd:Caption>
        <rasd:InstanceId>feeae0f9-ce77-4df7-8daf-0d128ef0e0e2</rasd:InstanceId>
        <rasd:ResourceType>0</rasd:ResourceType>
        <Type>rng</Type>
        <Device>virtio</Device>
      </Item>
      <Item>
        <rasd:Caption>Memory Ballooning Device</rasd:Caption>
        <rasd:InstanceId>b7015425-3e07-4f69-bac2-29649b2acea3</rasd:InstanceId>
        <rasd:ResourceType>0</rasd:ResourceType>
        <Type>balloon</Type>
        <Device>memballoon</Device>
      </Item>
....

2.2 Import guest after editing ovf from export domain to data domain
2.3 Power on guest,but guest can't be ran due to same error with bug
#cat engine.log
....
2018-05-07 18:03:56,966+08 ERROR [org.ovirt.engine.core.vdsbroker.CreateVDSCommand] (EE-ManagedThreadFactory-engine-Thread-118856) [9501d505-a1d3-4427-a2bf-cbee5af9f26c] Failed to create VM: java.lang.NullPointerException
	at org.ovirt.engine.core.vdsbroker.builder.vminfo.LibvirtVmXmlBuilder.writeRng(LibvirtVmXmlBuilder.java:1431) [vdsbroker.jar:]
....




Verify the bug with below builds:
virt-v2v-1.38.1-1.el7.x86_64
libguestfs-1.38.1-1.el7.x86_64

Steps:
1.Convert a rhel7 guest which has RNG device and Memory Ballooning Device by virt-v2v using -o rhv
2.Check converted guest's ovf after finishing conversion and there are RNG device and memory Ballooning Device in ovf, and these devices have specified spec-params.
# cat 207a4c12-072f-4873-8ada-efc448187c99/207a4c12-072f-4873-8ada-efc448187c99.ovf 
 
     <Item>
        <rasd:Caption>RNG Device</rasd:Caption>
        <rasd:InstanceId>12631d09-c5d9-4b6d-8926-670e4fe252b7</rasd:InstanceId>
        <rasd:ResourceType>0</rasd:ResourceType>
        <Type>rng</Type>
        <Device>virtio</Device>
        <SpecParams>
          <source>urandom</source>
        </SpecParams>
      </Item>
      <Item>
        <rasd:Caption>Memory Ballooning Device</rasd:Caption>
        <rasd:InstanceId>118d2331-8623-4aa5-a445-664f198d328f</rasd:InstanceId>
        <rasd:ResourceType>0</rasd:ResourceType>
        <Type>balloon</Type>
        <Device>memballoon</Device>
        <SpecParams>
          <model>virtio</model>
        </SpecParams>
      </Item>

3.Guest could boot into os normally after importing from export domain to data domain


Hi rjones,

   V2V will delete RNG and Memory Ballooning Device from guest during conversion if libguestfs<=1.38.0, so I have to add RNG and Memory Ballooning to converted guest on rhv4.2 manually as scenario2 to reproduce the bug, do you think these reproduce steps of scenario2 are reasonable?

Thanks

Comment 6 Pino Toscano 2018-05-07 11:00:41 UTC

Hi mxie,

virt-v2v outputs the RNG and memory balloon devices to the OVF if the kernel in the guest supports virtio-rng and virtio-balloon. The rng and balloon devices in the libvirt XML are ignored.
Can you please attach the log of virt-v2v at the step 1.2 with -v -x?

Comment 7 mxie@redhat.com 2018-05-08 03:31:25 UTC

Created attachment 1432952 [details]
v2v-memory-rng-7.5.z.log

Comment 9 Pino Toscano 2018-05-09 13:12:22 UTC

(In reply to mxie from comment #5)
>    V2V will delete RNG and Memory Ballooning Device from guest during
> conversion if libguestfs<=1.38.0, so I have to add RNG and Memory Ballooning
> to converted guest on rhv4.2 manually as scenario2 to reproduce the bug, do
> you think these reproduce steps of scenario2 are reasonable?

It is a bit stretched as test case, but could work.  Another option is to pick an older build from brew, for example libguestfs-1.38.0-1.el7, so it produces the OVF with memory balloon and random devices but without the fix.

Comment 11 mxie@redhat.com 2018-05-10 08:43:24 UTC

I can reproduce the bug with below builds:
virt-v2v-1.38.0-1.el7.x86_64
libguestfs-1.38.0-1.el7.x86_64

Reproduce steps:
1. Convert a rhel7 guest to rhv4.2 by virt-v2v using -o rhv
# virt-v2v rhel7.5 -o rhv -os 10.66.144.40:/home/nfs_export -on no-rng-mem
[   0.0] Opening the source -i libvirt rhel7.5
[   0.0] Creating an overlay to protect the source from being modified
[   0.8] Initializing the target -o rhv -os 10.66.144.40:/home/nfs_export
[   1.1] Opening the overlay
[ 439.7] Inspecting the overlay
[ 588.1] Checking for sufficient free disk space in the guest
[ 588.1] Estimating space required on target for each disk
[ 588.1] Converting Red Hat Enterprise Linux Server 7.5 (Maipo) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 824.0] Mapping filesystem data to avoid copying unused and blank areas
[ 826.4] Closing the overlay
[ 828.2] Checking if the guest needs BIOS or UEFI to boot
[ 828.2] Assigning disks to buses
[ 828.2] Copying disk 1/1 to /tmp/v2v.7SSPvV/ea9cb06f-8bf9-4fc8-a247-478e754d898a/images/1bfbce40-c01b-4cea-b7ff-ed5f112069ee/6d4177bf-1bd6-4776-8e4b-edc0e2ce5127 (qcow2)
    (100.00/100%)
[ 979.7] Creating output metadata
[ 979.8] Finishing off


2. Check converted guest's ovf after finishing conversion and could find RNG deivce and memory Ballooning Device without specified spec-params
....
     <Item>
        <rasd:Caption>RNG Device</rasd:Caption>
        <rasd:InstanceId>109c8226-6081-4437-8db2-de729f76260c</rasd:InstanceId>
        <rasd:ResourceType>0</rasd:ResourceType>
        <Type>rng</Type>
        <Device>virtio</Device>
      </Item>
      <Item>
        <rasd:Caption>Memory Ballooning Device</rasd:Caption>
        <rasd:InstanceId>59b27b33-0a24-47bb-bca8-336bded25c28</rasd:InstanceId>
        <rasd:ResourceType>0</rasd:ResourceType>
        <Type>balloon</Type>
        <Device>memballoon</Device>
      </Item>
      <Item>
.....

3.Failed to ran guest and get same error with bug in engine.log 
2018-05-10 16:35:33,880+08 ERROR [org.ovirt.engine.core.vdsbroker.CreateVDSCommand] (EE-ManagedThreadFactory-engine-Thread-186792) [f3946278-e735-4650-8bbf-16408fb3a712] Failed to create VM: java.lang.NullPointerException
	at org.ovirt.engine.core.vdsbroker.builder.vminfo.LibvirtVmXmlBuilder.writeRng(LibvirtVmXmlBuilder.java:1431) [vdsbroker.jar:]

Comment 12 mxie@redhat.com 2018-06-05 09:16:07 UTC

Verify the bug with below builds:
virt-v2v-1.38.2-3.el7.x86_64
libguestfs-1.38.2-3.el7.x86_64
libvirt-4.3.0-1.el7.x86_64
qemu-kvm-rhev-2.12.0-3.el7.x86_64

Steps:
1. Convert a rhel7 guest to rhv4.2 by virt-v2v 
#  virt-v2v -ic vpx://root.73.141/data/10.73.75.219?no_verify=1 esx6.7-rhel7.5-x86_64 -o rhv -os 10.66.144.40:/home/nfs_export -b ovirtmgmt --password-file /tmp/passwd
[   0.0] Opening the source -i libvirt -ic vpx://root.73.141/data/10.73.75.219?no_verify=1 esx6.7-rhel7.5-x86_64
[   2.0] Creating an overlay to protect the source from being modified
[   3.1] Initializing the target -o rhv -os 10.66.144.40:/home/nfs_export
[   3.4] Opening the overlay
[  26.6] Inspecting the overlay
[ 206.8] Checking for sufficient free disk space in the guest
[ 206.8] Estimating space required on target for each disk
[ 206.8] Converting Red Hat Enterprise Linux Server 7.5 (Maipo) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[1544.8] Mapping filesystem data to avoid copying unused and blank areas
[1551.4] Closing the overlay
[1554.8] Checking if the guest needs BIOS or UEFI to boot
[1554.8] Assigning disks to buses
[1554.8] Copying disk 1/1 to /tmp/v2v.7RYevN/ea9cb06f-8bf9-4fc8-a247-478e754d898a/images/934071de-bc9f-4f53-9eaa-57430651a46a/e5367eeb-1471-4335-a542-e816bc4cfb28 (raw)
    (100.00/100%)
[2332.5] Creating output metadata
[2332.6] Finishing off

2. Check converted guest's ovf at export domain after finishing conversion and could find RNG deivce and memory Ballooning Device with specified spec-params
#cat 355f261f-65f3-4ce1-9e46-4926c0e6dd11/355f261f-65f3-4ce1-9e46-4926c0e6dd11.ovf
....
    <Item>
        <rasd:Caption>RNG Device</rasd:Caption>
        <rasd:InstanceId>5022155a-3b1a-4a10-a6ff-ef80ba742f30</rasd:InstanceId>
        <rasd:ResourceType>0</rasd:ResourceType>
        <Type>rng</Type>
        <Device>virtio</Device>
        <SpecParams>
          <source>urandom</source>
        </SpecParams>
      </Item>
      <Item>
        <rasd:Caption>Memory Ballooning Device</rasd:Caption>
        <rasd:InstanceId>ec609589-b500-465b-8311-cd76e1bec452</rasd:InstanceId>
        <rasd:ResourceType>0</rasd:ResourceType>
        <Type>balloon</Type>
        <Device>memballoon</Device>
        <SpecParams>
          <model>virtio</model>
        </SpecParams>
      </Item>
....

3.Import guest from export domain to data domain, power on guest normally and checkpoints of guest are passed


Result:
   Virt-v2v could add RNG Device and Memory Ballooning Device with specified spec-params for guest during conversion, so move the bug from ON_QA to VERIFIED

Comment 14 errata-xmlrpc 2018-10-30 07:45:24 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:3021

Note You need to log in before you can comment on or make changes to this bug.