Bug 1266088

Summary: internal error: Invalid or not yet handled value 'emptyBackingString' for VMX entry 'ide1:0.fileName' for device type 'cdrom-image'
Product: [oVirt] vdsm Reporter: ian
Component: GeneralAssignee: Shahar Havivi <shavivi>
Status: CLOSED CURRENTRELEASE QA Contact: Nisim Simsolo <nsimsolo>
Severity: high Docs Contact:
Priority: high    
Version: 4.17.5CC: amureini, bugs, danken, eblake, fromani, jherrman, mavital, mgoldboi, michal.skrivanek, nsimsolo, nsoffer, shavivi, tjelinek, ylavi
Target Milestone: ovirt-3.6.4Flags: shavivi: needinfo-
rule-engine: ovirt-3.6.z+
mgoldboi: planning_ack+
michal.skrivanek: devel_ack+
mavital: testing_ack+
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Previously, when using the libvirt library with the VMWare ESXi hypervisor, loading a guest virtual machine that contained empty CD-ROM images failed with an "internal error" message. With this update, libvirt is able to successfully parse ESXi configuration values for empty CD-ROM images, and the described failure no longer occurs.
Story Points: ---
Clone Of:
: 1275039 (view as bug list) Environment:
Last Closed: 2016-04-05 13:53:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Virt RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1275039, 1301892    
Bug Blocks: 1317865    
Attachments:
Description Flags
vdsm.log file form host that queried the vcenter host.
none
vdsm log with reproduction of this bug.
none
https://bugzilla.redhat.com/show_bug.cgi?id=1266088#c14 vdsm.log none

Description ian 2015-09-24 12:49:14 UTC
Description of problem:
oVirt fails to load the list of VMs when querying a vCenter installation.

Version-Release number of selected component (if applicable):
3.6 rc5

How reproducible:
Seems only when certain VMs are present on the vCenter host queried.

Steps to Reproduce:
1. Log into administration portal, System->Virtual Machines. 
2. Click Import. Select VMware for source  and complete details for your vCenter environment or select a valid external provider (previously set).
3. Click the "Load" button.

Actual results:
The "Import Virtual Machine(s)" popup window in webui hangs forever. "internal error: Invalid or not yet handled value 'emptyBackingString' for VMX entry 'ide1:0.fileName' for device type 'cdrom-image'" appears in vdsm.log
Expected results:
list of VM's available for migration is show.


Additional info:

Comment 1 ian 2015-09-24 12:51:24 UTC
Created attachment 1076535 [details]
vdsm.log file form host that queried the vcenter host.

Comment 2 Nir Soffer 2015-09-24 14:20:51 UTC
This fails when vdsm try to get domain xml for a remote vm:

      File "/usr/share/vdsm/v2v.py", line 152, in get_external_vms
        root = ET.fromstring(vm.XMLDesc(0))
      File "/usr/lib64/python2.7/site-packages/libvirt.py", line 478, in XMLDesc
        if ret is None: raise libvirtError ('virDomainGetXMLDesc() failed', dom=self)
    libvirtError: internal error: Invalid or not yet handled value
    'emptyBackingString' for VMX entry 'ide1:0.fileName' for device type
    'cdrom-image'

Vdsm enumerate all the vms on the remote server; if it fail to get the
xml for one of them, the entire request is aborted, preventing import
of a vm which may be fine.

Comment 3 Nir Soffer 2015-09-24 14:27:21 UTC
Eric, who can help with the underlying libvirt error?

Comment 4 Red Hat Bugzilla Rules Engine 2015-10-19 11:03:02 UTC
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 5 Nir Soffer 2015-10-19 14:28:09 UTC
This is an issue with libvirt, we cannot do anything with this expect
filing a libvirt bug about this and block on it.

Comment 6 Yaniv Lavi 2015-10-20 14:55:10 UTC
Can you clone this bug to libvirt?

Comment 7 Sandro Bonazzola 2015-10-26 12:44:14 UTC
this is an automated message. oVirt 3.6.0 RC3 has been released and GA is targeted to next week, Nov 4th 2015.
Please review this bug and if not a blocker, please postpone to a later release.
All bugs not postponed on GA release will be automatically re-targeted to

- 3.6.1 if severity >= high
- 4.0 if severity < high

Comment 8 Red Hat Bugzilla Rules Engine 2015-10-26 18:17:17 UTC
This bug is not marked for z-stream, yet the milestone is for a z-stream version, therefore the milestone has been reset.
Please set the correct milestone or add the z-stream flag.

Comment 9 Francesco Romani 2015-10-28 12:58:43 UTC
Clearing NEEDINFO on me. This bug already depends on https://bugzilla.redhat.com/show_bug.cgi?id=1275039 which is on libvirt (thanks to Shahar).

Comment 10 Michal Skrivanek 2015-12-03 14:24:35 UTC
RHEV-side error is fixed. 
The libvirt bug 1275039 still needs to get addressed to completely resolve the issue

Comment 11 Nisim Simsolo 2016-02-23 08:37:37 UTC
Created attachment 1129674 [details]
vdsm log with reproduction of this bug.

Comment 14 Nisim Simsolo 2016-02-29 13:59:49 UTC
Reassigned.

Verification scenario:
1. Browse webadmin -> Virtual machines -> import: load VM from VMware.

Actual result:
webadmin "loading" action failed, source VM list is empty and the next error displayed in the import dialog page: 
"Failed to communicate with the external provider, see log for additional details."

VDSM log attached, issue reproduced at: 
jsonrpc.Executor/4::ERROR::2016-02-29 15:20:08,563::v2v::164::root::(get_external_vms) error getting domain xml for vm 'murmur-server': internal error: Invalid or not yet handled value 'emp
tyBackingString' for VMX entry 'ide1:0.fileName' for device type 'cdrom-image'

Comment 15 Red Hat Bugzilla Rules Engine 2016-02-29 13:59:55 UTC
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 17 Nisim Simsolo 2016-02-29 14:15:36 UTC
Verified using builds:
rhevm-3.6.3.3-0.1.el6
vdsm-4.17.23-0.el7ev.noarch
libvirt-client-1.2.17-13.el7_2.2.x86_64
qemu-kvm-rhev-2.3.0-31.el7_2.4.x86_64
sanlock-3.2.4-1.el7.x86_64

Comment 18 Shahar Havivi 2016-03-01 07:12:48 UTC
Looking at the log its looks like the patch didn't make it,
Nisim please point me to the host you are using.

Comment 19 Shahar Havivi 2016-03-01 08:03:03 UTC
After checking the source code libvirt raise the following error for all of the vms:

Traceback (most recent call last):
  File "/usr/share/vdsm/v2v.py", line 156, in get_external_vms
    _add_vm_info(vm, params)
  File "/usr/share/vdsm/v2v.py", line 662, in _add_vm_info
    if vm.state()[0] == libvirt.VIR_DOMAIN_SHUTOFF:
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2661, in state
    if ret is None: raise libvirtError ('virDomainGetState() failed', dom=self)
libvirtError: Domain not found: Could not find domain with UUID '4216bb0e-ffbd-3784-57b9-81a3ca40b5d0'

Comment 20 Shahar Havivi 2016-03-02 13:10:55 UTC
on libvirt version: 
===================
libvirt-daemon-1.2.17-13.el7_2.3.x86_64

its faling the following if statement:
======================================

for vm in conn.listAllDomains():
    params['vmName'] = vm.name()
    if vm.state()[0] == libvirt.VIR_DOMAIN_SHUTOFF:
        params['status'] = "Down"
    else:
        params['status'] = "Up"

with the exception:
===================
Traceback (most recent call last):
  File "/usr/share/vdsm/v2v.py", line 156, in get_external_vms
    _add_vm_info(vm, params)
  File "/usr/share/vdsm/v2v.py", line 662, in _add_vm_info
    if vm.state()[0] == libvirt.VIR_DOMAIN_SHUTOFF:
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2661, in state
    if ret is None: raise libvirtError ('virDomainGetState() failed', dom=self)
libvirtError: Domain not found: Could not find domain with UUID '4216df1f-d02a-adc9-d0c8-e2c9e9ab2ef2'


when I comment the if with statement I get error on the following code:
======================================================================
xml = vm.XMLDesc(0)

Comment 21 Shahar Havivi 2016-03-02 13:15:30 UTC
virsh on the same version raise the following error on some domains:

virsh # dumpxml RHEL7.1-Server
error: internal error: File name '/vmfs/volumes/524ea59a-dbd92f18-cd01-40f2e91d7188/RHEL-7.1-20150219.1-Server-x86_64-dvd1(1).iso' refers to non-existing datastore '524ea59a-dbd92f18-cd01-40f2e91d7188'

this is a different error but still a problem.

Comment 22 Yaniv Lavi 2016-03-02 14:11:29 UTC
Please scrub this bug to a valid milestone.

Comment 23 Moran Goldboim 2016-03-02 23:49:15 UTC
(In reply to Yaniv Dary from comment #22)
> Please scrub this bug to a valid milestone.

This is a high priority bug that should be fixed soon and a good candidate for a short release cycle.

Shahar,Nisim have you opened/reopened libvirt bug?

Comment 24 Shahar Havivi 2016-03-03 06:42:13 UTC
I comment on bug 1275039

Comment 25 Tomas Jelinek 2016-03-07 07:15:50 UTC
@Nisim: so according to https://bugzilla.redhat.com/show_bug.cgi?id=1275039#c23 it is a vsphere bug and you have succeeded to verify this fix in newer vsphere. Any reason why not to close this bug and continue by opening a new issue against vmware?

Comment 33 Shahar Havivi 2016-03-14 12:59:45 UTC
There is nothing to do in oVirt,
Looks like VMWare bug

Comment 35 Nisim Simsolo 2016-03-14 14:52:42 UTC
Verified. 
similar VM with empty datastore ISO file created on TLV VMware environment (Version 6.0.0 Build 2594327), and when "loading" VMs using webadmin import dialog, VMware VMs are listed except the problematic VM (FAD).

Currently, the original environment on which this bug was opened, cannot be accessed using virt-v2v. the assumption for this issue related to vSphere 5.5 which in some cases, VMware SOAP API is not communicating with libvirt. see bug: https://bugzilla.redhat.com/show_bug.cgi?id=1287681 for more details.

Verified using builds: 
rhevm-3.6.3.4-0.1.el6
vdsm-4.17.23-0.el7ev.noarch
qemu-kvm-rhev-2.3.0-31.el7_2.8.x86_64
libvirt-1.2.17-13.el7_2.3.x86_64