Bug 1266088 - internal error: Invalid or not yet handled value 'emptyBackingString' for VMX entry 'ide1:0.fileName' for device type 'cdrom-image'
internal error: Invalid or not yet handled value 'emptyBackingString' for VMX...
Status: CLOSED CURRENTRELEASE
Product: vdsm
Classification: oVirt
Component: General (Show other bugs)
4.17.5
Unspecified Unspecified
high Severity high (vote)
: ovirt-3.6.4
: ---
Assigned To: Shahar Havivi
Nisim Simsolo
:
Depends On: 1275039 1301892
Blocks: 1317865
  Show dependency treegraph
 
Reported: 2015-09-24 08:49 EDT by ian
Modified: 2016-04-05 09:53 EDT (History)
14 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously, when using the libvirt library with the VMWare ESXi hypervisor, loading a guest virtual machine that contained empty CD-ROM images failed with an "internal error" message. With this update, libvirt is able to successfully parse ESXi configuration values for empty CD-ROM images, and the described failure no longer occurs.
Story Points: ---
Clone Of:
: 1275039 (view as bug list)
Environment:
Last Closed: 2016-04-05 09:53:37 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Virt
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
shavivi: needinfo-
rule-engine: ovirt‑3.6.z+
mgoldboi: planning_ack+
michal.skrivanek: devel_ack+
mavital: testing_ack+


Attachments (Terms of Use)
vdsm.log file form host that queried the vcenter host. (1.61 MB, text/plain)
2015-09-24 08:51 EDT, ian
no flags Details
vdsm log with reproduction of this bug. (669.24 KB, application/x-gzip)
2016-02-23 03:37 EST, Nisim Simsolo
no flags Details
https://bugzilla.redhat.com/show_bug.cgi?id=1266088#c14 vdsm.log (659.59 KB, application/x-gzip)
2016-02-29 09:02 EST, Nisim Simsolo
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 46634 master MERGED v2v: Skip vm if XMLDesc() fails Never
oVirt gerrit 47363 ovirt-3.6 MERGED v2v: Skip vm if XMLDesc() fails Never

  None (edit)
Description ian 2015-09-24 08:49:14 EDT
Description of problem:
oVirt fails to load the list of VMs when querying a vCenter installation.

Version-Release number of selected component (if applicable):
3.6 rc5

How reproducible:
Seems only when certain VMs are present on the vCenter host queried.

Steps to Reproduce:
1. Log into administration portal, System->Virtual Machines. 
2. Click Import. Select VMware for source  and complete details for your vCenter environment or select a valid external provider (previously set).
3. Click the "Load" button.

Actual results:
The "Import Virtual Machine(s)" popup window in webui hangs forever. "internal error: Invalid or not yet handled value 'emptyBackingString' for VMX entry 'ide1:0.fileName' for device type 'cdrom-image'" appears in vdsm.log
Expected results:
list of VM's available for migration is show.


Additional info:
Comment 1 ian 2015-09-24 08:51 EDT
Created attachment 1076535 [details]
vdsm.log file form host that queried the vcenter host.
Comment 2 Nir Soffer 2015-09-24 10:20:51 EDT
This fails when vdsm try to get domain xml for a remote vm:

      File "/usr/share/vdsm/v2v.py", line 152, in get_external_vms
        root = ET.fromstring(vm.XMLDesc(0))
      File "/usr/lib64/python2.7/site-packages/libvirt.py", line 478, in XMLDesc
        if ret is None: raise libvirtError ('virDomainGetXMLDesc() failed', dom=self)
    libvirtError: internal error: Invalid or not yet handled value
    'emptyBackingString' for VMX entry 'ide1:0.fileName' for device type
    'cdrom-image'

Vdsm enumerate all the vms on the remote server; if it fail to get the
xml for one of them, the entire request is aborted, preventing import
of a vm which may be fine.
Comment 3 Nir Soffer 2015-09-24 10:27:21 EDT
Eric, who can help with the underlying libvirt error?
Comment 4 Red Hat Bugzilla Rules Engine 2015-10-19 07:03:02 EDT
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.
Comment 5 Nir Soffer 2015-10-19 10:28:09 EDT
This is an issue with libvirt, we cannot do anything with this expect
filing a libvirt bug about this and block on it.
Comment 6 Yaniv Lavi 2015-10-20 10:55:10 EDT
Can you clone this bug to libvirt?
Comment 7 Sandro Bonazzola 2015-10-26 08:44:14 EDT
this is an automated message. oVirt 3.6.0 RC3 has been released and GA is targeted to next week, Nov 4th 2015.
Please review this bug and if not a blocker, please postpone to a later release.
All bugs not postponed on GA release will be automatically re-targeted to

- 3.6.1 if severity >= high
- 4.0 if severity < high
Comment 8 Red Hat Bugzilla Rules Engine 2015-10-26 14:17:17 EDT
This bug is not marked for z-stream, yet the milestone is for a z-stream version, therefore the milestone has been reset.
Please set the correct milestone or add the z-stream flag.
Comment 9 Francesco Romani 2015-10-28 08:58:43 EDT
Clearing NEEDINFO on me. This bug already depends on https://bugzilla.redhat.com/show_bug.cgi?id=1275039 which is on libvirt (thanks to Shahar).
Comment 10 Michal Skrivanek 2015-12-03 09:24:35 EST
RHEV-side error is fixed. 
The libvirt bug 1275039 still needs to get addressed to completely resolve the issue
Comment 11 Nisim Simsolo 2016-02-23 03:37 EST
Created attachment 1129674 [details]
vdsm log with reproduction of this bug.
Comment 14 Nisim Simsolo 2016-02-29 08:59:49 EST
Reassigned.

Verification scenario:
1. Browse webadmin -> Virtual machines -> import: load VM from VMware.

Actual result:
webadmin "loading" action failed, source VM list is empty and the next error displayed in the import dialog page: 
"Failed to communicate with the external provider, see log for additional details."

VDSM log attached, issue reproduced at: 
jsonrpc.Executor/4::ERROR::2016-02-29 15:20:08,563::v2v::164::root::(get_external_vms) error getting domain xml for vm 'murmur-server': internal error: Invalid or not yet handled value 'emp
tyBackingString' for VMX entry 'ide1:0.fileName' for device type 'cdrom-image'
Comment 15 Red Hat Bugzilla Rules Engine 2016-02-29 08:59:55 EST
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.
Comment 17 Nisim Simsolo 2016-02-29 09:15:36 EST
Verified using builds:
rhevm-3.6.3.3-0.1.el6
vdsm-4.17.23-0.el7ev.noarch
libvirt-client-1.2.17-13.el7_2.2.x86_64
qemu-kvm-rhev-2.3.0-31.el7_2.4.x86_64
sanlock-3.2.4-1.el7.x86_64
Comment 18 Shahar Havivi 2016-03-01 02:12:48 EST
Looking at the log its looks like the patch didn't make it,
Nisim please point me to the host you are using.
Comment 19 Shahar Havivi 2016-03-01 03:03:03 EST
After checking the source code libvirt raise the following error for all of the vms:

Traceback (most recent call last):
  File "/usr/share/vdsm/v2v.py", line 156, in get_external_vms
    _add_vm_info(vm, params)
  File "/usr/share/vdsm/v2v.py", line 662, in _add_vm_info
    if vm.state()[0] == libvirt.VIR_DOMAIN_SHUTOFF:
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2661, in state
    if ret is None: raise libvirtError ('virDomainGetState() failed', dom=self)
libvirtError: Domain not found: Could not find domain with UUID '4216bb0e-ffbd-3784-57b9-81a3ca40b5d0'
Comment 20 Shahar Havivi 2016-03-02 08:10:55 EST
on libvirt version: 
===================
libvirt-daemon-1.2.17-13.el7_2.3.x86_64

its faling the following if statement:
======================================

for vm in conn.listAllDomains():
    params['vmName'] = vm.name()
    if vm.state()[0] == libvirt.VIR_DOMAIN_SHUTOFF:
        params['status'] = "Down"
    else:
        params['status'] = "Up"

with the exception:
===================
Traceback (most recent call last):
  File "/usr/share/vdsm/v2v.py", line 156, in get_external_vms
    _add_vm_info(vm, params)
  File "/usr/share/vdsm/v2v.py", line 662, in _add_vm_info
    if vm.state()[0] == libvirt.VIR_DOMAIN_SHUTOFF:
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2661, in state
    if ret is None: raise libvirtError ('virDomainGetState() failed', dom=self)
libvirtError: Domain not found: Could not find domain with UUID '4216df1f-d02a-adc9-d0c8-e2c9e9ab2ef2'


when I comment the if with statement I get error on the following code:
======================================================================
xml = vm.XMLDesc(0)
Comment 21 Shahar Havivi 2016-03-02 08:15:30 EST
virsh on the same version raise the following error on some domains:

virsh # dumpxml RHEL7.1-Server
error: internal error: File name '/vmfs/volumes/524ea59a-dbd92f18-cd01-40f2e91d7188/RHEL-7.1-20150219.1-Server-x86_64-dvd1(1).iso' refers to non-existing datastore '524ea59a-dbd92f18-cd01-40f2e91d7188'

this is a different error but still a problem.
Comment 22 Yaniv Lavi 2016-03-02 09:11:29 EST
Please scrub this bug to a valid milestone.
Comment 23 Moran Goldboim 2016-03-02 18:49:15 EST
(In reply to Yaniv Dary from comment #22)
> Please scrub this bug to a valid milestone.

This is a high priority bug that should be fixed soon and a good candidate for a short release cycle.

Shahar,Nisim have you opened/reopened libvirt bug?
Comment 24 Shahar Havivi 2016-03-03 01:42:13 EST
I comment on bug 1275039
Comment 25 Tomas Jelinek 2016-03-07 02:15:50 EST
@Nisim: so according to https://bugzilla.redhat.com/show_bug.cgi?id=1275039#c23 it is a vsphere bug and you have succeeded to verify this fix in newer vsphere. Any reason why not to close this bug and continue by opening a new issue against vmware?
Comment 33 Shahar Havivi 2016-03-14 08:59:45 EDT
There is nothing to do in oVirt,
Looks like VMWare bug
Comment 35 Nisim Simsolo 2016-03-14 10:52:42 EDT
Verified. 
similar VM with empty datastore ISO file created on TLV VMware environment (Version 6.0.0 Build 2594327), and when "loading" VMs using webadmin import dialog, VMware VMs are listed except the problematic VM (FAD).

Currently, the original environment on which this bug was opened, cannot be accessed using virt-v2v. the assumption for this issue related to vSphere 5.5 which in some cases, VMware SOAP API is not communicating with libvirt. see bug: https://bugzilla.redhat.com/show_bug.cgi?id=1287681 for more details.

Verified using builds: 
rhevm-3.6.3.4-0.1.el6
vdsm-4.17.23-0.el7ev.noarch
qemu-kvm-rhev-2.3.0-31.el7_2.8.x86_64
libvirt-1.2.17-13.el7_2.3.x86_64

Note You need to log in before you can comment on or make changes to this bug.