Bug 1075602

Summary: VM's can't be started
Product: Red Hat Enterprise Virtualization Manager Reporter: Lev Veyde <lveyde>
Component: ovirt-engineAssignee: Roy Golan <rgolan>
Status: CLOSED CURRENTRELEASE QA Contact: Pavel Novotny <pnovotny>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 3.4.0CC: acathrow, eedri, gklein, iheim, istein, knesenko, lpeer, lveyde, mavital, michal.skrivanek, obasan, ofrenkel, pnovotny, Rhev-m-bugs, yeylon
Target Milestone: ---   
Target Release: 3.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: virt
Fixed In Version: org.ovirt.engine-root-3.4.0-18 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1077300 (view as bug list) Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1077300    
Bug Blocks: 1078909, 1142926    
Attachments:
Description Flags
Engine log both before and after the restart none

Description Lev Veyde 2014-03-12 13:02:06 UTC
Created attachment 873534 [details]
Engine log both before and after the restart

Description of problem:
VM's can't be started from RHEVM WebAdmin with Failed to run VM <vm-name> (User: admin). message.

Version-Release number of selected component (if applicable):
rhevm-3.4.0-0.4.master.el6ev.noarch

How reproducible:
N/A

Steps to Reproduce:
1. Install rhevm-3.4.0-0.4.master.el6ev.noarch
2. Configure the env. adding host and storage
3. Create VMs
4. Attempt to start VMs

Actual results:
Start fails with error message of "Failed to run VM <vm-name> (User: admin)".

Expected results:
VMs should start normally.

Additional info:
The RHEVM engine logs contains the following messages:

2014-03-12 14:31:43,555 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-4-thread-12) [4e92c413] Command CreateVDSCommand(HostName = nari11.eng.lab.tlv.redhat.com, HostId = 8b1e989c-ccec-4ac6-b347-5f7df8ef3ccb, vmId=bc3328ae-ef45-4645-a858-b7e378785e80, vm=VM [desktop-from-server]) execution failed. Exception: VDSNetworkException: org.apache.xmlrpc.common.XmlRpcExtensionException: Null values aren't supported, if isEnabledForExtensions() == false
2014-03-12 14:31:43,555 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-4-thread-12) [4e92c413] FINISH, CreateVDSCommand, log id: 3a0e42b7
2014-03-12 14:31:43,555 ERROR [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-4-thread-12) [4e92c413] VDS::create Failed creating vm desktop-from-server in vds = 8b1e989c-ccec-4ac6-b347-5f7df8ef3ccb : nari11.eng.lab.tlv.redhat.com error = org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: org.apache.xmlrpc.common.XmlRpcExtensionException: Null values aren't supported, if isEnabledForExtensions() == false
2014-03-12 14:31:43,556 INFO  [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-4-thread-12) [4e92c413] FINISH, CreateVmVDSCommand, return: Down, log id: 6770d6d3
2014-03-12 14:31:43,557 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-4-thread-12) [4e92c413] Lock freed to object EngineLock [exclusiveLocks= key: bc3328ae-ef45-4645-a858-b7e378785e80 value: VM
, sharedLocks= ]
2014-03-12 14:31:43,557 ERROR [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-4-thread-12) [4e92c413] Command org.ovirt.engine.core.bll.RunVmCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: org.apache.xmlrpc.common.XmlRpcExtensionException: Null values aren't supported, if isEnabledForExtensions() == false (Failed with error VDS_NETWORK_ERROR and code 5022)
2014-03-12 14:31:43,595 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-4-thread-12) [4e92c413] Correlation ID: 4e92c413, Job ID: 55b44f79-a718-4f3b-b52c-ff1a11b55a41, Call Stack: null, Custom Event ID: -1, Message: Failed to run VM desktop-from-server (User: admin).


Important! : The workaround is to restart the ovirt-engine.
After the ovirt-engine is restarted VMs are started normally.

Comment 1 Roy Golan 2014-03-12 13:34:56 UTC
1. please be specific, whats the details of the vm that failed
2. provide the vdsm log, maybe we could see the xml there

Comment 2 Roy Golan 2014-03-12 15:12:38 UTC
(In reply to Roy Golan from comment #1)
> 1. please be specific, whats the details of the vm that failed
> 2. provide the vdsm log, maybe we could see the xml there

ignore 2. the call doesn't reach vdsm . it fails on the engine client (apache xmlRpc client)

Comment 5 Michal Skrivanek 2014-03-16 13:39:24 UTC
does it reproduce? or was it only for the first time after installation?
did you try to install again and it happened again?

Comment 6 Lev Veyde 2014-03-16 14:08:02 UTC
(In reply to Michal Skrivanek from comment #5)
> does it reproduce? or was it only for the first time after installation?
> did you try to install again and it happened again?

Hi Michal,

It seems to be quite reproducible and appears after the first setup.

Comment 7 Michal Skrivanek 2014-03-17 09:32:18 UTC
doesn't reproduce on upstream 3.4

Comment 8 Michal Skrivanek 2014-03-17 10:22:54 UTC
(In reply to Michal Skrivanek from comment #7)
eventually it does:)
though it happens only in the Default cluster (due to its arch initially created as Unknown)
New cluster always enforces a specific arch in the dialog.
Another workaround is to put the host to Maintenance and then Activate

Comment 12 Roy Golan 2014-03-18 06:46:54 UTC
analysis

setup:
 use Default cluster or any cluster with arch "Undefined"
 install a host, wait for it to go UP ; don't "Activate"

outcome:
 emulated_machine field of the cluster is never set (i.e null) by the host
 running a VM fails - XML rpc client refuses to send null values

cause:
 different flow between Activate and Install - 
 Activate:
  - we call VdsManager.refreshHost which set auto-detect treu (because arch == Undefined)
  - monitoring cycle starts, Host goes UP, we call VdsManager.refreshHost again,  the emulated machine is now set
 
 Install:
 - we put the host on status Initializing and wait for monitoring to kick in
 - monitoring ends, host goes Up, we call VdsManager.refreshHost, we set auto-detect to true and emulated machine is null

root cause:
 refreshHost first tries to detect emulated machine (proccesSoftwareCapabilities) and update the cluster and cpu and arch - which sets emulated machine to null if the arch has change 

monitoringStrategy.processHardwareCapabilities(vds);

Comment 13 Pavel Novotny 2014-03-18 19:46:56 UTC
Verified in rhevm-3.4.0-0.9.beta1.el6ev (av3.1).

Verification steps (same used for reproducing the bug in rhevm-3.4.0-0.4.master.el6ev):
1. On a clean RHEL 6.5 machine install RHEVM:
  # yum install rhevm-setup -y
  # rhevm-setup
  ...use defaults wherever possible...
2. In RHEVM Webadmin, add new host and wait until it's Up.
3. Add new storage (I used NFS), wait until it's Active.
4. Add new VM (fill in only the VM name). Add a disk to the VM so it can run (1GB, Thin Provision, Activate, Bootable).
5. Run VM.

Results: 
The VM is successfully launched, no errors in engine.log or vdsm.log.

Comment 15 Ilanit Stein 2014-03-19 08:34:29 UTC
Also had to chance to see start VM work fine on av3.1 (same flow as in comment #13)

Comment 16 Kiril Nesenko 2014-05-08 15:16:32 UTC
Issue still happens on 9cf2c62b6eea35bd5bf4186a040ccb7c2b4c9082

Comment 17 Michal Skrivanek 2014-05-09 05:59:32 UTC
(In reply to Kiril Nesenko from comment #16)
logs / access info / anything?
Is it exactly the same issue?
Does the workaround from bug 1077300 help?

Comment 21 Pavel Novotny 2014-05-12 12:53:54 UTC
Verified in rhevm-3.4.0-0.18.el6ev.noarch (av9).

Verified manaually and also in automated test scenario.
The verification steps were exactly the same as in comment 13, just copy&pasting:

1. On a clean RHEL 6.5 machine install RHEVM:
  # yum install rhevm-setup -y
  # rhevm-setup
  ...use defaults wherever possible...
2. In RHEVM Webadmin, add new host and wait until it's Up.
3. Add new storage (I used NFS), wait until it's Active.
4. Add new VM (fill in only the VM name). Add a disk to the VM so it can run (1GB, Thin Provision, Activate, Bootable).
5. Run VM.

Results: 
The VM is successfully launched, no errors in engine.log or vdsm.log.

Comment 23 Itamar Heim 2014-06-12 14:08:05 UTC
Closing as part of 3.4.0