Bug 1075602 - VM's can't be started
Summary: VM's can't be started
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 3.4.0
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 3.4.0
Assignee: Roy Golan
QA Contact: Pavel Novotny
URL:
Whiteboard: virt
Depends On: 1077300
Blocks: rhev3.4beta 1142926
TreeView+ depends on / blocked
 
Reported: 2014-03-12 13:02 UTC by Lev Veyde
Modified: 2014-09-18 12:24 UTC (History)
15 users (show)

Fixed In Version: org.ovirt.engine-root-3.4.0-18
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1077300 (view as bug list)
Environment:
Last Closed:
oVirt Team: ---
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Engine log both before and after the restart (28.14 KB, application/x-tar-gz)
2014-03-12 13:02 UTC, Lev Veyde
no flags Details


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 27539 0 master MERGED core: call prccessHardwareCaps after GetHardwareInfo Never

Description Lev Veyde 2014-03-12 13:02:06 UTC
Created attachment 873534 [details]
Engine log both before and after the restart

Description of problem:
VM's can't be started from RHEVM WebAdmin with Failed to run VM <vm-name> (User: admin). message.

Version-Release number of selected component (if applicable):
rhevm-3.4.0-0.4.master.el6ev.noarch

How reproducible:
N/A

Steps to Reproduce:
1. Install rhevm-3.4.0-0.4.master.el6ev.noarch
2. Configure the env. adding host and storage
3. Create VMs
4. Attempt to start VMs

Actual results:
Start fails with error message of "Failed to run VM <vm-name> (User: admin)".

Expected results:
VMs should start normally.

Additional info:
The RHEVM engine logs contains the following messages:

2014-03-12 14:31:43,555 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-4-thread-12) [4e92c413] Command CreateVDSCommand(HostName = nari11.eng.lab.tlv.redhat.com, HostId = 8b1e989c-ccec-4ac6-b347-5f7df8ef3ccb, vmId=bc3328ae-ef45-4645-a858-b7e378785e80, vm=VM [desktop-from-server]) execution failed. Exception: VDSNetworkException: org.apache.xmlrpc.common.XmlRpcExtensionException: Null values aren't supported, if isEnabledForExtensions() == false
2014-03-12 14:31:43,555 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-4-thread-12) [4e92c413] FINISH, CreateVDSCommand, log id: 3a0e42b7
2014-03-12 14:31:43,555 ERROR [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-4-thread-12) [4e92c413] VDS::create Failed creating vm desktop-from-server in vds = 8b1e989c-ccec-4ac6-b347-5f7df8ef3ccb : nari11.eng.lab.tlv.redhat.com error = org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: org.apache.xmlrpc.common.XmlRpcExtensionException: Null values aren't supported, if isEnabledForExtensions() == false
2014-03-12 14:31:43,556 INFO  [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-4-thread-12) [4e92c413] FINISH, CreateVmVDSCommand, return: Down, log id: 6770d6d3
2014-03-12 14:31:43,557 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-4-thread-12) [4e92c413] Lock freed to object EngineLock [exclusiveLocks= key: bc3328ae-ef45-4645-a858-b7e378785e80 value: VM
, sharedLocks= ]
2014-03-12 14:31:43,557 ERROR [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-4-thread-12) [4e92c413] Command org.ovirt.engine.core.bll.RunVmCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: org.apache.xmlrpc.common.XmlRpcExtensionException: Null values aren't supported, if isEnabledForExtensions() == false (Failed with error VDS_NETWORK_ERROR and code 5022)
2014-03-12 14:31:43,595 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-4-thread-12) [4e92c413] Correlation ID: 4e92c413, Job ID: 55b44f79-a718-4f3b-b52c-ff1a11b55a41, Call Stack: null, Custom Event ID: -1, Message: Failed to run VM desktop-from-server (User: admin).


Important! : The workaround is to restart the ovirt-engine.
After the ovirt-engine is restarted VMs are started normally.

Comment 1 Roy Golan 2014-03-12 13:34:56 UTC
1. please be specific, whats the details of the vm that failed
2. provide the vdsm log, maybe we could see the xml there

Comment 2 Roy Golan 2014-03-12 15:12:38 UTC
(In reply to Roy Golan from comment #1)
> 1. please be specific, whats the details of the vm that failed
> 2. provide the vdsm log, maybe we could see the xml there

ignore 2. the call doesn't reach vdsm . it fails on the engine client (apache xmlRpc client)

Comment 5 Michal Skrivanek 2014-03-16 13:39:24 UTC
does it reproduce? or was it only for the first time after installation?
did you try to install again and it happened again?

Comment 6 Lev Veyde 2014-03-16 14:08:02 UTC
(In reply to Michal Skrivanek from comment #5)
> does it reproduce? or was it only for the first time after installation?
> did you try to install again and it happened again?

Hi Michal,

It seems to be quite reproducible and appears after the first setup.

Comment 7 Michal Skrivanek 2014-03-17 09:32:18 UTC
doesn't reproduce on upstream 3.4

Comment 8 Michal Skrivanek 2014-03-17 10:22:54 UTC
(In reply to Michal Skrivanek from comment #7)
eventually it does:)
though it happens only in the Default cluster (due to its arch initially created as Unknown)
New cluster always enforces a specific arch in the dialog.
Another workaround is to put the host to Maintenance and then Activate

Comment 12 Roy Golan 2014-03-18 06:46:54 UTC
analysis

setup:
 use Default cluster or any cluster with arch "Undefined"
 install a host, wait for it to go UP ; don't "Activate"

outcome:
 emulated_machine field of the cluster is never set (i.e null) by the host
 running a VM fails - XML rpc client refuses to send null values

cause:
 different flow between Activate and Install - 
 Activate:
  - we call VdsManager.refreshHost which set auto-detect treu (because arch == Undefined)
  - monitoring cycle starts, Host goes UP, we call VdsManager.refreshHost again,  the emulated machine is now set
 
 Install:
 - we put the host on status Initializing and wait for monitoring to kick in
 - monitoring ends, host goes Up, we call VdsManager.refreshHost, we set auto-detect to true and emulated machine is null

root cause:
 refreshHost first tries to detect emulated machine (proccesSoftwareCapabilities) and update the cluster and cpu and arch - which sets emulated machine to null if the arch has change 

monitoringStrategy.processHardwareCapabilities(vds);

Comment 13 Pavel Novotny 2014-03-18 19:46:56 UTC
Verified in rhevm-3.4.0-0.9.beta1.el6ev (av3.1).

Verification steps (same used for reproducing the bug in rhevm-3.4.0-0.4.master.el6ev):
1. On a clean RHEL 6.5 machine install RHEVM:
  # yum install rhevm-setup -y
  # rhevm-setup
  ...use defaults wherever possible...
2. In RHEVM Webadmin, add new host and wait until it's Up.
3. Add new storage (I used NFS), wait until it's Active.
4. Add new VM (fill in only the VM name). Add a disk to the VM so it can run (1GB, Thin Provision, Activate, Bootable).
5. Run VM.

Results: 
The VM is successfully launched, no errors in engine.log or vdsm.log.

Comment 15 Ilanit Stein 2014-03-19 08:34:29 UTC
Also had to chance to see start VM work fine on av3.1 (same flow as in comment #13)

Comment 16 Kiril Nesenko 2014-05-08 15:16:32 UTC
Issue still happens on 9cf2c62b6eea35bd5bf4186a040ccb7c2b4c9082

Comment 17 Michal Skrivanek 2014-05-09 05:59:32 UTC
(In reply to Kiril Nesenko from comment #16)
logs / access info / anything?
Is it exactly the same issue?
Does the workaround from bug 1077300 help?

Comment 21 Pavel Novotny 2014-05-12 12:53:54 UTC
Verified in rhevm-3.4.0-0.18.el6ev.noarch (av9).

Verified manaually and also in automated test scenario.
The verification steps were exactly the same as in comment 13, just copy&pasting:

1. On a clean RHEL 6.5 machine install RHEVM:
  # yum install rhevm-setup -y
  # rhevm-setup
  ...use defaults wherever possible...
2. In RHEVM Webadmin, add new host and wait until it's Up.
3. Add new storage (I used NFS), wait until it's Active.
4. Add new VM (fill in only the VM name). Add a disk to the VM so it can run (1GB, Thin Provision, Activate, Bootable).
5. Run VM.

Results: 
The VM is successfully launched, no errors in engine.log or vdsm.log.

Comment 23 Itamar Heim 2014-06-12 14:08:05 UTC
Closing as part of 3.4.0


Note You need to log in before you can comment on or make changes to this bug.