Created attachment 873534 [details] Engine log both before and after the restart Description of problem: VM's can't be started from RHEVM WebAdmin with Failed to run VM <vm-name> (User: admin). message. Version-Release number of selected component (if applicable): rhevm-3.4.0-0.4.master.el6ev.noarch How reproducible: N/A Steps to Reproduce: 1. Install rhevm-3.4.0-0.4.master.el6ev.noarch 2. Configure the env. adding host and storage 3. Create VMs 4. Attempt to start VMs Actual results: Start fails with error message of "Failed to run VM <vm-name> (User: admin)". Expected results: VMs should start normally. Additional info: The RHEVM engine logs contains the following messages: 2014-03-12 14:31:43,555 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-4-thread-12) [4e92c413] Command CreateVDSCommand(HostName = nari11.eng.lab.tlv.redhat.com, HostId = 8b1e989c-ccec-4ac6-b347-5f7df8ef3ccb, vmId=bc3328ae-ef45-4645-a858-b7e378785e80, vm=VM [desktop-from-server]) execution failed. Exception: VDSNetworkException: org.apache.xmlrpc.common.XmlRpcExtensionException: Null values aren't supported, if isEnabledForExtensions() == false 2014-03-12 14:31:43,555 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-4-thread-12) [4e92c413] FINISH, CreateVDSCommand, log id: 3a0e42b7 2014-03-12 14:31:43,555 ERROR [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-4-thread-12) [4e92c413] VDS::create Failed creating vm desktop-from-server in vds = 8b1e989c-ccec-4ac6-b347-5f7df8ef3ccb : nari11.eng.lab.tlv.redhat.com error = org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: org.apache.xmlrpc.common.XmlRpcExtensionException: Null values aren't supported, if isEnabledForExtensions() == false 2014-03-12 14:31:43,556 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-4-thread-12) [4e92c413] FINISH, CreateVmVDSCommand, return: Down, log id: 6770d6d3 2014-03-12 14:31:43,557 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-4-thread-12) [4e92c413] Lock freed to object EngineLock [exclusiveLocks= key: bc3328ae-ef45-4645-a858-b7e378785e80 value: VM , sharedLocks= ] 2014-03-12 14:31:43,557 ERROR [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-4-thread-12) [4e92c413] Command org.ovirt.engine.core.bll.RunVmCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: org.apache.xmlrpc.common.XmlRpcExtensionException: Null values aren't supported, if isEnabledForExtensions() == false (Failed with error VDS_NETWORK_ERROR and code 5022) 2014-03-12 14:31:43,595 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-4-thread-12) [4e92c413] Correlation ID: 4e92c413, Job ID: 55b44f79-a718-4f3b-b52c-ff1a11b55a41, Call Stack: null, Custom Event ID: -1, Message: Failed to run VM desktop-from-server (User: admin). Important! : The workaround is to restart the ovirt-engine. After the ovirt-engine is restarted VMs are started normally.
1. please be specific, whats the details of the vm that failed 2. provide the vdsm log, maybe we could see the xml there
(In reply to Roy Golan from comment #1) > 1. please be specific, whats the details of the vm that failed > 2. provide the vdsm log, maybe we could see the xml there ignore 2. the call doesn't reach vdsm . it fails on the engine client (apache xmlRpc client)
does it reproduce? or was it only for the first time after installation? did you try to install again and it happened again?
(In reply to Michal Skrivanek from comment #5) > does it reproduce? or was it only for the first time after installation? > did you try to install again and it happened again? Hi Michal, It seems to be quite reproducible and appears after the first setup.
doesn't reproduce on upstream 3.4
(In reply to Michal Skrivanek from comment #7) eventually it does:) though it happens only in the Default cluster (due to its arch initially created as Unknown) New cluster always enforces a specific arch in the dialog. Another workaround is to put the host to Maintenance and then Activate
analysis setup: use Default cluster or any cluster with arch "Undefined" install a host, wait for it to go UP ; don't "Activate" outcome: emulated_machine field of the cluster is never set (i.e null) by the host running a VM fails - XML rpc client refuses to send null values cause: different flow between Activate and Install - Activate: - we call VdsManager.refreshHost which set auto-detect treu (because arch == Undefined) - monitoring cycle starts, Host goes UP, we call VdsManager.refreshHost again, the emulated machine is now set Install: - we put the host on status Initializing and wait for monitoring to kick in - monitoring ends, host goes Up, we call VdsManager.refreshHost, we set auto-detect to true and emulated machine is null root cause: refreshHost first tries to detect emulated machine (proccesSoftwareCapabilities) and update the cluster and cpu and arch - which sets emulated machine to null if the arch has change monitoringStrategy.processHardwareCapabilities(vds);
Verified in rhevm-3.4.0-0.9.beta1.el6ev (av3.1). Verification steps (same used for reproducing the bug in rhevm-3.4.0-0.4.master.el6ev): 1. On a clean RHEL 6.5 machine install RHEVM: # yum install rhevm-setup -y # rhevm-setup ...use defaults wherever possible... 2. In RHEVM Webadmin, add new host and wait until it's Up. 3. Add new storage (I used NFS), wait until it's Active. 4. Add new VM (fill in only the VM name). Add a disk to the VM so it can run (1GB, Thin Provision, Activate, Bootable). 5. Run VM. Results: The VM is successfully launched, no errors in engine.log or vdsm.log.
Also had to chance to see start VM work fine on av3.1 (same flow as in comment #13)
Issue still happens on 9cf2c62b6eea35bd5bf4186a040ccb7c2b4c9082
(In reply to Kiril Nesenko from comment #16) logs / access info / anything? Is it exactly the same issue? Does the workaround from bug 1077300 help?
Verified in rhevm-3.4.0-0.18.el6ev.noarch (av9). Verified manaually and also in automated test scenario. The verification steps were exactly the same as in comment 13, just copy&pasting: 1. On a clean RHEL 6.5 machine install RHEVM: # yum install rhevm-setup -y # rhevm-setup ...use defaults wherever possible... 2. In RHEVM Webadmin, add new host and wait until it's Up. 3. Add new storage (I used NFS), wait until it's Active. 4. Add new VM (fill in only the VM name). Add a disk to the VM so it can run (1GB, Thin Provision, Activate, Bootable). 5. Run VM. Results: The VM is successfully launched, no errors in engine.log or vdsm.log.
Closing as part of 3.4.0