Bug 1388989

Summary: RunVmOnce/RunVm fails with error 500
Product: [oVirt] ovirt-engine Reporter: Mor <mkalfon>
Component: BLL.VirtAssignee: Michal Skrivanek <michal.skrivanek>
Status: CLOSED UPSTREAM QA Contact: meital avital <mavital>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 4.1.0CC: bugs, mkalfon, myakove, tjelinek
Target Milestone: ---Keywords: AutomationBlocker, Regression
Target Release: ---Flags: mkalfon: planning_ack?
rule-engine: devel_ack?
mkalfon: testing_ack?
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-01 14:51:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Virt RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
server log
none
engine log
none
UI log
none
engine log from different environment
none
engine log , last version
none
engine.log with RunVm failure none

Description Mor 2016-10-26 15:48:10 UTC
Description of problem:
When using the run once option to start a new VM, an error is displayed:
"Error while executing action: A Request to the Server failed with the following Status Code: 500"

Version-Release number of selected component (if applicable):
oVirt Engine Version: 4.1.0-0.0.master.20161025231555.gitf10966b.el7.centos

How reproducible:
100%

Steps to Reproduce:
1. Create a new VM with disk attached.
2. Use the run once on the VM.
3. Press OK.

Actual results:
Error message: "Error while executing action: A Request to the Server failed with the following Status Code: 500"

Expected results:
VM should be started.

Additional info:
engine.log:
2016-10-26 18:42:57,454 ERROR [org.ovirt.engine.core.bll.CommandsFactory] (default task-1) [] Error in invocating CTOR of command 'RunVmOnce' with parameters 'RunVmOnceParams:{commandId='95ec51e8-dc88-4097-a58c-769dee66d05a', user='null', commandType='Unknown', vmId='e3fe2637-9dff-4f62-93de-9aea95effae6'}': null

Comment 1 Red Hat Bugzilla Rules Engine 2016-10-26 15:48:18 UTC
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 2 Meni Yakove 2016-10-26 18:16:56 UTC
please attach logs.

Comment 3 Mor 2016-10-26 18:31:30 UTC
Created attachment 1214382 [details]
server log

Comment 4 Mor 2016-10-26 18:32:16 UTC
Created attachment 1214383 [details]
engine log

Comment 5 Mor 2016-10-26 18:32:48 UTC
Created attachment 1214384 [details]
UI log

Comment 6 Yaniv Kaul 2016-10-27 07:24:38 UTC
1. Please compress logs.
2. There's something very wrong with your setup. The flood of:
2016-10-26 15:53:52,274 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogableBase] (VdsDeploy) [5bc4e53a] Failed to get vds 'db54b5b5-e430-4f04-8ad4-51991c21d85d', error: null
2016-10-26 15:53:52,274 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogableBase] (VdsDeploy) [5bc4e53a] Failed to get vds 'db54b5b5-e430-4f04-8ad4-51991c21d85d', error: null
2016-10-26 15:53:52,274 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogableBase] (VdsDeploy) [5bc4e53a] Failed to get vds 'db54b5b5-e430-4f04-8ad4-51991c21d85d', error null
2016-10-26 15:53:52,274 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogableBase] (VdsDeploy) [5bc4e53a] Failed to get vds 'db54b5b5-e430-4f04-8ad4-51991c21d85d', error null

Makes no sense. Please fix this first.

Comment 7 Yaniv Kaul 2016-10-27 07:26:49 UTC
Did you noticed the (unrelated) exception in server.log?
2016-10-26 16:14:06,720 INFO  [org.quartz.impl.StdSchedulerFactory] (ServerService Thread Pool -- 65) Quartz scheduler version: 2.1.2
2016-10-26 16:14:06,742 ERROR [io.undertow.request] (default task-7) UT005023: Exception handling request to /ovirt-engine/: java.lang.NullPointerException
	at org.ovirt.engine.core.common.config.Config.getValue(Config.java:22) [common.jar:]
	at org.ovirt.engine.core.common.config.Config.getValue(Config.java:18) [common.jar:]
	at org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper.getLocalesKeys(UnsupportedLocaleHelper.java:57) [utils.jar:]
	at org.ovirt.engine.core.utils.servlet.UnsupportedLocaleHelper.getDisplayedLocales(UnsupportedLocaleHelper.java:18) [utils.jar:]
	at org.ovirt.engine.core.WelcomeServlet.doGet(WelcomeServlet.java:142) [welcome.jar:]

Which locale are you using? Can you reproduce on a clean environment and report separately?

Comment 8 Mor 2016-10-27 08:16:15 UTC
locale settings:

LANG=en_US.UTF-8
LC_CTYPE=en_US.UTF-8
LC_NUMERIC=en_US.UTF-8
LC_TIME=en_US.UTF-8
LC_COLLATE="en_US.UTF-8"
LC_MONETARY=en_US.UTF-8
LC_MESSAGES="en_US.UTF-8"
LC_PAPER=en_US.UTF-8
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT=en_US.UTF-8
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

Yes, it's reproducible on several environments, also on a clean one.

Comment 9 Yaniv Kaul 2016-10-27 08:19:51 UTC
(In reply to Mor from comment #8)
> locale settings:
> 
> LANG=en_US.UTF-8
> LC_CTYPE=en_US.UTF-8
> LC_NUMERIC=en_US.UTF-8
> LC_TIME=en_US.UTF-8
> LC_COLLATE="en_US.UTF-8"
> LC_MONETARY=en_US.UTF-8
> LC_MESSAGES="en_US.UTF-8"
> LC_PAPER=en_US.UTF-8
> LC_NAME="en_US.UTF-8"
> LC_ADDRESS="en_US.UTF-8"
> LC_TELEPHONE="en_US.UTF-8"
> LC_MEASUREMENT=en_US.UTF-8
> LC_IDENTIFICATION="en_US.UTF-8"
> LC_ALL=
> 
> Yes, it's reproducible on several environments, also on a clean one.

That's your browser setting?

In any case, a separate bug please.

Comment 10 Mor 2016-10-27 08:34:49 UTC
No, its the locale setting on the engine.

Comment 11 Ori Ben Sasson 2016-10-27 08:42:17 UTC
Created attachment 1214525 [details]
engine log from different environment

Comment 12 Ori Ben Sasson 2016-10-27 08:49:52 UTC
Created attachment 1214526 [details]
engine log , last version

Comment 13 Mor 2016-10-27 14:57:07 UTC
Issue still exists in version 4.1.0-0.0.master.20161026211311.git00a22c4.el7.centos, and in addition, RunVm command now fails.
Fails with error message:
ed869c19642, Call Stack: null, Custom Event ID: -1, Message: Failed to run VM VM-1 (User: admin@internal-authz).
2016-10-27 17:52:10,673 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-7) [66572a6c] Lock freed to object 'EngineLock:{exclusiveLocks='[e0f2f1e7-1fef-4ba9-9be6-4fee0ae6fcc9=<VM, ACTION_TYPE_FAILED_OBJECT_LOCKED>]', sharedLocks='null'}'

Comment 14 Mor 2016-10-27 14:59:54 UTC
Created attachment 1214652 [details]
engine.log with RunVm failure

Comment 15 Tomas Jelinek 2016-11-01 10:07:47 UTC
It seems it has been fixed by https://gerrit.ovirt.org/#/c/65784/

@Mor: could you please check if it works when you use a build containing this patch?

Comment 16 Mor 2016-11-01 11:24:41 UTC
Fixed on oVirt Engine Version: 4.1.0-0.0.master.20161029211316.git1d74709.el7.centos

Comment 17 Tomas Jelinek 2016-11-01 14:51:29 UTC
great, closing