Bug 1802270

Summary: Cluster Upgrade fails to run, AnsibleServlet fails to start the upgrade ansible playbook
Product: [oVirt] ovirt-engine Reporter: Scott Dickerson <sdickers>
Component: Backend.CoreAssignee: Martin Necas <mnecas>
Status: CLOSED CURRENTRELEASE QA Contact: Petr Kubica <pkubica>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.4.0CC: bpelled, bugs, mnecas, mperina, rchikatw, sgratch
Target Milestone: ovirt-4.4.0Flags: pm-rhel: ovirt-4.4+
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ansible-runner-1.4.5, ovirt-engine-4.4.0_beta1 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-20 20:02:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1807777    
Bug Blocks:    

Description Scott Dickerson 2020-02-12 18:30:20 UTC 
Description of problem:

On webadmin, attempting to run a cluster upgrade fails on the engine.


Version-Release number of selected component (if applicable): engine 4.4


How reproducible: 100%


Steps to Reproduce:
1. In webadmin select a Cluster
2. Click 'Upgrade' and page through the Wizard
3. Click the final blue 'Upgrade' button

Actual results: Toast notifiction "Failed to start the cluster upgrade on XXX"


Expected results: Toast notification that the cluster upgrade has started.


Additional info:
The functionality stops working after BZ1759143 was completed.

While the cluster upgrade wizard is part of ovirt-engine-ui-extensions package, it does use the AnsibleServlet to run a playbook that, in turn, invokes the cluster upgrade role.  The servlet is responding error code 500 and no ansible logs are created.

Relevant server log:
2020-02-12 13:25:49,634-05 TRACE [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] (default task-27) [] Enter AnsibleExecutor::runCommand
2020-02-12 13:25:49,636-05 ERROR [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] (default task-27) [] Exception: null
2020-02-12 13:25:49,636-05 DEBUG [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] (default task-27) [] Exception: : java.lang.NullPointerException
	at deployment.engine.ear.bll.jar//org.ovirt.engine.core.common.utils.ansible.AnsibleRunnerHTTPClient.runPlaybook(AnsibleRunnerHTTPClient.java:103)
	at deployment.engine.ear.bll.jar//org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor.runCommand(AnsibleExecutor.java:111)
	at deployment.engine.ear.bll.jar//org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor.runCommand(AnsibleExecutor.java:58)
	at deployment.engine.ear.services.war//org.ovirt.engine.core.services.AnsibleServlet.lambda$doPost$0(AnsibleServlet.java:92)
	at io.undertow.servlet.26.Final//io.undertow.servlet.spec.ServletContextImpl$2.call(ServletContextImpl.java:181)
	at io.undertow.servlet.26.Final//io.undertow.servlet.spec.ServletContextImpl$2.call(ServletContextImpl.java:178)
	at io.undertow.servlet.26.Final//io.undertow.servlet.spec.ServletContextImpl.invokeRunnable(ServletContextImpl.java:1032)
	at io.undertow.servlet.26.Final//io.undertow.servlet.spec.AsyncContextImpl$4.run(AsyncContextImpl.java:319)
	at org.jboss.threads.3.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
	at org.jboss.threads.3.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)
	at org.jboss.threads.3.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
	at org.jboss.threads.3.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
	at java.base/java.lang.Thread.run(Thread.java:834)

2020-02-12 13:25:49,636-05 ERROR [org.ovirt.engine.core.services.AnsibleServlet] (default task-27) [] Error while executing ansible-playbook command.
Comment 3 Martin Necas 2020-03-08 16:30:22 UTC 
I'm waiting for mine PRs to get merged.
Comment 4 Martin Necas 2020-03-12 14:54:57 UTC 
Waiting for the release of ansible-runner.
Comment 5 Ritesh Chikatwar 2020-03-12 15:13:32 UTC 
Any idea that when will ansible-runner will release as it's blocker for me for this bug.no.-1721366
Comment 6 Martin Perina 2020-03-23 09:36:49 UTC 
Moving to 4.4.1, we need to wait for ansible-runner-1.4.5 release
Comment 7 Martin Perina 2020-03-26 09:08:06 UTC 
Requires ansible-runner 1.4.5 installed to work properly
Comment 8 Sandro Bonazzola 2020-03-30 13:43:29 UTC 
This bug is in modified state and targeted to 4.4.1. Can we re-target to 4.4.0 and move to QE?
Comment 9 Sandro Bonazzola 2020-04-09 08:33:31 UTC 
ansible-runner 1.4.5 is now available.
Comment 10 Petr Kubica 2020-04-23 23:31:19 UTC 
Verified in 4.4.0-0.33.master.el8ev
tested cluster upgrade

also ansible-runner-1.4.5-1.el8ar.noarch located on engine machine
Comment 11 Sandro Bonazzola 2020-05-20 20:02:04 UTC 
This bugzilla is included in oVirt 4.4.0 release, published on May 20th 2020.

Since the problem described in this bug report should be
resolved in oVirt 4.4.0 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.
Comment 12 Red Hat Bugzilla 2023-09-15 00:29:34 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days