Bug 1866780

Summary:	hosted-engine upgrade to 4.4 with dwh on remote machine fails due to grafana
Product:	[oVirt] ovirt-appliance	Reporter:	Yedidyah Bar David <didi>
Component:	General	Assignee:	Yedidyah Bar David <didi>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Pavel Novotny <pnovotny>
Severity:	high	Docs Contact:
Priority:	urgent
Version:	4.4.0	CC:	bugs, emarcus, lleistne
Target Milestone:	ovirt-4.4.2	Flags:	sbonazzo: ovirt-4.4? sbonazzo: planning_ack? sbonazzo: devel_ack? lleistne: testing_ack+
Target Release:	4.4.2
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	ovirt-engine-appliance-4.4-20200826172858.1.el8	Doc Type:	Bug Fix
Doc Text:	Cause: In engine-setup, if grafana is not configured, we defaulted to configure it. If DWH DB was remote, we then queried the user for credentials of a DWH db user for grafana. hosted-engine deploy runs engine-setup non-interactivley, so could not query the user. Consequence: Upgrade of a 4.3 hosted-engine setup where DWH DB was remote fails. Fix: The appliance now includes a change to the answer file provided to engine-setup, to not configure grafana. Result: 4.4 hosted-engine deploy now does not configure grafana, whether upgrade or new, whether dwh db is remote or local. For configuring grafana after deploy, user should connect to the engine machine, and run: 'engine-setup --reconfigure-optional-components'.	Story Points:	---
Clone Of:
Clones:	1870122 (view as bug list)		Environment:
Last Closed:	2020-09-18 07:11:56 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	Integration	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1870122, 1980315, 1986393

Description Yedidyah Bar David 2020-08-06 10:38:56 UTC

Description of problem:

If a 4.3 hosted engine is set up with a remote dwh, upgrading it to 4.4 fails. It repeatedly "asks" the user for grafana db user credentials, but because it runs with --accept-defaults, without user interaction, it does this in a loop. Since we recently limited this loop to 1000 times (bug 1840756), it eventually fails.

The log contains lots of copies of:

Grafana database host []: \n[ ERROR ] Host name is not valid:  did not resolve into an IP address\n          Grafana database host []: \n[ ERROR ] Host name is not valid:  did not resolve into an IP address\n          Grafana database host []: \n[ ERROR ] Host name is not valid:  did not resolve into an IP address\n          Grafana database host []: \n[ ERROR ] Host name is not valid:  did not resolve into an IP address\n          Grafana database host []: \n[ ERROR ] Host name is not valid:  did not resolve into an IP address\n          Grafana database host []: \n[ ERROR ] Host name is not valid:  did not resolve into an IP address\n          Grafana database host []: \n[ ERROR ] Host name is not valid:  did not resolve into an IP address\n          Grafana database host []:

Version-Release number of selected component (if applicable):
4.4

How reproducible:
Always

Steps to Reproduce:
1. Setup 4.3 hosted-engine
2. Migrate DWH to a separate machine
3. Upgrade to 4.4

Actual results:
See above

Expected results:
Succeeds

Additional info:

Not sure if we want to be "smart" and disable grafana setup only if dwh is not included in the backup, or simply always, for hosted-engine (and then the patch is just a single line in the appliance.

A workaround (didn't verify, trying it as of writing):

Create a file e.g. /usr/share/ansible/roles/ovirt.hosted_engine_setup/hooks/enginevm_before_engine_setup/disable-grafana.yml , with content:

- name: Disable grafana
  lineinfile:
    path: /root/ovirt-engine-answers
    line: "OVESETUP_GRAFANA_CORE/enable=bool:False"
    state: present

Comment 1 Yedidyah Bar David 2020-08-18 10:12:48 UTC

Had an internal discussion, and decided to patch the appliance - add a single line to the engine-setup answer file, to not configure grafana. Result:

Standalone setups (both new install and upgrade) will configure grafana by default

Hosted-engine setups (both new install and upgrade) will not configure grafana.

For configuring grafana later, login to the dwh machine (can be same as engine, or separate - if migrated etc.) and run:

engine-setup --reconfigure-optional-components

Comment 2 Pavel Novotny 2020-09-11 12:58:12 UTC

Verified in
ovirt-engine-appliance-4.4-20200906174136.1.el8

Deployment of hosted-engine passed, no errors encountered regarding engine-setup with answer file.
Grafana was not configured. Verified it's disabled in the answer file:

~~~ ovirt-engine-setup.log
...
2020-09-11 14:46:55,961+0300 DEBUG otopi.context context.dumpEnvironment:775 ENV OVESETUP_GRAFANA_CORE/enable=bool:'False'
...
~~~

Grafana can be additionally installed via `engine-setup --reconfigure-optional-components`.

Comment 3 Sandro Bonazzola 2020-09-18 07:11:56 UTC

This bugzilla is included in oVirt 4.4.2 release, published on September 17th 2020.

Since the problem described in this bug report should be resolved in oVirt 4.4.2 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.