Bug 1866780 - hosted-engine upgrade to 4.4 with dwh on remote machine fails due to grafana
Summary: hosted-engine upgrade to 4.4 with dwh on remote machine fails due to grafana
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-appliance
Classification: oVirt
Component: General
Version: 4.4.0
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: ovirt-4.4.2
: 4.4.2
Assignee: Yedidyah Bar David
QA Contact: Pavel Novotny
URL:
Whiteboard:
Depends On:
Blocks: 1870122 1980315 1986393
TreeView+ depends on / blocked
 
Reported: 2020-08-06 10:38 UTC by Yedidyah Bar David
Modified: 2021-07-27 12:47 UTC (History)
3 users (show)

Fixed In Version: ovirt-engine-appliance-4.4-20200826172858.1.el8
Doc Type: Bug Fix
Doc Text:
Cause: In engine-setup, if grafana is not configured, we defaulted to configure it. If DWH DB was remote, we then queried the user for credentials of a DWH db user for grafana. hosted-engine deploy runs engine-setup non-interactivley, so could not query the user. Consequence: Upgrade of a 4.3 hosted-engine setup where DWH DB was remote fails. Fix: The appliance now includes a change to the answer file provided to engine-setup, to not configure grafana. Result: 4.4 hosted-engine deploy now does not configure grafana, whether upgrade or new, whether dwh db is remote or local. For configuring grafana after deploy, user should connect to the engine machine, and run: 'engine-setup --reconfigure-optional-components'.
Clone Of:
: 1870122 (view as bug list)
Environment:
Last Closed: 2020-09-18 07:11:56 UTC
oVirt Team: Integration
Embargoed:
sbonazzo: ovirt-4.4?
sbonazzo: planning_ack?
sbonazzo: devel_ack?
lleistne: testing_ack+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 110869 0 master MERGED answer file: Disable grafana setup 2021-01-06 06:35:36 UTC
oVirt gerrit 110962 0 ovirt-4.4 MERGED answer file: Disable grafana setup 2021-01-06 06:36:12 UTC

Description Yedidyah Bar David 2020-08-06 10:38:56 UTC
Description of problem:

If a 4.3 hosted engine is set up with a remote dwh, upgrading it to 4.4 fails. It repeatedly "asks" the user for grafana db user credentials, but because it runs with --accept-defaults, without user interaction, it does this in a loop. Since we recently limited this loop to 1000 times (bug 1840756), it eventually fails.

The log contains lots of copies of:

Grafana database host []: \n[ ERROR ] Host name is not valid:  did not resolve into an IP address\n          Grafana database host []: \n[ ERROR ] Host name is not valid:  did not resolve into an IP address\n          Grafana database host []: \n[ ERROR ] Host name is not valid:  did not resolve into an IP address\n          Grafana database host []: \n[ ERROR ] Host name is not valid:  did not resolve into an IP address\n          Grafana database host []: \n[ ERROR ] Host name is not valid:  did not resolve into an IP address\n          Grafana database host []: \n[ ERROR ] Host name is not valid:  did not resolve into an IP address\n          Grafana database host []: \n[ ERROR ] Host name is not valid:  did not resolve into an IP address\n          Grafana database host []:

Version-Release number of selected component (if applicable):
4.4

How reproducible:
Always

Steps to Reproduce:
1. Setup 4.3 hosted-engine
2. Migrate DWH to a separate machine
3. Upgrade to 4.4

Actual results:
See above

Expected results:
Succeeds

Additional info:

Not sure if we want to be "smart" and disable grafana setup only if dwh is not included in the backup, or simply always, for hosted-engine (and then the patch is just a single line in the appliance.

A workaround (didn't verify, trying it as of writing):

Create a file e.g. /usr/share/ansible/roles/ovirt.hosted_engine_setup/hooks/enginevm_before_engine_setup/disable-grafana.yml , with content:

- name: Disable grafana
  lineinfile:
    path: /root/ovirt-engine-answers
    line: "OVESETUP_GRAFANA_CORE/enable=bool:False"
    state: present

Comment 1 Yedidyah Bar David 2020-08-18 10:12:48 UTC
Had an internal discussion, and decided to patch the appliance - add a single line to the engine-setup answer file, to not configure grafana. Result:

Standalone setups (both new install and upgrade) will configure grafana by default

Hosted-engine setups (both new install and upgrade) will not configure grafana.

For configuring grafana later, login to the dwh machine (can be same as engine, or separate - if migrated etc.) and run:

engine-setup --reconfigure-optional-components

Comment 2 Pavel Novotny 2020-09-11 12:58:12 UTC
Verified in
ovirt-engine-appliance-4.4-20200906174136.1.el8

Deployment of hosted-engine passed, no errors encountered regarding engine-setup with answer file.
Grafana was not configured. Verified it's disabled in the answer file:

~~~ ovirt-engine-setup.log
...
2020-09-11 14:46:55,961+0300 DEBUG otopi.context context.dumpEnvironment:775 ENV OVESETUP_GRAFANA_CORE/enable=bool:'False'
...
~~~

Grafana can be additionally installed via `engine-setup --reconfigure-optional-components`.

Comment 3 Sandro Bonazzola 2020-09-18 07:11:56 UTC
This bugzilla is included in oVirt 4.4.2 release, published on September 17th 2020.

Since the problem described in this bug report should be resolved in oVirt 4.4.2 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.