Bug 1846212 - After grafana setup, if next engine-setup fails, rollback fails
Summary: After grafana setup, if next engine-setup fails, rollback fails
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: Setup.EngineCommon
Version: 4.4.1
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: ovirt-4.4.1
: 4.4.1.4
Assignee: Yedidyah Bar David
QA Contact: Pavel Novotny
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-06-11 05:43 UTC by Yedidyah Bar David
Modified: 2020-08-05 06:25 UTC (History)
2 users (show)

Fixed In Version: ovirt-engine-4.4.1.4
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-08-05 06:25:00 UTC
oVirt Team: Metrics
Embargoed:
pm-rhel: ovirt-4.4+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 109615 0 master MERGED packaging: setup: rollback: Simplify PG errors filtering 2021-01-26 10:09:38 UTC
oVirt gerrit 109616 0 master MERGED packaging: setup: rollback: Ignore grafana user error 2021-01-26 10:09:38 UTC

Description Yedidyah Bar David 2020-06-11 05:43:35 UTC
Description of problem:

Creation of grafana db user is done in a way that causes pg_dump to dump statements that the dwh db user has no rights to execute, so if we need to rollback, we fail. This is in the setup log:

2020-06-11 08:31:05,641+0300 ERROR otopi.ovirt_engine_setup.engine_common.database database.restore:956 Errors while restoring ovirt_engine_history database, please check the log file for details
2020-06-11 08:31:05,641+0300 DEBUG otopi.ovirt_engine_setup.engine_common.database database.restore:961 Errors unfiltered during restore:

pg_restore: from TOC entry 3234; 826 16411 DEFAULT ACL DEFAULT PRIVILEGES FOR TABLES postgres
pg_restore: error: could not execute query: ERROR:  must be member of role "postgres"
Command was: ALTER DEFAULT PRIVILEGES FOR ROLE postgres IN SCHEMA public REVOKE ALL ON TABLES  FROM postgres;
ALTER DEFAULT PRIVILEGES FOR ROLE postgres IN SCHEMA public GRANT SELECT ON TABLES  TO ovirt_engine_history_grafana;

Version-Release number of selected component (if applicable):
Current master

How reproducible:
Always

Steps to Reproduce:
1. Install and setup engine+dwh+grafana
2. Run engine-setup again, make it fail after it updated dwh db schema
3.

Actual results:
Fails as above

Expected results:
Succeeds, by ignoring these errors - they can be ignored, because we create the grafana db user in a way that will still work even if these statements fail.

Additional info:
Noticed while checking bug 1839676, which is basically a duplicate except for grafana. Decided to also open current to not postpone verification of that one.

Comment 1 Pavel Novotny 2020-07-13 19:12:34 UTC
Verified in
ovirt-engine-4.4.1.8-0.7.el8ev.noarch
otopi-common-1.9.2-1.el8ev.noarch

Verification steps:
1. Install engine+dwh+grafana.
2. Run setup again and fail it after 'Creating/refreshing DWH database schema' step:
   - install otopi-debug-plugins
   - run: OTOPI_FORCE_FAIL_STAGE=STAGE_MISC OTOPI_FORCE_FAIL_PRIORITY=PRIORITY_LOW engine-setup

Result:
Setup fails after refreshing DWH DB schema step.
Rollback of DWH DB then succeeds, error 'pg_restore: error: could not execute query: ERROR:  must be member of role "postgres"' is ignored.

Console output:
-~-
...
[ INFO  ] Backing up database localhost:ovirt_engine_history to '/var/lib/ovirt-engine-dwh/backups/dwh-20200713205557.de65z02w.dump'.
[ INFO  ] Creating/refreshing DWH database schema
[ INFO  ] Configuring WebSocket Proxy
[ INFO  ] Backing up database localhost:engine to '/var/lib/ovirt-engine/backups/engine-20200713205611.dngmii2a.dump'.
[ INFO  ] Creating/refreshing Engine database schema
[ INFO  ] Creating/refreshing Engine 'internal' domain database schema
          Unregistering existing client registration info.
[ ERROR ] Failed to execute stage 'Misc configuration': Force Fail: stage 11 priority 9000
[ INFO  ] DNF Performing DNF transaction rollback
[WARNING] Rollback of DWH database postponed to Stage "Clean up"
[ INFO  ] Rolling back database schema
[ INFO  ] Clearing Engine database engine
[ INFO  ] Restoring Engine database engine
[ INFO  ] Restoring file '/var/lib/ovirt-engine/backups/engine-20200713205611.dngmii2a.dump' to database localhost:engine.
[ INFO  ] Stage: Clean up
          Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20200713204338-bxekez.log
[ INFO  ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20200713205757-setup.conf'
[WARNING] Rollback of DWH database started
          This might be a long process, but it should be safe to start the engine service before it finishes, if needed.
[ INFO  ] Clearing DWH database ovirt_engine_history
[ INFO  ] Restoring DWH database ovirt_engine_history
[ INFO  ] Restoring file '/var/lib/ovirt-engine-dwh/backups/dwh-20200713205557.de65z02w.dump' to database localhost:ovirt_engine_history.
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ ERROR ] Execution of setup failed
-~-

Comment 2 Sandro Bonazzola 2020-08-05 06:25:00 UTC
This bugzilla is included in oVirt 4.4.1 release, published on July 8th 2020.

Since the problem described in this bug report should be resolved in oVirt 4.4.1 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.