Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2026473

Summary: Cannot upgrade the Engine database schema due to wrong ownership of some database entities
Product: [oVirt] ovirt-engine Reporter: RaSca <rasca>
Component: Database.CoreAssignee: Eli Mesika <emesika>
Status: CLOSED CURRENTRELEASE QA Contact: Lucie Leistnerova <lleistne>
Severity: high Docs Contact:
Priority: high    
Version: 4.4.9.5CC: bugs, gdeolive, mavital, mperina
Target Milestone: ovirt-4.5.0Flags: mperina: ovirt-4.5+
Target Release: 4.5.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: ovirt-engine-4.5.0 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-05-30 06:42:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
ovirt-engine-setup log none

Description RaSca 2021-11-24 18:45:52 UTC
Created attachment 1843464 [details]
ovirt-engine-setup log

Description of problem:

While upgrading from oVirt 4.4.4 to 4.4.9 engine-setup command ends in error while doing the 'Setup validation' step.

Version-Release number of selected component (if applicable):

ovirt-engine-setup-base-4.4.9.5-1.el8.noarch

How reproducible:

Starting from a 4.4.4 version just follow the official upgrade between minor releases procedure here https://www.ovirt.org/documentation/upgrade_guide/#updates-between-minor-releases and launch engine-setup.

Actual results:

[ INFO  ] Stage: Setup validation
          During execution engine service will be stopped (OK, Cancel) [OK]: 
[ ERROR ] Failed to execute stage 'Setup validation': Cannot upgrade the Engine database schema due to wrong ownership of some database entities.
         
[ INFO  ] Stage: Clean up
          Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20211124175322-28e5wg.log
[ INFO  ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20211124175333-setup.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ ERROR ] Execution of setup failed

Expected results:

Successful setp.

Additional info:

Attached the /var/log/ovirt-engine/setup/ovirt-engine-setup-20211124175322-28e5wg.log produced by the installer.

Comment 1 RHEL Program Management 2021-11-24 18:45:59 UTC
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 2 RHEL Program Management 2021-11-25 07:20:00 UTC
The documentation text flag should only be set after 'doc text' field is provided. Please provide the documentation text and set the flag to '?' again.

Comment 3 Sandro Bonazzola 2021-11-25 07:22:27 UTC
2021-11-24 17:53:33,871+0100 DEBUG otopi.ovirt_engine_setup.engine_common.database database.execute:234 Database: 'None', Statement: '
                select
                    nsp.nspname as object_schema,
                    cls.relname as object_name,
                    rol.rolname as owner,
                    case cls.relkind
                        when 'r' then 'TABLE'
                        when 'i' then 'INDEX'
                        when 'S' then 'SEQUENCE'
                        when 'v' then 'VIEW'
                        when 'c' then 'TYPE'
                    else
                        cls.relkind::text
                    end as object_type
                from
                    pg_class cls join
                    pg_roles rol on rol.oid = cls.relowner join
                    pg_namespace nsp on nsp.oid = cls.relnamespace
                where
                    nsp.nspname not in ('information_schema', 'pg_catalog') and
                    nsp.nspname not like 'pg_%%' and
                    cls.relname not like 'pg_%%' and
                    rol.rolname != %(user)s
                order by
                    nsp.nspname,
                    cls.relname
            ', args: {'user': 'engine'}
2021-11-24 17:53:33,872+0100 DEBUG otopi.ovirt_engine_setup.engine_common.database database.execute:239 Creating own connection
2021-11-24 17:53:33,887+0100 DEBUG otopi.ovirt_engine_setup.engine_common.database database.execute:284 Result: [{'object_schema': 'public', 'object_name': 'fk_info_rs', 'owner': 'postgres', 'object_type': 'TYPE'}]

Comment 4 Eli Mesika 2021-11-30 07:57:28 UTC
A short update on investigation 

It seems that  DB validations plugin executed from packaging/setup/plugins/ovirt-engine-setup/ovirt-engine/upgrade/dbvalidations.py is executing the fkvalidator.sh with the "postgres" user instead of the "engine"
fkvalidator.sh creates some helper db objects on-the-fly 

As a result , the database validation _checkDatabaseOwnership function in packaging/setup/plugins/ovirt-engine-setup/ovirt-engine/db/schema.py fails because it finds some db objects owned by "postgres" instead of by "engine"

for example , the following TYPE:

[{'object_schema': 'public', 'object_name': 'fk_info_rs', 'owner': 'postgres', 'object_type': 'TYPE'}]

Please attach a database dump in order to verify that this is the problem

Comment 5 RaSca 2021-12-01 14:51:07 UTC
Hey Eli,
Thanks for your time.
We were able to fix our problem by first verifying the result of the query:

engine=# select
                    nsp.nspname as object_schema,
                    cls.relname as object_name,
                    rol.rolname as owner,
                    case cls.relkind
                        when 'r' then 'TABLE'
                        when 'i' then 'INDEX'
                        when 'S' then 'SEQUENCE'
                        when 'v' then 'VIEW'
                        when 'c' then 'TYPE'
                    else
                        cls.relkind::text
                    end as object_type
                from
                    pg_class cls join
                    pg_roles rol on rol.oid = cls.relowner join
                    pg_namespace nsp on nsp.oid = cls.relnamespace
                where
                    nsp.nspname not in ('information_schema', 'pg_catalog') and
                    nsp.nspname not like 'pg_%%' and
                    cls.relname not like 'pg_%%' and
                    rol.rolname != 'engine'
                order by
                    nsp.nspname,
                    cls.relname;

That was, as expected:

 object_schema | object_name |  owner   | object_type 
---------------+-------------+----------+-------------
 public        | fk_info_rs  | postgres | TYPE
(1 row)

And then fixing the type so that the owner was engine instead of postgres:

engine=# alter type fk_info_rs owner to engine;
ALTER TYPE

After verifying the query again:

engine=# select                                  
                    nsp.nspname as object_schema,
                    cls.relname as object_name,
                    rol.rolname as owner,
                    case cls.relkind
                        when 'r' then 'TABLE'
                        when 'i' then 'INDEX'
                        when 'S' then 'SEQUENCE'
                        when 'v' then 'VIEW'
                        when 'c' then 'TYPE'
                    else
                        cls.relkind::text
                    end as object_type
                from
                    pg_class cls join
                    pg_roles rol on rol.oid = cls.relowner join
                    pg_namespace nsp on nsp.oid = cls.relnamespace
                where
                    nsp.nspname not in ('information_schema', 'pg_catalog') and
                    nsp.nspname not like 'pg_%%' and
                    cls.relname not like 'pg_%%' and
                    rol.rolname != 'engine'
                order by
                    nsp.nspname,
                    cls.relname;

With no output:

 object_schema | object_name | owner | object_type 
---------------+-------------+-------+-------------
(0 rows)

We relaunched the engine config command and everything ended smoothly.

Thanks again for the help.

Comment 6 Eli Mesika 2021-12-02 09:11:51 UTC
Yes 

But the bug is still relevant 

This fk_info_rs TYPE which defines the fn_db_validate_fks that uses it are created by fkvalidator.sh and should be dropped afterwards 
(see that ./packaging/setup/dbutils/fkvalidator.sh has a cleanup method that is trapped and executed)

So, we must check why we have this fk_info_rs TYPE and it was not cleaned after fkvalidator.sh completes

Comment 7 Eli Mesika 2021-12-05 11:33:57 UTC
I did not succeed to reproduce the issue.
I suppose that this may occur only by manual termination,
so the DB remains with uncleaned temporary object.

I have added CASCADE to the function DROP statement just to be on the
safe side.

Comment 8 Eli Mesika 2021-12-06 12:32:13 UTC
Only way to try to reproduce this issue for QE :

While upgrading from oVirt 4.4.4 to 4.4.9 

1) Kill manually engine-setup
2) run engine-setup again

Comment 9 Lucie Leistnerova 2022-05-23 14:14:36 UTC
Tested in ovirt-engine-4.5.0.7-0.9.el8ev.noarch

Steps:
1. installed 4.4.4 engine
2. did run engine-upgrade-check - no object with postgres owner
3. interrupted engine-setup in Misc configuration (early) - no object with postgres owner
4. interrupted engine-setup in Misc configuration - no object with postgres owner
5. interrupted engine-setup in Creating/refreshing Engine 'internal' domain database schema - no object with postgres owner 
6. engine-setup - succeeded

So I did not reproduce the issue. Database user running the fkvalidator is defined by OVESETUP_DB/user and I did use the default value.
Feel free to reopen when it happens again.