Bug 2077387

So there is something special inside your database which prevents the upgrade, I've just tested upgrade from 4.4.10.7 to 4.5.0.4 and it was successful. Could you please run sos-log-collector to gather all required information from your ovirt-engine (including database dump) and atach it to the bug so we can fully investigate the issue? (feel free to use "Make attachment and comment private (visible only to members of the redhat group)" option on the attachment so it's visible only to Red Hat).

Thanks

Here is the link for log collector tool documentation: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/html/administration_guide/chap-utilities_rhv_backup_restore#sect-The_Log_Collector_Tool

So following options in vdc_options table contains NULL as a default value:

AdminDomain
AuthenticationMethod
AllowDuplicateMacAddresses
DefaultWorkgroup
AdUserName
AdUserPassword
DomainName
LdapQueryPageSize
LDAPQueryTimeout
LDAPConnectTimeout
LDAPOperationTimeout
LDAPServerPort
LicenseCertificateFingerPrint
MaxLDAPQueryPartsNumber
MacPoolRanges
MaxMacsCountInPool
LdapServers
LDAPProviderTypes
LDAPSecurityAuthentication
VdsFenceOptions
AdUserId
VMMinMemorySizeInMB
EnableVdsHaReservation
KeystoneAuthUrl
GlusterTunedProfile

That is caused by an error in our internal fn_db_change_column_type function, which didn't handle setting NULL/NOT NULL column property. The function has been fixed along in https://gerrit.ovirt.org/117448 but we have missed options which no longer exists. Those options should have been deleted from vdc_options in previous releases, but unfortunately due to unknown reason they still exists. We have no easy way how to remove them now, so we need to fix their default values before properlly marking default value as NON NULL value.

As a workaround following SQL commands should be applied before performing an upgrade to 4.5:

1. Connect to the machine where ovirt-engine is installed
2. Run following commands:

   /usr/share/ovirt-engine/dbscripts/engine-psql.sh -c "UPDATE vdc_options SET default_value=option_value WHERE default_value IS NULL AND option_value IS NOT NULL;"
   /usr/share/ovirt-engine/dbscripts/engine-psql.sh -c "UPDATE vdc_options SET default_value='' WHERE default_value IS NULL AND option_value IS NULL;"

When successfully applied please continue with upgrade to 4.5

Could you please chec

Sorry, clicked on Save changes too early.

Could you please check above workaround?

The first query changed 25 rows, the second 0 rows.

The engine-setup worked like a charm and now I'm on 4.5.0.4-1.el8

Thanks for the quick solution.

(In reply to Giorgio Biacchi from comment #6)
> The first query changed 25 rows, the second 0 rows.
> 
> The engine-setup worked like a charm and now I'm on 4.5.0.4-1.el8
> 
> Thanks for the quick solution.

Great to hear it works, thanks a lot for confirmation!

Now I face another issue...

I run a yum update after the engine-setup to update the rest of the system and I've done a reboot since one of the updates was the kernel. Now I got "500 Internal server error" if I try to reach the UI via web.

I'm still looking for clues in the logs

Quickest solution ever :) I can confirm that downgrading postgresql-jdbc brings the UI back to life.

Thanks again

I've been using ovirt for a very long time, using the same database, and I can confirm the problem:

[root@ovirt ovirt-engine]# /usr/share/ovirt-engine/dbscripts/engine-psql.sh -c "select *  from vdc_options where default_value is null ;"
 option_id |        option_name         |                  option_value                  | version | default_value
-----------+----------------------------+------------------------------------------------+---------+---------------
       626 | ConfigDir                  | /etc/ovirt-engine                              | general |
       627 | AllowDuplicateMacAddresses | false                                          | general |
       645 | MacPoolRanges              | 00:<snip>-00:<snip>            | general |
       646 | MaxMacsCountInPool         | 100000                                         | general |
       625 | GlusterTunedProfile        | virtual-host,rhgs-sequential-io,rhgs-random-io | 4.2     |
(5 rows)

I dont go mucking about in my DB, so at some point these got added by the system without default values.

/usr/share/ovirt-engine/dbscripts/engine-psql.sh -c "UPDATE vdc_options SET default_value=option_value WHERE default_value IS NULL AND option_value IS NOT NULL;" obviously will fix the schema problem. 

dnf downgrade postgresql-jdbc is also required on centos8 stream as you noted.

Adding the line: exclude=postgresql-jdbc-42.2.14-1.el8 to /etc/yum.repos.d/CentOS-Stream-AppStream.rep is a good idea as well

The JDBC section of this bug relates to: https://bugzilla.redhat.com/show_bug.cgi?id=2077794

Tested in ovirt-engine-4.5.0.7-0.9.el8ev.noarch and engine-setup failed on the same error

Steps:
1. installed 4.4.4 engine
2. added AdminDomain with default_value = NULL
3. added 4.5 repos
4. dnf update ovirt\*setup\*
5. engine-setup

2022-05-23 15:20:43,619+0300 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine_common.distro-rpm.packages packages.verbose:194 DNF     install   : ovirt-engine-dbscripts-4.5.0.7-0.9.el8ev.noarch
...
2022-05-23 15:25:18,210+0300 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.db.schema plugin.execute:926 execute-output: ['/usr/share/ovirt-engine/dbscripts/schema.sh', '-s', 'localhost', '-p', '5432', '-u', 'engine', '-d', 'engine', '-l', '/var/log/ovirt-engine/setup/ovirt-engine-setup-20220523152034-jyxc56.log', '-c', 'apply'] stderr:
psql:/usr/share/ovirt-engine/dbscripts/upgrade/pre_upgrade/0000_config.sql:1454: ERROR:  column "default_value" contains null values
CONTEXT:  SQL statement "ALTER TABLE vdc_options ALTER COLUMN default_value SET NOT NULL"
PL/pgSQL function fn_db_change_column_null(character varying,character varying,boolean) line 10 at EXECUTE
FATAL: Cannot execute sql command: --file=/usr/share/ovirt-engine/dbscripts/upgrade/pre_upgrade/0000_config.sql


No 04_05_0100_fix_04_04_0050_set_to_not_null.sql did run

Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

(In reply to Lucie Leistnerova from comment #16)
> 
> No 04_05_0100_fix_04_04_0050_set_to_not_null.sql did run

Please provide the output of the following query on the database :

select * from schema_version where script ILIKE '%04_05_%';

Reproduced on upgrade from 4.4.4 to master , no need for the info above

this will need another fix. However the workaround is simple (and 4.5.0-1 is out anyway)

Eli, any recommendations on what to do if a downstream user hits this and can't get the fix yet?

(In reply to Marina Kalinin from comment #23)
> Eli, any recommendations on what to do if a downstream user hits this and
> can't get the fix yet?

Sorry, I missed comment#4. All set now.

engine-setup successful (update from 4.4.2), AdminDomain default_value changed to ''
Verified in ovirt-engine-4.5.1.1-0.14.el8ev.noarch

*** Bug 2097232 has been marked as a duplicate of this bug. ***

This bugzilla is included in oVirt 4.5.1 release, published on June 22nd 2022.
Since the problem described in this bug report should be resolved in oVirt 4.5.1 release, it has been closed with a resolution of CURRENT RELEASE.
If the solution does not work for you, please open a new bug report.