Bug 1583664

Summary: After update "provider_binding_host_id" table doesn't exist, hosts are stuck in Activating state
Product: [oVirt] ovirt-engine Reporter: Shir Fishbain <sfishbai>
Component: Setup.EngineAssignee: Eli Mesika <emesika>
Status: CLOSED CURRENTRELEASE QA Contact: Lucie Leistnerova <lleistne>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.2.3CC: bugs, ebenahar, lsvaty, mperina, rduda, sfishbai
Target Milestone: ovirt-4.2.4Keywords: Regression, Reopened
Target Release: ---Flags: rule-engine: ovirt-4.2+
rule-engine: blocker+
Hardware: x86_64   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-06-26 08:46:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
Missing table none

Description Shir Fishbain 2018-05-29 12:37:21 UTC
Description of problem:
I've upgraded from version 4.2.3-8 to version 4.2.3-10 (passed). 
Later on, I've upgraded from 4.2.3-10 to 4.2.4-1, which seems to pass, but engine log shows that the table: "provider_binding_host_id" is missing.

Version-Release number of selected component (if applicable):
4.2.4-1

Steps to Reproduce:
Upgrade from 4.2.3-10 to 4.2.4-1.

Actual results:
engine.log:
2018-05-28 10:53:02,698+03 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-25) [] Unable to RefreshCapabilities beforeFirstRefreshTreatment: PSQLException: ERROR: relation "provider_binding_host_id" does not exist
  Where: PL/pgSQL function updatehostproviderbinding(uuid,character varying[],character varying[]) line 3 at PERFORM

2018-05-28 11:01:35,168+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CollectVdsNetworkDataAfterInstallationVDSCommand] (EE-ManagedThreadFactory-engine-Thread-12) [e9c7497c-187c-4938-947a-c932f00da923] Failed in 'CollectVdsNetworkDataAfterInstallationVDS' method, for vds: 'green-vdsc.qa.lab.tlv.redhat.com'; host: 'green-vdsc.qa.lab.tlv.redhat.com': CallableStatementCallback; bad SQL grammar [{call updatehostproviderbinding(?, ?, ?)}]; nested exception is org.postgresql.util.PSQLException: ERROR: relation "provider_binding_host_id" does not exist
  Where: PL/pgSQL function updatehostproviderbinding(uuid,character varying[],character varying[]) line 3 at PERFORM
2018-05-28 11:01:35,168+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CollectVdsNetworkDataAfterInstallationVDSCommand] (EE-ManagedThreadFactory-engine-Thread-12) [e9c7497c-187c-4938-947a-c932f00da923] Command 'CollectVdsNetworkDataAfterInstallationVDSCommand(HostName = green-vdsc.qa.lab.tlv.redhat.com, CollectHostNetworkDataVdsCommandParameters:{hostId='aa6c25cf-d256-408f-87b1-8f678f9d8a9e', vds='Host[green-vdsc.qa.lab.tlv.redhat.com,aa6c25cf-d256-408f-87b1-8f678f9d8a9e]'})' execution failed: CallableStatementCallback; bad SQL grammar [{call updatehostproviderbinding(?, ?, ?)}]; nested exception is org.postgresql.util.PSQLException: ERROR: relation "provider_binding_host_id" does not exist
  Where: PL/pgSQL function updatehostproviderbinding(uuid,character varying[],character varying[]) line 3 at PERFORM
2018-05-28 11:01:35,168+03 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-12) [e9c7497c-187c-4938-947a-c932f00da923] Exception: org.ovirt.engine.core.common.errors.EngineException: EngineException: org.springframework.jdbc.BadSqlGrammarException: CallableStatementCallback; bad SQL grammar [{call updatehostproviderbinding(?, ?, ?)}]; nested exception is org.postgresql.util.PSQLException: ERROR: relation "provider_binding_host_id" does not exist
  Where: PL/pgSQL function updatehostproviderbinding(uuid,character varying[],character varying[]) line 3 at PERFORM (Failed with error ENGINE and code 5001)


After consulting with Alona (conversion from Email attached) and trying to run engine setup again, the table is still missing.
output for "select * from schema_version order by id desc;"
shows the "enlarge_custom_cpu_type.sql" twice (rows 3 and 5) and
the add_host_provider_binding_ids.sql doesn't appear in the output at all.


The "provider binding host id table doesn't exist when running the select * from schema_version.

Expected results:
For upgrade to succeed (table provider_binding_host_id to be created)

Comment 1 Martin Perina 2018-05-30 11:58:58 UTC

*** This bug has been marked as a duplicate of bug 1583562 ***

Comment 2 Martin Perina 2018-05-30 12:00:02 UTC
Reopening to verify if this issue doesn't need to be fixed on several places

Comment 3 Eli Mesika 2018-06-03 10:07:33 UTC
(In reply to Shir Fishbain from comment #0)
> output for "select * from schema_version order by id desc;"
> shows the "enlarge_custom_cpu_type.sql" twice (rows 3 and 5) and
> the add_host_provider_binding_ids.sql doesn't appear in the output at all.
> 
> 
> The "provider binding host id table doesn't exist when running the select *
> from schema_version.

Can you please attach the full result of "select * from schema_version order by id desc;" 
I want to see all the information including the script name and numbering in order to see what caused this issue 

My "guess" right now is that "enlarge_custom_cpu_type.sql" was installed first as in your 5th row in the version_schema table (since it desc) , then it was changed and run again as your 3rd row  and because it has a different md5 it was successful and  add_host_provider_binding_ids.sql had the same numbering as the last "enlarge_custom_cpu_type.sql that was executed and therefor it was skipped assuming that it was already installed into the database.

Comment 4 Shir Fishbain 2018-06-03 11:57:37 UTC
Created attachment 1447199 [details]
Missing table

Comment 5 Shir Fishbain 2018-06-03 12:05:23 UTC
Attached a zip file with the full result of the query, the name of the file is
output.csv.

Comment 6 Eli Mesika 2018-06-03 14:13:00 UTC
From the attached I see that the original name of the missing script is 04_02_1090_add_host_provider_binding_ids.sql while the schema version shows that this numbering was used by 04_02_1090_enlarge_custom_cpu_type.sql that was changed later on (modified textually because scripts with eh same md5 are skipped) and was installed again as 04_02_1110_enlarge_custom_cpu_type.sql.

So, I see no bug here.

The upgrade script will skip any numbering that is recorded in the schema_version table as installed.

I recommend to close as NOT A BUG

Comment 7 Martin Perina 2018-06-04 14:09:18 UTC
(In reply to Eli Mesika from comment #6)
> From the attached I see that the original name of the missing script is
> 04_02_1090_add_host_provider_binding_ids.sql while the schema version shows
> that this numbering was used by 04_02_1090_enlarge_custom_cpu_type.sql that
> was changed later on (modified textually because scripts with eh same md5
> are skipped) and was installed again as
> 04_02_1110_enlarge_custom_cpu_type.sql.
> 
> So, I see no bug here.
> 
> The upgrade script will skip any numbering that is recorded in the
> schema_version table as installed.
> 
> I recommend to close as NOT A BUG

It's a bug, because we have caused it by backporting db scripts to 4.2.3.z and then not matching to changes in 4.2.3.z to 4.2 branch.

Comment 8 Lucie Leistnerova 2018-06-12 11:36:54 UTC
Upgrade 4.2.3.10 -> 4.2.4 skipped 04_02_1090_enlarge_custom_cpu_type.sql and ran newly installed scripts: 04_02_1100_add_host_provider_binding_ids, 04_02_1110_remove_test_imageio_connection_config_value.sql, 04_02_1120_add_confirmed_size_warning.sql ...
So the table provider_binding_host_id exists.

verified in ovirt-engine-setup-4.2.4.2-0.1.el7_3.noarch

Comment 9 Sandro Bonazzola 2018-06-26 08:46:24 UTC
This bugzilla is included in oVirt 4.2.4 release, published on June 26th 2018.

Since the problem described in this bug report should be
resolved in oVirt 4.2.4 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.