Bug 1721596

Summary:	Upgrading from 4.6 to 4.7 presents an error in the global region
Product:	Red Hat CloudForms Management Engine	Reporter:	Ryan Spagnola <rspagnol>
Component:	Appliance	Assignee:	Nick Carboni <ncarboni>
Status:	CLOSED WONTFIX	QA Contact:	Jaroslav Henner <jhenner>
Severity:	high	Docs Contact:	Red Hat CloudForms Documentation <cloudforms-docs>
Priority:	medium
Version:	5.10.5	CC:	abellott, mnadeem, ncarboni, obarenbo, simaishi, smallamp
Target Milestone:	GA	Keywords:	TestOnly, ZStream
Target Release:	5.11.0
Hardware:	All
OS:	All
Whiteboard:
Fixed In Version:	5.11.0.10	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1722540 (view as bug list)		Environment:
Last Closed:	2019-07-25 13:20:27 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	Bug
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	CFME Core	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1722540

Description Ryan Spagnola 2019-06-18 16:10:35 UTC

Description of problem:
The follow steps were taken to upgrade our DEV environment but we are seeing error logs similar to those is the BZ in our DEV global region:

2019-06-18 09:56:09 EDT::5d08ed79.3c73:@:[15475]:LOG:  starting apply for subscription region_0_subscription
2019-06-18 09:56:09 EDT::5d08ed79.3c73:@:[15475]:ERROR:  unknown column name product_name
2019-06-18 09:56:09 EDT::5d08ed79.3c73:@:[15475]:LOG:  apply worker [15475] at slot 1 generation 7 exiting with error
2019-06-18 09:56:09 EDT::5d08eae2.36ba:@:[14010]:LOG:  worker process: pglogical apply 16386:1658409966 (PID 15475) exited with exit code 1

Below outlines the steps to upgrade:

region 0 (remote)
1. stop evm
2. upgrade to 5.10.5.1
3. restart $APPLIANCE_PG_SERVICE
4. migrate
5. automate reset
6. restart $APPLIANCE_PG_SERVICE
7. reboot db
8. reboot all workers

region 99 (global)
same process as above

Its worth noting that our database appliances do not run evm, they are purely running $APPLIANCE_PG_SERVICE.


Version-Release number of selected component (if applicable):
5.10.5

How reproducible:
First upgrade attempt in customer environment has broken

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
We are looking for a solution which does not involve deleting and re-adding the replication, as doing this in production is very undesirable.

Comment 2 Brandon Dunne 2019-06-18 16:48:36 UTC

Hi Ryan

Were the remote regions upgraded (and migrations run) before the global region?  Has replication recovered after the global has migrated?  The error reported can occur during the time after the remote region has upgraded, but before the global region is upgraded.  Once the global region is upgraded, replication should resume and the errors should no longer show up in the logs.

Comment 3 Ryan Spagnola 2019-06-18 18:33:21 UTC

Hi Brandon,

I requested verification from the customer, they have responded with the following:

Yes, the remote regions were upgraded and migrated prior to the global. Once the remote regions completed, I then upgraded and migrated the global. 

After upgrading the global region, replication has not recovered and still seeing the error posted prior. In the global UI, the replication status shows down.

Comment 7 Ryan Spagnola 2019-06-18 19:38:37 UTC

Created attachment 1581875 [details]
Lower region pg logs

Comment 8 CFME Bot 2019-06-18 23:17:51 UTC

https://github.com/ManageIQ/manageiq-schema/pull/383

Comment 9 CFME Bot 2019-06-19 14:13:19 UTC

New commit detected on ManageIQ/manageiq-schema/master:

https://github.com/ManageIQ/manageiq-schema/commit/4f106e59a47bcca1e983bfaac9e83112080e40b2
commit 4f106e59a47bcca1e983bfaac9e83112080e40b2
Author:     Nick Carboni <ncarboni>
AuthorDate: Tue Jun 18 19:00:53 2019 -0400
Commit:     Nick Carboni <ncarboni>
CommitDate: Tue Jun 18 19:00:53 2019 -0400

    Don't sync the schema_migrations_ran table when adding it

    The purpose of the table is to track when migrations are run on
    remote regions. If we sync the whole table outside of the scope
    of the transaction based replication then we're violating the
    assumption that the migration version number only appears in the
    global region when the data from that migration was replicated
    up to the global region.

    This was causing a timing issue where the sync could take place
    just before the global region runs its migrations. Then the global
    would blaze past migrations without successfully replicating up
    the data needed for a particular schema version.

    This could lead to an irrecoverable situation in replication where
    the subscription would have to be removed and re-added.

    Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1721596

 db/migrate/20171031010000_add_schema_migrations_ran.rb | 4 +-
 1 file changed, 3 insertions(+), 1 deletion(-)

Comment 11 Jaroslav Henner 2019-07-10 12:58:26 UTC

This is said to be fixed in 5.11 so I assume for verification I should do migration from 5.10 or less to 5.11 with replication enabled. But I cannot do that as (as Satoe says) no in-situ upgrade to 5.11 is supported. Only upgrade using db dump & restore are supported.

So I think I cannot verify this in any way.

Comment 12 Sudhir Mallamprabhakara 2019-07-25 13:20:27 UTC

Based on BZ1722540 and https://bugzilla.redhat.com/show_bug.cgi?id=1721596#c11, I am marking this as closed since this upgrade method is not supported.

Comment 13 Jaroslav Henner 2019-11-29 13:49:07 UTC

Well this is WONTFIX but I think we might have manual coverage for this already or we may even have automation that have not been working previously. I need to check.