Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem: The follow steps were taken to upgrade our DEV environment but we are seeing error logs similar to those is the BZ in our DEV global region: 2019-06-18 09:56:09 EDT::5d08ed79.3c73:@:[15475]:LOG: starting apply for subscription region_0_subscription 2019-06-18 09:56:09 EDT::5d08ed79.3c73:@:[15475]:ERROR: unknown column name product_name 2019-06-18 09:56:09 EDT::5d08ed79.3c73:@:[15475]:LOG: apply worker [15475] at slot 1 generation 7 exiting with error 2019-06-18 09:56:09 EDT::5d08eae2.36ba:@:[14010]:LOG: worker process: pglogical apply 16386:1658409966 (PID 15475) exited with exit code 1 Below outlines the steps to upgrade: region 0 (remote) 1. stop evm 2. upgrade to 5.10.5.1 3. restart $APPLIANCE_PG_SERVICE 4. migrate 5. automate reset 6. restart $APPLIANCE_PG_SERVICE 7. reboot db 8. reboot all workers region 99 (global) same process as above Its worth noting that our database appliances do not run evm, they are purely running $APPLIANCE_PG_SERVICE. Version-Release number of selected component (if applicable): 5.10.5 How reproducible: First upgrade attempt in customer environment has broken Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: We are looking for a solution which does not involve deleting and re-adding the replication, as doing this in production is very undesirable.
Hi Ryan Were the remote regions upgraded (and migrations run) before the global region? Has replication recovered after the global has migrated? The error reported can occur during the time after the remote region has upgraded, but before the global region is upgraded. Once the global region is upgraded, replication should resume and the errors should no longer show up in the logs.
Hi Brandon, I requested verification from the customer, they have responded with the following: Yes, the remote regions were upgraded and migrated prior to the global. Once the remote regions completed, I then upgraded and migrated the global. After upgrading the global region, replication has not recovered and still seeing the error posted prior. In the global UI, the replication status shows down.
Created attachment 1581875 [details] Lower region pg logs
https://github.com/ManageIQ/manageiq-schema/pull/383
New commit detected on ManageIQ/manageiq-schema/master: https://github.com/ManageIQ/manageiq-schema/commit/4f106e59a47bcca1e983bfaac9e83112080e40b2 commit 4f106e59a47bcca1e983bfaac9e83112080e40b2 Author: Nick Carboni <ncarboni> AuthorDate: Tue Jun 18 19:00:53 2019 -0400 Commit: Nick Carboni <ncarboni> CommitDate: Tue Jun 18 19:00:53 2019 -0400 Don't sync the schema_migrations_ran table when adding it The purpose of the table is to track when migrations are run on remote regions. If we sync the whole table outside of the scope of the transaction based replication then we're violating the assumption that the migration version number only appears in the global region when the data from that migration was replicated up to the global region. This was causing a timing issue where the sync could take place just before the global region runs its migrations. Then the global would blaze past migrations without successfully replicating up the data needed for a particular schema version. This could lead to an irrecoverable situation in replication where the subscription would have to be removed and re-added. Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1721596 db/migrate/20171031010000_add_schema_migrations_ran.rb | 4 +- 1 file changed, 3 insertions(+), 1 deletion(-)
This is said to be fixed in 5.11 so I assume for verification I should do migration from 5.10 or less to 5.11 with replication enabled. But I cannot do that as (as Satoe says) no in-situ upgrade to 5.11 is supported. Only upgrade using db dump & restore are supported. So I think I cannot verify this in any way.
Based on BZ1722540 and https://bugzilla.redhat.com/show_bug.cgi?id=1721596#c11, I am marking this as closed since this upgrade method is not supported.
Well this is WONTFIX but I think we might have manual coverage for this already or we may even have automation that have not been working previously. I need to check.