Bug 1344456

Summary: replication errors after inplace upgrade from 5.5 to 5.6.0.10
Product: Red Hat CloudForms Management Engine Reporter: luke couzens <lcouzens>
Component: ApplianceAssignee: Nick Carboni <ncarboni>
Status: CLOSED ERRATA QA Contact: luke couzens <lcouzens>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 5.6.0CC: abellott, cpelland, jhardy, ncarboni, obarenbo, simaishi
Target Milestone: GA   
Target Release: 5.6.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: upgrade:black
Fixed In Version: 5.6.0.12 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-06-29 16:07:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description luke couzens 2016-06-09 18:06:46 UTC
Description of problem:replication errors after inplace upgrade from 5.5 to 5.6.0.10


Version-Release number of selected component (if applicable):5.6.0.10


How reproducible:100%


Steps to Reproduce:
1.2 appliance's in replicated setup
2.add update.repo with latest 5.6 repos to /etc/yum.repos.d/
3.stop replication sync in ui
4.run systemctl stop evmserverd 
5.run yum update following docs

Actual results:replication shows as active but backlog increases and errors show in logs.


Expected results: replication works correctly


Additional info:
[0]
Documentation https://access.redhat.com/articles/2297391
evm.log http://pastebin.test.redhat.com/382304

apps ips
rro 10.16.5.145
rr99 10.8.58.217

Comment 2 Nick Carboni 2016-06-16 21:07:12 UTC
https://github.com/ManageIQ/manageiq/pull/9264

Comment 3 CFME Bot 2016-06-20 18:36:05 UTC
New commit detected on ManageIQ/manageiq/master:
https://github.com/ManageIQ/manageiq/commit/b1cb72ee2c059864ff2fc6b7ee5f295153c63a83

commit b1cb72ee2c059864ff2fc6b7ee5f295153c63a83
Author:     Nick Carboni <ncarboni>
AuthorDate: Thu Jun 16 16:54:24 2016 -0400
Commit:     Nick Carboni <ncarboni>
CommitDate: Mon Jun 20 11:08:36 2016 -0400

    Fix replication installation after primary key addition
    
    In 7656212 a migration added "id" primary keys to some
    tables that were being replicated. The triggers used in
    rubyrep replication hardcode the "key" used to retrieve
    the row that has been changed. In tables without a proper PK
    these triggers were using multiple columns for this key.
    
    Before this change, a migrated environment would have the same (old)
    triggers firing on tables that now had a primary key which would
    write the old key into the pending changes table. This mismatch
    between what was recorded in the pending changes table and the
    actual primary key caused an infinate loop / timeout during
    rubyrep replication.
    
    This fixes the situation by removing the triggers and other
    replication state information for these tables and allowing
    them to be rebuilt when replication starts again.
    
    The new triggers will have the new primary key in them and
    will generate changes that can be properly processed.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1344456

 ..._rep_triggers_on_tables_with_new_primary_key.rb | 34 ++++++++++++++++++++++
 ...triggers_on_tables_with_new_primary_key_spec.rb | 27 +++++++++++++++++
 2 files changed, 61 insertions(+)
 create mode 100644 db/migrate/20160425161345_reset_ruby_rep_triggers_on_tables_with_new_primary_key.rb
 create mode 100644 spec/migrations/20160425161345_reset_ruby_rep_triggers_on_tables_with_new_primary_key_spec.rb

Comment 4 luke couzens 2016-06-23 15:00:55 UTC
Verified in 5.6.0.12

Comment 6 errata-xmlrpc 2016-06-29 16:07:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1348