Bug 1289156

Summary: Replication: host_storages table, duplicate key
Product: Red Hat CloudForms Management Engine Reporter: Jan Krocil <jkrocil>
Component: ApplianceAssignee: Nick Carboni <ncarboni>
Status: CLOSED ERRATA QA Contact: Alex Newman <anewman>
Severity: high Docs Contact:
Priority: high    
Version: 5.5.0CC: abellott, cpelland, dajohnso, jhardy, jocarter, jprause, mfeifer, ncarboni, obarenbo, ssainkar
Target Milestone: GAKeywords: ZStream
Target Release: 5.6.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: replication
Fixed In Version: 5.6.0.0 Doc Type: Bug Fix
Doc Text:
In the previous version of CloudForms Management Engine, replication failed with duplicate key present in the `host_storages` table on master. This was happening because a migration was adding a primary key to the host_storages table and in the process altering existing rows to have a key within the range defined by the current database region. This caused existing rows in the host_storages table to be assigned primary keys in the master database region which resulted in rows with ids from multiple regions (master and remote regions). Replication determines whether a row exists on the master side by comparing primary keys. Thereby, the primary key did not match for a row where it should. This caused replication to attempt to reinsert the row which failed because of a unique index on two other columns in host_storages. This patch adds migration to fix host_storages replication issue. This removes the entries in the host_storages table on the target region and resets the replication triggers on the source regions which has now resolved the issue.
Story Points: ---
Clone Of:
: 1289200 (view as bug list) Environment:
Last Closed: 2016-06-29 15:18:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1289200    

Comment 2 Nick Carboni 2015-12-07 15:09:50 UTC
This is happening because a migration (https://github.com/ManageIQ/manageiq/blob/master/db/migrate/20150921204114_add_vmware_ro_datastores_to_hosts_storages.rb) is adding a primary key to the host_storages table and in the process altering existing rows to have a key within the range defined by the current database region.

This causes existing rows in the host_storages table to be assigned primary keys in the master database region which will result in rows with ids from multiple regions (master and remote regions).

Replication determines whether a row exists on the master side by comparing primary keys. In this case the primary key will not match for a row where it should. This causes replication to attempt to reinsert the row which fails because of a unique index on two other columns in host_storages.

Comment 3 Marianne Feifer 2015-12-07 19:58:00 UTC
Added link to migration article

Comment 4 Nick Carboni 2015-12-08 21:44:07 UTC
https://github.com/ManageIQ/manageiq/pull/5760

Comment 5 CFME Bot 2015-12-09 15:03:52 UTC
New commit detected on ManageIQ/manageiq/master:
https://github.com/ManageIQ/manageiq/commit/04b785eee3771861e3d9a9e7cd78064a721fb0e9

commit 04b785eee3771861e3d9a9e7cd78064a721fb0e9
Author:     Nick Carboni <ncarboni>
AuthorDate: Tue Dec 8 16:06:55 2015 -0500
Commit:     Nick Carboni <ncarboni>
CommitDate: Wed Dec 9 08:52:56 2015 -0500

    Add migration to fix host_storages replication issue
    
    After we run the migration added in commit 04f2fa211b8ebeb0557ca3712af846c9b556869f
    the table host_storages will be assigned a primary key column and existing
    rows will be given a value within the current region's range.
    
    This causes an issue when run on a replication target as the id should
    match the one on the replication source region not the target one.
    
    This fix removes the entries in the host_storages table on the target
    region and resets the replication triggers on the source regions.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1289156

 ...0956_fix_host_storage_replication_on_upgrade.rb | 49 ++++++++++++++++++++++
 ...fix_host_storage_replication_on_upgrade_spec.rb | 47 +++++++++++++++++++++
 2 files changed, 96 insertions(+)
 create mode 100644 db/migrate/20151208150956_fix_host_storage_replication_on_upgrade.rb
 create mode 100644 spec/migrations/20151208150956_fix_host_storage_replication_on_upgrade_spec.rb

Comment 7 errata-xmlrpc 2016-06-29 15:18:04 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1348