Bug 1507323

Looking at the supplied postgres log it seems that a replication subscription was created to the same region. Meaning a subscription was created from region 7 to region 7. As part of creating a subscription, all data from the remote region is deleted from the current region. In this case it's the same region. That's why all the data was removed from region 7

We can see that this happened twice. Also the timestamp of the first one coincides with the exact time that we first see the error in the EVM log indicating that the server was unable to find its id in the db. (Note that the pg log is in UTC and the evm log in local time which is UTC +11)

2017-10-24 17:53:35 GMT:[local]:59ef7d8f.6a32:root@vmdb_production:[27186]:LOG:  duration: 21821.927 ms  execute <unnamed>: DELETE FROM event_streams WHERE id >= 7000000000000 AND id <= 7999999999999
2017-10-24 17:53:58 GMT:[local]:59ef7d8f.6a32:root@vmdb_production:[27186]:LOG:  duration: 15914.969 ms  execute <unnamed>: DELETE FROM metric_rollups WHERE id >= 7000000000000 AND id <= 7999999999999
2017-10-24 17:56:30 GMT:[local]:59ef7d8f.6a32:root@vmdb_production:[27186]:LOG:  duration: 132897.955 ms  execute <unnamed>: DELETE FROM vim_performance_states WHERE id >= 7000000000000 AND id <= 7999999999999
2017-10-24 17:57:51 GMT:[local]:59ef7d8f.6a32:root@vmdb_production:[27186]:LOG:  duration: 79110.368 ms  execute <unnamed>: DELETE FROM vmdb_metrics WHERE id >= 7000000000000 AND id <= 7999999999999
5783173 2017-10-24 17:57:54 GMT:[local]:59ef7d8f.6a32:root@vmdb_production:[27186]:STATEMENT:  SELECT pglogical.create_subscription($1, $2, $3, $4, $5, $6)


2017-10-24 18:33:50 GMT:[local]:59ef84e5.7624:root@vmdb_production:[30244]:LOG:  duration: 53801.086 ms  bind <unnamed>: DELETE FROM vim_performance_states WHERE id >= 7000000000000 AND id <= 7999999999999
2017-10-24 18:33:58 GMT:[local]:59ef84e5.7624:root@vmdb_production:[30244]:LOG:  duration: 6405.231 ms  execute <unnamed>: DELETE FROM vmdb_metrics WHERE id >= 7000000000000 AND id <= 7999999999999
2017-10-24 18:33:59 GMT:[local]:59ef84e5.7624:root@vmdb_production:[30244]:STATEMENT:  SELECT pglogical.create_subscription($1, $2, $3, $4, $5, $6)

https://github.com/ManageIQ/manageiq/pull/16446

New commit detected on ManageIQ/manageiq/master:
https://github.com/ManageIQ/manageiq/commit/f173fdce1ba57458d753150c83d06788c07d9e93

commit f173fdce1ba57458d753150c83d06788c07d9e93
Author:     Gregg Tanzillo <gtanzill>
AuthorDate: Fri Nov 10 14:52:58 2017 -0500
Commit:     Gregg Tanzillo <gtanzill>
CommitDate: Fri Nov 10 17:46:33 2017 -0500

    Prevent replication subscription to the same region as the current region
    
    This change protects a user from accidentally creating a replication subscription to the same region
    he is in which will result in deleting all the data from the current region.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1507323
    
    Fixed typo1
    
    change 1

 app/models/pglogical_subscription.rb       |  7 +++++++
 spec/models/pglogical_subscription_spec.rb | 26 ++++++++++++++++++++++++++
 2 files changed, 33 insertions(+)

New commit detected on ManageIQ/manageiq/master:
https://github.com/ManageIQ/manageiq/commit/192b14eaa2ba554b8bf6a56818fe0fe5cfb58bb1

commit 192b14eaa2ba554b8bf6a56818fe0fe5cfb58bb1
Author:     Gregg Tanzillo <gtanzill>
AuthorDate: Fri Nov 10 14:59:28 2017 -0500
Commit:     Gregg Tanzillo <gtanzill>
CommitDate: Fri Nov 10 17:46:35 2017 -0500

    Update existing tests to handle check for subscription being to a different region than the current region
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1507323
    
    Fixed typo2
    
    Change 2

 spec/models/pglogical_subscription_spec.rb | 19 +++++++++++++++----
 1 file changed, 15 insertions(+), 4 deletions(-)