Bug 1435004

Summary:	evmserverd on global appliance fails to restart after configuring replication
Product:	Red Hat CloudForms Management Engine	Reporter:	Nick Carboni <ncarboni>
Component:	Replication	Assignee:	Nick Carboni <ncarboni>
Status:	CLOSED ERRATA	QA Contact:	Alex Newman <anewman>
Severity:	high	Docs Contact:
Priority:	high
Version:	5.8.0	CC:	cpelland, jhardy, lcouzens, obarenbo, simaishi
Target Milestone:	GA
Target Release:	5.8.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:	replication
Fixed In Version:	5.8.0.8	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-05-31 14:43:35 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Nick Carboni 2017-03-22 21:34:47 UTC

Description of problem:
The seeding process on the global fails during a validation when trying to start up after the initial sync.

The issue is that there are rows that would fail a rails validation if they were saved by rails, but because there is no corresponding unique constraint we see the failure when we try to update one of the rows during seeding

Version-Release number of selected component (if applicable): master

Steps to Reproduce:
1. Configure region-to-region replication
2. Run `systemctl restart evmserverd` on the global appliance.

Actual results:

The server process fails to start up.

Additional info:
The following backtrace is seen in evm.log indicating a validation error:

[----] I, [2017-03-22T17:33:54.509691 #20005:781130]  INFO -- : MIQ(EvmDatabase.seed) Seeding ChargeableField
[----] E, [2017-03-22T17:33:54.528877 #20005:781130] ERROR -- : [ActiveRecord::RecordInvalid]: Validation failed: Metric has already been taken  Method:[rescue in block (2 levels) in seed]
[----] E, [2017-03-22T17:33:54.529072 #20005:781130] ERROR -- : /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/validations.rb:78:in `raise_validation_error'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/validations.rb:50:in `save!'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/attribute_methods/dirty.rb:30:in `save!'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/transactions.rb:324:in `block in save!'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/transactions.rb:395:in `block in with_transaction_returning_status'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/connection_adapters/abstract/database_statements.rb:230:in `transaction'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/transactions.rb:211:in `transaction'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/transactions.rb:392:in `with_transaction_returning_status'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/transactions.rb:324:in `save!'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/suppressor.rb:45:in `save!'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/persistence.rb:280:in `block in update!'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/transactions.rb:395:in `block in with_transaction_returning_status'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/connection_adapters/abstract/database_statements.rb:230:in `transaction'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/transactions.rb:211:in `transaction'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/transactions.rb:392:in `with_transaction_returning_status'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/persistence.rb:278:in `update!'
/var/www/miq/vmdb/app/models/chargeable_field.rb:73:in `block in seed'
/var/www/miq/vmdb/app/models/chargeable_field.rb:64:in `each'
/var/www/miq/vmdb/app/models/chargeable_field.rb:64:in `seed'
/var/www/miq/vmdb/lib/evm_database.rb:82:in `block (2 levels) in seed'
/var/www/miq/vmdb/lib/evm_database.rb:71:in `each'
/var/www/miq/vmdb/lib/evm_database.rb:71:in `block in seed'
/var/www/miq/vmdb/lib/extensions/ar_table_lock.rb:21:in `block (2 levels) in with_lock'
/opt/rubies/ruby-2.3.1/lib/ruby/2.3.0/timeout.rb:91:in `block in timeout'
/opt/rubies/ruby-2.3.1/lib/ruby/2.3.0/timeout.rb:33:in `block in catch'
/opt/rubies/ruby-2.3.1/lib/ruby/2.3.0/timeout.rb:33:in `catch'
/opt/rubies/ruby-2.3.1/lib/ruby/2.3.0/timeout.rb:33:in `catch'
/opt/rubies/ruby-2.3.1/lib/ruby/2.3.0/timeout.rb:106:in `timeout'
/var/www/miq/vmdb/lib/extensions/ar_table_lock.rb:21:in `block in with_lock'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/connection_adapters/abstract/database_statements.rb:232:in `block in transaction'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/connection_adapters/abstract/transaction.rb:189:in `within_new_transaction'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/connection_adapters/abstract/database_statements.rb:232:in `transaction'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/transactions.rb:211:in `transaction'
/var/www/miq/vmdb/lib/extensions/ar_table_lock.rb:15:in `with_lock'
/var/www/miq/vmdb/lib/evm_database.rb:70:in `seed'
/var/www/miq/vmdb/lib/evm_database.rb:57:in `seed_last'
/var/www/miq/vmdb/app/models/miq_server.rb:247:in `start'
/var/www/miq/vmdb/lib/workers/evm_server.rb:65:in `start'
/var/www/miq/vmdb/lib/workers/evm_server.rb:91:in `start'
/var/www/miq/vmdb/lib/workers/bin/evm_server.rb:4:in `<main>'

Comment 2 Nick Carboni 2017-03-22 21:36:07 UTC

In this case the culprit is chargeable_fields, which should probably be excluded.

Comment 3 CFME Bot 2017-03-22 21:51:34 UTC

https://github.com/ManageIQ/manageiq/pull/14466

Comment 4 CFME Bot 2017-03-23 14:21:04 UTC

New commit detected on ManageIQ/manageiq/master:
https://github.com/ManageIQ/manageiq/commit/259ab71287989db2735bd697205129ed63383e42

commit 259ab71287989db2735bd697205129ed63383e42
Author:     Nick Carboni <ncarboni>
AuthorDate: Wed Mar 22 17:38:56 2017 -0400
Commit:     Nick Carboni <ncarboni>
CommitDate: Wed Mar 22 17:38:56 2017 -0400

    Exclude chargeback lookup tables
    
    An AR validation which was being enforced during seeding was
    preventing the global server from starting up with failing
    rows in the database.
    
    This situation happens when the seeded rows get replicated up from
    a remote region and then the global appliance runs through seeding
    again.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1435004

 config/default_replication_exclude_tables.yml | 3 +++
 1 file changed, 3 insertions(+)

Comment 5 Nick Carboni 2017-03-23 15:08:50 UTC

Ah sorry, set the version wrong.

This doesn't need a backport.

Comment 6 Gregg Tanzillo 2017-04-13 20:48:42 UTC

*** Bug 1435644 has been marked as a duplicate of this bug. ***

Comment 8 errata-xmlrpc 2017-05-31 14:43:35 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:1367