Bug 1435004 - evmserverd on global appliance fails to restart after configuring replication
Summary: evmserverd on global appliance fails to restart after configuring replication
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Replication
Version: 5.8.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: GA
: 5.8.0
Assignee: Nick Carboni
QA Contact: Alex Newman
URL:
Whiteboard: replication
: 1435644 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-03-22 21:34 UTC by Nick Carboni
Modified: 2018-01-19 20:19 UTC (History)
5 users (show)

Fixed In Version: 5.8.0.8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-05-31 14:43:35 UTC
Category: ---
Cloudforms Team: ---
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:1367 normal SHIPPED_LIVE Moderate: CFME 5.8.0 security, bug, and enhancement update 2017-05-31 18:16:03 UTC

Description Nick Carboni 2017-03-22 21:34:47 UTC
Description of problem:
The seeding process on the global fails during a validation when trying to start up after the initial sync.

The issue is that there are rows that would fail a rails validation if they were saved by rails, but because there is no corresponding unique constraint we see the failure when we try to update one of the rows during seeding

Version-Release number of selected component (if applicable): master

Steps to Reproduce:
1. Configure region-to-region replication
2. Run `systemctl restart evmserverd` on the global appliance.

Actual results:

The server process fails to start up.

Additional info:
The following backtrace is seen in evm.log indicating a validation error:

[----] I, [2017-03-22T17:33:54.509691 #20005:781130]  INFO -- : MIQ(EvmDatabase.seed) Seeding ChargeableField
[----] E, [2017-03-22T17:33:54.528877 #20005:781130] ERROR -- : [ActiveRecord::RecordInvalid]: Validation failed: Metric has already been taken  Method:[rescue in block (2 levels) in seed]
[----] E, [2017-03-22T17:33:54.529072 #20005:781130] ERROR -- : /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/validations.rb:78:in `raise_validation_error'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/validations.rb:50:in `save!'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/attribute_methods/dirty.rb:30:in `save!'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/transactions.rb:324:in `block in save!'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/transactions.rb:395:in `block in with_transaction_returning_status'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/connection_adapters/abstract/database_statements.rb:230:in `transaction'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/transactions.rb:211:in `transaction'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/transactions.rb:392:in `with_transaction_returning_status'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/transactions.rb:324:in `save!'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/suppressor.rb:45:in `save!'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/persistence.rb:280:in `block in update!'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/transactions.rb:395:in `block in with_transaction_returning_status'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/connection_adapters/abstract/database_statements.rb:230:in `transaction'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/transactions.rb:211:in `transaction'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/transactions.rb:392:in `with_transaction_returning_status'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/persistence.rb:278:in `update!'
/var/www/miq/vmdb/app/models/chargeable_field.rb:73:in `block in seed'
/var/www/miq/vmdb/app/models/chargeable_field.rb:64:in `each'
/var/www/miq/vmdb/app/models/chargeable_field.rb:64:in `seed'
/var/www/miq/vmdb/lib/evm_database.rb:82:in `block (2 levels) in seed'
/var/www/miq/vmdb/lib/evm_database.rb:71:in `each'
/var/www/miq/vmdb/lib/evm_database.rb:71:in `block in seed'
/var/www/miq/vmdb/lib/extensions/ar_table_lock.rb:21:in `block (2 levels) in with_lock'
/opt/rubies/ruby-2.3.1/lib/ruby/2.3.0/timeout.rb:91:in `block in timeout'
/opt/rubies/ruby-2.3.1/lib/ruby/2.3.0/timeout.rb:33:in `block in catch'
/opt/rubies/ruby-2.3.1/lib/ruby/2.3.0/timeout.rb:33:in `catch'
/opt/rubies/ruby-2.3.1/lib/ruby/2.3.0/timeout.rb:33:in `catch'
/opt/rubies/ruby-2.3.1/lib/ruby/2.3.0/timeout.rb:106:in `timeout'
/var/www/miq/vmdb/lib/extensions/ar_table_lock.rb:21:in `block in with_lock'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/connection_adapters/abstract/database_statements.rb:232:in `block in transaction'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/connection_adapters/abstract/transaction.rb:189:in `within_new_transaction'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/connection_adapters/abstract/database_statements.rb:232:in `transaction'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/transactions.rb:211:in `transaction'
/var/www/miq/vmdb/lib/extensions/ar_table_lock.rb:15:in `with_lock'
/var/www/miq/vmdb/lib/evm_database.rb:70:in `seed'
/var/www/miq/vmdb/lib/evm_database.rb:57:in `seed_last'
/var/www/miq/vmdb/app/models/miq_server.rb:247:in `start'
/var/www/miq/vmdb/lib/workers/evm_server.rb:65:in `start'
/var/www/miq/vmdb/lib/workers/evm_server.rb:91:in `start'
/var/www/miq/vmdb/lib/workers/bin/evm_server.rb:4:in `<main>'

Comment 2 Nick Carboni 2017-03-22 21:36:07 UTC
In this case the culprit is chargeable_fields, which should probably be excluded.

Comment 4 CFME Bot 2017-03-23 14:21:04 UTC
New commit detected on ManageIQ/manageiq/master:
https://github.com/ManageIQ/manageiq/commit/259ab71287989db2735bd697205129ed63383e42

commit 259ab71287989db2735bd697205129ed63383e42
Author:     Nick Carboni <ncarboni@redhat.com>
AuthorDate: Wed Mar 22 17:38:56 2017 -0400
Commit:     Nick Carboni <ncarboni@redhat.com>
CommitDate: Wed Mar 22 17:38:56 2017 -0400

    Exclude chargeback lookup tables
    
    An AR validation which was being enforced during seeding was
    preventing the global server from starting up with failing
    rows in the database.
    
    This situation happens when the seeded rows get replicated up from
    a remote region and then the global appliance runs through seeding
    again.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1435004

 config/default_replication_exclude_tables.yml | 3 +++
 1 file changed, 3 insertions(+)

Comment 5 Nick Carboni 2017-03-23 15:08:50 UTC
Ah sorry, set the version wrong.

This doesn't need a backport.

Comment 6 Gregg Tanzillo 2017-04-13 20:48:42 UTC
*** Bug 1435644 has been marked as a duplicate of this bug. ***

Comment 8 errata-xmlrpc 2017-05-31 14:43:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:1367


Note You need to log in before you can comment on or make changes to this bug.