Description of problem: The seeding process on the global fails during a validation when trying to start up after the initial sync. The issue is that there are rows that would fail a rails validation if they were saved by rails, but because there is no corresponding unique constraint we see the failure when we try to update one of the rows during seeding Version-Release number of selected component (if applicable): master Steps to Reproduce: 1. Configure region-to-region replication 2. Run `systemctl restart evmserverd` on the global appliance. Actual results: The server process fails to start up. Additional info: The following backtrace is seen in evm.log indicating a validation error: [----] I, [2017-03-22T17:33:54.509691 #20005:781130] INFO -- : MIQ(EvmDatabase.seed) Seeding ChargeableField [----] E, [2017-03-22T17:33:54.528877 #20005:781130] ERROR -- : [ActiveRecord::RecordInvalid]: Validation failed: Metric has already been taken Method:[rescue in block (2 levels) in seed] [----] E, [2017-03-22T17:33:54.529072 #20005:781130] ERROR -- : /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/validations.rb:78:in `raise_validation_error' /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/validations.rb:50:in `save!' /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/attribute_methods/dirty.rb:30:in `save!' /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/transactions.rb:324:in `block in save!' /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/transactions.rb:395:in `block in with_transaction_returning_status' /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/connection_adapters/abstract/database_statements.rb:230:in `transaction' /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/transactions.rb:211:in `transaction' /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/transactions.rb:392:in `with_transaction_returning_status' /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/transactions.rb:324:in `save!' /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/suppressor.rb:45:in `save!' /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/persistence.rb:280:in `block in update!' /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/transactions.rb:395:in `block in with_transaction_returning_status' /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/connection_adapters/abstract/database_statements.rb:230:in `transaction' /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/transactions.rb:211:in `transaction' /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/transactions.rb:392:in `with_transaction_returning_status' /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/persistence.rb:278:in `update!' /var/www/miq/vmdb/app/models/chargeable_field.rb:73:in `block in seed' /var/www/miq/vmdb/app/models/chargeable_field.rb:64:in `each' /var/www/miq/vmdb/app/models/chargeable_field.rb:64:in `seed' /var/www/miq/vmdb/lib/evm_database.rb:82:in `block (2 levels) in seed' /var/www/miq/vmdb/lib/evm_database.rb:71:in `each' /var/www/miq/vmdb/lib/evm_database.rb:71:in `block in seed' /var/www/miq/vmdb/lib/extensions/ar_table_lock.rb:21:in `block (2 levels) in with_lock' /opt/rubies/ruby-2.3.1/lib/ruby/2.3.0/timeout.rb:91:in `block in timeout' /opt/rubies/ruby-2.3.1/lib/ruby/2.3.0/timeout.rb:33:in `block in catch' /opt/rubies/ruby-2.3.1/lib/ruby/2.3.0/timeout.rb:33:in `catch' /opt/rubies/ruby-2.3.1/lib/ruby/2.3.0/timeout.rb:33:in `catch' /opt/rubies/ruby-2.3.1/lib/ruby/2.3.0/timeout.rb:106:in `timeout' /var/www/miq/vmdb/lib/extensions/ar_table_lock.rb:21:in `block in with_lock' /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/connection_adapters/abstract/database_statements.rb:232:in `block in transaction' /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/connection_adapters/abstract/transaction.rb:189:in `within_new_transaction' /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/connection_adapters/abstract/database_statements.rb:232:in `transaction' /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.2/lib/active_record/transactions.rb:211:in `transaction' /var/www/miq/vmdb/lib/extensions/ar_table_lock.rb:15:in `with_lock' /var/www/miq/vmdb/lib/evm_database.rb:70:in `seed' /var/www/miq/vmdb/lib/evm_database.rb:57:in `seed_last' /var/www/miq/vmdb/app/models/miq_server.rb:247:in `start' /var/www/miq/vmdb/lib/workers/evm_server.rb:65:in `start' /var/www/miq/vmdb/lib/workers/evm_server.rb:91:in `start' /var/www/miq/vmdb/lib/workers/bin/evm_server.rb:4:in `<main>'
In this case the culprit is chargeable_fields, which should probably be excluded.
https://github.com/ManageIQ/manageiq/pull/14466
New commit detected on ManageIQ/manageiq/master: https://github.com/ManageIQ/manageiq/commit/259ab71287989db2735bd697205129ed63383e42 commit 259ab71287989db2735bd697205129ed63383e42 Author: Nick Carboni <ncarboni> AuthorDate: Wed Mar 22 17:38:56 2017 -0400 Commit: Nick Carboni <ncarboni> CommitDate: Wed Mar 22 17:38:56 2017 -0400 Exclude chargeback lookup tables An AR validation which was being enforced during seeding was preventing the global server from starting up with failing rows in the database. This situation happens when the seeded rows get replicated up from a remote region and then the global appliance runs through seeding again. https://bugzilla.redhat.com/show_bug.cgi?id=1435004 config/default_replication_exclude_tables.yml | 3 +++ 1 file changed, 3 insertions(+)
Ah sorry, set the version wrong. This doesn't need a backport.
*** Bug 1435644 has been marked as a duplicate of this bug. ***
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2017:1367