Description of problem: I went through the upgrade process in our other UK region. The upgrade ran successfully but CF fails to start. In the logs it appears to be an issue with reports. It was required that I revert to 3.2, remove all the widgets before the upgrade, and finally re-attempt upgrade again. Version-Release number of selected component (if applicable): cfme-5.6.1.2 How reproducible: need this vmdb+widgets Actual results: upgrade finishes, but when server starts up it hiccups and errors out Expected results: upgrade succeeds and the appliance starts up afterwords Additional info: log lines: [----] I, [2016-09-10T21:00:30.914867 #5154:c5b994] INFO -- : MIQ(RssFeed.sync_from_yml_file) [vmware_vms] file has been updated on disk, synchronizing with model [----] I, [2016-09-10T21:00:30.973106 #5154:c5b994] INFO -- : MIQ(EvmDatabase.seed) Seeding MiqWidget [----] I, [2016-09-10T21:00:31.691248 #5154:c5b994] INFO -- : Widget: [chart_hosts_summary_by_version] file has been updated on disk, synchronizing with model [----] I, [2016-09-10T21:00:31.765727 #5154:c5b994] INFO -- : Widget: [report_top_storage_consumers] file has been added to disk, adding to model [----] E, [2016-09-10T21:00:32.661837 #5154:c5b994] ERROR -- : [ActiveRecord::RecordInvalid]: Validation failed: Title has already been taken Method:[rescue in block (2 levels) in seed] [----] E, [2016-09-10T21:00:32.662101 #5154:c5b994] ERROR -- : /opt/rh/cfme-gemset/gems/activerecord-5.0.0/lib/active_record/validations.rb:78:in `raise_validation_error' /opt/rh/cfme-gemset/gems/activerecord-5.0.0/lib/active_record/validations.rb:50:in `save!' /opt/rh/cfme-gemset/gems/activerecord-5.0.0/lib/active_record/attribute_methods/dirty.rb:30:in `save!' /opt/rh/cfme-gemset/gems/activerecord-5.0.0/lib/active_record/transactions.rb:324:in `block in save!' /opt/rh/cfme-gemset/gems/activerecord-5.0.0/lib/active_record/transactions.rb:395:in `block in with_transaction_returning_status' /opt/rh/cfme-gemset/gems/activerecord-5.0.0/lib/active_record/connection_adapters/abstract/database_statements.rb:230:in `transaction' /opt/rh/cfme-gemset/gems/activerecord-5.0.0/lib/active_record/transactions.rb:211:in `transaction' /opt/rh/cfme-gemset/gems/activerecord-5.0.0/lib/active_record/transactions.rb:392:in `with_transaction_returning_status' /opt/rh/cfme-gemset/gems/activerecord-5.0.0/lib/active_record/transactions.rb:324:in `save!' /opt/rh/cfme-gemset/gems/activerecord-5.0.0/lib/active_record/suppressor.rb:45:in `save!' /opt/rh/cfme-gemset/gems/activerecord-5.0.0/lib/active_record/persistence.rb:51:in `create!' /var/www/miq/vmdb/app/models/miq_widget.rb:544:in `sync_from_hash' /var/www/miq/vmdb/app/models/miq_widget.rb:518:in `sync_from_file' /var/www/miq/vmdb/app/models/miq_widget.rb:513:in `block in sync_from_dir' /var/www/miq/vmdb/app/models/miq_widget.rb:513:in `each' /var/www/miq/vmdb/app/models/miq_widget.rb:513:in `sync_from_dir' /var/www/miq/vmdb/app/models/miq_widget.rb:604:in `seed' /var/www/miq/vmdb/lib/evm_database.rb:76:in `block (2 levels) in seed' /var/www/miq/vmdb/lib/evm_database.rb:65:in `each' /var/www/miq/vmdb/lib/evm_database.rb:65:in `block in seed' /var/www/miq/vmdb/lib/extensions/ar_table_lock.rb:21:in `block (2 levels) in with_lock' /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:89:in `block in timeout' /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:34:in `block in catch' /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:34:in `catch' /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:34:in `catch' /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:104:in `timeout' /var/www/miq/vmdb/lib/extensions/ar_table_lock.rb:21:in `block in with_lock' /opt/rh/cfme-gemset/gems/activerecord-5.0.0/lib/active_record/connection_adapters/abstract/database_statements.rb:232:in `block in transaction' /opt/rh/cfme-gemset/gems/activerecord-5.0.0/lib/active_record/connection_adapters/abstract/transaction.rb:189:in `within_new_transaction' /opt/rh/cfme-gemset/gems/activerecord-5.0.0/lib/active_record/connection_adapters/abstract/database_statements.rb:232:in `transaction' /opt/rh/cfme-gemset/gems/activerecord-5.0.0/lib/active_record/transactions.rb:211:in `transaction' /var/www/miq/vmdb/lib/extensions/ar_table_lock.rb:15:in `with_lock' /var/www/miq/vmdb/lib/evm_database.rb:64:in `seed' /var/www/miq/vmdb/lib/evm_database.rb:52:in `seed_last' /var/www/miq/vmdb/app/models/miq_server.rb:261:in `start' /var/www/miq/vmdb/lib/workers/evm_server.rb:65:in `start' /var/www/miq/vmdb/lib/workers/evm_server.rb:92:in `start' /var/www/miq/vmdb/lib/workers/bin/evm_server.rb:4:in `<main>' /opt/rh/cfme-gemset/gems/activerecord-5.0.0/lib/active_record/validations.rb:78:in `raise_validation_error': Validation failed: Title has already been taken (ActiveRecord::RecordInvalid) from /opt/rh/cfme-gemset/gems/activerecord-5.0.0/lib/active_record/validations.rb:50:in `save!' from /opt/rh/cfme-gemset/gems/activerecord-5.0.0/lib/active_record/attribute_methods/dirty.rb:30:in `save!' from /opt/rh/cfme-gemset/gems/activerecord-5.0.0/lib/active_record/transactions.rb:324:in `block in save!' from /opt/rh/cfme-gemset/gems/activerecord-5.0.0/lib/active_record/transactions.rb:395:in `block in with_transaction_returning_status' from /opt/rh/cfme-gemset/gems/activerecord-5.0.0/lib/active_record/connection_adapters/abstract/database_statements.rb:230:in `transaction' from /opt/rh/cfme-gemset/gems/activerecord-5.0.0/lib/active_record/transactions.rb:211:in `transaction' from /opt/rh/cfme-gemset/gems/activerecord-5.0.0/lib/active_record/transactions.rb:392:in `with_transaction_returning_status' from /opt/rh/cfme-gemset/gems/activerecord-5.0.0/lib/active_record/transactions.rb:324:in `save!' from /opt/rh/cfme-gemset/gems/activerecord-5.0.0/lib/active_record/suppressor.rb:45:in `save!' from /opt/rh/cfme-gemset/gems/activerecord-5.0.0/lib/active_record/persistence.rb:51:in `create!'
Okay, I was able to reproduce this by removing the default widget "Top Storage Consumers" and then recreating a widget with the same title, then restarting the server. The restart caused the seed to run which tried to recreate the default "Top Storage Consumers" widget which fails the unique validation on the title. We can remove the unique validation on the title, but for now just removing the "Top Storage Consumers" widget should do the trick (unless of course this has been done for other widgets as well).
https://github.com/ManageIQ/manageiq/pull/11431
New commit detected on ManageIQ/manageiq/master: https://github.com/ManageIQ/manageiq/commit/0f60e7de16f6a3874c495710a282d96243b25dbb commit 0f60e7de16f6a3874c495710a282d96243b25dbb Author: Nick Carboni <ncarboni> AuthorDate: Wed Sep 21 14:56:05 2016 -0400 Commit: Nick Carboni <ncarboni> CommitDate: Wed Sep 21 14:56:05 2016 -0400 Remove unique validation on miq_widget title This allows a user to remove a default widget and recreate a new one with the same title without having the seed blow up with a unique validation error when it runs again. https://bugzilla.redhat.com/show_bug.cgi?id=1375313 app/models/miq_widget.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
Verified fixed in 5.8.0.12 - 5.8.0.12-rc1.20170425180304_4f35996.