Description of problem:evm fails to start on remote region after upgrading from 5.6 rubyrep to 5.8 due to records left in miq_workers Version-Release number of selected component (if applicable):5.8.0.12 How reproducible:100% Steps to Reproduce: 1.provision 5.6 appliances 2.setup rubyrep 3.add a provider 4.check its replicated 5.stop db sync role (remote) 6.stop evm on (remote) 7.run rake evm:dbsync:uninstall (remote) 8.stop evm on global 9.add 5.8 repos 10.run yum update on both apps 11.reconnect ssh (reload ruby) 12.run vmdb 13.run rake db:migrate 14.run rake evm:automate:reset 15.run /usr/bin/miq_postgres_upgrade 16.make edits and start postgres 17.start evm Actual results:evm fails to start on remote region Expected results:evm to start correctly Additional info: vmdb_production=# select id, status, type from miq_workers; id | status | type ---------------+----------+---------------------- 1000000000012 | stopping | MiqReplicationWorker 1000000000019 | aborted | MiqPriorityWorker 1000000000020 | aborted | MiqPriorityWorker 1000000000017 | aborted | MiqGenericWorker 1000000000018 | aborted | MiqGenericWorker 1000000000021 | aborted | MiqScheduleWorker (6 rows) journalctl -u evmserverd -- Logs begin at Thu 2017-04-27 10:56:50 EDT, end at Thu 2017-04-27 14:56:19 EDT. -- Starting EVM server daemon... DEPRECATION WARNING: `config.serve_static_files` is deprecated and will be removed in Rails 5.1. Please use `config.public_file_server.enabled = false` instead. (called from block in <top (required)> at /var/www/miq/vmdb/config/environments/production.rb:15) Starting EVM... Running EVM in background... Started EVM server daemon. Stopping EVM server daemon... DEPRECATION WARNING: `config.serve_static_files` is deprecated and will be removed in Rails 5.1. Please use `config.public_file_server.enabled = false` instead. (called from block in <top (required)> at /var/www/miq/vmdb/config/environments/production.rb:15) Stopping EVM gracefully... evmserverd.service stopping timed out. Terminating. evmserverd.service stop-sigterm timed out. Killing. evmserverd.service: main process exited, code=killed, status=9/KILL Stopped EVM server daemon. Unit evmserverd.service entered failed state. evmserverd.service failed. Starting EVM server daemon... Starting EVM... rake aborted! ActiveRecord::SubclassNotFound: The single-table inheritance mechanism failed to locate the subclass: 'MiqReplicationWorker'. This error is raised /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/inheritance.rb:182:in `rescue in find_sti_class' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/inheritance.rb:175:in `find_sti_class' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/inheritance.rb:163:in `discriminate_class_for_record' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/persistence.rb:67:in `instantiate' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/querying.rb:50:in `block (2 levels) in find_by_sql' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/result.rb:52:in `block in each' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/result.rb:52:in `each' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/result.rb:52:in `each' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/querying.rb:50:in `map' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/querying.rb:50:in `block in find_by_sql' /opt/rh/cfme-gemset/gems/activesupport-5.0.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/querying.rb:49:in `find_by_sql' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/statement_cache.rb:109:in `execute' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/associations/collection_association.rb:481:in `get_records' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/associations/collection_association.rb:485:in `find_target' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/associations/collection_association.rb:416:in `load_target' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/associations/collection_proxy.rb:44:in `load_target' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/associations/collection_proxy.rb:986:in `records' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/relation/delegation.rb:38:in `each' /var/www/miq/vmdb/app/models/miq_server/worker_management/monitor/kill.rb:21:in `kill_all_workers' /var/www/miq/vmdb/app/models/miq_server/worker_management.rb:15:in `kill_all_workers' /var/www/miq/vmdb/lib/tasks/evm_application.rb:21:in `start' /var/www/miq/vmdb/lib/tasks/evm.rake:8:in `block (2 levels) in <top (required)>' /opt/rh/cfme-gemset/gems/rake-12.0.0/exe/rake:27:in `<top (required)>' NameError: uninitialized constant MiqReplicationWorker /opt/rh/cfme-gemset/gems/activesupport-5.0.2/lib/active_support/inflector/methods.rb:268:in `const_get' /opt/rh/cfme-gemset/gems/activesupport-5.0.2/lib/active_support/inflector/methods.rb:268:in `block in constantize' /opt/rh/cfme-gemset/gems/activesupport-5.0.2/lib/active_support/inflector/methods.rb:266:in `each' /opt/rh/cfme-gemset/gems/activesupport-5.0.2/lib/active_support/inflector/methods.rb:266:in `inject' /opt/rh/cfme-gemset/gems/activesupport-5.0.2/lib/active_support/inflector/methods.rb:266:in `constantize' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/inheritance.rb:177:in `find_sti_class' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/inheritance.rb:163:in `discriminate_class_for_record' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/persistence.rb:67:in `instantiate' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/querying.rb:50:in `block (2 levels) in find_by_sql' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/result.rb:52:in `block in each' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/result.rb:52:in `each' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/result.rb:52:in `each' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/querying.rb:50:in `map' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/querying.rb:50:in `block in find_by_sql' /opt/rh/cfme-gemset/gems/activesupport-5.0.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/querying.rb:49:in `find_by_sql' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/statement_cache.rb:109:in `execute' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/associations/collection_association.rb:481:in `get_records' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/associations/collection_association.rb:485:in `find_target' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/associations/collection_association.rb:416:in `load_target' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/associations/collection_proxy.rb:44:in `load_target' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/associations/collection_proxy.rb:986:in `records' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/relation/delegation.rb:38:in `each' /var/www/miq/vmdb/app/models/miq_server/worker_management/monitor/kill.rb:21:in `kill_all_workers' /var/www/miq/vmdb/app/models/miq_server/worker_management.rb:15:in `kill_all_workers' /var/www/miq/vmdb/lib/tasks/evm_application.rb:21:in `start' /var/www/miq/vmdb/lib/tasks/evm.rake:8:in `block (2 levels) in <top (required)>' /opt/rh/cfme-gemset/gems/rake-12.0.0/exe/rake:27:in `<top (required)>' Tasks: TOP => evm:start (See full trace by running task with --trace) evmserverd.service: control process exited, code=exited status=1 Failed to start EVM server daemon.
I did some research... we used to have a clever hack to allow us to load a no longer existing model by trying to load it using it's base_class. commit 17123af7d16ad87565d1c0b1efaaccbcd6790443 Author: Joe Rafaniello <jrafanie> Date: Wed Jul 14 22:21:52 2010 +0000 Provide a generic way to instantiate existing miq_workers rows created using a subclass of MiqWorker which no longer exists (such as MiqWorkerMonitor). Do this by aliasing ActiveRecord::Base.instantiate and adding a retry using the base class (MiqWorker). bugzid: 9497 git-svn-id: http://miq-ubuntusub.manageiq.com/svn/svnrepos/Manageiq/trunk@22166 3c68ef56-dcc3-11dc-9475-a42b84ecc76f diff --git a/vmdb/app/models/miq_worker.rb b/vmdb/app/models/miq_worker.rb index ee8c1c6d4f..c9511da472 100644 --- a/vmdb/app/models/miq_worker.rb +++ b/vmdb/app/models/miq_worker.rb @@ -204,6 +204,21 @@ class MiqWorker < ActiveRecord::Base end end + # Provides a generic way to instantiate existing miq_workers rows created + # using a subclass of MiqWorker which no longer exists (such as MiqWorkerMonitor). + class << self + def instantiate_with_retry_failed_subclass(record) + instantiate_without_retry_failed_subclass(record) + rescue ActiveRecord::SubclassNotFound => err + raise if record[inheritance_column] == self.base_class.name + $log.warn("MIQ(#{self.name}.instantiate_with_retry_failed_subclass) Retrying missing subclass: [#{record[inheritance_column]}] with base class: [#{self.base_class}]") + record[inheritance_column] = self.base_class.name + retry + end + + alias_method_chain :instantiate, :retry_failed_subclass + end + def self.status_update self.find_current.each { |w| w.status_update } end I then removed this hack as a rails 4.2 upgrade broke it. The decision was made to not do this cleverness and instead do a migration, where we can disable sti, to delete the row. commit 0c4f18f8f19befb43e7ecc9f15e338e6b6f9019e Author: Joe Rafaniello <jrafanie> Date: Fri Dec 19 16:51:19 2014 -0500 Remove code allowing missing models to be instantiated via a base class. Rails 4.2 doesn't like this monkey patch since instantiate's arity changed from Rails 3.2 (1) to Rails 4.2(2). We can do this better by using a migration to delete miq_workers rows with the offending values. When we delete models, we need to write a data migration to remove rows for that model. diff --git a/vmdb/app/models/miq_worker.rb b/vmdb/app/models/miq_worker.rb index 998ba38fb1..1d6fe2c54a 100644 --- a/vmdb/app/models/miq_worker.rb +++ b/vmdb/app/models/miq_worker.rb @@ -241,21 +241,6 @@ class MiqWorker < ActiveRecord::Base end end - # Provides a generic way to instantiate existing miq_workers rows created - # using a subclass of MiqWorker which no longer exists (such as MiqWorkerMonitor). - class << self - def instantiate_with_retry_failed_subclass(record) - instantiate_without_retry_failed_subclass(record) - rescue ActiveRecord::SubclassNotFound => err - raise if record[inheritance_column] == self.base_class.name - $log.warn("MIQ(#{self.name}.instantiate_with_retry_failed_subclass) Retrying missing subclass: [#{record[inheritance_column]}] with base class: [#{self.base_class}]") - record[inheritance_column] = self.base_class.name - retry - end - - alias_method_chain :instantiate, :retry_failed_subclass - end -
PR: https://github.com/ManageIQ/manageiq/pull/14956
New commit detected on ManageIQ/manageiq/master: https://github.com/ManageIQ/manageiq/commit/53661b747e02625e541abff08eebd9ccb11f0bab commit 53661b747e02625e541abff08eebd9ccb11f0bab Author: Gregg Tanzillo <gtanzill> AuthorDate: Mon May 1 16:14:39 2017 -0400 Commit: Gregg Tanzillo <gtanzill> CommitDate: Mon May 1 17:34:23 2017 -0400 Migration to delete legacy "MiqReplicationWorker" rows from the miq_workers table. Fixes "ActiveRecord::SubclassNotFound: The single-table inheritance mechanism failed to locate the subclass: 'MiqReplicationWorker'." error at server startup. https://bugzilla.redhat.com/show_bug.cgi?id=1446398 .../20170405192333_delete_miq_replication_worker.rb | 9 +++++++++ ...170405192333_delete_miq_replication_worker_spec.rb | 19 +++++++++++++++++++ 2 files changed, 28 insertions(+) create mode 100644 db/migrate/20170405192333_delete_miq_replication_worker.rb create mode 100644 spec/migrations/20170405192333_delete_miq_replication_worker_spec.rb
Verified in 5.9.0.2 from 5.6.4.2