Bug 1446398
| Summary: | evm fails to start on remote region after upgrading from 5.6 rubyrep to 5.8 | |||
|---|---|---|---|---|
| Product: | Red Hat CloudForms Management Engine | Reporter: | luke couzens <lcouzens> | |
| Component: | Appliance | Assignee: | Joe Rafaniello <jrafanie> | |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | luke couzens <lcouzens> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 5.8.0 | CC: | abellott, cpelland, dajohnso, gtanzill, jhardy, jrafanie, obarenbo, simaishi | |
| Target Milestone: | GA | Keywords: | TestOnly | |
| Target Release: | 5.9.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | black:replication:upgrade | |||
| Fixed In Version: | 5.9.0.1 | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1447350 (view as bug list) | Environment: | ||
| Last Closed: | 2018-03-06 14:55:40 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | CFME Core | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1447350 | |||
I did some research... we used to have a clever hack to allow us to load a no longer existing model by trying to load it using it's base_class.
commit 17123af7d16ad87565d1c0b1efaaccbcd6790443
Author: Joe Rafaniello <jrafanie>
Date: Wed Jul 14 22:21:52 2010 +0000
Provide a generic way to instantiate existing miq_workers rows created
using a subclass of MiqWorker which no longer exists (such as MiqWorkerMonitor).
Do this by aliasing ActiveRecord::Base.instantiate and adding a retry
using the base class (MiqWorker).
bugzid: 9497
git-svn-id: http://miq-ubuntusub.manageiq.com/svn/svnrepos/Manageiq/trunk@22166 3c68ef56-dcc3-11dc-9475-a42b84ecc76f
diff --git a/vmdb/app/models/miq_worker.rb b/vmdb/app/models/miq_worker.rb
index ee8c1c6d4f..c9511da472 100644
--- a/vmdb/app/models/miq_worker.rb
+++ b/vmdb/app/models/miq_worker.rb
@@ -204,6 +204,21 @@ class MiqWorker < ActiveRecord::Base
end
end
+ # Provides a generic way to instantiate existing miq_workers rows created
+ # using a subclass of MiqWorker which no longer exists (such as MiqWorkerMonitor).
+ class << self
+ def instantiate_with_retry_failed_subclass(record)
+ instantiate_without_retry_failed_subclass(record)
+ rescue ActiveRecord::SubclassNotFound => err
+ raise if record[inheritance_column] == self.base_class.name
+ $log.warn("MIQ(#{self.name}.instantiate_with_retry_failed_subclass) Retrying missing subclass: [#{record[inheritance_column]}] with base class: [#{self.base_class}]")
+ record[inheritance_column] = self.base_class.name
+ retry
+ end
+
+ alias_method_chain :instantiate, :retry_failed_subclass
+ end
+
def self.status_update
self.find_current.each { |w| w.status_update }
end
I then removed this hack as a rails 4.2 upgrade broke it. The decision was made to not do this cleverness and instead do a migration, where we can disable sti, to delete the row.
commit 0c4f18f8f19befb43e7ecc9f15e338e6b6f9019e
Author: Joe Rafaniello <jrafanie>
Date: Fri Dec 19 16:51:19 2014 -0500
Remove code allowing missing models to be instantiated via a base class.
Rails 4.2 doesn't like this monkey patch since instantiate's arity changed from Rails 3.2 (1) to Rails 4.2(2).
We can do this better by using a migration to delete miq_workers rows with the offending values.
When we delete models, we need to write a data migration to remove rows for that model.
diff --git a/vmdb/app/models/miq_worker.rb b/vmdb/app/models/miq_worker.rb
index 998ba38fb1..1d6fe2c54a 100644
--- a/vmdb/app/models/miq_worker.rb
+++ b/vmdb/app/models/miq_worker.rb
@@ -241,21 +241,6 @@ class MiqWorker < ActiveRecord::Base
end
end
- # Provides a generic way to instantiate existing miq_workers rows created
- # using a subclass of MiqWorker which no longer exists (such as MiqWorkerMonitor).
- class << self
- def instantiate_with_retry_failed_subclass(record)
- instantiate_without_retry_failed_subclass(record)
- rescue ActiveRecord::SubclassNotFound => err
- raise if record[inheritance_column] == self.base_class.name
- $log.warn("MIQ(#{self.name}.instantiate_with_retry_failed_subclass) Retrying missing subclass: [#{record[inheritance_column]}] with base class: [#{self.base_class}]")
- record[inheritance_column] = self.base_class.name
- retry
- end
-
- alias_method_chain :instantiate, :retry_failed_subclass
- end
-
New commit detected on ManageIQ/manageiq/master: https://github.com/ManageIQ/manageiq/commit/53661b747e02625e541abff08eebd9ccb11f0bab commit 53661b747e02625e541abff08eebd9ccb11f0bab Author: Gregg Tanzillo <gtanzill> AuthorDate: Mon May 1 16:14:39 2017 -0400 Commit: Gregg Tanzillo <gtanzill> CommitDate: Mon May 1 17:34:23 2017 -0400 Migration to delete legacy "MiqReplicationWorker" rows from the miq_workers table. Fixes "ActiveRecord::SubclassNotFound: The single-table inheritance mechanism failed to locate the subclass: 'MiqReplicationWorker'." error at server startup. https://bugzilla.redhat.com/show_bug.cgi?id=1446398 .../20170405192333_delete_miq_replication_worker.rb | 9 +++++++++ ...170405192333_delete_miq_replication_worker_spec.rb | 19 +++++++++++++++++++ 2 files changed, 28 insertions(+) create mode 100644 db/migrate/20170405192333_delete_miq_replication_worker.rb create mode 100644 spec/migrations/20170405192333_delete_miq_replication_worker_spec.rb Verified in 5.9.0.2 from 5.6.4.2 |
Description of problem:evm fails to start on remote region after upgrading from 5.6 rubyrep to 5.8 due to records left in miq_workers Version-Release number of selected component (if applicable):5.8.0.12 How reproducible:100% Steps to Reproduce: 1.provision 5.6 appliances 2.setup rubyrep 3.add a provider 4.check its replicated 5.stop db sync role (remote) 6.stop evm on (remote) 7.run rake evm:dbsync:uninstall (remote) 8.stop evm on global 9.add 5.8 repos 10.run yum update on both apps 11.reconnect ssh (reload ruby) 12.run vmdb 13.run rake db:migrate 14.run rake evm:automate:reset 15.run /usr/bin/miq_postgres_upgrade 16.make edits and start postgres 17.start evm Actual results:evm fails to start on remote region Expected results:evm to start correctly Additional info: vmdb_production=# select id, status, type from miq_workers; id | status | type ---------------+----------+---------------------- 1000000000012 | stopping | MiqReplicationWorker 1000000000019 | aborted | MiqPriorityWorker 1000000000020 | aborted | MiqPriorityWorker 1000000000017 | aborted | MiqGenericWorker 1000000000018 | aborted | MiqGenericWorker 1000000000021 | aborted | MiqScheduleWorker (6 rows) journalctl -u evmserverd -- Logs begin at Thu 2017-04-27 10:56:50 EDT, end at Thu 2017-04-27 14:56:19 EDT. -- Starting EVM server daemon... DEPRECATION WARNING: `config.serve_static_files` is deprecated and will be removed in Rails 5.1. Please use `config.public_file_server.enabled = false` instead. (called from block in <top (required)> at /var/www/miq/vmdb/config/environments/production.rb:15) Starting EVM... Running EVM in background... Started EVM server daemon. Stopping EVM server daemon... DEPRECATION WARNING: `config.serve_static_files` is deprecated and will be removed in Rails 5.1. Please use `config.public_file_server.enabled = false` instead. (called from block in <top (required)> at /var/www/miq/vmdb/config/environments/production.rb:15) Stopping EVM gracefully... evmserverd.service stopping timed out. Terminating. evmserverd.service stop-sigterm timed out. Killing. evmserverd.service: main process exited, code=killed, status=9/KILL Stopped EVM server daemon. Unit evmserverd.service entered failed state. evmserverd.service failed. Starting EVM server daemon... Starting EVM... rake aborted! ActiveRecord::SubclassNotFound: The single-table inheritance mechanism failed to locate the subclass: 'MiqReplicationWorker'. This error is raised /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/inheritance.rb:182:in `rescue in find_sti_class' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/inheritance.rb:175:in `find_sti_class' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/inheritance.rb:163:in `discriminate_class_for_record' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/persistence.rb:67:in `instantiate' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/querying.rb:50:in `block (2 levels) in find_by_sql' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/result.rb:52:in `block in each' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/result.rb:52:in `each' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/result.rb:52:in `each' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/querying.rb:50:in `map' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/querying.rb:50:in `block in find_by_sql' /opt/rh/cfme-gemset/gems/activesupport-5.0.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/querying.rb:49:in `find_by_sql' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/statement_cache.rb:109:in `execute' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/associations/collection_association.rb:481:in `get_records' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/associations/collection_association.rb:485:in `find_target' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/associations/collection_association.rb:416:in `load_target' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/associations/collection_proxy.rb:44:in `load_target' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/associations/collection_proxy.rb:986:in `records' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/relation/delegation.rb:38:in `each' /var/www/miq/vmdb/app/models/miq_server/worker_management/monitor/kill.rb:21:in `kill_all_workers' /var/www/miq/vmdb/app/models/miq_server/worker_management.rb:15:in `kill_all_workers' /var/www/miq/vmdb/lib/tasks/evm_application.rb:21:in `start' /var/www/miq/vmdb/lib/tasks/evm.rake:8:in `block (2 levels) in <top (required)>' /opt/rh/cfme-gemset/gems/rake-12.0.0/exe/rake:27:in `<top (required)>' NameError: uninitialized constant MiqReplicationWorker /opt/rh/cfme-gemset/gems/activesupport-5.0.2/lib/active_support/inflector/methods.rb:268:in `const_get' /opt/rh/cfme-gemset/gems/activesupport-5.0.2/lib/active_support/inflector/methods.rb:268:in `block in constantize' /opt/rh/cfme-gemset/gems/activesupport-5.0.2/lib/active_support/inflector/methods.rb:266:in `each' /opt/rh/cfme-gemset/gems/activesupport-5.0.2/lib/active_support/inflector/methods.rb:266:in `inject' /opt/rh/cfme-gemset/gems/activesupport-5.0.2/lib/active_support/inflector/methods.rb:266:in `constantize' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/inheritance.rb:177:in `find_sti_class' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/inheritance.rb:163:in `discriminate_class_for_record' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/persistence.rb:67:in `instantiate' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/querying.rb:50:in `block (2 levels) in find_by_sql' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/result.rb:52:in `block in each' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/result.rb:52:in `each' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/result.rb:52:in `each' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/querying.rb:50:in `map' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/querying.rb:50:in `block in find_by_sql' /opt/rh/cfme-gemset/gems/activesupport-5.0.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/querying.rb:49:in `find_by_sql' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/statement_cache.rb:109:in `execute' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/associations/collection_association.rb:481:in `get_records' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/associations/collection_association.rb:485:in `find_target' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/associations/collection_association.rb:416:in `load_target' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/associations/collection_proxy.rb:44:in `load_target' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/associations/collection_proxy.rb:986:in `records' /opt/rh/cfme-gemset/gems/activerecord-5.0.2/lib/active_record/relation/delegation.rb:38:in `each' /var/www/miq/vmdb/app/models/miq_server/worker_management/monitor/kill.rb:21:in `kill_all_workers' /var/www/miq/vmdb/app/models/miq_server/worker_management.rb:15:in `kill_all_workers' /var/www/miq/vmdb/lib/tasks/evm_application.rb:21:in `start' /var/www/miq/vmdb/lib/tasks/evm.rake:8:in `block (2 levels) in <top (required)>' /opt/rh/cfme-gemset/gems/rake-12.0.0/exe/rake:27:in `<top (required)>' Tasks: TOP => evm:start (See full trace by running task with --trace) evmserverd.service: control process exited, code=exited status=1 Failed to start EVM server daemon.