Bug 1661957 - evmserverd failed to start after upgrade from 4.5 to 4.6 with OpenStack 10 provider [NEEDINFO]
Summary: evmserverd failed to start after upgrade from 4.5 to 4.6 with OpenStack 10 pr...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Appliance
Version: 5.9.6
Hardware: x86_64
OS: Linux
high
high
Target Milestone: GA
: 5.9.8
Assignee: Julian Cheal
QA Contact: Md Nadeem
Red Hat CloudForms Documentation
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-12-24 22:00 UTC by Ahmed El-Rayess
Modified: 2020-02-20 15:52 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-01-28 17:16:27 UTC
Category: ---
Cloudforms Team: CFME Core
Target Upstream Version:
jcheal: needinfo? (jfrey)


Attachments (Terms of Use)

Description Ahmed El-Rayess 2018-12-24 22:00:26 UTC
Description of problem:
After doing the upgrade of CloudForms from version 4.5 to 4.6 the evmserverd fails to start.

rake evm:status displays the below error
Checking EVM status...
 Zone    | Server | Status  |              ID |  PID | SPID | URL                                               | Started On           | Last Heartbeat       | MB Usage | Master? | Active Roles
---------+--------+---------+-----------------+------+------+---------------------------------------------------+----------------------+----------------------+----------+---------+----------------
 default | DSG    | quiesce | 101000000000001 | 4882 | 5011 | drbunix:///tmp/worker_monitor20181225-4882-1j2zke | 2018-12-24T20:01:46Z | 2018-12-24T20:34:14Z |      370 | false   | database_owner

rake aborted!
ActiveRecord::SubclassNotFound: The single-table inheritance mechanism failed to locate the subclass: 'ManageIQ::Providers::StorageManager::CinderManager::EventCatcher'. This error is raised because the column 'type' is reserved for storing the class in case of inheritance. Please rename this column if you didn't intend it to be used for storing the inheritance class or overwrite MiqWorker.inheritance_column to use another column for that information.
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/inheritance.rb:182:in `rescue in find_sti_class'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/inheritance.rb:175:in `find_sti_class'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/inheritance.rb:163:in `discriminate_class_for_record'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/persistence.rb:67:in `instantiate'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/querying.rb:50:in `block (2 levels) in find_by_sql'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/result.rb:52:in `block in each'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/result.rb:52:in `each'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/result.rb:52:in `each'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/querying.rb:50:in `map'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/querying.rb:50:in `block in find_by_sql'
/opt/rh/cfme-gemset/gems/activesupport-5.0.6/lib/active_support/notifications/instrumenter.rb:21:in `instrument'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/querying.rb:49:in `find_by_sql'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/relation.rb:706:in `exec_queries'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/association_relation.rb:32:in `exec_queries'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/relation.rb:583:in `load'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/relation.rb:260:in `records'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/relation/delegation.rb:38:in `each'
/var/www/miq/vmdb/lib/tasks/evm_application.rb:96:in `block in output_workers_status'
/var/www/miq/vmdb/lib/tasks/evm_application.rb:95:in `output_workers_status'
/var/www/miq/vmdb/lib/tasks/evm_application.rb:69:in `status'
/var/www/miq/vmdb/lib/tasks/evm.rake:45:in `block (2 levels) in <top (required)>'
/opt/rh/cfme-gemset/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'

Caused by:
NameError: uninitialized constant ManageIQ::Providers::StorageManager::CinderManager::EventCatcher
/opt/rh/cfme-gemset/gems/activesupport-5.0.6/lib/active_support/inflector/methods.rb:270:in `const_get'
/opt/rh/cfme-gemset/gems/activesupport-5.0.6/lib/active_support/inflector/methods.rb:270:in `block in constantize'
/opt/rh/cfme-gemset/gems/activesupport-5.0.6/lib/active_support/inflector/methods.rb:266:in `each'
/opt/rh/cfme-gemset/gems/activesupport-5.0.6/lib/active_support/inflector/methods.rb:266:in `inject'
/opt/rh/cfme-gemset/gems/activesupport-5.0.6/lib/active_support/inflector/methods.rb:266:in `constantize'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/inheritance.rb:177:in `find_sti_class'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/inheritance.rb:163:in `discriminate_class_for_record'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/persistence.rb:67:in `instantiate'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/querying.rb:50:in `block (2 levels) in find_by_sql'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/result.rb:52:in `block in each'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/result.rb:52:in `each'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/result.rb:52:in `each'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/querying.rb:50:in `map'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/querying.rb:50:in `block in find_by_sql'
/opt/rh/cfme-gemset/gems/activesupport-5.0.6/lib/active_support/notifications/instrumenter.rb:21:in `instrument'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/querying.rb:49:in `find_by_sql'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/relation.rb:706:in `exec_queries'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/association_relation.rb:32:in `exec_queries'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/relation.rb:583:in `load'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/relation.rb:260:in `records'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/relation/delegation.rb:38:in `each'
/var/www/miq/vmdb/lib/tasks/evm_application.rb:96:in `block in output_workers_status'
/var/www/miq/vmdb/lib/tasks/evm_application.rb:95:in `output_workers_status'
/var/www/miq/vmdb/lib/tasks/evm_application.rb:69:in `status'
/var/www/miq/vmdb/lib/tasks/evm.rake:45:in `block (2 levels) in <top (required)>'
/opt/rh/cfme-gemset/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => evm:status
(See full trace by running task with --trace)


Version-Release number of selected component (if applicable):
cfme-5.9.6.5-4.el7cf.x86_64
cfme-appliance-5.9.6.5-1.el7cf.x86_64
cfme-gemset-5.9.6.5-3.el7cf.x86_64
cfme-appliance-common-5.9.6.5-1.el7cf.x86_64

How reproducible:


Steps to Reproduce:
1. Configure CF 4.5 with OpenStack 10 provider
2. Upgrade to CF 4.6 and apply hotfix 5.9.6.5-4 and gemset 5.9.6.5-3
3. reboot and attempt to start evmserverd

Actual results:
evmserverd is not starting and displaying the above error

Expected results:
evmserverd is able to start normally

Additional info:

Comment 2 Joe Rafaniello 2019-01-07 16:50:18 UTC
It looks like this class was removed here [1] but the miq_workers rows were not removed.  We'll need to write a migration to remove these rows.

Similar to bug 1644083 and bug 1655794.

Until we have the fixed data migration released, the user can open a rails console and do the following:

MiqWorker.where(:type => "ManageIQ::Providers::StorageManager::CinderManager::EventCatcher").delete_all


[1] https://github.com/ManageIQ/manageiq/pull/14962

Comment 3 Julian Cheal 2019-01-08 17:53:30 UTC
PR with data migration https://github.com/ManageIQ/manageiq-schema/pull/320

Comment 4 dmetzger 2019-01-09 17:42:04 UTC
Reverting the 5.9.z -> 5.9.8 change. Migrations do not run on on Z-Stream updates, so the proposed PR will not help. If there is a way to resolve this in the 5.9 branch it can be considered for a future errata.


Please verify whether this problem exists in 5.10.0.

Comment 5 Jason Frey 2019-01-17 17:27:44 UTC
IMO, we need to find a better general approach to this.  We have a number of migrations that fix specific problems, but I think we need to do a more strategic fix that clears all errant worker records, regardless of STI.


Note You need to log in before you can comment on or make changes to this bug.