Bug 1661957

Summary: evmserverd failed to start after upgrade from 4.5 to 4.6 with OpenStack 10 provider
Product: Red Hat CloudForms Management Engine Reporter: Ahmed El-Rayess <aelrayes>
Component: ApplianceAssignee: Julian Cheal <jcheal>
Status: CLOSED WONTFIX QA Contact: Md Nadeem <mnadeem>
Severity: high Docs Contact: Red Hat CloudForms Documentation <cloudforms-docs>
Priority: high    
Version: 5.9.6CC: abellott, dmetzger, jcheal, jfrey, jprause, jrafanie, mfeifer, obarenbo, rspagnol, simaishi, smallamp, tuado
Target Milestone: GA   
Target Release: 5.9.8   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-01-28 17:16:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: CFME Core Target Upstream Version:
Embargoed:

Description Ahmed El-Rayess 2018-12-24 22:00:26 UTC 
Description of problem:
After doing the upgrade of CloudForms from version 4.5 to 4.6 the evmserverd fails to start.

rake evm:status displays the below error
Checking EVM status...
 Zone    | Server | Status  |              ID |  PID | SPID | URL                                               | Started On           | Last Heartbeat       | MB Usage | Master? | Active Roles
---------+--------+---------+-----------------+------+------+---------------------------------------------------+----------------------+----------------------+----------+---------+----------------
 default | DSG    | quiesce | 101000000000001 | 4882 | 5011 | drbunix:///tmp/worker_monitor20181225-4882-1j2zke | 2018-12-24T20:01:46Z | 2018-12-24T20:34:14Z |      370 | false   | database_owner

rake aborted!
ActiveRecord::SubclassNotFound: The single-table inheritance mechanism failed to locate the subclass: 'ManageIQ::Providers::StorageManager::CinderManager::EventCatcher'. This error is raised because the column 'type' is reserved for storing the class in case of inheritance. Please rename this column if you didn't intend it to be used for storing the inheritance class or overwrite MiqWorker.inheritance_column to use another column for that information.
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/inheritance.rb:182:in `rescue in find_sti_class'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/inheritance.rb:175:in `find_sti_class'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/inheritance.rb:163:in `discriminate_class_for_record'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/persistence.rb:67:in `instantiate'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/querying.rb:50:in `block (2 levels) in find_by_sql'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/result.rb:52:in `block in each'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/result.rb:52:in `each'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/result.rb:52:in `each'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/querying.rb:50:in `map'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/querying.rb:50:in `block in find_by_sql'
/opt/rh/cfme-gemset/gems/activesupport-5.0.6/lib/active_support/notifications/instrumenter.rb:21:in `instrument'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/querying.rb:49:in `find_by_sql'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/relation.rb:706:in `exec_queries'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/association_relation.rb:32:in `exec_queries'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/relation.rb:583:in `load'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/relation.rb:260:in `records'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/relation/delegation.rb:38:in `each'
/var/www/miq/vmdb/lib/tasks/evm_application.rb:96:in `block in output_workers_status'
/var/www/miq/vmdb/lib/tasks/evm_application.rb:95:in `output_workers_status'
/var/www/miq/vmdb/lib/tasks/evm_application.rb:69:in `status'
/var/www/miq/vmdb/lib/tasks/evm.rake:45:in `block (2 levels) in <top (required)>'
/opt/rh/cfme-gemset/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'

Caused by:
NameError: uninitialized constant ManageIQ::Providers::StorageManager::CinderManager::EventCatcher
/opt/rh/cfme-gemset/gems/activesupport-5.0.6/lib/active_support/inflector/methods.rb:270:in `const_get'
/opt/rh/cfme-gemset/gems/activesupport-5.0.6/lib/active_support/inflector/methods.rb:270:in `block in constantize'
/opt/rh/cfme-gemset/gems/activesupport-5.0.6/lib/active_support/inflector/methods.rb:266:in `each'
/opt/rh/cfme-gemset/gems/activesupport-5.0.6/lib/active_support/inflector/methods.rb:266:in `inject'
/opt/rh/cfme-gemset/gems/activesupport-5.0.6/lib/active_support/inflector/methods.rb:266:in `constantize'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/inheritance.rb:177:in `find_sti_class'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/inheritance.rb:163:in `discriminate_class_for_record'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/persistence.rb:67:in `instantiate'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/querying.rb:50:in `block (2 levels) in find_by_sql'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/result.rb:52:in `block in each'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/result.rb:52:in `each'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/result.rb:52:in `each'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/querying.rb:50:in `map'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/querying.rb:50:in `block in find_by_sql'
/opt/rh/cfme-gemset/gems/activesupport-5.0.6/lib/active_support/notifications/instrumenter.rb:21:in `instrument'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/querying.rb:49:in `find_by_sql'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/relation.rb:706:in `exec_queries'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/association_relation.rb:32:in `exec_queries'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/relation.rb:583:in `load'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/relation.rb:260:in `records'
/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/relation/delegation.rb:38:in `each'
/var/www/miq/vmdb/lib/tasks/evm_application.rb:96:in `block in output_workers_status'
/var/www/miq/vmdb/lib/tasks/evm_application.rb:95:in `output_workers_status'
/var/www/miq/vmdb/lib/tasks/evm_application.rb:69:in `status'
/var/www/miq/vmdb/lib/tasks/evm.rake:45:in `block (2 levels) in <top (required)>'
/opt/rh/cfme-gemset/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => evm:status
(See full trace by running task with --trace)


Version-Release number of selected component (if applicable):
cfme-5.9.6.5-4.el7cf.x86_64
cfme-appliance-5.9.6.5-1.el7cf.x86_64
cfme-gemset-5.9.6.5-3.el7cf.x86_64
cfme-appliance-common-5.9.6.5-1.el7cf.x86_64

How reproducible:


Steps to Reproduce:
1. Configure CF 4.5 with OpenStack 10 provider
2. Upgrade to CF 4.6 and apply hotfix 5.9.6.5-4 and gemset 5.9.6.5-3
3. reboot and attempt to start evmserverd

Actual results:
evmserverd is not starting and displaying the above error

Expected results:
evmserverd is able to start normally

Additional info:
Comment 2 Joe Rafaniello 2019-01-07 16:50:18 UTC 
It looks like this class was removed here [1] but the miq_workers rows were not removed.  We'll need to write a migration to remove these rows.

Similar to bug 1644083 and bug 1655794.

Until we have the fixed data migration released, the user can open a rails console and do the following:

MiqWorker.where(:type => "ManageIQ::Providers::StorageManager::CinderManager::EventCatcher").delete_all


[1] https://github.com/ManageIQ/manageiq/pull/14962
Comment 3 Julian Cheal 2019-01-08 17:53:30 UTC 
PR with data migration https://github.com/ManageIQ/manageiq-schema/pull/320
Comment 4 dmetzger 2019-01-09 17:42:04 UTC 
Reverting the 5.9.z -> 5.9.8 change. Migrations do not run on on Z-Stream updates, so the proposed PR will not help. If there is a way to resolve this in the 5.9 branch it can be considered for a future errata.


Please verify whether this problem exists in 5.10.0.
Comment 5 Jason Frey 2019-01-17 17:27:44 UTC 
IMO, we need to find a better general approach to this.  We have a number of migrations that fix specific problems, but I think we need to do a more strategic fix that clears all errant worker records, regardless of STI.
Comment 9 Red Hat Bugzilla 2023-09-18 00:15:08 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days