Bug 1467337

Summary: Upgrade [6.2 -> 6.3]: Failed Upgrade Step: migrate_foreman
Product: Red Hat Satellite Reporter: Sanket Jagtap <sjagtap>
Component: UpgradesAssignee: Eric Helms <ehelms>
Status: CLOSED ERRATA QA Contact: Sachin Ghai <sghai>
Severity: high Docs Contact:
Priority: high    
Version: 6.3.0CC: bbuckingham, ehelms, inecas, mbacovsk, sghai, sjagtap
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: foreman-1.15.2 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-02-21 16:54:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
foreman-debug
none
foreman-debug
none
logs from console none

Description Sanket Jagtap 2017-07-03 12:37:32 UTC
Description of problem:
Build: Satellite 6.3 snap5

Version-Release number of selected component (if applicable):
sat6.2.9 -> sat6.3.0 snap5

How reproducible:


Steps to Reproduce:
1. yum update -y
2. satellite-installer --scenario satellite --upgrade
3. Failed Upgrade Step: migrate_foreman


Logs
[satellite.com] out: Upgrade Step: migrate_foreman...
[satellite.com] out: foreman-rake -- config -k use_pulp_oauth -v true >/dev/null failed! Check the output for error!
[satellite.com] out: API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
[satellite.com] out: rake aborted!
[satellite.com] out: NoMethodError: undefined method `override=' for #<Filter:0x000000093f9400>
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activemodel-4.2.6/lib/active_model/attribute_methods.rb:433:in `method_missing'
[satellite.com] out: /usr/share/foreman/app/models/filter.rb:221:in `enforce_override_flag'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:432:in `block in make_lambda'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:164:in `block in halting'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:504:in `block in call'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:504:in `each'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:504:in `call'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:92:in `__run_callbacks__'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:778:in `_run_validation_callbacks'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activemodel-4.2.6/lib/active_model/validations/callbacks.rb:113:in `run_validations!'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activemodel-4.2.6/lib/active_model/validations.rb:338:in `valid?'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/validations.rb:58:in `valid?'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/autosave_association.rb:324:in `association_valid?'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/autosave_association.rb:312:in `block in validate_collection_association'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/autosave_association.rb:312:in `each'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/autosave_association.rb:312:in `validate_collection_association'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/autosave_association.rb:217:in `block in define_autosave_validation_callbacks'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/autosave_association.rb:157:in `instance_eval'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/autosave_association.rb:157:in `block in define_non_cyclic_method'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:432:in `block in make_lambda'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:192:in `block in simple'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:504:in `block in call'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:504:in `each'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:504:in `call'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:92:in `__run_callbacks__'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:778:in `_run_validate_callbacks'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activemodel-4.2.6/lib/active_model/validations.rb:399:in `run_validations!'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activemodel-4.2.6/lib/active_model/validations/callbacks.rb:113:in `block in run_validations!'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:117:in `call'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:555:in `block (2 levels) in compile'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:505:in `call'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:92:in `__run_callbacks__'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:778:in `_run_validation_callbacks'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activemodel-4.2.6/lib/active_model/validations/callbacks.rb:113:in `run_validations!'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activemodel-4.2.6/lib/active_model/validations.rb:338:in `valid?'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/validations.rb:58:in `valid?'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/validations.rb:83:in `perform_validations'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/validations.rb:43:in `save!'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/attribute_methods/dirty.rb:29:in `save!'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/transactions.rb:291:in `block in save!'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/transactions.rb:351:in `block in with_transaction_returning_status'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/database_statements.rb:211:in `transaction'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/transactions.rb:220:in `transaction'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/transactions.rb:348:in `with_transaction_returning_status'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/transactions.rb:291:in `save!'
[satellite.com] out: /usr/share/foreman/app/models/role.rb:154:in `add_permissions!'
[satellite.com] out: /usr/share/foreman/app/services/foreman/plugin/rbac_support.rb:85:in `block (2 levels) in include_permissions_for_role'
[satellite.com] out: /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.4.1/lib/audited/auditor.rb:283:in `without_auditing'
[satellite.com] out: /usr/share/foreman/app/services/foreman/plugin/rbac_support.rb:82:in `block in include_permissions_for_role'
[satellite.com] out: /usr/share/foreman/app/models/role.rb:176:in `ignore_locking'
[satellite.com] out: /usr/share/foreman/app/services/foreman/plugin/rbac_support.rb:81:in `include_permissions_for_role'
[satellite.com] out: /usr/share/foreman/app/services/foreman/plugin/rbac_support.rb:93:in `add_all_permissions_to_role'
[satellite.com] out: /usr/share/foreman/app/services/foreman/plugin/rbac_support.rb:7:in `block in add_all_permissions_to_default_roles'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `block in transaction'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/transaction.rb:184:in `within_new_transaction'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `transaction'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/transactions.rb:220:in `transaction'
[satellite.com] out: /usr/share/foreman/app/services/foreman/plugin/rbac_support.rb:6:in `add_all_permissions_to_default_roles'
[satellite.com] out: /usr/share/foreman/app/services/foreman/plugin.rb:272:in `add_all_permissions_to_default_roles'
[satellite.com] out: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.3/lib/foreman_tasks/engine.rb:73:in `block (2 levels) in <class:Engine>'
[satellite.com] out: /usr/share/foreman/app/services/foreman/plugin.rb:66:in `instance_eval'
[satellite.com] out: /usr/share/foreman/app/services/foreman/plugin.rb:66:in `register'
[satellite.com] out: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.3/lib/foreman_tasks/engine.rb:43:in `block in <class:Engine>'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/railties-4.2.6/lib/rails/initializable.rb:30:in `instance_exec'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/railties-4.2.6/lib/rails/initializable.rb:30:in `run'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/railties-4.2.6/lib/rails/initializable.rb:55:in `block in run_initializers'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/railties-4.2.6/lib/rails/initializable.rb:54:in `run_initializers'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/railties-4.2.6/lib/rails/application.rb:352:in `initialize!'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/railties-4.2.6/lib/rails/railtie.rb:194:in `public_send'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/railties-4.2.6/lib/rails/railtie.rb:194:in `method_missing'
[satellite.com] out: /usr/share/foreman/config/environment.rb:5:in `<top (required)>'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/polyglot-0.3.4/lib/polyglot.rb:65:in `require'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in `block in require'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:240:in `load_dependency'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in `require'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/railties-4.2.6/lib/rails/application.rb:328:in `require_environment!'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/railties-4.2.6/lib/rails/application.rb:457:in `block in run_tasks_blocks'
[satellite.com] out: Tasks: TOP => db:migrate => environment
[satellite.com] out: (See full trace by running task with --trace)
[satellite.com] out: foreman-rake db:migrate failed! Check the output for error!
[satellite.com] out: foreman-rake -- config -k use_pulp_oauth -v false >/dev/null failed! Check the output for error!
[satellite.com] out: Upgrade step migrate_foreman failed. Check logs for more information.
[satellite.com] out:

Fatal error: run() received nonzero return code 1 while executing!

Requested: satellite-installer --scenario satellite --upgrade
Executed: /bin/bash -l -c "satellite-installer --scenario satellite --upgrade"

Aborting.




Actual results:


Expected results:


Additional info:

Comment 1 Sachin Ghai 2017-07-03 12:53:03 UTC
Created attachment 1293884 [details]
foreman-debug

Comment 3 Sanket Jagtap 2017-07-17 06:35:15 UTC
Build: Satellite 6.3 snap 7

Upgraded from 6.2.10 -> satellite 6.3 snap7

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: 
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: == 20170208215148 AddDockerRepoName: migrating ================================
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: -- add_column(:katello_repositories, :container_repository_name, :string)
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out:    -> 0.0007s
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: foreman-rake db:migrate failed! Check the output for error!
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: foreman-rake -- config -k use_pulp_oauth -v false >/dev/null finished successfully!
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade step migrate_foreman failed. Check logs for more information.

This still fails , with same error.

I can see the related fix in the file, but this is still reproducible

Comment 5 Satellite Program 2017-07-17 08:02:32 UTC
Upstream bug assigned to ehelms

Comment 7 Sachin Ghai 2017-08-03 08:23:40 UTC
Created attachment 1308624 [details]
foreman-debug

Comment 8 Sachin Ghai 2017-08-03 08:40:06 UTC
Created attachment 1308642 [details]
logs from console

Comment 9 Sachin Ghai 2017-08-03 08:41:43 UTC
If you see the attached log from comment8, here are the errors:

[SAT_SERVER] out: == 20161102194100 CreateContentViewDockerFilterRules: migrating ===============
	[SAT_SERVER] out: -- create_table(:katello_content_view_docker_filter_rules)
	[SAT_SERVER] out:    -> 0.0103s
	[SAT_SERVER] out: -- add_foreign_key(:katello_content_view_docker_filter_rules, :katello_content_view_filters, {:name=>"katello_content_view_docker_filter_rules_filter_fk", :column=>"content_view_filter_id"})rake aborted!
	[SAT_SERVER] out: StandardError: An error has occurred, this and all later migrations canceled:
	[SAT_SERVER] out: 
	[SAT_SERVER] out: Could not find the inverse association for content_view_version (:repositories in Katello::ContentViewVersion)/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/reflection.rb:184:in `check_validity_of_inverse!'
	[SAT_SERVER] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/reflection.rb:355:in `check_validity!'
	[SAT_SERVER] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/associations/association.rb:25:in `initialize'
	[SAT_SERVER] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/associations.rb:162:in `new'
	[SAT_SERVER] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/associations.rb:162:in `association'
	[SAT_SERVER] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/associations/builder/association.rb:115:in `content_view_version'
	[SAT_SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.2/app/models/katello/repository.rb:190:in `content_view'
	[SAT_SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.2/app/models/katello/repository.rb:178:in `set_container_repository_name'

Comment 10 Sachin Ghai 2017-09-06 10:54:08 UTC
I see similar issue w/ sat6.3 snap14. 

: Upgrade Step: migrate_foreman...
[qe-upgrade] out: foreman-rake -- config -k use_pulp_oauth -v true >/dev/null failed! Check the output for error!
[qe-upgrade] out: API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
[qe-upgrade] out: == 20151210143537 AddTypeToMailNotification: migrating ========================
[qe-upgrade] out: -- add_column(:mail_notifications, :type, :string, {:limit=>255})
[qe-upgrade]out:

Comment 11 Sachin Ghai 2017-09-27 09:08:11 UTC
The reported issue is fixed w/ Sat 6.3 snap16. upgrade was completed successfully.

Comment 13 Satellite Program 2018-02-21 16:54:37 UTC
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.
> > 
> > For information on the advisory, and where to find the updated files, follow the link below.
> > 
> > If the solution does not work for you, open a new bug report.
> > 
> > https://access.redhat.com/errata/RHSA-2018:0336