Bug 1467337 - Upgrade [6.2 -> 6.3]: Failed Upgrade Step: migrate_foreman
Upgrade [6.2 -> 6.3]: Failed Upgrade Step: migrate_foreman
Status: VERIFIED
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Upgrades (Show other bugs)
6.3.0
Unspecified Unspecified
high Severity high (vote)
: Beta
: --
Assigned To: Eric Helms
Sachin Ghai
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-03 08:37 EDT by Sanket Jagtap
Modified: 2017-09-27 05:08 EDT (History)
6 users (show)

See Also:
Fixed In Version: foreman-1.15.2
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
foreman-debug (6.43 MB, application/x-xz)
2017-07-03 08:53 EDT, Sachin Ghai
no flags Details
foreman-debug (6.08 MB, application/x-xz)
2017-08-03 04:23 EDT, Sachin Ghai
no flags Details
logs from console (65.15 KB, text/plain)
2017-08-03 04:40 EDT, Sachin Ghai
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Foreman Issue Tracker 19930 None None None 2017-07-05 11:07 EDT
Foreman Issue Tracker 20551 None None None 2017-08-10 13:45 EDT

  None (edit)
Description Sanket Jagtap 2017-07-03 08:37:32 EDT
Description of problem:
Build: Satellite 6.3 snap5

Version-Release number of selected component (if applicable):
sat6.2.9 -> sat6.3.0 snap5

How reproducible:


Steps to Reproduce:
1. yum update -y
2. satellite-installer --scenario satellite --upgrade
3. Failed Upgrade Step: migrate_foreman


Logs
[satellite.com] out: Upgrade Step: migrate_foreman...
[satellite.com] out: foreman-rake -- config -k use_pulp_oauth -v true >/dev/null failed! Check the output for error!
[satellite.com] out: API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
[satellite.com] out: rake aborted!
[satellite.com] out: NoMethodError: undefined method `override=' for #<Filter:0x000000093f9400>
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activemodel-4.2.6/lib/active_model/attribute_methods.rb:433:in `method_missing'
[satellite.com] out: /usr/share/foreman/app/models/filter.rb:221:in `enforce_override_flag'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:432:in `block in make_lambda'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:164:in `block in halting'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:504:in `block in call'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:504:in `each'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:504:in `call'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:92:in `__run_callbacks__'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:778:in `_run_validation_callbacks'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activemodel-4.2.6/lib/active_model/validations/callbacks.rb:113:in `run_validations!'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activemodel-4.2.6/lib/active_model/validations.rb:338:in `valid?'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/validations.rb:58:in `valid?'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/autosave_association.rb:324:in `association_valid?'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/autosave_association.rb:312:in `block in validate_collection_association'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/autosave_association.rb:312:in `each'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/autosave_association.rb:312:in `validate_collection_association'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/autosave_association.rb:217:in `block in define_autosave_validation_callbacks'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/autosave_association.rb:157:in `instance_eval'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/autosave_association.rb:157:in `block in define_non_cyclic_method'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:432:in `block in make_lambda'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:192:in `block in simple'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:504:in `block in call'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:504:in `each'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:504:in `call'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:92:in `__run_callbacks__'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:778:in `_run_validate_callbacks'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activemodel-4.2.6/lib/active_model/validations.rb:399:in `run_validations!'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activemodel-4.2.6/lib/active_model/validations/callbacks.rb:113:in `block in run_validations!'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:117:in `call'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:555:in `block (2 levels) in compile'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:505:in `call'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:92:in `__run_callbacks__'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/callbacks.rb:778:in `_run_validation_callbacks'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activemodel-4.2.6/lib/active_model/validations/callbacks.rb:113:in `run_validations!'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activemodel-4.2.6/lib/active_model/validations.rb:338:in `valid?'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/validations.rb:58:in `valid?'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/validations.rb:83:in `perform_validations'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/validations.rb:43:in `save!'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/attribute_methods/dirty.rb:29:in `save!'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/transactions.rb:291:in `block in save!'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/transactions.rb:351:in `block in with_transaction_returning_status'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/database_statements.rb:211:in `transaction'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/transactions.rb:220:in `transaction'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/transactions.rb:348:in `with_transaction_returning_status'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/transactions.rb:291:in `save!'
[satellite.com] out: /usr/share/foreman/app/models/role.rb:154:in `add_permissions!'
[satellite.com] out: /usr/share/foreman/app/services/foreman/plugin/rbac_support.rb:85:in `block (2 levels) in include_permissions_for_role'
[satellite.com] out: /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.4.1/lib/audited/auditor.rb:283:in `without_auditing'
[satellite.com] out: /usr/share/foreman/app/services/foreman/plugin/rbac_support.rb:82:in `block in include_permissions_for_role'
[satellite.com] out: /usr/share/foreman/app/models/role.rb:176:in `ignore_locking'
[satellite.com] out: /usr/share/foreman/app/services/foreman/plugin/rbac_support.rb:81:in `include_permissions_for_role'
[satellite.com] out: /usr/share/foreman/app/services/foreman/plugin/rbac_support.rb:93:in `add_all_permissions_to_role'
[satellite.com] out: /usr/share/foreman/app/services/foreman/plugin/rbac_support.rb:7:in `block in add_all_permissions_to_default_roles'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `block in transaction'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/transaction.rb:184:in `within_new_transaction'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `transaction'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/transactions.rb:220:in `transaction'
[satellite.com] out: /usr/share/foreman/app/services/foreman/plugin/rbac_support.rb:6:in `add_all_permissions_to_default_roles'
[satellite.com] out: /usr/share/foreman/app/services/foreman/plugin.rb:272:in `add_all_permissions_to_default_roles'
[satellite.com] out: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.3/lib/foreman_tasks/engine.rb:73:in `block (2 levels) in <class:Engine>'
[satellite.com] out: /usr/share/foreman/app/services/foreman/plugin.rb:66:in `instance_eval'
[satellite.com] out: /usr/share/foreman/app/services/foreman/plugin.rb:66:in `register'
[satellite.com] out: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.3/lib/foreman_tasks/engine.rb:43:in `block in <class:Engine>'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/railties-4.2.6/lib/rails/initializable.rb:30:in `instance_exec'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/railties-4.2.6/lib/rails/initializable.rb:30:in `run'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/railties-4.2.6/lib/rails/initializable.rb:55:in `block in run_initializers'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/railties-4.2.6/lib/rails/initializable.rb:54:in `run_initializers'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/railties-4.2.6/lib/rails/application.rb:352:in `initialize!'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/railties-4.2.6/lib/rails/railtie.rb:194:in `public_send'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/railties-4.2.6/lib/rails/railtie.rb:194:in `method_missing'
[satellite.com] out: /usr/share/foreman/config/environment.rb:5:in `<top (required)>'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/polyglot-0.3.4/lib/polyglot.rb:65:in `require'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in `block in require'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:240:in `load_dependency'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in `require'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/railties-4.2.6/lib/rails/application.rb:328:in `require_environment!'
[satellite.com] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/railties-4.2.6/lib/rails/application.rb:457:in `block in run_tasks_blocks'
[satellite.com] out: Tasks: TOP => db:migrate => environment
[satellite.com] out: (See full trace by running task with --trace)
[satellite.com] out: foreman-rake db:migrate failed! Check the output for error!
[satellite.com] out: foreman-rake -- config -k use_pulp_oauth -v false >/dev/null failed! Check the output for error!
[satellite.com] out: Upgrade step migrate_foreman failed. Check logs for more information.
[satellite.com] out:

Fatal error: run() received nonzero return code 1 while executing!

Requested: satellite-installer --scenario satellite --upgrade
Executed: /bin/bash -l -c "satellite-installer --scenario satellite --upgrade"

Aborting.




Actual results:


Expected results:


Additional info:
Comment 1 Sachin Ghai 2017-07-03 08:53 EDT
Created attachment 1293884 [details]
foreman-debug
Comment 3 Sanket Jagtap 2017-07-17 02:35:15 EDT
Build: Satellite 6.3 snap 7

Upgraded from 6.2.10 -> satellite 6.3 snap7

[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: 
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: == 20170208215148 AddDockerRepoName: migrating ================================
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: -- add_column(:katello_repositories, :container_repository_name, :string)
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out:    -> 0.0007s
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: foreman-rake db:migrate failed! Check the output for error!
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: foreman-rake -- config -k use_pulp_oauth -v false >/dev/null finished successfully!
[qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com] out: Upgrade step migrate_foreman failed. Check logs for more information.

This still fails , with same error.

I can see the related fix in the file, but this is still reproducible
Comment 5 pm-sat@redhat.com 2017-07-17 04:02:32 EDT
Upstream bug assigned to ehelms@redhat.com
Comment 7 Sachin Ghai 2017-08-03 04:23 EDT
Created attachment 1308624 [details]
foreman-debug
Comment 8 Sachin Ghai 2017-08-03 04:40 EDT
Created attachment 1308642 [details]
logs from console
Comment 9 Sachin Ghai 2017-08-03 04:41:43 EDT
If you see the attached log from comment8, here are the errors:

[SAT_SERVER] out: == 20161102194100 CreateContentViewDockerFilterRules: migrating ===============
	[SAT_SERVER] out: -- create_table(:katello_content_view_docker_filter_rules)
	[SAT_SERVER] out:    -> 0.0103s
	[SAT_SERVER] out: -- add_foreign_key(:katello_content_view_docker_filter_rules, :katello_content_view_filters, {:name=>"katello_content_view_docker_filter_rules_filter_fk", :column=>"content_view_filter_id"})rake aborted!
	[SAT_SERVER] out: StandardError: An error has occurred, this and all later migrations canceled:
	[SAT_SERVER] out: 
	[SAT_SERVER] out: Could not find the inverse association for content_view_version (:repositories in Katello::ContentViewVersion)/opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/reflection.rb:184:in `check_validity_of_inverse!'
	[SAT_SERVER] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/reflection.rb:355:in `check_validity!'
	[SAT_SERVER] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/associations/association.rb:25:in `initialize'
	[SAT_SERVER] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/associations.rb:162:in `new'
	[SAT_SERVER] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/associations.rb:162:in `association'
	[SAT_SERVER] out: /opt/rh/rh-ror42/root/usr/share/gems/gems/activerecord-4.2.6/lib/active_record/associations/builder/association.rb:115:in `content_view_version'
	[SAT_SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.2/app/models/katello/repository.rb:190:in `content_view'
	[SAT_SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.2/app/models/katello/repository.rb:178:in `set_container_repository_name'
Comment 10 Sachin Ghai 2017-09-06 06:54:08 EDT
I see similar issue w/ sat6.3 snap14. 

: Upgrade Step: migrate_foreman...
[qe-upgrade] out: foreman-rake -- config -k use_pulp_oauth -v true >/dev/null failed! Check the output for error!
[qe-upgrade] out: API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
[qe-upgrade] out: == 20151210143537 AddTypeToMailNotification: migrating ========================
[qe-upgrade] out: -- add_column(:mail_notifications, :type, :string, {:limit=>255})
[qe-upgrade]out:
Comment 11 Sachin Ghai 2017-09-27 05:08:11 EDT
The reported issue is fixed w/ Sat 6.3 snap16. upgrade was completed successfully.

Note You need to log in before you can comment on or make changes to this bug.