Bug 1655277

Summary: Upgrade step katello:upgrades:3.9:migrate_sync_plans failed while 6.4 to 6.5 upgrade
Product: Red Hat Satellite Reporter: Nikhil Kathole <nkathole>
Component: Sync PlansAssignee: Samir Jha <sajha>
Status: CLOSED ERRATA QA Contact: Nikhil Kathole <nkathole>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.5.0CC: ehelms, sajha
Target Milestone: 6.5.0Keywords: Regression, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: tfm-rubygem-katello-3.10.0 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-05-14 12:39:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1653153    

Description Nikhil Kathole 2018-12-02 10:06:12 UTC
Description of problem:

Upgrading 6.4.0 to 6.5.0 showed upgrade step katello:upgrades:3.9:migrate_sync_plans failed.

=============================================
 Upgrade Step 5/5: katello:upgrades:3.9:migrate_sync_plans. Starting recurring logic for migrated sync plans and deleting Pulp schedules
 Failed upgrade task: katello:upgrades:3.9:migrate_sync_plans, see logs for more information.
 foreman-rake upgrade:run finished successfully!
 Upgrade completed!

foreman-rake trace showed

NoMethodError: undefined method `importer_type' for #<Katello::Repository:0x000000000d438ef8>
Did you mean?  importers


Version-Release number of selected component (if applicable):
Satellite 6.5 snap 6


How reproducible: always


Steps to Reproduce:
1. foreman-maintain upgrade run --target-version 6.5


Actual results: Upgrade step failed 

# foreman-rake katello:upgrades:3.9:migrate_sync_plans --trace
/usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
/usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
** Invoke katello:upgrades:3.9:migrate_sync_plans (first_time)
** Invoke environment (first_time)
** Execute environment
/usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
/usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
** Invoke katello:upgrades:3.9:migrate_sync_plans (first_time)
** Invoke environment (first_time)
** Execute environment
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/models/katello/concerns/content_facet_host_extensions.rb:7: warning: already initialized constant Katello::Concerns::ContentFacetHostExtensions::ERRATA_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/models/katello/concerns/content_facet_host_extensions.rb:7: warning: previous definition of ERRATA_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/models/katello/concerns/content_facet_host_extensions.rb:14: warning: already initialized constant Katello::Concerns::ContentFacetHostExtensions::TRACE_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/models/katello/concerns/content_facet_host_extensions.rb:14: warning: previous definition of TRACE_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/models/katello/concerns/subscription_facet_host_extensions.rb:13: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::SUBSCRIPTION_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/models/katello/concerns/subscription_facet_host_extensions.rb:13: warning: previous definition of SUBSCRIPTION_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/models/katello/concerns/subscription_facet_host_extensions.rb:21: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::SLA_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/models/katello/concerns/subscription_facet_host_extensions.rb:21: warning: previous definition of SLA_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/models/katello/concerns/subscription_facet_host_extensions.rb:26: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::USAGE_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/models/katello/concerns/subscription_facet_host_extensions.rb:26: warning: previous definition of USAGE_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/models/katello/concerns/subscription_facet_host_extensions.rb:31: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::ROLE_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/models/katello/concerns/subscription_facet_host_extensions.rb:31: warning: previous definition of ROLE_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/models/katello/concerns/subscription_facet_host_extensions.rb:36: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::ADDONS_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/models/katello/concerns/subscription_facet_host_extensions.rb:36: warning: previous definition of ADDONS_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/models/katello/concerns/subscription_facet_host_extensions.rb:41: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::PURPOSE_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/models/katello/concerns/subscription_facet_host_extensions.rb:41: warning: previous definition of PURPOSE_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/models/katello/concerns/content_facet_host_extensions.rb:7: warning: already initialized constant Katello::Concerns::ContentFacetHostExtensions::ERRATA_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/models/katello/concerns/content_facet_host_extensions.rb:7: warning: previous definition of ERRATA_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/models/katello/concerns/content_facet_host_extensions.rb:14: warning: already initialized constant Katello::Concerns::ContentFacetHostExtensions::TRACE_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/models/katello/concerns/content_facet_host_extensions.rb:14: warning: previous definition of TRACE_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/models/katello/concerns/subscription_facet_host_extensions.rb:13: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::SUBSCRIPTION_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/models/katello/concerns/subscription_facet_host_extensions.rb:13: warning: previous definition of SUBSCRIPTION_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/models/katello/concerns/subscription_facet_host_extensions.rb:21: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::SLA_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/models/katello/concerns/subscription_facet_host_extensions.rb:21: warning: previous definition of SLA_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/models/katello/concerns/subscription_facet_host_extensions.rb:26: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::USAGE_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/models/katello/concerns/subscription_facet_host_extensions.rb:26: warning: previous definition of USAGE_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/models/katello/concerns/subscription_facet_host_extensions.rb:31: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::ROLE_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/models/katello/concerns/subscription_facet_host_extensions.rb:31: warning: previous definition of ROLE_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/models/katello/concerns/subscription_facet_host_extensions.rb:36: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::ADDONS_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/models/katello/concerns/subscription_facet_host_extensions.rb:36: warning: previous definition of ADDONS_STATUS_MAP was here
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/models/katello/concerns/subscription_facet_host_extensions.rb:41: warning: already initialized constant Katello::Concerns::SubscriptionFacetHostExtensions::PURPOSE_STATUS_MAP
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/models/katello/concerns/subscription_facet_host_extensions.rb:41: warning: previous definition of PURPOSE_STATUS_MAP was here
** Execute katello:upgrades:3.9:migrate_sync_plans
Starting recurring logic for migrated sync plans and deleting Pulp schedules
rake aborted!
NoMethodError: undefined method `importer_type' for #<Katello::Repository:0x000000000d438ef8>
Did you mean?  importers
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activemodel-5.2.1/lib/active_model/attribute_methods.rb:430:in `method_missing'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/lib/katello/tasks/upgrades/3.9/migrate_sync_plans.rake:21:in `block (7 levels) in <top (required)>'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation/delegation.rb:41:in `each'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation/delegation.rb:41:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/lib/katello/tasks/upgrades/3.9/migrate_sync_plans.rake:18:in `block (6 levels) in <top (required)>'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation/delegation.rb:41:in `each'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation/delegation.rb:41:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/lib/katello/tasks/upgrades/3.9/migrate_sync_plans.rake:17:in `block (5 levels) in <top (required)>'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation/batches.rb:70:in `block (2 levels) in find_each'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation/batches.rb:70:in `each'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation/batches.rb:70:in `block in find_each'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation/batches.rb:136:in `block in find_in_batches'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation/batches.rb:238:in `block in in_batches'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation/batches.rb:222:in `loop'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation/batches.rb:222:in `in_batches'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation/batches.rb:135:in `find_in_batches'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation/batches.rb:69:in `find_each'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/querying.rb:11:in `find_each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/lib/katello/tasks/upgrades/3.9/migrate_sync_plans.rake:9:in `block (4 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `block in execute'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `each'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `execute'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:195:in `block in invoke_with_call_chain'
/opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in `mon_synchronize'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:188:in `invoke_with_call_chain'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:181:in `invoke'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:160:in `invoke_task'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `each'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `block in top_level'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:125:in `run_with_threads'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:110:in `top_level'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:83:in `block in run'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:80:in `run'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
/opt/rh/rh-ruby25/root/usr/bin/rake:23:in `load'
/opt/rh/rh-ruby25/root/usr/bin/rake:23:in `<main>'
Tasks: TOP => katello:upgrades:3.9:migrate_sync_plans




Expected results: Successful upgrade.


Additional info: This is blocker for upgrading satellite as foreman-maintain unable to enable sync-plans after upgrade.

 | re-enabling sync plans
                                                                                 
 / re-enabling sync plans                                              [FAIL]
 Could not update the sync plan:
   undefined method `cancel' for nil:NilClass
 --------------------------------------------------------------------------------
 Scenario [Procedures after migrating to Satellite 6.5] failed.
 
 The following steps ended up in failing state:
 
   [sync-plans-enable]
 
 Resolve the failed steps and rerun
 the command. In case the failures are false positives,
 use --whitelist="sync-plans-enable"

and production.log showed

 2018-12-02T05:05:20 [I|app|] Started PUT "/katello/api/sync_plans/2" for IP at 2018-12-02 05:05:20 -0500
2018-12-02T05:05:20 [I|app|7784d] Processing by Katello::Api::V2::SyncPlansController#update as JSON
2018-12-02T05:05:20 [I|app|7784d]   Parameters: {"enabled"=>true, "api_version"=>"v2", "id"=>"2", "sync_plan"=>{"enabled"=>true}}
2018-12-02T05:05:20 [I|app|7784d] Authorized user admin(Admin User)
2018-12-02T05:05:20 [I|app|7784d] Current user set to admin (admin)
2018-12-02T05:05:20 [I|aud|7784d] Katello::SyncPlan (2) update event on enabled false, true
2018-12-02T05:05:20 [I|aud|7784d] Katello::SyncPlan (2) update event on cron_expression , 
2018-12-02T05:05:21 [I|aud|7784d] Katello::SyncPlan (2) update event on foreman_tasks_recurring_logic_id , 8
2018-12-02T05:05:21 [E|app|7784d] NoMethodError: undefined method `cancel' for nil:NilClass
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/models/katello/sync_plan.rb:63:in `block in update_attributes_with_logics!'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:254:in `block in transaction'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/transaction.rb:239:in `block in within_new_transaction'
/opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in `mon_synchronize'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/transaction.rb:236:in `within_new_transaction'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:254:in `transaction'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/transactions.rb:212:in `transaction'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/transactions.rb:301:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/models/katello/sync_plan.rb:56:in `update_attributes_with_logics!'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/controllers/katello/api/v2/sync_plans_controller.rb:69:in `update'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:194:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
/usr/share/foreman/app/controllers/api/v2/base_controller.rb:163:in `disable_json_root'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/application_shared.rb:15:in `set_timezone'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:136:in `run_callbacks'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/app/controllers/katello/concerns/api/api_controller.rb:39:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:134:in `process'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:32:in `process'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:191:in `dispatch'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:252:in `dispatch'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:34:in `serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/static_dispatcher.rb:65:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/extractor/recorder.rb:137:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/usr/share/foreman/lib/middleware/telemetry.rb:9:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
/usr/share/foreman/lib/middleware/catch_json_parse_errors.rb:8:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/tempfile_reaper.rb:15:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/etag.rb:25:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/conditional_get.rb:38:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/head.rb:12:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
/usr/share/foreman/lib/middleware/logging_context.rb:23:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:232:in `context'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:226:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/cookies.rb:670:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:98:in `run_callbacks'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:38:in `call_app'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:28:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.10.0.rc1/lib/katello/prevent_json_parsing.rb:12:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/method_override.rb:22:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/runtime.rb:22:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/sendfile.rb:111:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.0.0/lib/secure_headers/middleware.rb:13:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:68:in `block in call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:53:in `each'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:53:in `call'
/usr/share/gems/gems/passenger-4.0.18/lib/phusion_passenger/rack/thread_handler_extension.rb:77:in `process_request'
/usr/share/gems/gems/passenger-4.0.18/lib/phusion_passenger/request_handler/thread_handler.rb:140:in `accept_and_process_next_request'
/usr/share/gems/gems/passenger-4.0.18/lib/phusion_passenger/request_handler/thread_handler.rb:108:in `main_loop'
/usr/share/gems/gems/passenger-4.0.18/lib/phusion_passenger/request_handler.rb:441:in `block (3 levels) in start_threads'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2018-12-02T05:05:21 [I|app|7784d] Completed 500 Internal Server Error in 738ms (Views: 1.2ms | ActiveRecord: 153.9ms)

Comment 5 Samir Jha 2018-12-05 21:51:52 UTC
Hey Nikhil,

Would you be able to run foreman-rake db:migrate on the system that failed to upgrade and provide the output?

Thanks & Regards,
Samir Jha

Comment 6 Samir Jha 2018-12-05 22:06:14 UTC
Ignore the previous comment..Failed to notice 3.10.0.rc1..

Comment 7 Samir Jha 2018-12-06 15:46:01 UTC
Created redmine issue http://projects.theforeman.org/issues/25639 from this bug

Comment 8 Satellite Program 2018-12-10 23:03:10 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue https://projects.theforeman.org/issues/25639 has been resolved.

Comment 9 Nikhil Kathole 2018-12-26 09:03:14 UTC
VERIFIED

version tested:
upgrade from 6.4 to 6.5 snap 9

=============================================
 Upgrade Step 4/6: katello:upgrades:3.8:clear_checksum_type. =============================================
 Upgrade Step 5/6: katello:upgrades:3.9:migrate_sync_plans. Starting recurring logic for migrated sync plans and deleting Pulp schedules
 =============================================
 Upgrade Step 6/6: katello:upgrades:3.10:clear_invalid_repo_credentials. foreman-rake upgrade:run finished successfully!
 Upgrade completed!

Comment 11 errata-xmlrpc 2019-05-14 12:39:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:1222