Bug 1320794

Summary: Task fails with "undefined method `cp_config' for"
Product: Red Hat Satellite Reporter: Pradeep Kumar Surisetty <psuriset>
Component: Content ManagementAssignee: Ivan Necas <inecas>
Status: CLOSED ERRATA QA Contact: Lukas Zapletal <lzap>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.2.0CC: ahuchcha, bbuckingham, brubisch, egolov, ehelms, erik-fedora, ggatward, inecas, jhutar, jmatthew, jmontleo, jsherril, lzap, mburgerh, mmccune, omaciel, oshtaier, peter.vreman, psuriset, sghai, sreber, tpapaioa, xdmoon
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: rubygem-katello-3.0.0.38-1 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-07-27 11:09:45 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1115190, 1122832, 1212602, 1335807    
Attachments:
Description Flags
dynflow error
none
Dynflow crash none

Description Pradeep Kumar Surisetty 2016-03-24 02:47:17 UTC
Description of problem:

1) Installed 6.2 snap5 
2) Upload manifest, enable 5 repos ( local CDN rsync repo)
3) Sync all 5 repos concurrently.  
4) Create 8 cvs and added all 5 repos to all cvs
5) Concurrently publish all 10 cvs
6) Out of 8, 4 cv's published successfully. Rest 4 failed due to abnormal termination

There is a sudden drop in dynflow threads noticed in middle of 

Publish fails with below error: 

---


Task 360dfc47-f584-4af2-87a3-b8d5eacde3f6 error: 0.13025210084033614/1, 13%, 0.0/s, elapsed: 00:00:38
Task 360dfc47-f584-4af2-87a3-b8d5eacde3f6 error: 0.13025210084033614/1, 13%, 0.0/s, elapsed: 00:00:38
undefined method `cp_config' for #<Class:0x000000076decd0>
undefined method `cp_config' for #<Class:0x000000076decd0>
undefined method `cp_config' for #<Class:0x000000076decd0>
undefined method `cp_config' for #<Class:0x000000076decd0>
undefined method `cp_config' for #<Class:0x000000076decd0>
---

This is due to DynFlow error. 

foreman/dynflow_executor.output:E, [2016-03-23T23:22:49.115925 #29152] ERROR -- /connector-database-core: No executor available (Dynflow::Error)
foreman/dynflow_executor.output:E, [2016-03-23T23:22:49.145976 #29152] ERROR -- /connector-database-core: No executor available (Dynflow::Error)
foreman/dynflow_executor.output:E, [2016-03-23T23:22:49.157515 #29152] ERROR -- /connector-database-core: No executor available (Dynflow::Error)
foreman/dynflow_executor.output:E, [2016-03-23T23:22:49.167901 #29152] ERROR -- /connector-database-core: No executor available (Dynflow::Error)
foreman/dynflow_executor.output:E, [2016-03-23T23:22:49.179427 #29152] ERROR -- /connector-database-core: No executor available (Dynflow::Error)
foreman/dynflow_executor.output:E, [2016-03-23T23:22:49.374086 #29152] ERROR -- /connector-database-core: No executor available (Dynflow::Error)
foreman/dynflow_executor.output:E, [2016-03-23T23:22:49.648997 #29152] ERROR -- /connector-database-core: No executor available (Dynflow::Error)
foreman/dynflow_executor.output:E, [2016-03-23T23:22:49.884759 #29152] ERROR -- /connector-database-core: No executor available (Dynflow::Error)
foreman/dynflow_executor.output:E, [2016-03-23T23:22:50.089132 #29152] ERROR -- /connector-database-core: No executor available (Dynflow::Error)
foreman/dynflow_executor.output:E, [2016-03-23T23:22:50.269215 #29152] ERROR -- /connector-database-core: No executor available (Dynflow::Error)
foreman/dynflow_executor.output:E, [2016-03-23T23:22:50.444887 #29152] ERROR -- /connector-database-core: No executor available (Dynflow::Error)



Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:

1) Installed 6.2 snap5 
2) Upload manifest, enable 5 repos ( local CDN rsync repo)
3) Sync all 5 repos concurrently.  
4) Create 10 cvs and added all 5 repos to all cvs
5) Concurrently publish all 100 cvs

Actual results:

cv publish fails. 

Expected results:

Publish cv successfully. 

Additional info:

Comment 1 Pradeep Kumar Surisetty 2016-03-24 02:48:02 UTC
Created attachment 1139819 [details]
dynflow error

Comment 3 Ivan Necas 2016-04-20 10:36:35 UTC
Would it be possible to upload the whole sos report or foreman-debug from the issue: the screenshot is too little info to investigate

Comment 4 Ivan Necas 2016-04-20 10:41:21 UTC
Also, any chance that service restart happened in the meantime?

Comment 5 Ivan Necas 2016-04-20 10:41:43 UTC
I mean for the foreman-tasks service

Comment 6 Pradeep Kumar Surisetty 2016-04-20 10:45:09 UTC
I havent restarted services.  

foreman debug , i will get it

Comment 7 Ivan Necas 2016-04-20 10:53:03 UTC
I was able to reproduce the issue when the task was triggered before the foreman-tasks service started, which lead to the `cp_config` method not being defined, because the starting happened before the class was fully extended.

Comment 8 Stephen Benjamin 2016-04-20 13:33:32 UTC
I was able to reproduce this once so far, and also did not restart any services.  After the first time I haven't been able to reproduce it though, and I've tried the reproducer steps about a dozen times now.

Comment 10 Brad Buckingham 2016-04-21 17:53:10 UTC
WORKAROUND:  The following are a couple of options:

- When simultaneously publishing multiple content views, perform them in smaller batches.  For example, rather than publish 10 at a time, publish 5.
- If the failure raised by this bug does occur, go in to Monitor->Tasks and perform a 'Resume' on any failed content view publish tasks.

Comment 11 Justin Sherrill 2016-04-26 13:34:50 UTC
I've seen this issue as well, when i've seen it it was after a really really long amount of polling such as when a pulp worker is not responding properly or a gofer client picks up the task and doesn't respond (due to another bz).

However ori saw this as well almost immediately after starting a CV publish though.  So it seems quite random.

Comment 12 Justin Sherrill 2016-05-02 15:07:26 UTC
Created attachment 1152985 [details]
Dynflow crash

Comment 13 Ivan Necas 2016-05-02 15:40:53 UTC
This report give us much more info on what's going on there. The suspicious issue seems to be related to this patch in sequel https://github.com/jeremyevans/sequel/pull/1083/commits/6fc930d8a40e6f8b1630866e833d97a6df16c2fd

I will try to get a solid reproducer and trying out a sequel upgrade after that to see, if it moves us forward with the issue

Comment 14 Justin Sherrill 2016-05-02 15:41:30 UTC
*** Bug 1331732 has been marked as a duplicate of this bug. ***

Comment 15 Og Maciel 2016-05-02 15:50:11 UTC
I've been able to hit the same "undefined method `cp_config'" error but while synchronizing repositories (both custom and RH repositories) on a Satellite 6.2.0 Beta system.

Action:

    Actions::Pulp::Repository::Sync

Exception:

    NoMethodError: undefined method `cp_config' for #<Class:0x00000009631010>

Backtrace:

/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/dynamic_matchers.rb:26:in `method_missing'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.24/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.24/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.24/app/lib/actions/middleware/remote_action.rb:16:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:17:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action/progress.rb:30:in `with_progress_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action/progress.rb:16:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:17:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.24/app/lib/actions/middleware/keep_locale.rb:11:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.24/app/lib/actions/middleware/keep_locale.rb:22:in `with_locale'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.24/app/lib/actions/middleware/keep_locale.rb:11:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:17:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:30:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/world.rb:30:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:505:in `block (2 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:504:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:504:in `block in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:419:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:419:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:419:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:419:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:499:in `execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:260:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/executors/parallel/worker.rb:15:in `block in on_message'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in `block in assigns'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `tap'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `assigns'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in `match_value'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in `block in match?'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `match?'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in `match'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/executors/parallel/worker.rb:12:in `on_message'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/context.rb:46:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/actor.rb:26:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:161:in `process_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:95:in `block in on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:118:in `block (2 levels) in schedule_execution'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `block in synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:115:in `block in schedule_execution'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:96:in `work'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:77:in `block in call_job'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in `run_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:322:in `block (3 levels) in create_worker'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in `loop'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in `block (2 levels) in create_worker'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in `block in create_worker'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'

Comment 16 Geoff Gatward 2016-05-09 23:06:37 UTC
Also seeing this error on 6.2 Beta (customer installation) during sync tasks initiated from a daily sync plan.  Seems to be various repos fail to sync with error:

undefined method `cp_config' for #<Class:0x00000002ebcc18>
Trying to use logger katello/pulp_rest which has not been configured.


Output from Errors tab on tasks viewer:


Action:

Actions::Pulp::Repository::Sync

Input:

{"pulp_id"=>
  "Default_Organization-Red_Hat_Satellite_Capsule-Red_Hat_Satellite_Capsule_6_1_for_RHEL_7_Server_RPMs_x86_64",
 "task_id"=>"7d66cdf0-c60d-453c-90fb-de254868b44a",
 "source_url"=>nil,
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin",
 "locale"=>"en"}

Output:

{"contents_changed"=>false,
 "pulp_tasks"=>
  [{"exception"=>nil,
    "task_type"=>"pulp.server.managers.repo.sync.sync",
    "_href"=>"/pulp/api/v2/tasks/7d66cdf0-c60d-453c-90fb-de254868b44a/",
    "task_id"=>"7d66cdf0-c60d-453c-90fb-de254868b44a",
    "tags"=>
     ["pulp:repository:Default_Organization-Red_Hat_Satellite_Capsule-Red_Hat_Satellite_Capsule_6_1_for_RHEL_7_Server_RPMs_x86_64",
      "pulp:action:sync"],
    "finish_time"=>"2016-05-09T14:10:47Z",
    "_ns"=>"task_status",
    "start_time"=>"2016-05-09T14:10:39Z",
    "traceback"=>nil,
    "spawned_tasks"=>
     [{"_href"=>"/pulp/api/v2/tasks/7a8c15fa-3e18-41b2-b3ef-31cdf117e6d3/",
       "task_id"=>"7a8c15fa-3e18-41b2-b3ef-31cdf117e6d3"}],
    "progress_report"=>
     {"yum_importer"=>
       {"content"=>
         {"items_total"=>0,
          "state"=>"FINISHED",
          "error_details"=>[],
          "details"=>
           {"rpm_total"=>0, "rpm_done"=>0, "drpm_total"=>0, "drpm_done"=>0},
          "size_total"=>0,
          "size_left"=>0,
          "items_left"=>0},
        "comps"=>{"state"=>"FINISHED"},
        "purge_duplicates"=>{"state"=>"FINISHED"},
        "distribution"=>
         {"items_total"=>0,
          "state"=>"FINISHED",
          "error_details"=>[],
          "items_left"=>0},
        "errata"=>{"state"=>"FINISHED"},
        "metadata"=>{"state"=>"FINISHED"}}},
    "queue"=>"reserved_resource_worker-0.org.dq",
    "state"=>"finished",
    "worker_name"=>"reserved_resource_worker-0.org",
    "result"=>
     {"result"=>"success",
      "importer_id"=>"yum_importer",
      "exception"=>nil,
      "repo_id"=>
       "Default_Organization-Red_Hat_Satellite_Capsule-Red_Hat_Satellite_Capsule_6_1_for_RHEL_7_Server_RPMs_x86_64",
      "traceback"=>nil,
      "started"=>"2016-05-09T14:10:39Z",
      "_ns"=>"repo_sync_results",
      "completed"=>"2016-05-09T14:10:47Z",
      "importer_type_id"=>"yum_importer",
      "error_message"=>nil,
      "summary"=>
       {"content"=>{"state"=>"FINISHED"},
        "comps"=>{"state"=>"FINISHED"},
        "purge_duplicates"=>{"state"=>"FINISHED"},
        "distribution"=>{"state"=>"FINISHED"},
        "errata"=>{"state"=>"FINISHED"},
        "metadata"=>{"state"=>"FINISHED"}},
      "added_count"=>0,
      "removed_count"=>0,
      "updated_count"=>0,
      "id"=>"57309a679d9422570bb514b6",
      "details"=>
       {"content"=>
         {"size_total"=>0,
          "items_left"=>0,
          "items_total"=>0,
          "state"=>"FINISHED",
          "size_left"=>0,
          "details"=>
           {"rpm_total"=>0, "rpm_done"=>0, "drpm_total"=>0, "drpm_done"=>0},
          "error_details"=>[]},
        "comps"=>{"state"=>"FINISHED"},
        "purge_duplicates"=>{"state"=>"FINISHED"},
        "distribution"=>
         {"items_total"=>0,
          "state"=>"FINISHED",
          "error_details"=>[],
          "items_left"=>0},
        "errata"=>{"state"=>"FINISHED"},
        "metadata"=>{"state"=>"FINISHED"}}},
    "error"=>nil,
    "_id"=>{"$oid"=>"573097e92d723599562be2ef"},
    "id"=>"573097e92d723599562be2ef"},
   {"exception"=>nil,
    "task_type"=>"pulp.server.managers.repo.publish.publish",
    "_href"=>"/pulp/api/v2/tasks/7a8c15fa-3e18-41b2-b3ef-31cdf117e6d3/",
    "task_id"=>"7a8c15fa-3e18-41b2-b3ef-31cdf117e6d3",
    "tags"=>
     ["pulp:repository:Default_Organization-Red_Hat_Satellite_Capsule-Red_Hat_Satellite_Capsule_6_1_for_RHEL_7_Server_RPMs_x86_64",
      "pulp:action:publish"],
    "finish_time"=>nil,
    "_ns"=>"task_status",
    "start_time"=>nil,
    "traceback"=>nil,
    "spawned_tasks"=>[],
    "progress_report"=>{},
    "queue"=>"None.dq",
    "state"=>"waiting",
    "worker_name"=>nil,
    "result"=>nil,
    "error"=>nil,
    "_id"=>{"$oid"=>"57309a672d723599562be365"},
    "id"=>"57309a672d723599562be365"}],
 "poll_attempts"=>{"total"=>90, "failed"=>0}}

Exception:

NoMethodError: undefined method `cp_config' for #<Class:0x00000002ebcc18>

Backtrace:

/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/dynamic_matchers.rb:26:in `method_missing'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.24/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.24/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.24/app/lib/actions/middleware/remote_action.rb:16:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:17:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action/progress.rb:30:in `with_progress_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action/progress.rb:16:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:17:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.24/app/lib/actions/middleware/keep_locale.rb:11:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.24/app/lib/actions/middleware/keep_locale.rb:22:in `with_locale'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.24/app/lib/actions/middleware/keep_locale.rb:11:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:17:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:30:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/world.rb:30:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:505:in `block (2 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:504:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:504:in `block in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:419:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:419:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:419:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:419:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:499:in `execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:260:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/executors/parallel/worker.rb:15:in `block in on_message'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in `block in assigns'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `tap'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `assigns'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in `match_value'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in `block in match?'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `match?'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in `match'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/executors/parallel/worker.rb:12:in `on_message'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/context.rb:46:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/actor.rb:26:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:161:in `process_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:95:in `block in on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:118:in `block (2 levels) in schedule_execution'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `block in synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:115:in `block in schedule_execution'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:96:in `work'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:77:in `block in call_job'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in `run_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:322:in `block (3 levels) in create_worker'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in `loop'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in `block (2 levels) in create_worker'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in `block in create_worker'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'

Action:

Actions::Pulp::Repository::Sync

Input:

{"pulp_id"=>
  "Default_Organization-Red_Hat_Satellite_Capsule-Red_Hat_Satellite_Capsule_6_1_for_RHEL_7_Server_RPMs_x86_64",
 "task_id"=>"7d66cdf0-c60d-453c-90fb-de254868b44a",
 "source_url"=>nil,
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin",
 "locale"=>"en"}

Output:

{"contents_changed"=>false,
 "pulp_tasks"=>
  [{"exception"=>nil,
    "task_type"=>"pulp.server.managers.repo.sync.sync",
    "_href"=>"/pulp/api/v2/tasks/7d66cdf0-c60d-453c-90fb-de254868b44a/",
    "task_id"=>"7d66cdf0-c60d-453c-90fb-de254868b44a",
    "tags"=>
     ["pulp:repository:Default_Organization-Red_Hat_Satellite_Capsule-Red_Hat_Satellite_Capsule_6_1_for_RHEL_7_Server_RPMs_x86_64",
      "pulp:action:sync"],
    "finish_time"=>"2016-05-09T14:10:47Z",
    "_ns"=>"task_status",
    "start_time"=>"2016-05-09T14:10:39Z",
    "traceback"=>nil,
    "spawned_tasks"=>
     [{"_href"=>"/pulp/api/v2/tasks/7a8c15fa-3e18-41b2-b3ef-31cdf117e6d3/",
       "task_id"=>"7a8c15fa-3e18-41b2-b3ef-31cdf117e6d3"}],
    "progress_report"=>
     {"yum_importer"=>
       {"content"=>
         {"items_total"=>0,
          "state"=>"FINISHED",
          "error_details"=>[],
          "details"=>
           {"rpm_total"=>0, "rpm_done"=>0, "drpm_total"=>0, "drpm_done"=>0},
          "size_total"=>0,
          "size_left"=>0,
          "items_left"=>0},
        "comps"=>{"state"=>"FINISHED"},
        "purge_duplicates"=>{"state"=>"FINISHED"},
        "distribution"=>
         {"items_total"=>0,
          "state"=>"FINISHED",
          "error_details"=>[],
          "items_left"=>0},
        "errata"=>{"state"=>"FINISHED"},
        "metadata"=>{"state"=>"FINISHED"}}},
    "queue"=>"reserved_resource_worker-0.org.dq",
    "state"=>"finished",
    "worker_name"=>"reserved_resource_worker-0.org",
    "result"=>
     {"result"=>"success",
      "importer_id"=>"yum_importer",
      "exception"=>nil,
      "repo_id"=>
       "Default_Organization-Red_Hat_Satellite_Capsule-Red_Hat_Satellite_Capsule_6_1_for_RHEL_7_Server_RPMs_x86_64",
      "traceback"=>nil,
      "started"=>"2016-05-09T14:10:39Z",
      "_ns"=>"repo_sync_results",
      "completed"=>"2016-05-09T14:10:47Z",
      "importer_type_id"=>"yum_importer",
      "error_message"=>nil,
      "summary"=>
       {"content"=>{"state"=>"FINISHED"},
        "comps"=>{"state"=>"FINISHED"},
        "purge_duplicates"=>{"state"=>"FINISHED"},
        "distribution"=>{"state"=>"FINISHED"},
        "errata"=>{"state"=>"FINISHED"},
        "metadata"=>{"state"=>"FINISHED"}},
      "added_count"=>0,
      "removed_count"=>0,
      "updated_count"=>0,
      "id"=>"57309a679d9422570bb514b6",
      "details"=>
       {"content"=>
         {"size_total"=>0,
          "items_left"=>0,
          "items_total"=>0,
          "state"=>"FINISHED",
          "size_left"=>0,
          "details"=>
           {"rpm_total"=>0, "rpm_done"=>0, "drpm_total"=>0, "drpm_done"=>0},
          "error_details"=>[]},
        "comps"=>{"state"=>"FINISHED"},
        "purge_duplicates"=>{"state"=>"FINISHED"},
        "distribution"=>
         {"items_total"=>0,
          "state"=>"FINISHED",
          "error_details"=>[],
          "items_left"=>0},
        "errata"=>{"state"=>"FINISHED"},
        "metadata"=>{"state"=>"FINISHED"}}},
    "error"=>nil,
    "_id"=>{"$oid"=>"573097e92d723599562be2ef"},
    "id"=>"573097e92d723599562be2ef"},
   {"exception"=>nil,
    "task_type"=>"pulp.server.managers.repo.publish.publish",
    "_href"=>"/pulp/api/v2/tasks/7a8c15fa-3e18-41b2-b3ef-31cdf117e6d3/",
    "task_id"=>"7a8c15fa-3e18-41b2-b3ef-31cdf117e6d3",
    "tags"=>
     ["pulp:repository:Default_Organization-Red_Hat_Satellite_Capsule-Red_Hat_Satellite_Capsule_6_1_for_RHEL_7_Server_RPMs_x86_64",
      "pulp:action:publish"],
    "finish_time"=>nil,
    "_ns"=>"task_status",
    "start_time"=>nil,
    "traceback"=>nil,
    "spawned_tasks"=>[],
    "progress_report"=>{},
    "queue"=>"None.dq",
    "state"=>"waiting",
    "worker_name"=>nil,
    "result"=>nil,
    "error"=>nil,
    "_id"=>{"$oid"=>"57309a672d723599562be365"},
    "id"=>"57309a672d723599562be365"}],
 "poll_attempts"=>{"total"=>90, "failed"=>0}}

Action:

Actions::Katello::Repository::IndexContent

Input:

{"id"=>22,
 "contents_changed"=>Step(3).output[:contents_changed],
 "current_user_id"=>1,
 "locale"=>"en"}

Output:

{}

Action:

Actions::Katello::Repository::CorrectChecksum

Input:

{"repo_id"=>22, "locale"=>"en"}

Output:

{}

Exception:

RuntimeError: Trying to use logger katello/pulp_rest which has not been configured.

Backtrace:

/usr/share/foreman/lib/foreman/logging.rb:54:in `logger'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.24/app/services/katello/pulp/server.rb:14:in `config'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.24/config/initializers/runcible.rb:3:in `pulp_server'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.24/app/models/katello/glue/pulp/repo.rb:15:in `block (2 levels) in included'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.24/app/lib/katello/lazy_accessor.rb:160:in `instance_eval'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.24/app/lib/katello/lazy_accessor.rb:160:in `run_initializer'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.24/app/lib/katello/lazy_accessor.rb:139:in `lazy_attribute_get'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.24/app/lib/katello/lazy_accessor.rb:60:in `block (2 levels) in lazy_accessor'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.24/app/models/katello/glue/pulp/repo.rb:26:in `block (2 levels) in included'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.24/app/lib/katello/lazy_accessor.rb:160:in `instance_eval'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.24/app/lib/katello/lazy_accessor.rb:160:in `run_initializer'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.24/app/lib/katello/lazy_accessor.rb:139:in `lazy_attribute_get'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.24/app/lib/katello/lazy_accessor.rb:60:in `block (2 levels) in lazy_accessor'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.24/app/models/katello/glue/pulp/repo.rb:678:in `find_distributor'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.24/app/models/katello/glue/pulp/repo.rb:97:in `pulp_checksum_type'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.24/app/lib/actions/katello/repository/correct_checksum.rb:12:in `finalize'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:528:in `block (2 levels) in execute_finalize'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:17:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action/progress.rb:30:in `with_progress_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action/progress.rb:22:in `finalize'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:17:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.24/app/lib/actions/middleware/keep_locale.rb:15:in `block in finalize'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.24/app/lib/actions/middleware/keep_locale.rb:22:in `with_locale'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.24/app/lib/actions/middleware/keep_locale.rb:15:in `finalize'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:17:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:38:in `finalize'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/world.rb:30:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:527:in `block in execute_finalize'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:419:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:419:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:419:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:419:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:526:in `execute_finalize'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:260:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/executors/parallel/sequential_manager.rb:68:in `run_step'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/executors/parallel/sequential_manager.rb:53:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/executors/parallel/sequential_manager.rb:60:in `block in run_in_sequence'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/executors/parallel/sequential_manager.rb:60:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/executors/parallel/sequential_manager.rb:60:in `all?'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/executors/parallel/sequential_manager.rb:60:in `run_in_sequence'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/executors/parallel/sequential_manager.rb:49:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/executors/parallel/sequential_manager.rb:27:in `block in finalize'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:17:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:46:in `finalize_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:17:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:46:in `finalize_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:17:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:46:in `finalize_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:17:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:46:in `finalize_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:17:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/common/transaction.rb:16:in `block in rollback_on_error'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in `block in transaction'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:209:in `within_new_transaction'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in `transaction'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/transactions.rb:208:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/transaction_adapters/active_record.rb:5:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/common/transaction.rb:15:in `rollback_on_error'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/common/transaction.rb:9:in `finalize_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/world.rb:30:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/executors/parallel/sequential_manager.rb:26:in `finalize'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/executors/parallel/worker.rb:18:in `block in on_message'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in `block in assigns'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `tap'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `assigns'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in `match_value'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in `block in match?'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `match?'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in `match'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/executors/parallel/worker.rb:12:in `on_message'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/context.rb:46:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/actor.rb:26:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:161:in `process_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:95:in `block in on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:118:in `block (2 levels) in schedule_execution'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `block in synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:115:in `block in schedule_execution'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:96:in `work'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:77:in `block in call_job'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in `run_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:322:in `block (3 levels) in create_worker'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in `loop'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in `block (2 levels) in create_worker'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in `block in create_worker'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'

Action:

Actions::Katello::Repository::UpdateMedia

Input:

{"repo_id"=>22,
 "contents_changed"=>Step(3).output[:contents_changed],
 "locale"=>"en"}

Output:

{}

Action:

Actions::Katello::Repository::ErrataMail

Input:

{"repo"=>22,
 "last_updated"=>"2016-05-02 06:47:59 UTC",
 "contents_changed"=>Step(3).output[:contents_changed],
 "locale"=>"en"}

Output:

{}

Action:

Actions::Katello::Repository::ErrataMail

Input:

{"repo"=>22,
 "last_updated"=>"2016-05-02 06:47:59 UTC",
 "contents_changed"=>Step(3).output[:contents_changed],
 "locale"=>"en"}

Output:

{}

Action:

Actions::Katello::Repository::Sync

Input:

{"repository"=>
  {"id"=>22,
   "name"=>"Red Hat Satellite Capsule 6.1 for RHEL 7 Server RPMs x86_64",
   "label"=>"Red_Hat_Satellite_Capsule_6_1_for_RHEL_7_Server_RPMs_x86_64"},
 "product"=>
  {"id"=>5,
   "name"=>"Red Hat Satellite Capsule",
   "label"=>"Red_Hat_Satellite_Capsule",
   "cp_id"=>"269"},
 "provider"=>{"id"=>2, "name"=>"Red Hat"},
 "organization"=>{"id"=>1, "name"=>"Default_Organization", "label"=>"Default_Organization"},
 "services_checked"=>["pulp", "pulp_auth"],
 "id"=>22,
 "sync_result"=>Step(3).output,
 "user_id"=>1,
 "contents_changed"=>Step(3).output[:contents_changed],
 "current_user_id"=>1,
 "locale"=>"en"}

Output:

{}

Action:

Actions::Pulp::Repository::RegenerateApplicability

Input:

{"pulp_id"=>
  "Default_Organization-Red_Hat_Satellite_Capsule-Red_Hat_Satellite_Capsule_6_1_for_RHEL_7_Server_RPMs_x86_64",
 "contents_changed"=>Step(3).output[:contents_changed],
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin",
 "locale"=>"en"}

Output:

{}

Action:

Actions::Katello::Repository::ImportApplicability

Input:

{"repo_id"=>22,
 "contents_changed"=>Step(3).output[:contents_changed],
 "locale"=>"en"}

Output:

{}

Comment 17 Peter Vreman 2016-05-10 13:43:30 UTC
I also having this issue by RedHat Repository sync of 34 Repositories

Additional to the error above there is also an error on the dynflow world:

2016-05-10 13:36:30 [foreman-tasks/dynflow] [W] Error on on_execution_plan_save event
 | RuntimeError: The Dynflow world was not initialized yet. If your plugin uses it, make sure to call ForemanTasks.dynflow.require! in some initializer
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.14.3/lib/foreman_tasks/dynflow.rb:73:in `world'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.14.3/app/models/foreman_tasks/task/dynflow_task.rb:48:in `execution_plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.14.3/app/models/foreman_tasks/task/dynflow_task.rb:82:in `main_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.14.3/app/models/foreman_tasks/task/dynflow_task.rb:17:in `update_from_dynflow'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.14.3/lib/foreman_tasks/dynflow/persistence.rb:46:in `on_execution_plan_save'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.14.3/lib/foreman_tasks/dynflow/persistence.rb:14:in `block in save_execution_plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.14.3/lib/foreman_tasks/dynflow/persistence.rb:12:in `tap'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.14.3/lib/foreman_tasks/dynflow/persistence.rb:12:in `save_execution_plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/persistence.rb:49:in `save_execution_plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan.rb:331:in `save'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan.rb:83:in `update_state'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/world.rb:294:in `invalidate_execution_lock'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/world.rb:274:in `block (2 levels) in invalidate'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/world.rb:273:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/world.rb:273:in `block in invalidate'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/coordinator.rb:229:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/coordinator.rb:229:in `acquire'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/world.rb:266:in `invalidate'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/world.rb:322:in `block in worlds_validity_check'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/world.rb:316:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/world.rb:316:in `worlds_validity_check'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/world.rb:40:in `initialize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.14.3/lib/foreman_tasks/dynflow/configuration.rb:58:in `new'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.14.3/lib/foreman_tasks/dynflow/configuration.rb:58:in `initialize_world'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.14.3/lib/foreman_tasks/dynflow.rb:40:in `initialize!'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.14.3/lib/foreman_tasks/engine.rb:138:in `block in <class:Engine>'

Comment 18 Ivan Necas 2016-05-10 18:55:59 UTC
I'm trying to reproduce this issue locally. Does this issue occur both on rhel6 and rhel7?

Comment 19 Pradeep Kumar Surisetty 2016-05-10 18:58:55 UTC
(In reply to Ivan Necas from comment #18)
> I'm trying to reproduce this issue locally. Does this issue occur both on
> rhel6 and rhel7?

tried on rhel7.2 only

I thought sat6.2 supports only on rhel7

Comment 20 Peter Vreman 2016-05-11 07:37:26 UTC
I see the issue on RHEL6.7

Comment 21 Ivan Necas 2016-05-11 08:44:57 UTC
Thanks. I have a local reproducer now (it's an intermitted issue, so the reproducer is not 100% but it gives us statistical information if the fix worked).

I tried bisecting sequel and pg versions as I had some suspicions around thread safety there, but unfortunately I was able to reproduce this even on the newest sequel and pg versions. I have one more candidate fix to give a try before I search for other candidates for the segmentation fault.

Comment 22 Ivan Necas 2016-05-11 11:33:48 UTC
I've opened a BZ for the segfault

https://bugzilla.redhat.com/show_bug.cgi?id=1335105

The proposed fix is submitted upstream. I've been able to reproduce this against pure dynflow and after the change, I haven't hit the issue with the same reproducer so far, so it looks promising.

I will open different fix in foreman-tasks for the cp_config issue, that this BZ is mentioning.

Comment 23 Peter Vreman 2016-05-11 12:16:39 UTC
In my case the proposed fix https://github.com/Dynflow/dynflow/pull/185/commits/1510982b3d2b641ee4248cf6aa0ad9bd5392f828 solves the problem.

Comment 25 Ivan Necas 2016-05-11 13:36:41 UTC
`hammer task resume` should resume paused tasks and make continue when it got stuck

Comment 26 Ivan Necas 2016-05-16 10:25:41 UTC
Created redmine issue http://projects.theforeman.org/issues/15055 from this bug

Comment 27 Ivan Necas 2016-05-16 10:39:52 UTC
The fix for the undefined method cp_config propsed in foreman-tasks https://github.com/theforeman/foreman-tasks/pull/185

How to reproduce without running into the segfault:

1. `service foreman-tasks stop`
2. trigger repository synchronization
3. wait until it gets to planned/pending stated
4. `service foreman-tasks start`

The error might not appear every time with the reproducer, but after serveral times it should show up, when the fix is not applied.

The fix makes sure dynflow starts being active after other code is initialized 
- before the fix, there might be some code not initialize yet, as the cp_config method definition.

Comment 28 Erik van Pienbroek 2016-05-17 16:04:58 UTC
I just tried to apply your patch against the Sat 6.2 beta, but after restarting everything (with katello-service restart) the webui only shows a fatal exception:

2016-05-17 17:51:08 [app] [W] Failed running foreman-tasks daemon
 | NameError: uninitialized constant Concerns::PulpDatabaseUnit
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_hooks-0.3.9/lib/foreman_hooks.rb:83:in `load_missing_constant_with_hooks'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.24/app/models/katello/erratum.rb:3:in `<class:Erratum>'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.24/app/models/katello/erratum.rb:2:in `<module:Katello>'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.24/app/models/katello/erratum.rb:1:in `<top (required)>'
 | /opt/rh/rh-ruby22/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:54:in `require'
<<snip>>

The complete trace can be found at https://paste.fedoraproject.org/367629/63500844/raw/

Once I reverted the patch (and restarted everything) the webui became operational again without exceptions

Comment 29 Ivan Necas 2016-05-18 11:48:23 UTC
Thanks @Erik trying that and reporting feedback, I've tested it with slightly older version of satellite, where this issue wasn't.

Here is an additional patch that should address the issue you've described:

   https://github.com/theforeman/foreman-tasks/pull/185/commits/96fd4af976ff026ec8aad1c8cf0095da66012bb9

Comment 30 Ivan Necas 2016-05-18 12:04:26 UTC
More testing revealed other issue that might happen occasionally

http://pastie.org/10842225

It's being caused by Katello initializing the plugin later than expected

It was introduced in this commit https://github.com/Katello/katello/commit/d637aa2b48ea02b3b73efbd8168eab9629be8afe#diff-cf74e596d7887011413ee9d06b16b753R198

@eric any background on why this change was needed in the first place and if there is another way to handle this. The problem is we need to resume tasks, that are pending execution once the foreman-tasks executor starts and after_initialize is the last instance we can hook in in the load process to start the execution. Since the katello's plugin initializes itself in after_initialize as well, we occasionally hit issues such as pulp logger being not initialized yet by that time.

Comment 31 Erik van Pienbroek 2016-05-18 16:07:53 UTC
@Ivan: thanks for the updated patch. I've just applied it to my Sat 6.2 beta testing environment and it seems to work fine so far (though more testing is needed). 

One thing I did notice is that after I applied the patch the Actions::Candlepin::ListenOnCandlepinEvents task was created and started automatically (according to the task history this task wasn't executed before on my local testing environment). It's probably unrelated to this patch, but it's a good thing that this task was created and executed. (recently I was having issues where this specific task was not being created on a Katello 3.0RC environment).

Comment 32 Ivan Necas 2016-05-18 16:11:21 UTC
The `Actions::Candlepin::ListenOnCandlepinEvents` task is a long-running task listening on QPID for subscription events: it's expected to be running there continuously and doesn't block anything else.

Comment 33 Ivan Necas 2016-05-20 14:46:29 UTC
I've opened additional PR in Katello to address the loggers not being initialized issue. https://github.com/Katello/katello/pull/6059

Comment 34 Bryan Kearney 2016-05-20 16:13:25 UTC
Upstream bug component is Content Management

Comment 35 Ivan Necas 2016-05-23 20:25:26 UTC
Foreman-tasks part of the fix merged https://github.com/theforeman/foreman-tasks/pull/185

Comment 36 Ivan Necas 2016-05-24 15:11:45 UTC
Katello's part of the fix got in https://github.com/Katello/katello/pull/6059

@eric: how to make sure both PRs get cherry-picked as part of the fix?

Comment 37 Ivan Necas 2016-05-26 12:59:43 UTC
*** Bug 1339911 has been marked as a duplicate of this bug. ***

Comment 38 Lukas Zapletal 2016-05-31 08:13:12 UTC
*** Bug 1341062 has been marked as a duplicate of this bug. ***

Comment 43 Lukas Zapletal 2016-06-01 11:39:34 UTC
I can't see any exceptions when syncing with the upgraded packages on snap 13.1. Just for the record.

Comment 44 Sachin Ghai 2016-06-02 09:33:41 UTC
I can't see the reported issue mentioned in duplicate of this bz 1339911 while performing upgrade from snap 6.1.9 -> sat 6.2 snap14 on rhel7

Comment 45 Lukas Zapletal 2016-06-06 12:24:06 UTC
VERIFIED

Synced 42 repositories (all RHEL platforms and Tools) without this issue:

[root@ibm-ls22-03 ~]# rpm -q satellite katello tfm-rubygem-foreman-tasks tfm-rubygem-katello
satellite-6.2.0-14.2.el7sat.noarch
katello-3.0.0-6.el7sat.noarch
tfm-rubygem-foreman-tasks-0.7.14.6-1.el7sat.noarch
tfm-rubygem-katello-3.0.0.42-2.el7sat.noarch

[root@ibm-ls22-03 ~]# grep cp_config /var/log/foreman/production.log

Comment 46 Bryan Kearney 2016-07-27 11:09:45 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1501