Bug 1662181 - Content view removal fails with an Exception "Katello::Errors::PulpError: Task canceled"
Summary: Content view removal fails with an Exception "Katello::Errors::PulpError: Tas...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Content Views
Version: 6.4.0
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Lai
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-12-26 19:32 UTC by Sam Yangsao
Modified: 2020-02-03 16:30 UTC (History)
0 users

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-02-03 16:30:14 UTC
Target Upstream Version:

Attachments (Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

Description Sam Yangsao 2018-12-26 19:32:30 UTC 
### Description of problem:

Content view removal fails with an Exception "Katello::Errors::PulpError: Task canceled"

Seems to occur frequently when a task is running to remove a Content View (CV) version and Satellite capsule sync kicks off, the capsule sync job causes the task to be cancelled.

### Version-Release number of selected component (if applicable):

Satellite 6.4.1
RHEL 7.6

### How reproducible:

Always?

### Steps to Reproduce:

1.  Upgrade to Satellite 6.4.1 from 6.3.5 on RHEL 7.6

2.  Add the 6.4 tools repository to the a regular RHEL CV (creating version 5)

3.  Publish, Promote RHEL CV (version 5)

4.  Remove the older RHEL CV versions (version 1 through version 4)

5.  While the older CV versions are being removed, a capsule synchronization kicks off (because of step 3), this capsule synchronization seems to terminate the deletion of the older CV version removals that are in flight.

### Actual results:

See error output in 'Additional info' section below.

Do capsule synchronizations take precedence and so it terminates other running tasks?

I've seen similar behaviour where the tasks are cancelled (not while running capsule synchronizations) [1].

Is this workaround [2] still valid?  

Since I don't see PULP_* related tuneables via the constants.py file.

What are the defaults with 6.4.x?

# Other observations 

In 6.3.z I've run these jobs before and even with high IOWAIT, the tasks still completed (CV version removal + capsule synchronization).

Resuming the job that failed with the 'Task error' completes (after the capsule synchronization is completed).

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1551636
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1551636#c5

### Expected results:

Running Tasks (capsule synchronizations) should not be terminating other running Tasks (Content View version removals).

### Additional info:

Output from the task itself.

Action:

Actions::Pulp::Repository::Destroy

Input:

{"pulp_id"=>"1-OCP32-v3_0-9d089d53-3b57-41d6-8ad0-add83c589cbf",
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin"}

Output:

{"pulp_tasks"=>
  [{"exception"=>nil,
    "task_type"=>"pulp.server.tasks.repository.delete",
    "_href"=>"/pulp/api/v2/tasks/04909029-3473-41e8-b65e-097c4be90a46/",
    "task_id"=>"04909029-3473-41e8-b65e-097c4be90a46",
    "tags"=>
     ["pulp:repository:1-OCP32-v3_0-9d089d53-3b57-41d6-8ad0-add83c589cbf",
      "pulp:action:delete"],
    "finish_time"=>nil,
    "_ns"=>"task_status",
    "start_time"=>"2018-12-26T18:37:53Z",
    "traceback"=>nil,
    "spawned_tasks"=>[],
    "progress_report"=>{},
    "queue"=>"reserved_resource_worker-5@sat6.lab.msp.redhat.com.dq2",
    "state"=>"canceled",
    "worker_name"=>"reserved_resource_worker-5@sat6.lab.msp.redhat.com",
    "result"=>nil,
    "error"=>nil,
    "_id"=>{"$oid"=>"5c23ca80ee2911532e45cb7c"},
    "id"=>"5c23ca80ee2911532e45cb7c"}],
 "poll_attempts"=>{"total"=>14, "failed"=>1}}

Exception:

Katello::Errors::PulpError: Task canceled

Backtrace:

/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.42/app/lib/actions/pulp/abstract_async_task.rb:122:in `block in external_task='
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.42/app/lib/actions/pulp/abstract_async_task.rb:120:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.42/app/lib/actions/pulp/abstract_async_task.rb:120:in `external_task='
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/action/polling.rb:98:in `poll_external_task_with_rescue'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/action/polling.rb:21:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/action/cancellable.rb:13:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.42/app/lib/actions/pulp/abstract_async_task.rb:45:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/action.rb:538:in `block (3 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware.rb:31:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.42/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.42/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.42/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.42/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.42/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.42/app/lib/actions/middleware/remote_action.rb:16:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.4.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/execution_wrapper.rb:85:in `wrap'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.4.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/action/progress.rb:30:in `with_progress_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/action/progress.rb:16:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.42/app/lib/actions/middleware/keep_locale.rb:11:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.42/app/lib/actions/middleware/keep_locale.rb:22:in `with_locale'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.42/app/lib/actions/middleware/keep_locale.rb:11:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.4.2/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.4.2/app/lib/actions/middleware/keep_current_user.rb:43:in `restore_curent_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.4.2/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware.rb:31:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware/world.rb:30:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/action.rb:537:in `block (2 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/action.rb:536:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/action.rb:536:in `block in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/action.rb:451:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/action.rb:451:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/action.rb:451:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/action.rb:531:in `execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/action.rb:278:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block (2 levels) in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/execution_plan/steps/abstract.rb:162:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:30:in `open_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:15:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/director.rb:56:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/executors/parallel/worker.rb:12:in `block in on_message'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/executors.rb:12:in `run_user_code'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/executors/parallel/worker.rb:11:in `on_message'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/context.rb:46:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/actor.rb:26:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:161:in `process_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:95:in `block in on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:118:in `block (2 levels) in schedule_execution'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `block in synchronize'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:115:in `block in schedule_execution'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/serialized_execution.rb:18:in `call'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/serialized_execution.rb:96:in `work'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/serialized_execution.rb:77:in `block in call_job'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:348:in `run_task'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:337:in `block (3 levels) in create_worker'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `loop'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `block (2 levels) in create_worker'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `catch'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `block in create_worker'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
Comment 3 Bryan Kearney 2020-01-15 21:01:00 UTC 
The Satellite Team is attempting to provide an accurate backlog of bugzilla requests which we feel will be resolved in the next few releases. We do not believe this bugzilla will meet that criteria, and have plans to close it out in 1 month. This is not a reflection on the validity of the request, but a reflection of the many priorities for the product. If you have any concerns about this, feel free to contact Red Hat Technical Support or your account team. If we do not hear from you, we will close this bug out. Thank you.
Comment 4 Bryan Kearney 2020-02-03 16:30:14 UTC 
Thank you for your interest in Satellite 6. We have evaluated this request, and while we recognize that it is a valid request, we do not expect this to be implemented in the product in the foreseeable future. This is due to other priorities for the product, and not a reflection on the request itself. We are therefore closing this out as WONTFIX. If you have any concerns about this, please do not reopen. Instead, feel free to contact Red Hat Technical Support. Thank you.
Note You need to log in before you can comment on or make changes to this bug.