2068527 – Unable to plan ansible tasks with large inventories and concurrency control - undefined method `wait' for nil:NilClass (NoMethodError)

Bug 2068527 - Unable to plan ansible tasks with large inventories and concurrency control - undefined method `wait' for nil:NilClass (NoMethodError)

Summary: Unable to plan ansible tasks with large inventories and concurrency control -...

Keywords:
Status:	NEW
Alias:	None
Product:	Red Hat Satellite
Classification:	Red Hat
Component:	Ansible
Sub Component:
Version:	6.9.7
Hardware:	Unspecified
OS:	Unspecified
Priority:	urgent
Severity:	urgent
Target Milestone:	Unspecified
Assignee:	satellite6-bugs
QA Contact:	Sam Bible
Docs Contact:
URL:
Whiteboard:
Duplicates (1):	2135790 (view as bug list)
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2022-03-25 15:11 UTC by Jan Jansky
Modified:	2023-08-14 08:41 UTC (History)
CC List:	22 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Github	Dynflow dynflow issues 415	None	open	Throttle limiter assumes a single process environment	2023-08-07 09:04:33 UTC
Red Hat Issue Tracker	SAT-15989	None	None	None	2023-02-16 15:43:32 UTC
Red Hat Knowledge Base (Solution)	6963591	None	None	None	2023-04-27 06:23:59 UTC
Red Hat Knowledge Base (Solution)	6986273	None	None	None	2022-12-07 11:58:25 UTC

Description Jan Jansky 2022-03-25 15:11:07 UTC

Description of problem:
When planning ansible task on large batch of host multiple errors appears. We are talking about 300 and more hosts. 

Version-Release number of selected component (if applicable):
satellite-6.9.7-1.el7sat.noarch

How reproducible:
I was not able to reproduce myself with 500+ hosts.

Steps to Reproduce:
1. have over 300 hosts
2. schedule ansible job, no matter of complexity

Actual results:
some hosts will stay forever with N/A status


Expected results:
Execution on large inventory possible

Additional info:
/var/log/messages
Mar 23 16:39:52 satellite smart-proxy: xxx.xxx.xxx.xxx - - [23/Mar/2022:16:39:52 CET] "GET /dynflow/tasks/count?state=running HTTP/1.1" 200 29
Mar 23 16:39:52 satellite smart-proxy: - -> /dynflow/tasks/count?state=running
Mar 23 16:39:53 satellite smart-proxy: xxx.xxx.xxx.xxx - - [23/Mar/2022:16:39:53 CET] "GET /version HTTP/1.1" 200 282
Mar 23 16:39:53 satellite smart-proxy: - -> /version
Mar 23 16:39:53 satellite smart-proxy: xxx.xxx.xxx.xxx - - [23/Mar/2022:16:39:53 CET] "POST /dynflow/tasks/launch HTTP/1.1" 200 110
Mar 23 16:39:53 satellite smart-proxy: - -> /dynflow/tasks/launch
Mar 23 16:39:54 satellite smart-proxy: xxx.xxx.xxx.xxx - - [23/Mar/2022:16:39:53 CET] "POST /dynflow/tasks/launch HTTP/1.1" 200 110
Mar 23 16:39:54 satellite smart-proxy: - -> /dynflow/tasks/launch
Mar 23 16:39:54 satellite smart-proxy: xxx.xxx.xxx.xxx - - [23/Mar/2022:16:39:54 CET] "POST /dynflow/tasks/launch HTTP/1.1" 200 110
Mar 23 16:39:54 satellite smart-proxy: - -> /dynflow/tasks/launch
Mar 23 16:39:55 satellite smart-proxy: xxx.xxx.xxx.xxx - - [23/Mar/2022:16:39:55 CET] "POST /dynflow/tasks/launch HTTP/1.1" 200 110
Mar 23 16:39:55 satellite smart-proxy: - -> /dynflow/tasks/launch
Mar 23 16:39:55 satellite smart-proxy: xxx.xxx.xxx.xxx - - [23/Mar/2022:16:39:55 CET] "POST /dynflow/tasks/launch HTTP/1.1" 200 110
Mar 23 16:39:55 satellite smart-proxy: - -> /dynflow/tasks/launch
Mar 23 16:39:56 satellite smart-proxy: xxx.xxx.xxx.xxx - - [23/Mar/2022:16:39:56 CET] "POST /dynflow/tasks/launch HTTP/1.1" 200 110
Mar 23 16:39:56 satellite smart-proxy: - -> /dynflow/tasks/launch
Mar 23 16:39:57 satellite smart-proxy: xxx.xxx.xxx.xxx - - [23/Mar/2022:16:39:56 CET] "POST /dynflow/tasks/launch HTTP/1.1" 200 110
Mar 23 16:39:57 satellite smart-proxy: - -> /dynflow/tasks/launch
Mar 23 16:39:57 satellite smart-proxy: xxx.xxx.xxx.xxx - - [23/Mar/2022:16:39:57 CET] "POST /dynflow/tasks/launch HTTP/1.1" 200 110
Mar 23 16:39:57 satellite smart-proxy: - -> /dynflow/tasks/launch
Mar 23 16:39:58 satellite smart-proxy: xxx.xxx.xxx.xxx - - [23/Mar/2022:16:39:58 CET] "POST /dynflow/tasks/launch HTTP/1.1" 200 110
Mar 23 16:39:58 satellite smart-proxy: - -> /dynflow/tasks/launch
Mar 23 16:39:59 satellite smart-proxy: xxx.xxx.xxx.xxx - - [23/Mar/2022:16:39:58 CET] "POST /dynflow/tasks/launch HTTP/1.1" 200 110
Mar 23 16:39:59 satellite smart-proxy: - -> /dynflow/tasks/launch
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: E, [2022-03-23T16:40:42.383222 #61768] ERROR -- /parallel-executor-core/pool default/worker-0: "\xC3" from ASCII-8BIT to UTF-8 (Encoding::UndefinedConversionError)
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/multi_json-1.14.1/lib/multi_json/adapters/json_common.rb:19:in `encode'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/multi_json-1.14.1/lib/multi_json/adapters/json_common.rb:19:in `to_json'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/multi_json-1.14.1/lib/multi_json/adapters/json_common.rb:19:in `dump'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/multi_json-1.14.1/lib/multi_json/adapter.rb:25:in `dump'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/multi_json-1.14.1/lib/multi_json.rb:139:in `dump'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/persistence_adapters/sequel.rb:397:in `dump_data'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/persistence_adapters/sequel.rb:300:in `block in serialize_columns'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/persistence_adapters/sequel.rb:298:in `each'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/persistence_adapters/sequel.rb:298:in `reduce'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/persistence_adapters/sequel.rb:298:in `serialize_columns'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/persistence_adapters/sequel.rb:288:in `prepare_record'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/persistence_adapters/sequel.rb:312:in `save'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/persistence_adapters/sequel.rb:173:in `save_action'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/persistence.rb:46:in `save_action'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:33:in `open_action'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/director.rb:93:in `execute'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/parallel/worker.rb:15:in `block in on_message'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors.rb:18:in `run_user_code'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/parallel/worker.rb:14:in `on_message'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: [ concurrent-ruby ]
Mar 23 16:41:13 satellite smart_proxy_dynflow_core: E, [2022-03-23T16:41:13.261767 #61768] ERROR -- /client-dispatcher: Could not find an executor for Dynflow::Dispatcher::Envelope[request_id: ac1bae45-fd9b-4391-9507-58ccfe0a91bc-319838, sender_id: ac1bae45-fd9b-4391-9507-58ccfe0a91bc, receiver_id: Dynflow::Dispatcher::UnknownWorld, message: Dynflow::Dispatcher::Event[execution_plan_id: 002c6216-8a0f-4767-a15e-09083206dde2, step_id: 2, event: #<ForemanTasksCore::Runner::Update:0x0000000004093f18>, time: ]] (Dynflow::Error)
Mar 23 16:41:13 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/dispatcher/client_dispatcher.rb:147:in `dispatch_request'
Mar 23 16:41:13 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/dispatcher/client_dispatcher.rb:118:in `block (2 levels) in publish_request'
Mar 23 16:41:13 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/dispatcher/client_dispatcher.rb:206:in `track_request'
Mar 23 16:41:13 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/dispatcher/client_dispatcher.rb:117:in `block in publish_request'
Mar 23 16:41:13 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/dispatcher/client_dispatcher.rb:248:in `with_ping_request_caching'
Mar 23 16:41:13 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/dispatcher/client_dispatcher.rb:116:in `publish_request'
Mar 23 16:41:13 satellite smart_proxy_dynflow_core: [ concurrent-ruby ]
...
Mar 23 16:49:52 satellite dynflow-sidekiq@orchestrator: E, [2022-03-23T16:49:52.395648 #62075] ERROR -- /parallel-executor-core: no manager for Dynflow::Director::Event[request_id: a2a59f6c-9a5e-4853-8b3a-f2eeb495e389-231, execution_plan_id: b73e2c8a-165a-47af-a84d-0f2831832948, step_id: 2, event: Actions::Middleware::WatchDelegatedProxySubTasks::CheckOnProxyActions, result: #<Concurrent::Promises::ResolvableFuture:0x00007f791ac15af8 pending>] (Dynflow::Error)
Mar 23 16:49:52 satellite dynflow-sidekiq@orchestrator: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/director.rb:167:in `handle_event'
Mar 23 16:49:52 satellite dynflow-sidekiq@orchestrator: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/abstract/core.rb:35:in `handle_event'
Mar 23 16:49:52 satellite dynflow-sidekiq@orchestrator: [ concurrent-ruby ]
Mar 23 16:49:52 satellite dynflow-sidekiq@orchestrator: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/abstract/core.rb:113:in `on_message'
Mar 23 16:49:52 satellite dynflow-sidekiq@orchestrator: [ concurrent-ruby ]

/var/log/foreman/production.log
2022-03-23T16:41:02 [E|bac|] undefined method `wait' for nil:NilClass (NoMethodError)
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/throttle_limiter.rb:71:in `block (2 levels) in handle_plans'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/throttle_limiter.rb:69:in `tap'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/throttle_limiter.rb:69:in `block in handle_plans'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/throttle_limiter.rb:68:in `map'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/throttle_limiter.rb:68:in `handle_plans'
 | [ concurrent-ruby ]
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/throttle_limiter.rb:21:in `handle_plans!'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/with_sub_plans.rb:92:in `trigger_with_concurrency_control'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/with_sub_plans.rb:82:in `trigger'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-4.2.3.1/app/lib/actions/remote_execution/run_hosts_job.rb:45:in `block in create_sub_plans'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/delegation.rb:87:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-4.2.3.1/app/lib/actions/remote_execution/run_hosts_job.rb:40:in `map'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-4.2.3.1/app/lib/actions/remote_execution/run_hosts_job.rb:40:in `create_sub_plans'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/with_sub_plans.rb:41:in `spawn_plans'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/with_bulk_sub_plans.rb:71:in `spawn_plans'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/with_bulk_sub_plans.rb:18:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/with_polling_sub_plans.rb:13:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-4.2.3.1/app/lib/actions/remote_execution/run_hosts_job.rb:97:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/proxy_batch_triggering.rb:9:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/watch_delegated_proxy_sub_tasks.rb:17:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in `wrap'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:17:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_request_id.rb:49:in `restore_current_request_id'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_user.rb:44:in `restore_curent_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/world.rb:31:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:570:in `block (2 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:569:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:569:in `block in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `block in with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:564:in `execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:285:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/director.rb:93:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors.rb:18:in `run_user_code'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
 | [ sidekiq ]
 | [ concurrent-ruby ]

Comment 4 Adam Ruzicka 2022-06-07 12:36:53 UTC

The json parse error should be fixed by [1], I would expect 6.11 to be free from this bug, although I can't say for certain as I have not managed to reproduce it myself.

[1] - https://github.com/ansible/ansible-runner/pull/638

Comment 11 Adam Ruzicka 2022-12-07 11:58:25 UTC

*** Bug 2135790 has been marked as a duplicate of this bug. ***

Comment 14 Pranjal 2022-12-13 16:59:38 UTC

In one of the case of satellite 6.11, user has 3 sidekiq workers and each of them have the exact same default configuration, causing same issue.

Upon following solution, manual changes made inside /etc/foreman/dynflow/worker-*.yml files gets overwritten with every installer run ( which is expected ). 

To resolve this, executing installer option i.e. <--foreman-dynflow-manage-services false> will unmanage dynflow configuration and allow manual modifications of dynflow workers to remain intact. But It's not an acceptable solution for the end-user 

User rather would like to be able to control the remote_execution queue assignment for individual workers with other method, instead of manually updating the worker configurations.

Comment 20 Peter Vreman 2023-03-02 14:41:44 UTC

Copying from BZ https://bugzilla.redhat.com/show_bug.cgi?id=2135790 that is closed as duplicated.

The issue on Satellite version 6.11 is not only the the undefined method or nilclass, but even worser that it has also tasks of the job in Pending state that cannot be cleaned up by a regular user in the UI.


Job execution with concurrency set broken with nilclass at 100 servers out of 116 if batch size >= 100 is having all servers > 100 in a pending task state and the job is hangs/fails.
How reproducible:
Always.

Steps to Reproduce:

Job details for reproducer:
- Type: SSH Command
- Query '*'  (make sure > 100 servers are there)
- Command: 'date'
- Concurrency: 30


Actual results:
Job on ~120 servers with concurrency 30 and the following Capsule batch sizes:
- 50: ok
- 99; ok
- 100: FAIL - Pending tasks without progress
- 101: FAIL - Pending tasks without progress
- 150: FAIL - Pending tasks without progress


Expected results:
It should not fail

Additional info:

Issue related to 'Concurrency level limited to: 30 tasks at a time'.
If do not set the concurrency level it successfully processes all ~116 servers. 
If set the concurrency to 30 tasks then it exactly hangs/fails again at ~100

Note You need to log in before you can comment on or make changes to this bug.

ahumbe
aruzicka
bshahu
dhjoshi
ekohlvan
hakon.gislason
hasingh
jbhatia
jkastnin
lstejska
mkalyat
nmohite
oezr
patalber
pbadguja
pdwyer
peter.vreman
pmendezh
saydas
sigbjorn.lie
smeyer
wpinheir