2068527 – Unable to plan ansible tasks with large inventories and concurrency control - undefined method `wait' for nil:NilClass (NoMethodError)

Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2068527 - Unable to plan ansible tasks with large inventories and concurrency control - undefined method `wait' for nil:NilClass (NoMethodError)

Summary: Unable to plan ansible tasks with large inventories and concurrency control -...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Satellite
Classification:	Red Hat
Component:	Ansible - Configuration Management
Sub Component:
Version:	6.9.7
Hardware:	Unspecified
OS:	Unspecified
Priority:	urgent
Severity:	urgent
Target Milestone:	6.15.0
Assignee:	Adam Ruzicka
QA Contact:	Satellite QE Team
Docs Contact:
URL:
Whiteboard:
Duplicates (2):	2135790 2250346 (view as bug list)
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2022-03-25 15:11 UTC by Jan Jansky
Modified:	2024-05-02 07:33 UTC (History)
CC List:	30 users (show)
Fixed In Version:	rubygem-dynflow-1.8.0, rubygem-foreman-tasks-8.2.0, rubygem-foreman_remote_execution-11.0.0
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Clones:	2250346 (view as bug list)
Environment:
Last Closed:	2024-04-23 17:11:05 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Github	Dynflow dynflow issues 415	None	open	Throttle limiter assumes a single process environment	2023-09-18 08:44:54 UTC
Red Hat Issue Tracker	SAT-15227	None	None	None	2023-09-18 08:45:42 UTC
Red Hat Issue Tracker	SAT-15989	None	None	None	2023-02-16 15:43:32 UTC
Red Hat Knowledge Base (Solution)	6963591	None	None	None	2023-04-27 06:23:59 UTC
Red Hat Knowledge Base (Solution)	6986273	None	None	None	2022-12-07 11:58:25 UTC
Red Hat Product Errata	RHSA-2024:2010	None	None	None	2024-04-23 17:11:09 UTC

Description Jan Jansky 2022-03-25 15:11:07 UTC

Description of problem:
When planning ansible task on large batch of host multiple errors appears. We are talking about 300 and more hosts. 

Version-Release number of selected component (if applicable):
satellite-6.9.7-1.el7sat.noarch

How reproducible:
I was not able to reproduce myself with 500+ hosts.

Steps to Reproduce:
1. have over 300 hosts
2. schedule ansible job, no matter of complexity

Actual results:
some hosts will stay forever with N/A status


Expected results:
Execution on large inventory possible

Additional info:
/var/log/messages
Mar 23 16:39:52 satellite smart-proxy: xxx.xxx.xxx.xxx - - [23/Mar/2022:16:39:52 CET] "GET /dynflow/tasks/count?state=running HTTP/1.1" 200 29
Mar 23 16:39:52 satellite smart-proxy: - -> /dynflow/tasks/count?state=running
Mar 23 16:39:53 satellite smart-proxy: xxx.xxx.xxx.xxx - - [23/Mar/2022:16:39:53 CET] "GET /version HTTP/1.1" 200 282
Mar 23 16:39:53 satellite smart-proxy: - -> /version
Mar 23 16:39:53 satellite smart-proxy: xxx.xxx.xxx.xxx - - [23/Mar/2022:16:39:53 CET] "POST /dynflow/tasks/launch HTTP/1.1" 200 110
Mar 23 16:39:53 satellite smart-proxy: - -> /dynflow/tasks/launch
Mar 23 16:39:54 satellite smart-proxy: xxx.xxx.xxx.xxx - - [23/Mar/2022:16:39:53 CET] "POST /dynflow/tasks/launch HTTP/1.1" 200 110
Mar 23 16:39:54 satellite smart-proxy: - -> /dynflow/tasks/launch
Mar 23 16:39:54 satellite smart-proxy: xxx.xxx.xxx.xxx - - [23/Mar/2022:16:39:54 CET] "POST /dynflow/tasks/launch HTTP/1.1" 200 110
Mar 23 16:39:54 satellite smart-proxy: - -> /dynflow/tasks/launch
Mar 23 16:39:55 satellite smart-proxy: xxx.xxx.xxx.xxx - - [23/Mar/2022:16:39:55 CET] "POST /dynflow/tasks/launch HTTP/1.1" 200 110
Mar 23 16:39:55 satellite smart-proxy: - -> /dynflow/tasks/launch
Mar 23 16:39:55 satellite smart-proxy: xxx.xxx.xxx.xxx - - [23/Mar/2022:16:39:55 CET] "POST /dynflow/tasks/launch HTTP/1.1" 200 110
Mar 23 16:39:55 satellite smart-proxy: - -> /dynflow/tasks/launch
Mar 23 16:39:56 satellite smart-proxy: xxx.xxx.xxx.xxx - - [23/Mar/2022:16:39:56 CET] "POST /dynflow/tasks/launch HTTP/1.1" 200 110
Mar 23 16:39:56 satellite smart-proxy: - -> /dynflow/tasks/launch
Mar 23 16:39:57 satellite smart-proxy: xxx.xxx.xxx.xxx - - [23/Mar/2022:16:39:56 CET] "POST /dynflow/tasks/launch HTTP/1.1" 200 110
Mar 23 16:39:57 satellite smart-proxy: - -> /dynflow/tasks/launch
Mar 23 16:39:57 satellite smart-proxy: xxx.xxx.xxx.xxx - - [23/Mar/2022:16:39:57 CET] "POST /dynflow/tasks/launch HTTP/1.1" 200 110
Mar 23 16:39:57 satellite smart-proxy: - -> /dynflow/tasks/launch
Mar 23 16:39:58 satellite smart-proxy: xxx.xxx.xxx.xxx - - [23/Mar/2022:16:39:58 CET] "POST /dynflow/tasks/launch HTTP/1.1" 200 110
Mar 23 16:39:58 satellite smart-proxy: - -> /dynflow/tasks/launch
Mar 23 16:39:59 satellite smart-proxy: xxx.xxx.xxx.xxx - - [23/Mar/2022:16:39:58 CET] "POST /dynflow/tasks/launch HTTP/1.1" 200 110
Mar 23 16:39:59 satellite smart-proxy: - -> /dynflow/tasks/launch
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: E, [2022-03-23T16:40:42.383222 #61768] ERROR -- /parallel-executor-core/pool default/worker-0: "\xC3" from ASCII-8BIT to UTF-8 (Encoding::UndefinedConversionError)
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/multi_json-1.14.1/lib/multi_json/adapters/json_common.rb:19:in `encode'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/multi_json-1.14.1/lib/multi_json/adapters/json_common.rb:19:in `to_json'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/multi_json-1.14.1/lib/multi_json/adapters/json_common.rb:19:in `dump'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/multi_json-1.14.1/lib/multi_json/adapter.rb:25:in `dump'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/multi_json-1.14.1/lib/multi_json.rb:139:in `dump'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/persistence_adapters/sequel.rb:397:in `dump_data'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/persistence_adapters/sequel.rb:300:in `block in serialize_columns'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/persistence_adapters/sequel.rb:298:in `each'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/persistence_adapters/sequel.rb:298:in `reduce'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/persistence_adapters/sequel.rb:298:in `serialize_columns'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/persistence_adapters/sequel.rb:288:in `prepare_record'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/persistence_adapters/sequel.rb:312:in `save'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/persistence_adapters/sequel.rb:173:in `save_action'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/persistence.rb:46:in `save_action'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:33:in `open_action'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/director.rb:93:in `execute'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/parallel/worker.rb:15:in `block in on_message'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors.rb:18:in `run_user_code'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/parallel/worker.rb:14:in `on_message'
Mar 23 16:40:42 satellite smart_proxy_dynflow_core: [ concurrent-ruby ]
Mar 23 16:41:13 satellite smart_proxy_dynflow_core: E, [2022-03-23T16:41:13.261767 #61768] ERROR -- /client-dispatcher: Could not find an executor for Dynflow::Dispatcher::Envelope[request_id: ac1bae45-fd9b-4391-9507-58ccfe0a91bc-319838, sender_id: ac1bae45-fd9b-4391-9507-58ccfe0a91bc, receiver_id: Dynflow::Dispatcher::UnknownWorld, message: Dynflow::Dispatcher::Event[execution_plan_id: 002c6216-8a0f-4767-a15e-09083206dde2, step_id: 2, event: #<ForemanTasksCore::Runner::Update:0x0000000004093f18>, time: ]] (Dynflow::Error)
Mar 23 16:41:13 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/dispatcher/client_dispatcher.rb:147:in `dispatch_request'
Mar 23 16:41:13 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/dispatcher/client_dispatcher.rb:118:in `block (2 levels) in publish_request'
Mar 23 16:41:13 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/dispatcher/client_dispatcher.rb:206:in `track_request'
Mar 23 16:41:13 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/dispatcher/client_dispatcher.rb:117:in `block in publish_request'
Mar 23 16:41:13 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/dispatcher/client_dispatcher.rb:248:in `with_ping_request_caching'
Mar 23 16:41:13 satellite smart_proxy_dynflow_core: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/dispatcher/client_dispatcher.rb:116:in `publish_request'
Mar 23 16:41:13 satellite smart_proxy_dynflow_core: [ concurrent-ruby ]
...
Mar 23 16:49:52 satellite dynflow-sidekiq@orchestrator: E, [2022-03-23T16:49:52.395648 #62075] ERROR -- /parallel-executor-core: no manager for Dynflow::Director::Event[request_id: a2a59f6c-9a5e-4853-8b3a-f2eeb495e389-231, execution_plan_id: b73e2c8a-165a-47af-a84d-0f2831832948, step_id: 2, event: Actions::Middleware::WatchDelegatedProxySubTasks::CheckOnProxyActions, result: #<Concurrent::Promises::ResolvableFuture:0x00007f791ac15af8 pending>] (Dynflow::Error)
Mar 23 16:49:52 satellite dynflow-sidekiq@orchestrator: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/director.rb:167:in `handle_event'
Mar 23 16:49:52 satellite dynflow-sidekiq@orchestrator: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/abstract/core.rb:35:in `handle_event'
Mar 23 16:49:52 satellite dynflow-sidekiq@orchestrator: [ concurrent-ruby ]
Mar 23 16:49:52 satellite dynflow-sidekiq@orchestrator: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/abstract/core.rb:113:in `on_message'
Mar 23 16:49:52 satellite dynflow-sidekiq@orchestrator: [ concurrent-ruby ]

/var/log/foreman/production.log
2022-03-23T16:41:02 [E|bac|] undefined method `wait' for nil:NilClass (NoMethodError)
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/throttle_limiter.rb:71:in `block (2 levels) in handle_plans'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/throttle_limiter.rb:69:in `tap'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/throttle_limiter.rb:69:in `block in handle_plans'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/throttle_limiter.rb:68:in `map'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/throttle_limiter.rb:68:in `handle_plans'
 | [ concurrent-ruby ]
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/throttle_limiter.rb:21:in `handle_plans!'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/with_sub_plans.rb:92:in `trigger_with_concurrency_control'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/with_sub_plans.rb:82:in `trigger'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-4.2.3.1/app/lib/actions/remote_execution/run_hosts_job.rb:45:in `block in create_sub_plans'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/delegation.rb:87:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-4.2.3.1/app/lib/actions/remote_execution/run_hosts_job.rb:40:in `map'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-4.2.3.1/app/lib/actions/remote_execution/run_hosts_job.rb:40:in `create_sub_plans'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/with_sub_plans.rb:41:in `spawn_plans'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/with_bulk_sub_plans.rb:71:in `spawn_plans'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/with_bulk_sub_plans.rb:18:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/with_polling_sub_plans.rb:13:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-4.2.3.1/app/lib/actions/remote_execution/run_hosts_job.rb:97:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/proxy_batch_triggering.rb:9:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/watch_delegated_proxy_sub_tasks.rb:17:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in `wrap'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:17:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_request_id.rb:49:in `restore_current_request_id'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_user.rb:44:in `restore_curent_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/world.rb:31:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:570:in `block (2 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:569:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:569:in `block in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `block in with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:564:in `execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:285:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/director.rb:93:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors.rb:18:in `run_user_code'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
 | [ sidekiq ]
 | [ concurrent-ruby ]

Comment 4 Adam Ruzicka 2022-06-07 12:36:53 UTC

The json parse error should be fixed by [1], I would expect 6.11 to be free from this bug, although I can't say for certain as I have not managed to reproduce it myself.

[1] - https://github.com/ansible/ansible-runner/pull/638

Comment 11 Adam Ruzicka 2022-12-07 11:58:25 UTC

*** Bug 2135790 has been marked as a duplicate of this bug. ***

Comment 14 Pranjal 2022-12-13 16:59:38 UTC

In one of the case of satellite 6.11, user has 3 sidekiq workers and each of them have the exact same default configuration, causing same issue.

Upon following solution, manual changes made inside /etc/foreman/dynflow/worker-*.yml files gets overwritten with every installer run ( which is expected ). 

To resolve this, executing installer option i.e. <--foreman-dynflow-manage-services false> will unmanage dynflow configuration and allow manual modifications of dynflow workers to remain intact. But It's not an acceptable solution for the end-user 

User rather would like to be able to control the remote_execution queue assignment for individual workers with other method, instead of manually updating the worker configurations.

Comment 20 Peter Vreman 2023-03-02 14:41:44 UTC

Copying from BZ https://bugzilla.redhat.com/show_bug.cgi?id=2135790 that is closed as duplicated.

The issue on Satellite version 6.11 is not only the the undefined method or nilclass, but even worser that it has also tasks of the job in Pending state that cannot be cleaned up by a regular user in the UI.


Job execution with concurrency set broken with nilclass at 100 servers out of 116 if batch size >= 100 is having all servers > 100 in a pending task state and the job is hangs/fails.
How reproducible:
Always.

Steps to Reproduce:

Job details for reproducer:
- Type: SSH Command
- Query '*'  (make sure > 100 servers are there)
- Command: 'date'
- Concurrency: 30


Actual results:
Job on ~120 servers with concurrency 30 and the following Capsule batch sizes:
- 50: ok
- 99; ok
- 100: FAIL - Pending tasks without progress
- 101: FAIL - Pending tasks without progress
- 150: FAIL - Pending tasks without progress


Expected results:
It should not fail

Additional info:

Issue related to 'Concurrency level limited to: 30 tasks at a time'.
If do not set the concurrency level it successfully processes all ~116 servers. 
If set the concurrency to 30 tasks then it exactly hangs/fails again at ~100

Comment 35 Adam Ruzicka 2023-10-06 07:06:02 UTC

Propagating information from jira back here

Comment 37 Brad Buckingham 2023-10-30 11:29:29 UTC

Bulk setting Target Milestone = 6.15.0 where sat-6.15.0+ is set.

Comment 39 Brad Buckingham 2024-01-29 20:18:42 UTC

*** Bug 2250346 has been marked as a duplicate of this bug. ***

Comment 42 errata-xmlrpc 2024-04-23 17:11:05 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.15.0 release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2024:2010

Note You need to log in before you can comment on or make changes to this bug.

ahumbe
aruzicka
bshahu
dhjoshi
ekohlvan
hakon.gislason
hasingh
ikaur
jbhatia
jkastnin
jsenkyri
juwatts
lstejska
lvrtelov
mkalyat
nmohite
oezr
osousa
patalber
pbadguja
pcreech
pdwyer
peter.vreman
pmendezh
saydas
shwsingh
sigbjorn.lie
smeyer
torkil
wpinheir