Hide Forgot
Description of problem: Due to the following setting, the jobs in the "remote_execution" queue will only be processed only if the "default" queue is empty. In other words, if the "default" queue is always busy polling the capsule task status, the remote execution jobs will never be processed until the capsule sync is finished. --------------------- /etc/foreman/dynflow/worker.yml : :concurrency: 5 :queues: - default - remote_execution --------------------- We can see check the queues' size and latency in the following url. Note: you need to login to the satellite first before you can access this url. https://<satellite fqdn>/foreman_tasks/sidekiq/queues In my Satellite: As you can see below, there are 86 jobs queuing in the "default" queue and the only job in the "remote execution" queue has been waiting for more than 5 minutes. ---------------------------------------- Queue Size Latency Actions default 86 31.36 dynflow_orchestrator 0 0 hosts_queue 0 0 remote_execution 1 346.36 (5 minutes ago) ---------------------------------------- To fix this issue, we need to add the equal weight to both queues so the jobs will have equal chance to be processed. Steps: 1) Edit "/etc/foreman/dynflow/worker.yml" and make the following change. :concurrency: 5 :queues: - ["default", 1] - ["remote_execution", 1] 2) Restart dynflow workers then wait for a few minutes. The jobs should be processed slowly. systemctl restart dynflow-sidekiq Steps to Reproduce: 1. Perform an optimized or full Capsule sync. It is ok as long as the task will take awhile and have something to sync. 2. Trigger a REX or Ansible job against 1 or more hosts Actual results: The REX or Ansible job is not triggering sub tasks until the Capsule sync is finished. Expected results: The REX or Ansible job can be run while Capsule is syncing
Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/31964 has been resolved.
Verified on Satellite 6.10 snap 5, remote execution jobs are no longer blocked by capsule sync
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Satellite 6.10 Release), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:4702