Bug 1932700

Description of problem:
Due to the following setting, the jobs in the "remote_execution" queue will only be processed only if the "default" queue is empty. In other words, if the "default" queue is always busy polling the capsule task status, the remote execution jobs will never be processed until the capsule sync is finished.

---------------------
/etc/foreman/dynflow/worker.yml :
:concurrency: 5
:queues:
  - default
  - remote_execution
---------------------

We can see check the queues' size and latency in the following url.  Note: you need to login to the satellite first before you can access this url.

https://<satellite fqdn>/foreman_tasks/sidekiq/queues

In my Satellite:
As you can see below, there are 86 jobs queuing in the "default" queue and the only job in the "remote execution" queue has been waiting for more than 5 minutes.

----------------------------------------
Queue	Size	Latency	Actions
default	86	31.36	
dynflow_orchestrator	0	0	
hosts_queue	0	0	
remote_execution	1	346.36 (5 minutes ago)
----------------------------------------

To fix this issue, we need to add the equal weight to both queues so the jobs will have equal chance to be processed.

Steps:
1) Edit "/etc/foreman/dynflow/worker.yml" and make the following change.

:concurrency: 5
:queues:
  - ["default", 1]
  - ["remote_execution", 1]

2) Restart dynflow workers then wait for a few minutes. The jobs should be processed slowly.

systemctl restart dynflow-sidekiq


Steps to Reproduce:
1. Perform an optimized or full Capsule sync. It is ok as long as the task will take awhile and have something to sync.
2. Trigger a REX or Ansible job against 1 or more hosts

Actual results:
The REX or Ansible job is not triggering sub tasks until the Capsule sync is finished.


Expected results:
The REX or Ansible job can be run while Capsule is syncing