Bug 1932700 - REX or Ansible jobs are not getting processed while syncing the Capsule.
Summary: REX or Ansible jobs are not getting processed while syncing the Capsule.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Tasks Plugin
Version: 6.8.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: 6.10.0
Assignee: Adam Ruzicka
QA Contact: Peter Ondrejka
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-02-25 00:31 UTC by Hao Chang Yu
Modified: 2021-11-16 14:10 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-11-16 14:10:12 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 31964 0 Normal Ready For Testing Sidekiq workers are configured to process queues sequentially 2021-03-01 09:45:20 UTC
Red Hat Product Errata RHSA-2021:4702 0 None None None 2021-11-16 14:10:21 UTC

Description Hao Chang Yu 2021-02-25 00:31:19 UTC
Description of problem:
Due to the following setting, the jobs in the "remote_execution" queue will only be processed only if the "default" queue is empty. In other words, if the "default" queue is always busy polling the capsule task status, the remote execution jobs will never be processed until the capsule sync is finished.

---------------------
/etc/foreman/dynflow/worker.yml :
:concurrency: 5
:queues:
  - default
  - remote_execution
---------------------

We can see check the queues' size and latency in the following url.  Note: you need to login to the satellite first before you can access this url.

https://<satellite fqdn>/foreman_tasks/sidekiq/queues

In my Satellite:
As you can see below, there are 86 jobs queuing in the "default" queue and the only job in the "remote execution" queue has been waiting for more than 5 minutes.

----------------------------------------
Queue	Size	Latency	Actions
default	86	31.36	
dynflow_orchestrator	0	0	
hosts_queue	0	0	
remote_execution	1	346.36 (5 minutes ago)
----------------------------------------

To fix this issue, we need to add the equal weight to both queues so the jobs will have equal chance to be processed.

Steps:
1) Edit "/etc/foreman/dynflow/worker.yml" and make the following change.

:concurrency: 5
:queues:
  - ["default", 1]
  - ["remote_execution", 1]

2) Restart dynflow workers then wait for a few minutes. The jobs should be processed slowly.

systemctl restart dynflow-sidekiq


Steps to Reproduce:
1. Perform an optimized or full Capsule sync. It is ok as long as the task will take awhile and have something to sync.
2. Trigger a REX or Ansible job against 1 or more hosts

Actual results:
The REX or Ansible job is not triggering sub tasks until the Capsule sync is finished.


Expected results:
The REX or Ansible job can be run while Capsule is syncing

Comment 3 Bryan Kearney 2021-04-13 10:16:28 UTC
Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/31964 has been resolved.

Comment 4 Peter Ondrejka 2021-06-17 14:07:32 UTC
Verified on Satellite 6.10 snap 5, remote execution jobs are no longer blocked by capsule sync

Comment 7 errata-xmlrpc 2021-11-16 14:10:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.10 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:4702


Note You need to log in before you can comment on or make changes to this bug.