Bug 2165469

Summary: Delayed tasks are not indicating their actual start time. Add a "sleeping" indication to tasks.
Product: Red Hat Satellite Reporter: Nikhil Gupta <ngupta>
Component: RH CloudAssignee: Shimon Shtein <sshtein>
Status: CLOSED MIGRATED QA Contact: Satellite QE Team <sat-qe-bz-list>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.12.1CC: aruzicka, ehelms, peter.vreman
Target Milestone: UnspecifiedKeywords: MigratedToJIRA, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2024-06-06 16:05:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Nikhil Gupta 2023-01-30 07:45:42 UTC
Description of problem:
With the new delay feature the inventory/insights sync jobs can take 5 hours instead of <5 seconds. This is not good from a troubleshooting/analyzing perspective because you cannot determine if the 5 hours is expected.

Also for the 5 hours a dynflow worker is occupied

See attached screenshot.

Version-Release number of selected component (if applicable):
Satellite-6.12.1

How reproducible:
Always

Steps to Reproduce:
1. Insights scheduled sync

Actual results:
The inventory/insights scheduled sync jobs can take 5 hours

Expected results:
Should be completed in few seconds.

Additional info:
Expectation is that the delay is used in the planning step and scheduling the job at time "X+<randomdelay>"

Comment 3 Shimon Shtein 2023-01-30 09:16:58 UTC
While there is a running task in the Dynflow worker, it's not consuming any resources. It's dehydrated and waiting for a timer event.
From debugging perspective, opening the steps will show you whether the task is waiting the delay or tries to perform: there is a separate "DelayAction" step that generates the delayed start effect. If the action is executing this step, it means it waits. If the action is performing InventoryFullSync - it's actually working.

There is an option to force the delay instead of having it randomized for you: you can set SATELLITE_RH_CLOUD_REQUESTS_DELAY=<seconds> ENV variable to force the delay to any value. This is a good option if you want to run the task synchronously:

~~~
echo "::ForemanTasks.sync_task(InventorySync::Async::InventoryScheduledSync)" | SATELLITE_RH_CLOUD_REQUESTS_DELAY=0 foreman-rake console
~~~

For technical reference, the commit that introduced the delay lives here: https://github.com/theforeman/foreman_rh_cloud/commit/8c38ff76b38f146baea755bc85d333da09d0c327.

If it would make more sense, we can add a parameter to the InventoryScheduledSync task directly to set the delay value once. So the call would look like:

~~~
echo "::ForemanTasks.sync_task(InventorySync::Async::InventoryScheduledSync, delay: 0)" | foreman-rake console
~~~


We have considered an option of randomizing the actual start time, but it created a lot of edge cases, so the behavior became unpredictable. The preferred solution was to introduce the delay step, which is apparent to the user (including the delay time, it's an input parameter passed to DelayAction and is available in the dynflow UI).

Comment 6 Eric Helms 2024-06-06 16:05:44 UTC
This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "SAT-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.