Bug 2165469

Summary: Delayed tasks are not indicating their actual start time. Add a "sleeping" indication to tasks.
Product: Red Hat Satellite Reporter: Nikhil Gupta <ngupta>
Component: RH Cloud - InventoryAssignee: Shimon Shtein <sshtein>
Status: NEW --- QA Contact: Satellite QE Team <sat-qe-bz-list>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.12.1CC: aruzicka, ehelms, peter.vreman
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Nikhil Gupta 2023-01-30 07:45:42 UTC 
Description of problem:
With the new delay feature the inventory/insights sync jobs can take 5 hours instead of <5 seconds. This is not good from a troubleshooting/analyzing perspective because you cannot determine if the 5 hours is expected.

Also for the 5 hours a dynflow worker is occupied

See attached screenshot.

Version-Release number of selected component (if applicable):
Satellite-6.12.1

How reproducible:
Always

Steps to Reproduce:
1. Insights scheduled sync

Actual results:
The inventory/insights scheduled sync jobs can take 5 hours

Expected results:
Should be completed in few seconds.

Additional info:
Expectation is that the delay is used in the planning step and scheduling the job at time "X+<randomdelay>"
Comment 3 Shimon Shtein 2023-01-30 09:16:58 UTC 
While there is a running task in the Dynflow worker, it's not consuming any resources. It's dehydrated and waiting for a timer event.
From debugging perspective, opening the steps will show you whether the task is waiting the delay or tries to perform: there is a separate "DelayAction" step that generates the delayed start effect. If the action is executing this step, it means it waits. If the action is performing InventoryFullSync - it's actually working.

There is an option to force the delay instead of having it randomized for you: you can set SATELLITE_RH_CLOUD_REQUESTS_DELAY=<seconds> ENV variable to force the delay to any value. This is a good option if you want to run the task synchronously:

~~~
echo "::ForemanTasks.sync_task(InventorySync::Async::InventoryScheduledSync)" | SATELLITE_RH_CLOUD_REQUESTS_DELAY=0 foreman-rake console
~~~

For technical reference, the commit that introduced the delay lives here: https://github.com/theforeman/foreman_rh_cloud/commit/8c38ff76b38f146baea755bc85d333da09d0c327.

If it would make more sense, we can add a parameter to the InventoryScheduledSync task directly to set the delay value once. So the call would look like:

~~~
echo "::ForemanTasks.sync_task(InventorySync::Async::InventoryScheduledSync, delay: 0)" | foreman-rake console
~~~


We have considered an option of randomizing the actual start time, but it created a lot of edge cases, so the behavior became unpredictable. The preferred solution was to introduce the delay step, which is apparent to the user (including the delay time, it's an input parameter passed to DelayAction and is available in the dynflow UI).