Bug 1969551 - [4.8.0] Assisted service times out on GetNextSteps due to `oc adm release info` taking too long
Summary: [4.8.0] Assisted service times out on GetNextSteps due to `oc adm release inf...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: assisted-installer
Version: 4.8
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.8.0
Assignee: Daniel Erez
QA Contact: Yuri Obshansky
URL:
Whiteboard: KNI-EDGE-JUKE-4.8 AI-Team-Core
Depends On: 1969477
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-06-08 15:14 UTC by Ronnie Lazar
Modified: 2021-07-27 23:12 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1969477
Environment:
Last Closed: 2021-07-27 23:12:15 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift assisted-service pull 2037 0 None open Bug 1969551: cache image fetched from release 2021-06-21 11:55:25 UTC
Red Hat Bugzilla 1969477 1 high CLOSED [master] Assisted service times out on GetNextSteps due to `oc adm release info` taking too long 2021-10-18 17:33:36 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 23:12:29 UTC

Description Ronnie Lazar 2021-06-08 15:14:13 UTC
+++ This bug was initially created as a clone of Bug #1969477 +++

Description of problem:
Agent calls GetNextSteps API, the request arrives at the service, while processing the request the service runs `oc adm release info`, the command may take a long time and the request eventually times out

Version-Release number of selected component (if applicable):
Latest (2021/06/08)

How reproducible:
It seems to happen when there's connectivity issues to the registry, it hangs for 15 seconds, then says

```
$ oc adm release info

error: unable to read image registry.ci...: Get "...": context deadline exceeded
```

Steps to Reproduce:
See above

Actual results:
Request takes a long time, seems like some intermediate HTTP proxy eventually gives up and returns a 504. This is unexpected in the swagger definition so the agent gives a cryptic error message about swagger and 504.

Expected results:
Service should be able to tell that the command will fail more quickly. Either by caching in-advance, or having a more strict internal timeout.

Additional info:

--- Additional comment from itsoiref on 20210608T14:31:55

We saw that assisted-service is running "oc adm release info" for 2 images "machine-config-operator" and "must-gather".
Current mirrored registry (disconnected env) was slow and it took ~15 seconds for each command to run. 
On the agent side we have 30 seconds timeout as default in assisted-client. This combination caused agent to timeout all the time and installation failed to start and moved from preparing back to ready and vicer-versa all the time.

We should add cache per ocp version per image as there is no benefit to run it all the time.

--- Additional comment from alazar on 20210608T14:59:33

Besides caching, maybe should we also enlarge the timeout on the agent's side?

Comment 3 nshidlin 2021-06-29 17:41:56 UTC
this bug doesn't block OCP4.8.0 -release -  the fix will be delivered the the ACM channel on the AI-operator add verified then

Comment 9 errata-xmlrpc 2021-07-27 23:12:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438


Note You need to log in before you can comment on or make changes to this bug.