Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1969477

Summary:	[master] Assisted service times out on GetNextSteps due to `oc adm release info` taking too long
Product:	OpenShift Container Platform	Reporter:	Omer Tuchfeld <otuchfel>
Component:	assisted-installer	Assignee:	Daniel Erez <derez>
assisted-installer sub component:	Deployment Operator	QA Contact:	bjacot
Status:	CLOSED ERRATA	Docs Contact:
Severity:	high
Priority:	high	CC:	aos-bugs, itsoiref, mfilanov, nshidlin, yprokule
Version:	4.8	Keywords:	Triaged
Target Milestone:	---	Flags:	alazar: needinfo-
Target Release:	4.9.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:	KNI-EDGE-JUKE-4.8 AI-Team-Core
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:
Clones:	1969551 (view as bug list)		Environment:
Last Closed:	2021-10-18 17:33:16 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1969551

Description Omer Tuchfeld 2021-06-08 13:51:56 UTC

Description of problem:
Agent calls GetNextSteps API, the request arrives at the service, while processing the request the service runs `oc adm release info`, the command may take a long time and the request eventually times out

Version-Release number of selected component (if applicable):
Latest (2021/06/08)

How reproducible:
It seems to happen when there's connectivity issues to the registry, it hangs for 15 seconds, then says

```
$ oc adm release info

error: unable to read image registry.ci...: Get "...": context deadline exceeded
```

Steps to Reproduce:
See above

Actual results:
Request takes a long time, seems like some intermediate HTTP proxy eventually gives up and returns a 504. This is unexpected in the swagger definition so the agent gives a cryptic error message about swagger and 504.

Expected results:
Service should be able to tell that the command will fail more quickly. Either by caching in-advance, or having a more strict internal timeout.

Additional info:

Comment 1 Igal Tsoiref 2021-06-08 14:31:55 UTC

We saw that assisted-service is running "oc adm release info" for 2 images "machine-config-operator" and "must-gather".
Current mirrored registry (disconnected env) was slow and it took ~15 seconds for each command to run. 
On the agent side we have 30 seconds timeout as default in assisted-client. This combination caused agent to timeout all the time and installation failed to start and moved from preparing back to ready and vicer-versa all the time.

We should add cache per ocp version per image as there is no benefit to run it all the time.

Comment 2 Ronnie Lazar 2021-06-08 14:59:33 UTC

Besides caching, maybe should we also enlarge the timeout on the agent's side?

Comment 6 errata-xmlrpc 2021-10-18 17:33:16 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759