Bug 1777890

Summary: [OCPv4.2] "oc adm release extract ... " command doesn't work in a real disconnected environment
Product: OpenShift Container Platform Reporter: Angelo Gabrieli <agabriel>
Component: InstallerAssignee: Abhinav Dahiya <adahiya>
Installer sub component: openshift-installer QA Contact: Johnny Liu <jialiu>
Status: CLOSED WONTFIX Docs Contact:
Severity: unspecified    
Priority: unspecified CC: agabriel, dtrainor, jialiu, kalexand, mfuruta, nchoudhu, oarribas
Version: 4.2.0   
Target Milestone: ---   
Target Release: 4.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-02-03 18:34:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Angelo Gabrieli 2019-11-28 15:38:35 UTC 
Created attachment 1640420 [details]
oc adm release extract command in a proxied restricted environment

Description of problem:
In an OCPv4.2 real restricted network installation (completely isolated from the internet) the command "oc adm release extract --command=openshift-install "${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}" " doesn't work because the command needs to connect to quay.io


Version-Release number of the following components:
Openshift 4.2


How reproducible:
Install an actual "restricted network" Openshift 4.2 cluster and extract the "openshift-install" binary


Steps to Reproduce:
1. Install an actual "restricted network" Openshift 4.2 cluster without access to the internet
2. Extract the "openshift-install" binary: oc adm release extract --command=openshift-install "${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}" "
3.


Actual results:
End of the oc adm release extract --loglevel=10

worqueue.go:143] about to send work queue error: <nil>
extract_tools:go:308] Skipping openshift-install-mac-%s.tar.gz, does not match current OS darwin
extract_tools:go316] Will extract usr/bin/openshift-install from quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256...
config.go:131] looking for config.json at /home/user/.docker/config.json
config.go:139] found valid config.json at /home/user/.docker/config.json
workqueue.go:54] worker 0 stopping
workqueue.go:60] work queue exiting
round_tripper.go:419] curl -k -v -XGET 'https://quay.io/v2/'
round_tripper.go:438] GET https://quay.io/v2/ in 10001 milliseconds
round_trippers.go:444] Response Headers:
workqueue.go:143] about to send work queue error: unable to connect to image repository quay.io/openshift-release-dev/ocp-v4.0-art-dev.....
helpers.go:114] error: unable to connect to image repository quay.io/openshift-release-dev/ocp-v4.0-art-dev..... io/timeout


Expected results:
The "oc adm release extract" command should be completed successfully and the "openshift-install" binary successfully extracted


Additional info:
See attachment for the same command performed in a "restricted" network behind a proxy: there are multiple calls to quay.io
Comment 1 Johnny Liu 2019-11-29 04:19:31 UTC 
Reproduce this bug.

# time curl -L quay.io
^C

real	0m49.246s
user	0m0.002s
sys	0m0.007s

# oc version
Client Version: v4.3.0
Server Version: 4.3.0-0.nightly-2019-11-28-190856
Kubernetes Version: v1.16.2

# oc adm release extract --command=openshift-install upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release:4.2 --loglevel=10
<--snip-->
I1128 23:15:22.990954   19884 extract_tools.go:319] Will extract usr/bin/openshift-install from quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:545959553564050f7784167b8caf8cfa9d35d0e46fc7bec0c5a089810b96ddc1
I1128 23:15:22.991034   19884 config.go:137] looking for config.json at /root/.docker/config.json
I1128 23:15:22.991097   19884 config.go:145] found valid config.json at /root/.docker/config.json
I1128 23:15:22.991133   19884 workqueue.go:54] worker 0 stopping
I1128 23:15:22.991148   19884 workqueue.go:60] work queue exiting
I1128 23:15:22.991204   19884 round_trippers.go:423] curl -k -v -XGET  'https://quay.io/v2/'
I1128 23:15:37.991387   19884 round_trippers.go:443] GET https://quay.io/v2/  in 15000 milliseconds
I1128 23:15:37.991413   19884 round_trippers.go:449] Response Headers:
I1128 23:15:37.991457   19884 workqueue.go:143] about to send work queue error: unable to connect to image repository quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:545959553564050f7784167b8caf8cfa9d35d0e46fc7bec0c5a089810b96ddc1: Get https://quay.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I1128 23:15:37.991492   19884 workqueue.go:54] worker 0 stopping
F1128 23:15:37.991511   19884 helpers.go:114] error: unable to connect to image repository quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:545959553564050f7784167b8caf8cfa9d35d0e46fc7bec0c5a089810b96ddc1: Get https://quay.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Because the release payload image is mirrored, no any content is changed in itself. So quay.io is still referenced, but the local host did not update registry mirror setting in container.conf, so it is trying to go outside (the real quay.io) to pull the installer image.


This issue should be already noticed, and fixed in https://github.com/openshift/openshift-docs/pull/17569
Comment 2 Scott Dodson 2019-12-02 18:29:37 UTC 
With the documentation change referenced in comment 1 this should now be clearer, we'll improve this in 4.4
Comment 3 Scott Dodson 2020-02-03 18:34:57 UTC 
This should no longer happen when following the process described in the documentation.