Bug 1773480 - Failed to run `oc adm catalog build` in a restricted cluster
Summary: Failed to run `oc adm catalog build` in a restricted cluster
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.3.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.3.0
Assignee: Kevin Rizza
QA Contact: Jian Zhang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-11-18 09:07 UTC by Jian Zhang
Modified: 2020-01-23 11:13 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-01-23 11:13:05 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2020:0062 0 None None None 2020-01-23 11:13:26 UTC

Description Jian Zhang 2019-11-18 09:07:45 UTC
Description of problem:
Got below errors when running the `oc adm catalog build --appregistry-org=redhat-operators-art --to=quay.io/redhat-operators-art/art:v1` command:
error: unable to parse image quay.io/operator-framework/operator-registry-server:latest: cannot retrieve image configuration for manifest sha256:a90004a32cb71af42b2d5b1bf24c6e054a240a1e985ebd896092a09672073774: Get https://d3uo42mtx6z2cr.cloudfront.net/sha256/38/38e840c2b9331a01f41c35e001b9800dc7d49b3a3dae38cb5f73bb0d55bd0281?Expires=1574067578&Signature=cb79cmMdP0Jyf~vRnjS1o9NTTLqqFrnIX233xqKoMK9T31ubJT1TfX6pjnOAiZqYQrh7Rz2M7nDaRYqA2NiGizrTpTUG-AmeKGbQ4lUc11GxYzFEocT1wHPi6wE881XZYl4DREAh81lE4QpKwUQVoKDf-aTUqgx~6PWwq7Dn1R5vaunBeQBIcrsjZ0Bk0imqfMoCPTe2nIqQpin0MH1VuXQlRiorX9dXT8gGTOivir1yNZ12eZGRJC~L-w73KbOiOcZ4AUn2Yk8XEq3OWsDDbbMOX-a6yC6bKLoEicsxthBARfMLoybbqMYZumWQ4UYZVK~UMTgqVhl2Yxhnfnd5UA__&Key-Pair-Id=APKAJ67PQLWGCSP66DGA: dial tcp: lookup d3uo42mtx6z2cr.cloudfront.net on 127.0.0.1:53: server misbehaving

And, seems like got the same error when running `oc version`
[root@dhcp-140-36 ~]# oc version
Client Version: openshift-clients-4.3.0-201910250623-48-g8c30708e
Unable to connect to the server: dial tcp: lookup api.qe-yapei1835.qe.devcluster.openshift.com on 127.0.0.1:53: server misbehaving


Version-Release number of selected component (if applicable):
openshift-clients-4.3.0-201910250623-48-g8c30708e

How reproducible:
always

Steps to Reproduce:
1. Install a restricted OCP 4.3 cluster. For example: 
https://openshift-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/Launch%20Environment%20Flexy/71785/artifact/workdir/install-dir/auth/kubeconfig/*view*/

2. Login the Quay.io, make sure the user has the permission of the "redhat-operators-art" registry namespace.

3. Run "oc adm catalog build --appregistry-org=redhat-operators-art --to=quay.io/redhat-operators-art/art:v1" command.

Actual results:
[root@dhcp-140-36 ~]# oc adm catalog build --appregistry-org=redhat-operators-art --to=quay.io/redhat-operators-art/art:v1
INFO[0001] loading Bundles                               dir=/tmp/manifests-269764601
INFO[0001] directory                                     dir=/tmp/manifests-269764601 file=manifests-269764601 load=bundles
INFO[0001] loading Packages and Entries                  dir=/tmp/manifests-269764601
INFO[0001] directory                                     dir=/tmp/manifests-269764601 file=manifests-269764601 load=package
error: unable to parse image quay.io/operator-framework/operator-registry-server:latest: cannot retrieve image configuration for manifest sha256:a90004a32cb71af42b2d5b1bf24c6e054a240a1e985ebd896092a09672073774: Get https://d3uo42mtx6z2cr.cloudfront.net/sha256/38/38e840c2b9331a01f41c35e001b9800dc7d49b3a3dae38cb5f73bb0d55bd0281?Expires=1574067578&Signature=cb79cmMdP0Jyf~vRnjS1o9NTTLqqFrnIX233xqKoMK9T31ubJT1TfX6pjnOAiZqYQrh7Rz2M7nDaRYqA2NiGizrTpTUG-AmeKGbQ4lUc11GxYzFEocT1wHPi6wE881XZYl4DREAh81lE4QpKwUQVoKDf-aTUqgx~6PWwq7Dn1R5vaunBeQBIcrsjZ0Bk0imqfMoCPTe2nIqQpin0MH1VuXQlRiorX9dXT8gGTOivir1yNZ12eZGRJC~L-w73KbOiOcZ4AUn2Yk8XEq3OWsDDbbMOX-a6yC6bKLoEicsxthBARfMLoybbqMYZumWQ4UYZVK~UMTgqVhl2Yxhnfnd5UA__&Key-Pair-Id=APKAJ67PQLWGCSP66DGA: dial tcp: lookup d3uo42mtx6z2cr.cloudfront.net on 127.0.0.1:53: server misbehaving

Expected results:
This command can work well in the restricted cluster.

Additional info:
[root@dhcp-140-36 ~]# oc version --loglevel=8
I1118 17:00:56.035705   29899 loader.go:375] Config loaded from file:  /root/43-disconnect-kubeconfig
I1118 17:00:56.036319   29899 round_trippers.go:420] GET https://api.qe-yapei1835.qe.devcluster.openshift.com:6443/version?timeout=32s
I1118 17:00:56.036327   29899 round_trippers.go:427] Request Headers:
I1118 17:00:56.036331   29899 round_trippers.go:431]     Accept: application/json, */*
I1118 17:00:56.036334   29899 round_trippers.go:431]     User-Agent: oc/v0.0.0 (linux/amd64) kubernetes/$Format
I1118 17:00:56.956312   29899 round_trippers.go:446] Response Status:  in 919 milliseconds
I1118 17:00:56.956346   29899 round_trippers.go:449] Response Headers:
I1118 17:00:56.956527   29899 round_trippers.go:420] GET https://api.qe-yapei1835.qe.devcluster.openshift.com:6443/apis/config.openshift.io/v1/clusteroperators/openshift-apiserver
I1118 17:00:56.956556   29899 round_trippers.go:427] Request Headers:
I1118 17:00:56.956580   29899 round_trippers.go:431]     Accept: application/json, */*
I1118 17:00:56.956595   29899 round_trippers.go:431]     User-Agent: oc/v0.0.0 (linux/amd64) kubernetes/$Format
I1118 17:00:57.886491   29899 round_trippers.go:446] Response Status:  in 929 milliseconds
I1118 17:00:57.886523   29899 round_trippers.go:449] Response Headers:
Client Version: openshift-clients-4.3.0-201910250623-48-g8c30708e
I1118 17:00:57.886601   29899 helpers.go:217] Connection error: Get https://api.qe-yapei1835.qe.devcluster.openshift.com:6443/apis/config.openshift.io/v1/clusteroperators/openshift-apiserver: dial tcp: lookup api.qe-yapei1835.qe.devcluster.openshift.com on 127.0.0.1:53: server misbehaving
F1118 17:00:57.886645   29899 helpers.go:114] Unable to connect to the server: dial tcp: lookup api.qe-yapei1835.qe.devcluster.openshift.com on 127.0.0.1:53: server misbehaving

Comment 1 Jian Zhang 2019-11-18 09:28:14 UTC
When I logout the cluster, and rerun it. It failed to pull, push from the private registry namespace: redhat-operators-art/art. But, it works after login the cluster.
[root@dhcp-140-36 ~]# oc get clusterversion
error: You must be logged in to the server (Unauthorized)
[root@dhcp-140-36 ~]# docker login quay.io
Username (jiazha): 
Password: 
Login Succeeded
[root@dhcp-140-36 ~]# oc adm catalog build --appregistry-org=redhat-operators-art --to=quay.io/redhat-operators-art/art:v1 --loglevel=8
INFO[0001] loading Bundles                               dir=/tmp/manifests-429810255
INFO[0001] directory                                     dir=/tmp/manifests-429810255 file=manifests-429810255 load=bundles
INFO[0001] loading Packages and Entries                  dir=/tmp/manifests-429810255
INFO[0001] directory                                     dir=/tmp/manifests-429810255 file=manifests-429810255 load=package
...
...
Uploading ... I1118 17:21:19.906344   30111 credentials.go:108] Found secret to match https://quay.io/v2/auth (quay.io/auth): 
I1118 17:21:19.906386   30111 round_trippers.go:420] GET https://quay.io/v2/auth?account=jiazha&scope=repository%3Aredhat-operators-art%2Fart%3Apull%2Cpush&service=quay.io
I1118 17:21:19.906400   30111 round_trippers.go:427] Request Headers:
I1118 17:21:19.906416   30111 round_trippers.go:431]     Authorization: Basic <masked>
I1118 17:21:20.459863   30111 round_trippers.go:446] Response Status: 200 OK in 553 milliseconds
I1118 17:21:20.459881   30111 round_trippers.go:449] Response Headers:
I1118 17:21:20.459889   30111 round_trippers.go:452]     Cache-Control: no-cache, no-store, must-revalidate
I1118 17:21:20.459899   30111 round_trippers.go:452]     X-Frame-Options: DENY
I1118 17:21:20.459903   30111 round_trippers.go:452]     Strict-Transport-Security: max-age=63072000; preload
I1118 17:21:20.459909   30111 round_trippers.go:452]     Server: nginx/1.12.1
I1118 17:21:20.459917   30111 round_trippers.go:452]     Date: Mon, 18 Nov 2019 09:21:20 GMT
I1118 17:21:20.459922   30111 round_trippers.go:452]     Content-Type: application/json
I1118 17:21:20.459932   30111 round_trippers.go:452]     Content-Length: 1031
I1118 17:21:20.460000   30111 round_trippers.go:420] POST https://quay.io/v2/redhat-operators-art/art/blobs/uploads/
I1118 17:21:20.460012   30111 round_trippers.go:427] Request Headers:
I1118 17:21:20.460022   30111 round_trippers.go:431]     Content-Type: 
I1118 17:21:20.460032   30111 round_trippers.go:431]     Authorization: Bearer <masked>
I1118 17:21:20.725834   30111 round_trippers.go:446] Response Status: 401 Unauthorized in 265 milliseconds
I1118 17:21:20.725865   30111 round_trippers.go:449] Response Headers:
I1118 17:21:20.725884   30111 round_trippers.go:452]     Docker-Distribution-Api-Version: registry/2.0
I1118 17:21:20.725900   30111 round_trippers.go:452]     Www-Authenticate: Bearer realm="https://quay.io/v2/auth",service="quay.io",scope="repository:redhat-operators-art/art:pull,push"
I1118 17:21:20.725914   30111 round_trippers.go:452]     Server: nginx/1.12.1
I1118 17:21:20.725927   30111 round_trippers.go:452]     Date: Mon, 18 Nov 2019 09:21:20 GMT
I1118 17:21:20.725940   30111 round_trippers.go:452]     Content-Type: application/json
I1118 17:21:20.725954   30111 round_trippers.go:452]     Content-Length: 112
failed
F1118 17:21:20.726201   30111 helpers.go:114] error: unauthorized: access to the requested resource is not authorized

Comment 4 Kevin Rizza 2019-11-21 14:52:22 UTC
Jian,

Yes, that image needs to be mirrored to the cluster as well. The `oc adm catalog` commands are built on top of the concepts from `oc adm release mirror` commands. If you want to run this catalog build *inside* the disconnected cluster, you would need to do what Evan described above and mirror that image into your disconnected registry.

Is there a reason why you are trying to do the catalog build from inside a disconnected environment? My assumption is that this command to build the disconnected catalog should be run from *outside* the disconnected environment (so that it has access to the required builder images as well as access to quay.io's appregistry. Then, once the image is built using `oc adm catalog build`, you can mirror that image into the disconnected environment's registry and attempt to create a catalogsource pointing to it.

Am I missing something here?

https://docs.openshift.com/container-platform/4.2/installing/installing_restricted_networks/installing-restricted-networks-preparations.html?extIdCarryOver=true&sc_cid=701f2000001Css5AAC
https://github.com/operator-framework/olm-book/pull/13

Comment 6 Jian Zhang 2019-11-25 02:26:02 UTC
Hi, Kevin

Thanks for your information.

> Is there a reason why you are trying to do the catalog build from inside a disconnected environment?

You know, some customers only have one disconnected environment. Previously, I hope this `oc adm catalog` command can work well too in the disconnected cluster to avoid the users do unneeded steps.
I see now, this `oc adm catalog` command is a precondition before starting to use the disconnected cluster. The users should use it to build the images first, right? If yes, the correct steps for the users as follows, right? If yes, I think I can verify this issue, thanks!
1, Logout the cluster(Don't log in to the disconnected cluster, otherwise, you cannot get the depended image: quay.io/operator-framework/operator-registry-server:latest).
2, Run the `oc adm catalog build --auth-token="basic xxx" --appregistry-org=<your registry> --to=<your image>` command to build your images

Comment 8 Jian Zhang 2019-11-26 02:14:48 UTC
Hi, Nick

Thanks for your information! Verify it per comment 4.

Comment 10 errata-xmlrpc 2020-01-23 11:13:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0062


Note You need to log in before you can comment on or make changes to this bug.