Bug 1823143 - oc adm release extract --command, --tools doesn't pull from localregistry when given a localregistry/image
Summary: oc adm release extract --command, --tools doesn't pull from localregistry whe...
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: oc
Version: 4.4
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.11.0
Assignee: Maciej Szulik
QA Contact: zhou ying
Whiteboard: KNI-EDGE-4.8
: 1812814 1814495 1827101 1957126 1972660 (view as bug list)
Depends On:
Blocks: 2069976
TreeView+ depends on / blocked
Reported: 2020-04-11 23:34 UTC by Sally
Modified: 2022-08-10 10:36 UTC (History)
43 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Last Closed: 2022-08-10 10:35:34 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift oc pull 1169 0 None open Bug 1823143: add --icsp-file option to adm release subcommands 2022-06-14 19:25:16 UTC
Github openshift oc pull 395 0 None closed Bug 1823143: Try user-given registry/repository for oc adm release extract|info|mirror before defaulting to referenced ... 2021-02-16 06:48:40 UTC
Github openshift oc pull 404 0 None closed Bug 1823143: Try user-given registry/repository for extract before defaulting to referenced image (take 2) 2021-02-16 06:48:40 UTC
Github openshift oc pull 829 0 None Merged Bug 1823143: wire ICSP lookups to oc image info 2022-05-23 18:30:51 UTC
Red Hat Knowledge Base (Solution) 5380841 0 None None None 2020-09-08 15:07:38 UTC
Red Hat Product Errata RHSA-2022:5069 0 None None None 2022-08-10 10:36:06 UTC

Internal Links: 1777890 2069976

Description Sally 2020-04-11 23:34:50 UTC
To reproduce: 
start a local registry
$ podman run -d -p 5000:5000 --restart=always --name registry registry:2

$ mkdir mirror_4.4.0
mirror the release to local disk
$ oc adm release mirror -a ~/pull-secret --from=quay.io/openshift-release-dev/ocp-release:4.4.0-rc.6-x86_64 --to file://mirror_4.4.0

mirror the release to local registry
$ oc image mirror --insecure=true 'file://mirror_4.4.0:4.4.0-rc.6*' localhost:5000/ocp4/openshift4

verify image is in localregistry
$ oc adm release --insecure=true info localhost:5000/ocp4/openshift4:4.4.0-rc.6

extract tools from localregistry/release 
$ oc adm release extract --tools --insecure=true --from=localhost:5000/ocp4/openshift4:4.4.0-rc.6 

extract fails, asks for quay credentials - succeeds to pull from quay when passed quay pull-secret

oc adm release extract --tools --from localregistry/release:local should pull from localregistry.

Comment 2 Sally 2020-04-20 18:51:01 UTC
I'm investigating this, but here is a workaround:
(relisting steps from original description, as there has been change due to code merge):

1) start local registry
$ podman run -p 5000:5000 -d registry:2

2) mirror a nightly release to local disk
$ oc adm release mirror -a ~/your/pull-secret --from=registry.svc.ci.openshift.org/ocp/release:4.5.0-0.nightly-2020-04-18-093630 --to file://mirror_4.5.0 --release-image-signature-to-dir=test

from now only disconnected steps
3) mirror localdisk nightly release to local registry 
$ oc image mirror --insecure=true 'file://mirror_4.5.0:4.5.0-0.nightly-2020-04-18-093630*' localhost:5000/ocp/release

4) extract openshift-install binary from local registry (sudo necessary with mounts)
$ ctr=$(sudo podman run -d --rm --entrypoint /bin/sh localhost:5000/ocp/release:4.5.0-0.nightly-2020-04-18-093630-installer)
$ mnt=$(sudo podman mount $ctr)  
$ sudo cp -R ${mnt}/bin/openshift-install /some/localdir
$ sudo chown $(whoami):$(whoami) /some/localdir/openshift-install

Comment 3 Sally 2020-04-20 19:37:19 UTC
Please disregard that above comment, that will not extract the binary necessary for a disconnected install- 'oc adm release extract' modifies the binary to pin the correct release image, while the above does not.  still investigating...

Comment 6 Maciej Szulik 2020-04-23 11:33:53 UTC
*** Bug 1827101 has been marked as a duplicate of this bug. ***

Comment 7 Maciej Szulik 2020-05-05 13:32:20 UTC
*** Bug 1812814 has been marked as a duplicate of this bug. ***

Comment 8 Maciej Szulik 2020-05-11 15:57:03 UTC
*** Bug 1814495 has been marked as a duplicate of this bug. ***

Comment 10 Sally 2020-05-20 18:07:29 UTC
The PR attached here: https://github.com/openshift/oc/pull/427  while it resolves the issue here it is not the correct path forward.

In order to resolve this properly, we're going to add ImageContentSources awareness to oc, I've opened a jira for this here: https://issues.redhat.com/browse/WRKLDS-174

Please track this through jira

Comment 11 Pablo Alonso Rodriguez 2020-05-21 07:27:15 UTC
Not being properly aware of a local registry is a bug, not a RFE. 

Whether the way to fix it is the pull request or making oc aware of ImageContentSources is a detail on how the fix will be implemented, it does not mean it is not a bug.

Hence I am reopening this bug.

Comment 12 Maciej Szulik 2020-05-21 10:19:37 UTC
Moving then to the next release since we're not going to be able to address this at this point in time.

Comment 14 Sally 2020-06-18 14:30:05 UTC
This bug is actively being worked on.

Comment 15 Sally 2020-07-10 19:15:40 UTC
I’m adding UpcomingSprint, because there are PRs for this bug that are waiting for review.  This is actively being worked on.

Comment 16 Sally 2020-07-30 21:36:56 UTC
I’m adding UpcomingSprint, because there are PRs for this bug that are waiting for review.  This is actively being worked on.

Comment 17 Michal Fojtik 2020-08-20 11:47:18 UTC
This bug hasn't had any activity in the last 30 days. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. As such, we're marking this bug as "LifecycleStale" and decreasing the severity/priority. If you have further information on the current state of the bug, please update it, otherwise this bug can be closed in about 7 days. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant.

Comment 18 Sally 2020-08-20 21:01:08 UTC
The PR to resolve this issue is waiting for review/merge https://github.com/openshift/oc/pull/439.  The necessary enhancement has been approved. Removing LifecycleStale as it's actively being worked on.

Comment 19 Michal Fojtik 2020-08-24 08:32:22 UTC
The LifecycleStale keyword was removed because the bug got commented on recently.
The bug assignee was notified.

Comment 21 Sally 2020-09-11 15:07:21 UTC
This is a large change, so the review process is taking time.  I'm working through the review process and will update here and this PR accordingly: https://github.com/openshift/oc/pull/439

Comment 22 Maciej Szulik 2020-09-15 13:32:59 UTC
ICSP topic is being deferred until 4.7.

Comment 23 Sally 2020-10-01 21:24:00 UTC
Other bugs took precedence last sprint and this continues to wait for review - adding UpcomingSprint tag.

Comment 24 Sally 2020-10-23 18:02:09 UTC
Waiting on PR review, as other bugs took priority last sprint, adding UpcomingSprint.

Comment 25 Sally 2020-11-12 17:02:41 UTC
Actively working on this, PR is under review, adding UpcomingSprint.

Comment 26 Sally 2020-12-05 00:43:35 UTC
PR is actively being reviewed, adding upcoming sprint

Comment 27 Sally 2021-01-15 15:20:22 UTC
PR actively being reviewed (still), adding upcoming sprint

Comment 28 Maciej Szulik 2021-02-08 12:23:22 UTC
Pushing this over to 4.8 since 4.7 is closed.

Comment 36 Yu Qi Zhang 2021-05-19 22:04:20 UTC
*** Bug 1957126 has been marked as a duplicate of this bug. ***

Comment 46 Andreas Karis 2021-06-16 11:55:29 UTC
*** Bug 1972660 has been marked as a duplicate of this bug. ***

Comment 49 Ken Young 2021-07-05 21:32:26 UTC
What is the status of the review in progress?

Comment 50 Maciej Szulik 2021-07-06 15:06:54 UTC
(In reply to Ken Young from comment #49)
> What is the status of the review in progress?

It's still work-in-progress.

Comment 58 Maciej Szulik 2021-10-20 18:31:40 UTC
I've updated https://github.com/openshift/oc/pull/829 with all comments addressed, waiting for reviews.

Comment 71 zhou ying 2022-04-15 07:51:50 UTC
Still could reproduce the issue :

oc version --client
Client Version: 4.11.0-202204141741.p0.g1b0e16c.assembly.stream-1b0e16c

1. create the local registry:
    `podman run --volume /home/registry:/var/lib/registry/docker/registry -d -p 5000:5000 --restart=always --privileged --name registry registry`
2. create dir and mirror the release to local disk:
    ` mkdir mirror_4.10.9` ; `cd mirror_4.10.9`, `oc adm release mirror  --from=quay.io/openshift-release-dev/ocp-release:4.10.9-x86_64 --to file://openshift/release`
3. mirror the release to local registry :
   `oc image mirror --insecure=true 'file://openshift/release:4.10.9-x86_64*'  localhost:5000/ocp4`
4. remove the quay credentials:
    `mv /root/.docker/config.json /root/.docker/config.json.13`
5. try to extract from local registry , but still failed:
 oc adm release extract --tools --insecure=true --from=localhost:5000/ocp4@sha256:39f360002b9b5c730d1167879ad6437352d51e72acc9fe80add3ec2a0d20400d
Warning: the default reading order of registry auth file will be changed from "${HOME}/.docker/config.json" to podman registry config locations in the future version. "${HOME}/.docker/config.json" is deprecated, but can still be used for storing credentials as a fallback. See https://github.com/containers/image/blob/main/docs/containers-auth.json.5.md for the order of podman registry config locations.
error: unable to read image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9456cdc44def5f31fc42f7534eade5b7ef45b6e6c9e83730eb03eb3f35196c56: unauthorized: access to the requested resource is not authorized

Comment 72 Eran Cohen 2022-04-24 12:16:45 UTC
I have the same issue as well, it seems that oc is trying to access quay.io instead of the mirror registry:
./oc adm -v=3 release extract --command=openshift-install --to=./ --insecure=false edge-01.edge.lab.eng.rdu2.redhat.com:5000/ocp:4.8.18 --registry-config=/tmp/registry.json
I0424 08:14:25.430489  304682 extract_tools.go:380] Skipping openshift-install-mac-%s.tar.gz, does not match current OS darwin
I0424 08:14:25.430530  304682 extract_tools.go:380] Skipping openshift-install-mac-arm64-%s.tar.gz, does not match current OS darwin
I0424 08:14:25.430535  304682 extract_tools.go:398] Will extract usr/bin/openshift-install from quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:18d97ab33fd127a72e508ce220e06d0c3657657e9f923bc9f7b98ea0be5a8bf1
I0424 08:14:25.430552  304682 extract_tools.go:390] Skipping duplicate openshift-install-linux-amd64-%s.tar.gz
error: unable to read image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:18d97ab33fd127a72e508ce220e06d0c3657657e9f923bc9f7b98ea0be5a8bf1: Get "https://quay.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Also, from the linked PR https://github.com/openshift/oc/pull/829 it seems that the ICSP option was added to `oc image info` and not `oc adm release extract`

Comment 73 Venkat B 2022-04-25 17:16:01 UTC
I see the same issue in our environment where Assisted Installer is used in disconnected environment.
We are using OCP Version: 4.8.29-assembly.art3875-x86_64.

Following are the versions of the AI Container Images in use =>

$ podman images
REPOSITORY                                                 TAG         IMAGE ID      CREATED       SIZE
quay.io/edge-infrastructure/assisted-service               latest      7a9a88cff3a4  2 weeks ago   849 MB
quay.io/edge-infrastructure/assisted-installer-ui          latest      debd6d4eec22  2 weeks ago   488 MB
quay.io/centos7/postgresql-12-centos7                      latest      d57e7b296b6b  2 weeks ago   382 MB
quay.io/edge-infrastructure/assisted-installer-controller  latest      19284a03a80d  2 weeks ago   731 MB
quay.io/edge-infrastructure/assisted-installer             latest      401d7e12fb74  2 weeks ago   470 MB
quay.io/edge-infrastructure/assisted-installer-agent       latest      e170df2ccc3e  2 weeks ago   1.03 GB
quay.io/edge-infrastructure/assisted-image-service         latest      3df5716c00ae  4 weeks ago   417 MB

Installation failed saying (from Cluster Events logs) =>

Failed to prepare the installation due to an unexpected error: failed generating install config for cluster 2f1de184-eec0-42ff-b286-6bdfe3b33aef: failed to get installer path: command 'oc adm release extract --command=openshift-baremetal-install --to=/data/install-config-generate/installercache/iss.zarya.net:5000/ocp-release:4.8.29-assembly.art3875-x86_64 --insecure=false iss.zarya.net:5000/ocp-release:4.8.29-assembly.art3875-x86_64 --registry-config=/tmp/registry-config1255408649' exited with non-zero exit code 1: error: unable to read image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9708c9c905e87646d606c4dcac1deee6314305379465b7a9646360be173e074e: Get "https://quay.io/v2/": dial tcp: lookup quay.io on server misbehaving . Please retry later

Comment 74 Eran Cohen 2022-04-26 09:06:39 UTC
venkatasubramanian.b note that a fix allowing assisted-service to workaround for this issue merged 2 days ago - https://github.com/openshift/assisted-service/pull/3700
So it should work once you update the assisted-service image or redeploy.

Comment 75 Venkat B 2022-04-27 09:07:58 UTC
Thankyou Eran. I have taken the latest assisted-service image (also all other needed images as well) and I confirm that I no more see that issue. Our cluster is now successfully commissioned.

Comment 78 Zane Bitter 2022-05-24 18:40:40 UTC
As mentioned by comment 71 and comment 72, we also need this implemented for the `oc adm release extract` command.

The assisted-service is currently using a forked version of oc to work around this problem, but we also need it urgently in 4.11 for the ephemeral agent installer, and that cannot use anything other than the version of oc shipped in the release payload.

Comment 79 Michael Filanov 2022-06-09 12:55:10 UTC
Any updates about this effort?

Comment 81 Maciej Szulik 2022-06-21 10:14:50 UTC
I think we have all the commands we cared about, moving to modified.

Comment 84 zhou ying 2022-06-23 12:11:22 UTC
verified with latest oc client:

oc version --client
Client Version: 4.11.0-0.nightly-2022-06-23-092832
Kustomize Version: v4.5.4

1. create the local registry:
    `podman run --volume /home/registry:/var/lib/registry/docker/registry -d -p 5000:5000 --restart=always --privileged --name registry registry`
2. mirror image to local registry :
    `oc adm release mirror registry.ci.openshift.org/ocp/release:4.11.0-0.nightly-2022-06-23-044003  --to='localhost:5000/ocp4' --insecure`

3. Create the icsp file from the output of `oc adm release mirror` command ;

apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
  name: example
  - mirrors:
    - localhost:5000/ocp4
    source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
  - mirrors:
    - localhost:5000/ocp4
    source: registry.ci.openshift.org/ocp/release
4. Use the icsp file to extract from local registry:

oc adm release extract --command='oc'   --icsp-file=../icsp.yaml  localhost:5000/ocp4@sha256:5cc4c2912c03b49bf9b8b2b4e36319c467c1483f30689291a87e05097e6844bb  -v 5 --insecure
I0623 20:09:10.345600   74658 config.go:127] looking for config.json at /root/.docker/config.json
I0623 20:09:10.345679   74658 config.go:93] looking for .dockercfg at /root/.dockercfg
I0623 20:09:10.345973   74658 client_mirrored.go:174] Attempting to connect to localhost:5000/ocp4
I0623 20:09:10.347256   74658 client.go:322] Falling back to an HTTP check for an insecure registry https://localhost:5000: Get "https://localhost:5000/v2/": http: server gave HTTP response to HTTPS client
I0623 20:09:10.352634   74658 client_mirrored.go:412] get manifest for sha256:5cc4c2912c03b49bf9b8b2b4e36319c467c1483f30689291a87e05097e6844bb served from registryclient.retryManifest{ManifestService:registryclient.manifestServiceVerifier{ManifestService:(*client.manifests)(0xc000911800)}, repo:(*registryclient.retryRepository)(0xc0000fa800)}: <nil>
I0623 20:09:10.352685   74658 client_mirrored.go:174] Attempting to connect to localhost:5000/ocp4
I0623 20:09:10.354882   74658 client_mirrored.go:445] get for sha256:412338ccddcdb725ecd375371d5488d0290be6f52e36b6e14cbd3107c0bf1770 served from ocp4: <nil>
I0623 20:09:10.354928   74658 manifest.go:312] Raw image config json:
{"id":"","created":"2022-06-23T04:45:27Z","container_config":{},"docker_version":"1.13.1","config":{"Hostname":"d300d7450dc1","Env":["__doozer=merge","BUILD_RELEASE=202206221808.p0.gdc927a4.assembly.stream","BUILD_VERSION=v4.11.0","OS_GIT_MAJOR=4","OS_GIT_MINOR=11","OS_GIT_PATCH=0","OS_GIT_TREE_STATE=clean","OS_GIT_VERSION=4.11.0-202206221808.p0.gdc927a4.assembly.stream-dc927a4","SOURCE_GIT_TREE_STATE=clean","OS_GIT_COMMIT=dc927a4","SOURCE_DATE_EPOCH=1655917064","SOURCE_GIT_COMMIT=dc927a4c63e2d9fb7f469ecb77503687a60c6564","SOURCE_GIT_TAG=v1.0.0-842-gdc927a4c","SOURCE_GIT_URL=https://github.com/openshift/cluster-version-operator","GODEBUG=x509ignoreCN=0,madvdontneed=1","PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","container=oci"],"Entrypoint":["/usr/bin/cluster-version-operator"],"Labels":{"io.openshift.release":"4.11.0-0.nightly-2022-06-23-044003","io.openshift.release.base-image-digest":"sha256:c4e86576fd8f3e02db9b0d7d883dbc966caf87336609bbcfbdf166b057d8b822"}},"architecture":"amd64","size":118249335,"rootfs":{"type":"layers","diff_ids":["sha256:5bf135c4a0de07e52c11282c0954e3e6b7c7ddc6c8834a7fd2803c3dc6a31a69","sha256:773711fd02f009e3bc5f9e2b1e859bf2103ba7318b3eb73390490afb3a3a8848","sha256:0c728cbdd06d5172e1e62858b7e8ed70b73457271b4aa4d8dd3718ef1dee8944","sha256:06ea8291551e7828cf59b048353083ca5dbee0299735699b065228677221e267","sha256:dbfca6502f224f1f14c180c5e44ab6accb743eae622b7e0c047bc4d3f59e7842","sha256:225fd4443357f1160f99cd07b7b46df71c8bf391421a1a57e91f088c572477eb"]},"history":[{"created":"2022-06-23T04:45:27Z","comment":"Release image for OpenShift"},{"created":"2022-06-23T04:45:27Z"},{"created":"2022-06-23T04:45:27Z"},{"created":"2022-06-23T04:45:27Z"},{"created":"2022-06-23T04:45:27Z"},{"created":"2022-06-23T04:45:27Z"}],"os":"linux"}
I0623 20:09:10.355192   74658 extract.go:484] Extracting from layer: distribution.Descriptor{MediaType:"application/vnd.docker.image.rootfs.diff.tar.gzip", Size:877142, Digest:"sha256:5dce837222ce844794f18ab4f4774210b255d885512863050246978421a9ce80", URLs:[]string(nil), Annotations:map[string]string(nil), Platform:(*v1.Platform)(nil)}
I0623 20:09:10.355227   74658 client_mirrored.go:174] Attempting to connect to localhost:5000/ocp4
I0623 20:09:10.357457   74658 client_mirrored.go:485] open (read) sha256:5dce837222ce844794f18ab4f4774210b255d885512863050246978421a9ce80 from ocp4: <nil>
I0623 20:09:10.358134   74658 extract.go:679] Exclude release-manifests due to missing prefix release-manifests/
I0623 20:09:10.358156   74658 extract.go:551] Exclude entry release-manifests 35 0

Comment 85 errata-xmlrpc 2022-08-10 10:35:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.