Hide Forgot
To reproduce: start a local registry $ podman run -d -p 5000:5000 --restart=always --name registry registry:2 $ mkdir mirror_4.4.0 mirror the release to local disk $ oc adm release mirror -a ~/pull-secret --from=quay.io/openshift-release-dev/ocp-release:4.4.0-rc.6-x86_64 --to file://mirror_4.4.0 mirror the release to local registry $ oc image mirror --insecure=true 'file://mirror_4.4.0:4.4.0-rc.6*' localhost:5000/ocp4/openshift4 verify image is in localregistry $ oc adm release --insecure=true info localhost:5000/ocp4/openshift4:4.4.0-rc.6 extract tools from localregistry/release $ oc adm release extract --tools --insecure=true --from=localhost:5000/ocp4/openshift4:4.4.0-rc.6 extract fails, asks for quay credentials - succeeds to pull from quay when passed quay pull-secret oc adm release extract --tools --from localregistry/release:local should pull from localregistry.
I'm investigating this, but here is a workaround: (relisting steps from original description, as there has been change due to code merge): 1) start local registry $ podman run -p 5000:5000 -d registry:2 2) mirror a nightly release to local disk $ oc adm release mirror -a ~/your/pull-secret --from=registry.svc.ci.openshift.org/ocp/release:4.5.0-0.nightly-2020-04-18-093630 --to file://mirror_4.5.0 --release-image-signature-to-dir=test from now only disconnected steps 3) mirror localdisk nightly release to local registry $ oc image mirror --insecure=true 'file://mirror_4.5.0:4.5.0-0.nightly-2020-04-18-093630*' localhost:5000/ocp/release 4) extract openshift-install binary from local registry (sudo necessary with mounts) $ ctr=$(sudo podman run -d --rm --entrypoint /bin/sh localhost:5000/ocp/release:4.5.0-0.nightly-2020-04-18-093630-installer) $ mnt=$(sudo podman mount $ctr) $ sudo cp -R ${mnt}/bin/openshift-install /some/localdir $ sudo chown $(whoami):$(whoami) /some/localdir/openshift-install
Please disregard that above comment, that will not extract the binary necessary for a disconnected install- 'oc adm release extract' modifies the binary to pin the correct release image, while the above does not. still investigating...
*** Bug 1827101 has been marked as a duplicate of this bug. ***
*** Bug 1812814 has been marked as a duplicate of this bug. ***
*** Bug 1814495 has been marked as a duplicate of this bug. ***
The PR attached here: https://github.com/openshift/oc/pull/427 while it resolves the issue here it is not the correct path forward. In order to resolve this properly, we're going to add ImageContentSources awareness to oc, I've opened a jira for this here: https://issues.redhat.com/browse/WRKLDS-174 Please track this through jira
Not being properly aware of a local registry is a bug, not a RFE. Whether the way to fix it is the pull request or making oc aware of ImageContentSources is a detail on how the fix will be implemented, it does not mean it is not a bug. Hence I am reopening this bug.
Moving then to the next release since we're not going to be able to address this at this point in time.
This bug is actively being worked on.
I’m adding UpcomingSprint, because there are PRs for this bug that are waiting for review. This is actively being worked on.
This bug hasn't had any activity in the last 30 days. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. As such, we're marking this bug as "LifecycleStale" and decreasing the severity/priority. If you have further information on the current state of the bug, please update it, otherwise this bug can be closed in about 7 days. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant.
The PR to resolve this issue is waiting for review/merge https://github.com/openshift/oc/pull/439. The necessary enhancement has been approved. Removing LifecycleStale as it's actively being worked on.
The LifecycleStale keyword was removed because the bug got commented on recently. The bug assignee was notified.
This is a large change, so the review process is taking time. I'm working through the review process and will update here and this PR accordingly: https://github.com/openshift/oc/pull/439
ICSP topic is being deferred until 4.7.
Other bugs took precedence last sprint and this continues to wait for review - adding UpcomingSprint tag.
Waiting on PR review, as other bugs took priority last sprint, adding UpcomingSprint.
Actively working on this, PR is under review, adding UpcomingSprint.
PR is actively being reviewed, adding upcoming sprint
PR actively being reviewed (still), adding upcoming sprint
Pushing this over to 4.8 since 4.7 is closed.
*** Bug 1957126 has been marked as a duplicate of this bug. ***
*** Bug 1972660 has been marked as a duplicate of this bug. ***
What is the status of the review in progress?
(In reply to Ken Young from comment #49) > What is the status of the review in progress? It's still work-in-progress.
I've updated https://github.com/openshift/oc/pull/829 with all comments addressed, waiting for reviews.
Still could reproduce the issue : oc version --client Client Version: 4.11.0-202204141741.p0.g1b0e16c.assembly.stream-1b0e16c 1. create the local registry: `podman run --volume /home/registry:/var/lib/registry/docker/registry -d -p 5000:5000 --restart=always --privileged --name registry registry` 2. create dir and mirror the release to local disk: ` mkdir mirror_4.10.9` ; `cd mirror_4.10.9`, `oc adm release mirror --from=quay.io/openshift-release-dev/ocp-release:4.10.9-x86_64 --to file://openshift/release` 3. mirror the release to local registry : `oc image mirror --insecure=true 'file://openshift/release:4.10.9-x86_64*' localhost:5000/ocp4` 4. remove the quay credentials: `mv /root/.docker/config.json /root/.docker/config.json.13` 5. try to extract from local registry , but still failed: oc adm release extract --tools --insecure=true --from=localhost:5000/ocp4@sha256:39f360002b9b5c730d1167879ad6437352d51e72acc9fe80add3ec2a0d20400d Warning: the default reading order of registry auth file will be changed from "${HOME}/.docker/config.json" to podman registry config locations in the future version. "${HOME}/.docker/config.json" is deprecated, but can still be used for storing credentials as a fallback. See https://github.com/containers/image/blob/main/docs/containers-auth.json.5.md for the order of podman registry config locations. error: unable to read image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9456cdc44def5f31fc42f7534eade5b7ef45b6e6c9e83730eb03eb3f35196c56: unauthorized: access to the requested resource is not authorized
I have the same issue as well, it seems that oc is trying to access quay.io instead of the mirror registry: ./oc adm -v=3 release extract --command=openshift-install --to=./ --insecure=false edge-01.edge.lab.eng.rdu2.redhat.com:5000/ocp:4.8.18 --registry-config=/tmp/registry.json I0424 08:14:25.430489 304682 extract_tools.go:380] Skipping openshift-install-mac-%s.tar.gz, does not match current OS darwin I0424 08:14:25.430530 304682 extract_tools.go:380] Skipping openshift-install-mac-arm64-%s.tar.gz, does not match current OS darwin I0424 08:14:25.430535 304682 extract_tools.go:398] Will extract usr/bin/openshift-install from quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:18d97ab33fd127a72e508ce220e06d0c3657657e9f923bc9f7b98ea0be5a8bf1 I0424 08:14:25.430552 304682 extract_tools.go:390] Skipping duplicate openshift-install-linux-amd64-%s.tar.gz error: unable to read image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:18d97ab33fd127a72e508ce220e06d0c3657657e9f923bc9f7b98ea0be5a8bf1: Get "https://quay.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Also, from the linked PR https://github.com/openshift/oc/pull/829 it seems that the ICSP option was added to `oc image info` and not `oc adm release extract`
I see the same issue in our environment where Assisted Installer is used in disconnected environment. We are using OCP Version: 4.8.29-assembly.art3875-x86_64. Following are the versions of the AI Container Images in use => $ podman images REPOSITORY TAG IMAGE ID CREATED SIZE quay.io/edge-infrastructure/assisted-service latest 7a9a88cff3a4 2 weeks ago 849 MB quay.io/edge-infrastructure/assisted-installer-ui latest debd6d4eec22 2 weeks ago 488 MB quay.io/centos7/postgresql-12-centos7 latest d57e7b296b6b 2 weeks ago 382 MB quay.io/edge-infrastructure/assisted-installer-controller latest 19284a03a80d 2 weeks ago 731 MB quay.io/edge-infrastructure/assisted-installer latest 401d7e12fb74 2 weeks ago 470 MB quay.io/edge-infrastructure/assisted-installer-agent latest e170df2ccc3e 2 weeks ago 1.03 GB quay.io/edge-infrastructure/assisted-image-service latest 3df5716c00ae 4 weeks ago 417 MB Installation failed saying (from Cluster Events logs) => Failed to prepare the installation due to an unexpected error: failed generating install config for cluster 2f1de184-eec0-42ff-b286-6bdfe3b33aef: failed to get installer path: command 'oc adm release extract --command=openshift-baremetal-install --to=/data/install-config-generate/installercache/iss.zarya.net:5000/ocp-release:4.8.29-assembly.art3875-x86_64 --insecure=false iss.zarya.net:5000/ocp-release:4.8.29-assembly.art3875-x86_64 --registry-config=/tmp/registry-config1255408649' exited with non-zero exit code 1: error: unable to read image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9708c9c905e87646d606c4dcac1deee6314305379465b7a9646360be173e074e: Get "https://quay.io/v2/": dial tcp: lookup quay.io on 100.2.1.9:53: server misbehaving . Please retry later
venkatasubramanian.b note that a fix allowing assisted-service to workaround for this issue merged 2 days ago - https://github.com/openshift/assisted-service/pull/3700 So it should work once you update the assisted-service image or redeploy.
Thankyou Eran. I have taken the latest assisted-service image (also all other needed images as well) and I confirm that I no more see that issue. Our cluster is now successfully commissioned.
As mentioned by comment 71 and comment 72, we also need this implemented for the `oc adm release extract` command. The assisted-service is currently using a forked version of oc to work around this problem, but we also need it urgently in 4.11 for the ephemeral agent installer, and that cannot use anything other than the version of oc shipped in the release payload.
Any updates about this effort?
I think we have all the commands we cared about, moving to modified.
verified with latest oc client: oc version --client Client Version: 4.11.0-0.nightly-2022-06-23-092832 Kustomize Version: v4.5.4 1. create the local registry: `podman run --volume /home/registry:/var/lib/registry/docker/registry -d -p 5000:5000 --restart=always --privileged --name registry registry` 2. mirror image to local registry : `oc adm release mirror registry.ci.openshift.org/ocp/release:4.11.0-0.nightly-2022-06-23-044003 --to='localhost:5000/ocp4' --insecure` 3. Create the icsp file from the output of `oc adm release mirror` command ; apiVersion: operator.openshift.io/v1alpha1 kind: ImageContentSourcePolicy metadata: name: example spec: repositoryDigestMirrors: - mirrors: - localhost:5000/ocp4 source: quay.io/openshift-release-dev/ocp-v4.0-art-dev - mirrors: - localhost:5000/ocp4 source: registry.ci.openshift.org/ocp/release 4. Use the icsp file to extract from local registry: oc adm release extract --command='oc' --icsp-file=../icsp.yaml localhost:5000/ocp4@sha256:5cc4c2912c03b49bf9b8b2b4e36319c467c1483f30689291a87e05097e6844bb -v 5 --insecure I0623 20:09:10.345600 74658 config.go:127] looking for config.json at /root/.docker/config.json I0623 20:09:10.345679 74658 config.go:93] looking for .dockercfg at /root/.dockercfg I0623 20:09:10.345973 74658 client_mirrored.go:174] Attempting to connect to localhost:5000/ocp4 I0623 20:09:10.347256 74658 client.go:322] Falling back to an HTTP check for an insecure registry https://localhost:5000: Get "https://localhost:5000/v2/": http: server gave HTTP response to HTTPS client I0623 20:09:10.352634 74658 client_mirrored.go:412] get manifest for sha256:5cc4c2912c03b49bf9b8b2b4e36319c467c1483f30689291a87e05097e6844bb served from registryclient.retryManifest{ManifestService:registryclient.manifestServiceVerifier{ManifestService:(*client.manifests)(0xc000911800)}, repo:(*registryclient.retryRepository)(0xc0000fa800)}: <nil> I0623 20:09:10.352685 74658 client_mirrored.go:174] Attempting to connect to localhost:5000/ocp4 I0623 20:09:10.354882 74658 client_mirrored.go:445] get for sha256:412338ccddcdb725ecd375371d5488d0290be6f52e36b6e14cbd3107c0bf1770 served from ocp4: <nil> I0623 20:09:10.354928 74658 manifest.go:312] Raw image config json: {"id":"","created":"2022-06-23T04:45:27Z","container_config":{},"docker_version":"1.13.1","config":{"Hostname":"d300d7450dc1","Env":["__doozer=merge","BUILD_RELEASE=202206221808.p0.gdc927a4.assembly.stream","BUILD_VERSION=v4.11.0","OS_GIT_MAJOR=4","OS_GIT_MINOR=11","OS_GIT_PATCH=0","OS_GIT_TREE_STATE=clean","OS_GIT_VERSION=4.11.0-202206221808.p0.gdc927a4.assembly.stream-dc927a4","SOURCE_GIT_TREE_STATE=clean","OS_GIT_COMMIT=dc927a4","SOURCE_DATE_EPOCH=1655917064","SOURCE_GIT_COMMIT=dc927a4c63e2d9fb7f469ecb77503687a60c6564","SOURCE_GIT_TAG=v1.0.0-842-gdc927a4c","SOURCE_GIT_URL=https://github.com/openshift/cluster-version-operator","GODEBUG=x509ignoreCN=0,madvdontneed=1","PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","container=oci"],"Entrypoint":["/usr/bin/cluster-version-operator"],"Labels":{"io.openshift.release":"4.11.0-0.nightly-2022-06-23-044003","io.openshift.release.base-image-digest":"sha256:c4e86576fd8f3e02db9b0d7d883dbc966caf87336609bbcfbdf166b057d8b822"}},"architecture":"amd64","size":118249335,"rootfs":{"type":"layers","diff_ids":["sha256:5bf135c4a0de07e52c11282c0954e3e6b7c7ddc6c8834a7fd2803c3dc6a31a69","sha256:773711fd02f009e3bc5f9e2b1e859bf2103ba7318b3eb73390490afb3a3a8848","sha256:0c728cbdd06d5172e1e62858b7e8ed70b73457271b4aa4d8dd3718ef1dee8944","sha256:06ea8291551e7828cf59b048353083ca5dbee0299735699b065228677221e267","sha256:dbfca6502f224f1f14c180c5e44ab6accb743eae622b7e0c047bc4d3f59e7842","sha256:225fd4443357f1160f99cd07b7b46df71c8bf391421a1a57e91f088c572477eb"]},"history":[{"created":"2022-06-23T04:45:27Z","comment":"Release image for OpenShift"},{"created":"2022-06-23T04:45:27Z"},{"created":"2022-06-23T04:45:27Z"},{"created":"2022-06-23T04:45:27Z"},{"created":"2022-06-23T04:45:27Z"},{"created":"2022-06-23T04:45:27Z"}],"os":"linux"} I0623 20:09:10.355192 74658 extract.go:484] Extracting from layer: distribution.Descriptor{MediaType:"application/vnd.docker.image.rootfs.diff.tar.gzip", Size:877142, Digest:"sha256:5dce837222ce844794f18ab4f4774210b255d885512863050246978421a9ce80", URLs:[]string(nil), Annotations:map[string]string(nil), Platform:(*v1.Platform)(nil)} I0623 20:09:10.355227 74658 client_mirrored.go:174] Attempting to connect to localhost:5000/ocp4 I0623 20:09:10.357457 74658 client_mirrored.go:485] open (read) sha256:5dce837222ce844794f18ab4f4774210b255d885512863050246978421a9ce80 from ocp4: <nil> I0623 20:09:10.358134 74658 extract.go:679] Exclude release-manifests due to missing prefix release-manifests/ I0623 20:09:10.358156 74658 extract.go:551] Exclude entry release-manifests 35 0 ....
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:5069