Bug 1823143

I'm investigating this, but here is a workaround:
(relisting steps from original description, as there has been change due to code merge):

1) start local registry
$ podman run -p 5000:5000 -d registry:2

2) mirror a nightly release to local disk
$ oc adm release mirror -a ~/your/pull-secret --from=registry.svc.ci.openshift.org/ocp/release:4.5.0-0.nightly-2020-04-18-093630 --to file://mirror_4.5.0 --release-image-signature-to-dir=test

from now only disconnected steps
3) mirror localdisk nightly release to local registry 
$ oc image mirror --insecure=true 'file://mirror_4.5.0:4.5.0-0.nightly-2020-04-18-093630*' localhost:5000/ocp/release

4) extract openshift-install binary from local registry (sudo necessary with mounts)
$ ctr=$(sudo podman run -d --rm --entrypoint /bin/sh localhost:5000/ocp/release:4.5.0-0.nightly-2020-04-18-093630-installer)
$ mnt=$(sudo podman mount $ctr)  
$ sudo cp -R ${mnt}/bin/openshift-install /some/localdir
$ sudo chown $(whoami):$(whoami) /some/localdir/openshift-install

Please disregard that above comment, that will not extract the binary necessary for a disconnected install- 'oc adm release extract' modifies the binary to pin the correct release image, while the above does not.  still investigating...

*** Bug 1827101 has been marked as a duplicate of this bug. ***

*** Bug 1812814 has been marked as a duplicate of this bug. ***

*** Bug 1814495 has been marked as a duplicate of this bug. ***

The PR attached here: https://github.com/openshift/oc/pull/427  while it resolves the issue here it is not the correct path forward.

In order to resolve this properly, we're going to add ImageContentSources awareness to oc, I've opened a jira for this here: https://issues.redhat.com/browse/WRKLDS-174

Please track this through jira

Not being properly aware of a local registry is a bug, not a RFE. 

Whether the way to fix it is the pull request or making oc aware of ImageContentSources is a detail on how the fix will be implemented, it does not mean it is not a bug.

Hence I am reopening this bug.

Moving then to the next release since we're not going to be able to address this at this point in time.

This bug is actively being worked on.

I’m adding UpcomingSprint, because there are PRs for this bug that are waiting for review.  This is actively being worked on.

I’m adding UpcomingSprint, because there are PRs for this bug that are waiting for review.  This is actively being worked on.

This bug hasn't had any activity in the last 30 days. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. As such, we're marking this bug as "LifecycleStale" and decreasing the severity/priority. If you have further information on the current state of the bug, please update it, otherwise this bug can be closed in about 7 days. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant.

The PR to resolve this issue is waiting for review/merge https://github.com/openshift/oc/pull/439.  The necessary enhancement has been approved. Removing LifecycleStale as it's actively being worked on.

The LifecycleStale keyword was removed because the bug got commented on recently.
The bug assignee was notified.

This is a large change, so the review process is taking time.  I'm working through the review process and will update here and this PR accordingly: https://github.com/openshift/oc/pull/439

ICSP topic is being deferred until 4.7.

Other bugs took precedence last sprint and this continues to wait for review - adding UpcomingSprint tag.

Waiting on PR review, as other bugs took priority last sprint, adding UpcomingSprint.

Actively working on this, PR is under review, adding UpcomingSprint.

PR is actively being reviewed, adding upcoming sprint

PR actively being reviewed (still), adding upcoming sprint

Pushing this over to 4.8 since 4.7 is closed.

*** Bug 1957126 has been marked as a duplicate of this bug. ***

*** Bug 1972660 has been marked as a duplicate of this bug. ***

What is the status of the review in progress?

(In reply to Ken Young from comment #49)
> What is the status of the review in progress?

It's still work-in-progress.

I've updated https://github.com/openshift/oc/pull/829 with all comments addressed, waiting for reviews.

Still could reproduce the issue :

oc version --client
Client Version: 4.11.0-202204141741.p0.g1b0e16c.assembly.stream-1b0e16c

1. create the local registry:
    `podman run --volume /home/registry:/var/lib/registry/docker/registry -d -p 5000:5000 --restart=always --privileged --name registry registry`
2. create dir and mirror the release to local disk:
    ` mkdir mirror_4.10.9` ; `cd mirror_4.10.9`, `oc adm release mirror  --from=quay.io/openshift-release-dev/ocp-release:4.10.9-x86_64 --to file://openshift/release`
3. mirror the release to local registry :
   `oc image mirror --insecure=true 'file://openshift/release:4.10.9-x86_64*'  localhost:5000/ocp4`
4. remove the quay credentials:
    `mv /root/.docker/config.json /root/.docker/config.json.13`
5. try to extract from local registry , but still failed:
 oc adm release extract --tools --insecure=true --from=localhost:5000/ocp4@sha256:39f360002b9b5c730d1167879ad6437352d51e72acc9fe80add3ec2a0d20400d
Warning: the default reading order of registry auth file will be changed from "${HOME}/.docker/config.json" to podman registry config locations in the future version. "${HOME}/.docker/config.json" is deprecated, but can still be used for storing credentials as a fallback. See https://github.com/containers/image/blob/main/docs/containers-auth.json.5.md for the order of podman registry config locations.
error: unable to read image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9456cdc44def5f31fc42f7534eade5b7ef45b6e6c9e83730eb03eb3f35196c56: unauthorized: access to the requested resource is not authorized

I have the same issue as well, it seems that oc is trying to access quay.io instead of the mirror registry:
./oc adm -v=3 release extract --command=openshift-install --to=./ --insecure=false edge-01.edge.lab.eng.rdu2.redhat.com:5000/ocp:4.8.18 --registry-config=/tmp/registry.json
I0424 08:14:25.430489  304682 extract_tools.go:380] Skipping openshift-install-mac-%s.tar.gz, does not match current OS darwin
I0424 08:14:25.430530  304682 extract_tools.go:380] Skipping openshift-install-mac-arm64-%s.tar.gz, does not match current OS darwin
I0424 08:14:25.430535  304682 extract_tools.go:398] Will extract usr/bin/openshift-install from quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:18d97ab33fd127a72e508ce220e06d0c3657657e9f923bc9f7b98ea0be5a8bf1
I0424 08:14:25.430552  304682 extract_tools.go:390] Skipping duplicate openshift-install-linux-amd64-%s.tar.gz
error: unable to read image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:18d97ab33fd127a72e508ce220e06d0c3657657e9f923bc9f7b98ea0be5a8bf1: Get "https://quay.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Also, from the linked PR https://github.com/openshift/oc/pull/829 it seems that the ICSP option was added to `oc image info` and not `oc adm release extract`

I see the same issue in our environment where Assisted Installer is used in disconnected environment.
We are using OCP Version: 4.8.29-assembly.art3875-x86_64.

Following are the versions of the AI Container Images in use =>

$ podman images
REPOSITORY                                                 TAG         IMAGE ID      CREATED       SIZE
quay.io/edge-infrastructure/assisted-service               latest      7a9a88cff3a4  2 weeks ago   849 MB
quay.io/edge-infrastructure/assisted-installer-ui          latest      debd6d4eec22  2 weeks ago   488 MB
quay.io/centos7/postgresql-12-centos7                      latest      d57e7b296b6b  2 weeks ago   382 MB
quay.io/edge-infrastructure/assisted-installer-controller  latest      19284a03a80d  2 weeks ago   731 MB
quay.io/edge-infrastructure/assisted-installer             latest      401d7e12fb74  2 weeks ago   470 MB
quay.io/edge-infrastructure/assisted-installer-agent       latest      e170df2ccc3e  2 weeks ago   1.03 GB
quay.io/edge-infrastructure/assisted-image-service         latest      3df5716c00ae  4 weeks ago   417 MB

Installation failed saying (from Cluster Events logs) =>

Failed to prepare the installation due to an unexpected error: failed generating install config for cluster 2f1de184-eec0-42ff-b286-6bdfe3b33aef: failed to get installer path: command 'oc adm release extract --command=openshift-baremetal-install --to=/data/install-config-generate/installercache/iss.zarya.net:5000/ocp-release:4.8.29-assembly.art3875-x86_64 --insecure=false iss.zarya.net:5000/ocp-release:4.8.29-assembly.art3875-x86_64 --registry-config=/tmp/registry-config1255408649' exited with non-zero exit code 1: error: unable to read image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9708c9c905e87646d606c4dcac1deee6314305379465b7a9646360be173e074e: Get "https://quay.io/v2/": dial tcp: lookup quay.io on 100.2.1.9:53: server misbehaving . Please retry later

venkatasubramanian.b note that a fix allowing assisted-service to workaround for this issue merged 2 days ago - https://github.com/openshift/assisted-service/pull/3700
So it should work once you update the assisted-service image or redeploy.

Thankyou Eran. I have taken the latest assisted-service image (also all other needed images as well) and I confirm that I no more see that issue. Our cluster is now successfully commissioned.

As mentioned by comment 71 and comment 72, we also need this implemented for the `oc adm release extract` command.

The assisted-service is currently using a forked version of oc to work around this problem, but we also need it urgently in 4.11 for the ephemeral agent installer, and that cannot use anything other than the version of oc shipped in the release payload.

Any updates about this effort?

I think we have all the commands we cared about, moving to modified.

verified with latest oc client:

oc version --client
Client Version: 4.11.0-0.nightly-2022-06-23-092832
Kustomize Version: v4.5.4


1. create the local registry:
    `podman run --volume /home/registry:/var/lib/registry/docker/registry -d -p 5000:5000 --restart=always --privileged --name registry registry`
2. mirror image to local registry :
    `oc adm release mirror registry.ci.openshift.org/ocp/release:4.11.0-0.nightly-2022-06-23-044003  --to='localhost:5000/ocp4' --insecure`

3. Create the icsp file from the output of `oc adm release mirror` command ;

apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
  name: example
spec:
  repositoryDigestMirrors:
  - mirrors:
    - localhost:5000/ocp4
    source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
  - mirrors:
    - localhost:5000/ocp4
    source: registry.ci.openshift.org/ocp/release
4. Use the icsp file to extract from local registry:

oc adm release extract --command='oc'   --icsp-file=../icsp.yaml  localhost:5000/ocp4@sha256:5cc4c2912c03b49bf9b8b2b4e36319c467c1483f30689291a87e05097e6844bb  -v 5 --insecure
I0623 20:09:10.345600   74658 config.go:127] looking for config.json at /root/.docker/config.json
I0623 20:09:10.345679   74658 config.go:93] looking for .dockercfg at /root/.dockercfg
I0623 20:09:10.345973   74658 client_mirrored.go:174] Attempting to connect to localhost:5000/ocp4
I0623 20:09:10.347256   74658 client.go:322] Falling back to an HTTP check for an insecure registry https://localhost:5000: Get "https://localhost:5000/v2/": http: server gave HTTP response to HTTPS client
I0623 20:09:10.352634   74658 client_mirrored.go:412] get manifest for sha256:5cc4c2912c03b49bf9b8b2b4e36319c467c1483f30689291a87e05097e6844bb served from registryclient.retryManifest{ManifestService:registryclient.manifestServiceVerifier{ManifestService:(*client.manifests)(0xc000911800)}, repo:(*registryclient.retryRepository)(0xc0000fa800)}: <nil>
I0623 20:09:10.352685   74658 client_mirrored.go:174] Attempting to connect to localhost:5000/ocp4
I0623 20:09:10.354882   74658 client_mirrored.go:445] get for sha256:412338ccddcdb725ecd375371d5488d0290be6f52e36b6e14cbd3107c0bf1770 served from ocp4: <nil>
I0623 20:09:10.354928   74658 manifest.go:312] Raw image config json:
{"id":"","created":"2022-06-23T04:45:27Z","container_config":{},"docker_version":"1.13.1","config":{"Hostname":"d300d7450dc1","Env":["__doozer=merge","BUILD_RELEASE=202206221808.p0.gdc927a4.assembly.stream","BUILD_VERSION=v4.11.0","OS_GIT_MAJOR=4","OS_GIT_MINOR=11","OS_GIT_PATCH=0","OS_GIT_TREE_STATE=clean","OS_GIT_VERSION=4.11.0-202206221808.p0.gdc927a4.assembly.stream-dc927a4","SOURCE_GIT_TREE_STATE=clean","OS_GIT_COMMIT=dc927a4","SOURCE_DATE_EPOCH=1655917064","SOURCE_GIT_COMMIT=dc927a4c63e2d9fb7f469ecb77503687a60c6564","SOURCE_GIT_TAG=v1.0.0-842-gdc927a4c","SOURCE_GIT_URL=https://github.com/openshift/cluster-version-operator","GODEBUG=x509ignoreCN=0,madvdontneed=1","PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","container=oci"],"Entrypoint":["/usr/bin/cluster-version-operator"],"Labels":{"io.openshift.release":"4.11.0-0.nightly-2022-06-23-044003","io.openshift.release.base-image-digest":"sha256:c4e86576fd8f3e02db9b0d7d883dbc966caf87336609bbcfbdf166b057d8b822"}},"architecture":"amd64","size":118249335,"rootfs":{"type":"layers","diff_ids":["sha256:5bf135c4a0de07e52c11282c0954e3e6b7c7ddc6c8834a7fd2803c3dc6a31a69","sha256:773711fd02f009e3bc5f9e2b1e859bf2103ba7318b3eb73390490afb3a3a8848","sha256:0c728cbdd06d5172e1e62858b7e8ed70b73457271b4aa4d8dd3718ef1dee8944","sha256:06ea8291551e7828cf59b048353083ca5dbee0299735699b065228677221e267","sha256:dbfca6502f224f1f14c180c5e44ab6accb743eae622b7e0c047bc4d3f59e7842","sha256:225fd4443357f1160f99cd07b7b46df71c8bf391421a1a57e91f088c572477eb"]},"history":[{"created":"2022-06-23T04:45:27Z","comment":"Release image for OpenShift"},{"created":"2022-06-23T04:45:27Z"},{"created":"2022-06-23T04:45:27Z"},{"created":"2022-06-23T04:45:27Z"},{"created":"2022-06-23T04:45:27Z"},{"created":"2022-06-23T04:45:27Z"}],"os":"linux"}
I0623 20:09:10.355192   74658 extract.go:484] Extracting from layer: distribution.Descriptor{MediaType:"application/vnd.docker.image.rootfs.diff.tar.gzip", Size:877142, Digest:"sha256:5dce837222ce844794f18ab4f4774210b255d885512863050246978421a9ce80", URLs:[]string(nil), Annotations:map[string]string(nil), Platform:(*v1.Platform)(nil)}
I0623 20:09:10.355227   74658 client_mirrored.go:174] Attempting to connect to localhost:5000/ocp4
I0623 20:09:10.357457   74658 client_mirrored.go:485] open (read) sha256:5dce837222ce844794f18ab4f4774210b255d885512863050246978421a9ce80 from ocp4: <nil>
I0623 20:09:10.358134   74658 extract.go:679] Exclude release-manifests due to missing prefix release-manifests/
I0623 20:09:10.358156   74658 extract.go:551] Exclude entry release-manifests 35 0
....

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069