While verifying that the latest Kubernetes test images could be mirrored, the mirror command failed with a "blob not found" error. After inspection it was determined the upstream e2e tests reference a windows image with layers in its manifest of the "foreign layer" media type, which means that the blob is not stored in the current registry but is instead locatable via URL. The current mirror logic (and several other parts of our structure) do not correctly handle foreign layers because we flag the blob into a list of resources to copy but don't preserve the source (which is a URL, not a location). Since foreign URLs are primarily used for windows containers and only the base images qualify, we can probably afford to do a quick fix (skip mirroring foreign blobs) and then return to this later and support mirroring of the foreign layer by associating the foreign URL with the mirroring mapping. However, there may be other consequences to foreign layer support in our registry (when mirroring an external image to the internal registry) that may have to be tested to find scope. For now, fix this by skipping foreign layers by default (clients should still be able to access the root image except in disconnected environments).
After some investigation and discussion with Oleg, we decided that given the Docker client's behavior of checking the registry first, then the URLs, and the doc in the spec (that URLs are alternate sources), that mirror should attempt to copy the blob by default. This fits expectations customers would have in disconnected environments and matches the intent of the field in the spec. The linked PR has been tested between gcr and the local registry and back and the original image that triggede the issue was mirrored from gcr to quay successfully: > oc image info quay.io/openshift/community-e2e-images@sha256:193356114f94f20b09a80fb0b4f822ae44b5761c3a99d50b026718621723cc02
Checked with latest oc , can't reproduce the issue now: [root@dhcp-140-138 ~]# oc image mirror quay.io/openshift/community-e2e-images@sha256:193356114f94f20b09a80fb0b4f822ae44b5761c3a99d50b026718621723cc02 docker.io/zhouying7780/mirrortest:latest docker.io/ zhouying7780/mirrortest blobs: quay.io/openshift/community-e2e-images sha256:206038661c9f3d80d3ddfe2a69ca9d12458d2f85a79c3b6e7f75ea5dee152dff 1.064KiB quay.io/openshift/community-e2e-images sha256:401dae14c835ca7cbf977c6ff95452570f320ad4e6c2f8431d963229dc91783f 1.064KiB quay.io/openshift/community-e2e-images sha256:09635ba14e57b1fc4b8abb1721f399c7a5782dd05959f73fe7f5a08db5c4a6f4 1.066KiB quay.io/openshift/community-e2e-images sha256:2de59e3266a5fb32407235b69ef4933b83dfb018911250ae9d9ba8fd66d9fb7c 1.066KiB quay.io/openshift/community-e2e-images sha256:a036a1efd1fa7ad68872a46522ccf76c8a71029d3fe057c67b98275903cac200 1.231KiB quay.io/openshift/community-e2e-images sha256:43e04586603ddd3f8269754b7e704ecd545a63c280d37c568064440db94a360a 1.254KiB quay.io/openshift/community-e2e-images sha256:858f866aa8bfc232777d7bd6461a9a25c98a3340dc247dcb9a60119b38573db0 2.07KiB quay.io/openshift/community-e2e-images sha256:a2c997f0257e154afe71d8bd4a2ae630ef8fe0f7c5aefa25dac4b37c3a81b20a 2.077KiB quay.io/openshift/community-e2e-images sha256:f846e8f9e25144f3676be7d6766ca0f47e4bf669a3a271bf0a66755b57167fde 2.565KiB quay.io/openshift/community-e2e-images sha256:6f8a6457b7a568ebe3ffe460b9863a69d99e695ef5a3e37b0448f69afa2bc260 6.354KiB quay.io/openshift/community-e2e-images sha256:022722865e68b57e7e48f926756acee753f2f246779e113b6286fe42c91496ca 16.4KiB quay.io/openshift/community-e2e-images sha256:83962641236162769d7223cedbd595d8767143d5c6913d00774db799534d6236 279.9KiB quay.io/openshift/community-e2e-images sha256:b01cdb426940b0c97fa603e2f4ccdee7fa8fcfa1c91ba0ac5fe92c5482c17efc 332.1KiB quay.io/openshift/community-e2e-images sha256:4d35a7ea989baa725c3446cdb3d36d1f48bd86ee2b2b06d82a93925dfc499bfb 866.8KiB quay.io/openshift/community-e2e-images sha256:cb5823db1787016ef25f8b16bd8c13c17f5986158c14cda23ba80d853a0d7cea 904.1KiB quay.io/openshift/community-e2e-images sha256:83d66aa1074a4b7b5588c3aded51a6cf36d843f82bcf2ced68b1d02797ff01c8 3.766MiB quay.io/openshift/community-e2e-images sha256:11bb2636f1ab9848e2c2888a8061710edb8c7528c32280b288a3cb531d59efc5 10.28MiB quay.io/openshift/community-e2e-images sha256:46b1bcb83494017f4b4e69f11586a8c2cb14184dcafe65cae87575f72d5f39ec 33.53MiB quay.io/openshift/community-e2e-images sha256:747e2d83e9ae0589a2c6180b75f7f15854c1092a1639adfe13650d254ed12cbb 33.56MiB quay.io/openshift/community-e2e-images sha256:ab033e23ee48f85bf33cac4f5ddda4c44f288d04d29b25917b257f148cc8b6ec 35.62MiB quay.io/openshift/community-e2e-images sha256:2bbc6ef33df8a58874aa8a6ab3fb2440342cff233bdaae36895f0d511afac421 46.34MiB quay.io/openshift/community-e2e-images sha256:aebaeb8e30b6ff23a978c66b2dc7b1ab0f8081f820bb03e5b34861bf863380b7 70.09MiB quay.io/openshift/community-e2e-images sha256:b4c5c65ae0c4e5a8a28d35af9a4273d11133061874011e8c7bd68d9af8781e9f 665.6MiB quay.io/openshift/community-e2e-images sha256:9038b92872bc268d5c975e84dd94e69848564b222ad116ee652c62e0c2f894b2 1.502GiB manifests: sha256:193356114f94f20b09a80fb0b4f822ae44b5761c3a99d50b026718621723cc02 -> latest stats: shared=0 unique=24 size=2.382GiB ratio=1.00 phase 0: docker.io zhouying7780/mirrortest blobs=24 mounts=0 manifests=1 shared=0 info: Planning completed in 5.14s I0902 08:57:20.190413 3653 mirror.go:787] Attempting to retrieve blob sha256:b4c5c65ae0c4e5a8a28d35af9a4273d11133061874011e8c7bd68d9af8781e9f from registry or urls [https://mcr.microsoft.com/v2/windows/servercore/blobs/sha256:b4c5c65ae0c4e5a8a28d35af9a4273d11133061874011e8c7bd68d9af8781e9f] uploading: docker.io/zhouying7780/mirrortest sha256:b4c5c65ae0c4e5a8a28d35af9a4273d11133061874011e8c7bd68d9af8781e9f 665.6MiB sha256:193356114f94f20b09a80fb0b4f822ae44b5761c3a99d50b026718621723cc02 docker.io/zhouying7780/mirrortest:latest info: Mirroring completed in 5m49.07s (1.999MB/s)
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4196