Bug 1872709 - Images that contain foreign layers (identified by URL in manifest) cannot be mirrored by oc image mirror
Summary: Images that contain foreign layers (identified by URL in manifest) cannot be ...
Keywords:
Status: VERIFIED
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: oc
Version: 4.6
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.6.0
Assignee: Clayton Coleman
QA Contact: zhou ying
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-08-26 13:28 UTC by Clayton Coleman
Modified: 2020-09-02 07:32 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift oc pull 530 None closed Bug 1872709: Copy foreign layers from external sources during mirroring 2020-09-01 01:52:43 UTC

Description Clayton Coleman 2020-08-26 13:28:26 UTC
While verifying that the latest Kubernetes test images could be mirrored, the mirror command failed with a "blob not found" error. After inspection it was determined the upstream e2e tests reference a windows image with layers in its manifest of the "foreign layer" media type, which means that the blob is not stored in the current registry but is instead locatable via URL.

The current mirror logic (and several other parts of our structure) do not correctly handle foreign layers because we flag the blob into a list of resources to copy but don't preserve the source (which is a URL, not a location).  Since foreign URLs are primarily used for windows containers and only the base images qualify, we can probably afford to do a quick fix (skip mirroring foreign blobs) and then return to this later and support mirroring of the foreign layer by associating the foreign URL with the mirroring mapping.  However, there may be other consequences to foreign layer support in our registry (when mirroring an external image to the internal registry) that may have to be tested to find scope.

For now, fix this by skipping foreign layers by default (clients should still be able to access the root image except in disconnected environments).

Comment 1 Clayton Coleman 2020-08-26 21:07:50 UTC
After some investigation and discussion with Oleg, we decided that given the Docker client's behavior of checking the registry first, then the URLs, and the doc in the spec (that URLs are alternate sources), that mirror should attempt to copy the blob by default. This fits expectations customers would have in disconnected environments and matches the intent of the field in the spec.

The linked PR has been tested between gcr and the local registry and back and the original image that triggede the issue was mirrored from gcr to quay successfully:

> oc image info quay.io/openshift/community-e2e-images@sha256:193356114f94f20b09a80fb0b4f822ae44b5761c3a99d50b026718621723cc02

Comment 4 zhou ying 2020-09-02 07:31:01 UTC
Checked with latest oc , can't reproduce the issue now: 

[root@dhcp-140-138 ~]# oc image mirror quay.io/openshift/community-e2e-images@sha256:193356114f94f20b09a80fb0b4f822ae44b5761c3a99d50b026718621723cc02 docker.io/zhouying7780/mirrortest:latest
docker.io/
  zhouying7780/mirrortest
    blobs:
      quay.io/openshift/community-e2e-images sha256:206038661c9f3d80d3ddfe2a69ca9d12458d2f85a79c3b6e7f75ea5dee152dff 1.064KiB
      quay.io/openshift/community-e2e-images sha256:401dae14c835ca7cbf977c6ff95452570f320ad4e6c2f8431d963229dc91783f 1.064KiB
      quay.io/openshift/community-e2e-images sha256:09635ba14e57b1fc4b8abb1721f399c7a5782dd05959f73fe7f5a08db5c4a6f4 1.066KiB
      quay.io/openshift/community-e2e-images sha256:2de59e3266a5fb32407235b69ef4933b83dfb018911250ae9d9ba8fd66d9fb7c 1.066KiB
      quay.io/openshift/community-e2e-images sha256:a036a1efd1fa7ad68872a46522ccf76c8a71029d3fe057c67b98275903cac200 1.231KiB
      quay.io/openshift/community-e2e-images sha256:43e04586603ddd3f8269754b7e704ecd545a63c280d37c568064440db94a360a 1.254KiB
      quay.io/openshift/community-e2e-images sha256:858f866aa8bfc232777d7bd6461a9a25c98a3340dc247dcb9a60119b38573db0 2.07KiB
      quay.io/openshift/community-e2e-images sha256:a2c997f0257e154afe71d8bd4a2ae630ef8fe0f7c5aefa25dac4b37c3a81b20a 2.077KiB
      quay.io/openshift/community-e2e-images sha256:f846e8f9e25144f3676be7d6766ca0f47e4bf669a3a271bf0a66755b57167fde 2.565KiB
      quay.io/openshift/community-e2e-images sha256:6f8a6457b7a568ebe3ffe460b9863a69d99e695ef5a3e37b0448f69afa2bc260 6.354KiB
      quay.io/openshift/community-e2e-images sha256:022722865e68b57e7e48f926756acee753f2f246779e113b6286fe42c91496ca 16.4KiB
      quay.io/openshift/community-e2e-images sha256:83962641236162769d7223cedbd595d8767143d5c6913d00774db799534d6236 279.9KiB
      quay.io/openshift/community-e2e-images sha256:b01cdb426940b0c97fa603e2f4ccdee7fa8fcfa1c91ba0ac5fe92c5482c17efc 332.1KiB
      quay.io/openshift/community-e2e-images sha256:4d35a7ea989baa725c3446cdb3d36d1f48bd86ee2b2b06d82a93925dfc499bfb 866.8KiB
      quay.io/openshift/community-e2e-images sha256:cb5823db1787016ef25f8b16bd8c13c17f5986158c14cda23ba80d853a0d7cea 904.1KiB
      quay.io/openshift/community-e2e-images sha256:83d66aa1074a4b7b5588c3aded51a6cf36d843f82bcf2ced68b1d02797ff01c8 3.766MiB
      quay.io/openshift/community-e2e-images sha256:11bb2636f1ab9848e2c2888a8061710edb8c7528c32280b288a3cb531d59efc5 10.28MiB
      quay.io/openshift/community-e2e-images sha256:46b1bcb83494017f4b4e69f11586a8c2cb14184dcafe65cae87575f72d5f39ec 33.53MiB
      quay.io/openshift/community-e2e-images sha256:747e2d83e9ae0589a2c6180b75f7f15854c1092a1639adfe13650d254ed12cbb 33.56MiB
      quay.io/openshift/community-e2e-images sha256:ab033e23ee48f85bf33cac4f5ddda4c44f288d04d29b25917b257f148cc8b6ec 35.62MiB
      quay.io/openshift/community-e2e-images sha256:2bbc6ef33df8a58874aa8a6ab3fb2440342cff233bdaae36895f0d511afac421 46.34MiB
      quay.io/openshift/community-e2e-images sha256:aebaeb8e30b6ff23a978c66b2dc7b1ab0f8081f820bb03e5b34861bf863380b7 70.09MiB
      quay.io/openshift/community-e2e-images sha256:b4c5c65ae0c4e5a8a28d35af9a4273d11133061874011e8c7bd68d9af8781e9f 665.6MiB
      quay.io/openshift/community-e2e-images sha256:9038b92872bc268d5c975e84dd94e69848564b222ad116ee652c62e0c2f894b2 1.502GiB
    manifests:
      sha256:193356114f94f20b09a80fb0b4f822ae44b5761c3a99d50b026718621723cc02 -> latest
  stats: shared=0 unique=24 size=2.382GiB ratio=1.00

phase 0:
  docker.io zhouying7780/mirrortest blobs=24 mounts=0 manifests=1 shared=0

info: Planning completed in 5.14s
I0902 08:57:20.190413    3653 mirror.go:787] Attempting to retrieve blob sha256:b4c5c65ae0c4e5a8a28d35af9a4273d11133061874011e8c7bd68d9af8781e9f from registry or urls [https://mcr.microsoft.com/v2/windows/servercore/blobs/sha256:b4c5c65ae0c4e5a8a28d35af9a4273d11133061874011e8c7bd68d9af8781e9f]
uploading: docker.io/zhouying7780/mirrortest sha256:b4c5c65ae0c4e5a8a28d35af9a4273d11133061874011e8c7bd68d9af8781e9f 665.6MiB
  sha256:193356114f94f20b09a80fb0b4f822ae44b5761c3a99d50b026718621723cc02 docker.io/zhouying7780/mirrortest:latest
info: Mirroring completed in 5m49.07s (1.999MB/s)


Note You need to log in before you can comment on or make changes to this bug.