Bug 1947229
| Summary: | oc image mirror produces new SHAs for Operator OLM images | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Will C <wcushen> |
| Component: | oc | Assignee: | Maciej Szulik <maszulik> |
| Status: | CLOSED NOTABUG | QA Contact: | zhou ying <yinzhou> |
| Severity: | high | Docs Contact: | |
| Priority: | medium | ||
| Version: | 4.6.z | CC: | aos-bugs, dbewley, dmesser, jokerman, kjanania, mfojtik, tkatarki |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-07-01 14:23:38 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
This bug hasn't had any activity in the last 30 days. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. As such, we're marking this bug as "LifecycleStale" and decreasing the severity/priority. If you have further information on the current state of the bug, please update it, otherwise this bug can be closed in about 7 days. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. Additionally, you can add LifecycleFrozen into Keywords if you think this bug should never be marked as stale. Please consult with bug assignee before you do that. I believe this is still an issue in 4.7. We've been encountering this recently as well and have been using skopeo copy as a workaround. The LifecycleStale keyword was removed because the bug got commented on recently. The bug assignee was notified. @maszulik this impacting several customers. so bumping the severity and priority. @maszulik this impacting several customers. so bumping the severity and priority. I'll double check what's possible, but currently overall mirroring story for disconnected has higher priority. This bug hasn't had any activity in the last 30 days. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. As such, we're marking this bug as "LifecycleStale" and decreasing the severity/priority. If you have further information on the current state of the bug, please update it, otherwise this bug can be closed in about 7 days. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. Additionally, you can add LifecycleFrozen into Keywords if you think this bug should never be marked as stale. Please consult with bug assignee before you do that. I've spent most of my day today digging through this topic and here's what I found: 1. I initially had issues when mirroring and pulling by ID which are actually coming from quay limitation which does not support pulling by ID, if you don't specify tag (see https://issues.redhat.com/browse/PROJQUAY-2181), which slowed down my testing and lead this issue to be looked at more than it should. 2. oc image mirror has 2 flags: - --filter-by-os='': A regular expression to control which images are considered when multiple variants are available. Images will be passed as '<platform>/<architecture>[/<variant>]'. - --keep-manifest-list=false: If an image is part of a manifest list, always mirror the list even if only one image is found. The default is to mirror the specific image unless unless --filter-by-os is passed. This flag is equivalent to setting --filter-by-os to '.*' since you cannot preserve the manifest list digest while filtering out any of the manifests included in the list. The problem and solution to the problem you're facing is using either of these flags. That is because when mirroring only certain images it very frequently happens that they are part of a ManifestList which contains information specific to platform architecture. That's why when mirroring you need to either specify to keep those manifest in place (--keep-manifest-list) or to make sure that all architectures and thus the whole ManifestList are copied (--filter-by-os='.*'). Looks like the --filter-by-os flag was deprecated for this reason. https://docs.openshift.com/container-platform/4.7/release_notes/ocp-4-7-release-notes.html#ocp-4-7-filterbyos-deprecated (In reply to Dale Bewley from comment #10) > Looks like the --filter-by-os flag was deprecated for this reason. > > https://docs.openshift.com/container-platform/4.7/release_notes/ocp-4-7- > release-notes.html#ocp-4-7-filterbyos-deprecated Not quite, that applies to oc adm catalog mirror, and the bug above was talking about oc image mirror ;-) |
Description of problem: Attempting to mirror a subset of the OLM Operator Catalog in a restricted network (via mapping.txt files) oc image mirror -f ../redhat-operators/mapping-subset-1.txt --insecure=true -a /run/user/1000/containers/auth.json As expected, the mirroring completed without errors, yet the SHAs are different from the mapping.txt file to what is resulting in the internal Quay repo. [redhat@quay manifests-redhat-operator-index-1616414956]$ cat mapping-compliance.txt registry.redhat.io/openshift4/compliance-content-rhel8@sha256:0e7a2c857a368ad5c19f9cdf213e3f971ec6cfb86de19be60e34b369a97980e3=10.204.248.104:8080/olmocp4/openshift4-compliance-content-rhel8:3ce71937 registry.redhat.io/openshift4/compliance-rhel8-operator-metadata@sha256:0f537ae9f4edc9830df58d841729c5628fc00829a64bade53adca8a4405d3d48=10.204.248.104:8080/olmocp4/openshift4-compliance-rhel8-operator-metadata:c9adb8e4 registry.redhat.io/openshift4/compliance-openscap-rhel8@sha256:6c066f4cb7b98f731a558e1800bc86f4a6631973366a54a8999422b44ef721cc=10.204.248.104:8080/olmocp4/openshift4-compliance-openscap-rhel8:26f7aaa7 registry.redhat.io/openshift4/compliance-rhel8-operator@sha256:027fa89c55c7a0e3fa65ce6111f24ee688e3b50a2fee918a60aaa9962008bcd3=10.204.248.104:8080/olmocp4/openshift4-compliance-rhel8-operator:6109f215 OC image mirroring: [redhat@quay manifests-redhat-operator-index-1616414956]$ ../../oc image mirror -f mapping-compliance.txt --insecure=true -a /run/user/1000/containers/auth.json W0329 08:16:25.643391 660955 manifest.go:440] Chose linux/amd64 manifest from the manifest list. W0329 08:16:26.413386 660955 manifest.go:440] Chose linux/amd64 manifest from the manifest list. W0329 08:16:26.448295 660955 manifest.go:440] Chose linux/amd64 manifest from the manifest list. 10.204.248.104:8080/ olmocp4/openshift4-compliance-content-rhel8 manifests: sha256:d419afa6e7579d32177fd24dc0cbfb3f8b1afe00d0f69d0678151fcf4650cf43 -> 3ce71937 olmocp4/openshift4-compliance-openscap-rhel8 manifests: sha256:54914618a94e2b88489b4c01f27920fdf0f07630e225cf10b9b4714c741baf28 -> 26f7aaa7 olmocp4/openshift4-compliance-rhel8-operator manifests: sha256:a70ec5158217951c142c5f3198fb1e897050eaed39bd4c76a0a1cfde8e5375c7 -> 6109f215 olmocp4/openshift4-compliance-rhel8-operator-metadata manifests: sha256:0f537ae9f4edc9830df58d841729c5628fc00829a64bade53adca8a4405d3d48 -> c9adb8e4 stats: shared=0 unique=0 size=0B phase 0: 10.204.248.104:8080 olmocp4/openshift4-compliance-openscap-rhel8 blobs=0 mounts=0 manifests=1 shared=0 10.204.248.104:8080 olmocp4/openshift4-compliance-rhel8-operator blobs=0 mounts=0 manifests=1 shared=0 10.204.248.104:8080 olmocp4/openshift4-compliance-content-rhel8 blobs=0 mounts=0 manifests=1 shared=0 10.204.248.104:8080 olmocp4/openshift4-compliance-rhel8-operator-metadata blobs=0 mounts=0 manifests=1 shared=0 info: Planning completed in 4.04s sha256:a70ec5158217951c142c5f3198fb1e897050eaed39bd4c76a0a1cfde8e5375c7 10.204.248.104:8080/olmocp4/openshift4-compliance-rhel8-operator:6109f215 sha256:d419afa6e7579d32177fd24dc0cbfb3f8b1afe00d0f69d0678151fcf4650cf43 10.204.248.104:8080/olmocp4/openshift4-compliance-content-rhel8:3ce71937 sha256:0f537ae9f4edc9830df58d841729c5628fc00829a64bade53adca8a4405d3d48 10.204.248.104:8080/olmocp4/openshift4-compliance-rhel8-operator-metadata:c9adb8e4 sha256:54914618a94e2b88489b4c01f27920fdf0f07630e225cf10b9b4714c741baf28 10.204.248.104:8080/olmocp4/openshift4-compliance-openscap-rhel8:26f7aaa7 info: Mirroring completed in 20ms (0B/s) We have been forced to run oc image mirror of a subset because we can't mirror the entire catalog at once with oc adm catalog mirror, see: https://bugzilla.redhat.com/show_bug.cgi?id=1938298 Version-Release number of selected component (if applicable): 4.6.12 How reproducible: Every time Steps to Reproduce: 1. oc image mirror -f ../redhat-operators/mapping-subset-1.txt --insecure=true -a /run/user/1000/containers/auth.json 2. Where auth.json has registry.rehat.io and internal Quay (insecure) credentials 3. Actual results: Operator images can't be installed from the disconnected custom CatalogSource because they point to an image with a different SHA from that in mapping.txt file. OCP is expecting the SHA from the latter. Incorrect SHA, means we have to manually skopeo copy every single image required for the Operator to run. Once we do this, Operators can be installed/run successfully. Expected results: Images are mirrored correctly to internal disconnected Quay and we are able to install Operators freely. Additional info: