Bug 1874410
| Summary: | Failed to pull correctly mirrored image with correct ImageContentSourcePolicy | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Filip Brychta <fbrychta> | ||||||||
| Component: | oc | Assignee: | Sally <somalley> | ||||||||
| Status: | CLOSED NOTABUG | QA Contact: | zhou ying <yinzhou> | ||||||||
| Severity: | high | Docs Contact: | |||||||||
| Priority: | high | ||||||||||
| Version: | 4.6 | CC: | afield, aos-bugs, ccoleman, dwalsh, eparis, gbaufake, jokerman, kconner, maszulik, mfojtik, mitr, mpatel, nagrawal, stchen, tsweeney | ||||||||
| Target Milestone: | --- | Keywords: | TestBlocker, TestBlockerForLayeredProduct | ||||||||
| Target Release: | 4.6.0 | ||||||||||
| Hardware: | Unspecified | ||||||||||
| OS: | Unspecified | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | Doc Type: | Known Issue | |||||||||
| Doc Text: | Story Points: | --- | |||||||||
| Clone Of: | Environment: | ||||||||||
| Last Closed: | 2020-09-18 11:02:43 UTC | Type: | Bug | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Embargoed: | |||||||||||
| Attachments: |
|
||||||||||
|
Description
Filip Brychta
2020-09-01 10:15:14 UTC
Created attachment 1713274 [details]
output of oc adm catalog mirror
Created attachment 1713275 [details]
mapping.txt
This is hapenning also in an AWS/GCP cluster. It seems the catalog doesn't mirror correctly what is on iib We hit this problem again on OCP 4.5.7 with ose-elasticsearch-operator.
The image was mirrored correctly with following in mapping.txt:
registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:b1463becf264e75e383d8cc33eaa407002499583b40a90eed8b071b627215ed3=bastion.ocp-jqe4-disconnected.maistra.upshift.redhat.com:55555/openshift4/ose-elasticsearch-operator:2f060a82
Following in imageContentSourcePolicy.yaml:
- mirrors:
- bastion.ocp-jqe4-disconnected.maistra.upshift.redhat.com:55555/openshift4/ose-elasticsearch-operator
source: registry.redhat.io/openshift4/ose-elasticsearch-operator
But installation of ES operators still fails with:
Failed to pull image "registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:b1463becf264e75e383d8cc33eaa407002499583b40a90eed8b071b627215ed3": rpc error: code = Unknown desc = (Mirrors also failed: [bastion.ocp-jqe4-disconnected.maistra.upshift.redhat.com:55555/openshift4/ose-elasticsearch-operator@sha256:b1463becf264e75e383d8cc33eaa407002499583b40a90eed8b071b627215ed3: Error reading manifest sha256:b1463becf264e75e383d8cc33eaa407002499583b40a90eed8b071b627215ed3 in bastion.ocp-jqe4-disconnected.maistra.upshift.redhat.com:55555/openshift4/ose-elasticsearch-operator: manifest unknown: manifest unknown]): registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:b1463becf264e75e383d8cc33eaa407002499583b40a90eed8b071b627215ed3: error pinging docker registry registry.redhat.io: Get https://registry.redhat.io/v2/: dial tcp 104.113.9.148:443: i/o timeout
Sending to Sally, she's dealing with ICSP topic from our side. ICSP topic is being deferred until 4.7. Does this mean OCP 4.6 will not support Disconnected installations? (In reply to kconner from comment #11) > Does this mean OCP 4.6 will not support Disconnected installations? The support will similar to the current so not 100%, we've mad an assessment on it and it seems too risky to land this feature now and we haven't completed full review, yet. Digest not being preserved has nothing to do with the ICSP feature. Generally digest not being preserved is an incorrect mirror call. Sally, this may be the filter-by-os confusion (stripping out manifest lists if they are in use) or it could be a registry that does not support schema v2. @Clayton this is an approach we use successfully with the older operator metadata format, with the same registry, so it's likely to be the mirroring which is failing. This seems to be confirmed by some of the additional comments. hi Filip, sorry just am digging into this now.. yes, this isn't the case of `oc` not preserving digests, just that the images are multi-arch so whatever is mirrored is a single manifest from a manifest list unless you specify to include all with the --filter-by-os=.* Please run this command w/ the steps listed in the bz description, and report back! $ oc image mirror quay.io/maistra/iib@sha256:dc43ea4c510742b6888ceca954be659449b70cee5deed2d823c3525b5d38ff27 `hostname`:55555/olm/iib-my:mytag -a auth.json --insecure --filter-by-os=.* Hi Sally, thank you very much. Adding --filter-by-os=.* really seems to solve the problem. I verified that using different scenario which was showing the same type of error so I still need to verify the scenario from this BZ to be 100% sure. I will do that tomorrow. I'm closing this as not a bug. The cause of the problem was missing --filter-by-os=.* parameter in step 3.b from the original description. Thank you Sally and Miloslav for your help. One thing to consider. We hit this issue because we were using existing mirroring scripts which were working fine without --filter-by-os=.* before the multi-arch images were introduced. Customers with existing mirroring scripts which are not using --filter-by-os=.* will hit the same issue. Would it make sense to change the default behaviour in a way that the oc adm catalog mirror command would continue to work without --filter-by-os=.* even for multi-arch images? @Filip, I was thinking the same, perhaps we can always include the manifestlist when mirroring. I'll discuss with the team, thank you! |