Created attachment 1686234 [details] Dockerfile Description of problem: I created my custom registry based on manifests generated from following command: oc adm catalog build --appregistry-org redhat-operators --from=registry.redhat.io/openshift4/ose-operator-registry:v4.4 --to=<mirror_registry_host>:55555/olm/redhat-operators:v1 -a auth.json --insecure --manifest-dir='/home/cloud-user' oc adm catalog mirror fails with: W0507 11:42:27.072825 21414 mirror.go:258] errors during mirroring. the full contents of the catalog may not have been mirrored: image does not exist Version-Release number of selected component (if applicable): OCP 4.4 How reproducible: Always Steps to Reproduce: 1. prepare mirror registry following [1] 2. podman login to both mirror registry and registry.redhat.io 3. build catalog (note that I will not actually use this catalog, I will build my own based only on some manifests generated by this command) oc adm catalog build --appregistry-org redhat-operators --from=registry.redhat.io/openshift4/ose-operator-registry:v4.4 --to=<mirror_registry_host>:55555/olm/redhat-operators:v1 -a auth.json --insecure --manifest-dir='/home/cloud-user' 4. mkdir -p manifests/jaeger-product 5. copy generated manifests from step 3 for jaeger product e.g.: cp -r manifests-843937294/jaeger-product/jaeger-product-4rcw0o35/* manifests/jaeger-product/ 6. build custom catalog using attached Dockerfile: podman build -f Dockerfile -t <mirror_registry_host>:55555/olm/custom-reg:v1 7. push catalog image podman push <mirror_registry_host>:55555/olm/custom-reg:v1 8. mirror oc adm catalog mirror <mirror_registry_host>:55555/olm/custom-reg:v1 <mirror_registry_host>:55555 -a auth.json Actual results: Fails with: W0507 11:59:03.424681 21429 mirror.go:258] errors during mirroring. the full contents of the catalog may not have been mirrored: image does not exist I0507 11:59:03.425613 21429 mirror.go:329] wrote mirroring manifests to custom-reg-manifests No images are mirrored. Expected results: Images successfully mirrored. Additional info: Used Dockerfile attached. Content of manifests dir attached. [1] - https://docs.openshift.com/container-platform/4.4/installing/install_config/installing-restricted-networks-preparations.html Mirroring of redhat-operators:v1 actually works fine: oc adm catalog mirror <mirror_registry_host>:55555/olm/redhat-operators:v1 <mirror_registry_host>:55555 -a auth.json Reason to do all mentioned steps is to create custom catalog which contains only a few manifests, not all from redhat-operators.
Created attachment 1686235 [details] manifests directory
The attached Dockerfile is referencing latest, it should be referencing the target version of ocp (4.4 in this case). Do you see the same problem if you build with the 4.4 tag?
(In reply to Evan Cordell from comment #3) > The attached Dockerfile is referencing latest, it should be referencing the > target version of ocp (4.4 in this case). Do you see the same problem if you > build with the 4.4 tag? Yes, I tried also with: registry.redhat.io/openshift4/ose-operator-registry:v4.4 quay.io/operator-framework/upstream-registry-builder both with the same error. It would be very helpful if anybody could try to reproduce and provide some workaround.
The issue is visible only when using podman. I tried to build and push (steps 6 and 7) via docker and then the 'oc adm catalog mirror ...' (step 8) works fine. Tested podman 1.9.1 and 1.6.4 Docker version 19.03.8, build afacb8b
Hi Filip, I was able to reproduce the issue you're experiencing. A few behaviors seem to be involved: - by default, "podman build" produces OCI images - oc's requests to the registry manifests endpoint don't indicate that it can handle the OCI manifest media type via the Accept header - quay.io will convert manifests on-the-fly to a media type understood by the client (https://github.com/quay/quay/blob/312717c7891b56ce5433dd573d06d3f579bce944/endpoints/v2/manifest.py#L126-L129) - the docker registry implementation will only convert manifests from docker v2->v1 -- if it's an OCI manifest, and the request doesn't accept OCI manifests, an error is returned (https://github.com/docker/distribution/blob/244d5246c2f193d50b2f8fd220784958a5e1318b/registry/handlers/manifests.go#L168-L171) The error being surfaced as "image not found" looks like this on the wire: { "errors": [ { "code": "MANIFEST_UNKNOWN", "message": "OCI manifest found, but accept header does not support OCI manifests" } ] } To work around this with podman, try repeating the process with the invocation of "podman build ..." changed to to "podman build --format docker ...". There are at least two open questions: 1) Is there a reason oc can't or shouldn't support OCI image manifests? 2) Should the referenced documentation include instructions for deploying Quay in addition to or instead of the Docker registry?
Using "podman build --format docker ..." works fine. Thank you very much for providing the workaround.
We should document that, when using podman directly, --format docker is required in order for `oc` commands to work.
Closing this ticket as there is not sufficient data on which documentation pages are impacted by this bug. Please reopen the bug with specific documentation pages that are impacted by this bug.