Bug 1833008

Summary: Should set the `--format docker` for podman when building image if want to use `oc adm catalog mirror`
Product: OpenShift Container Platform Reporter: Filip Brychta <fbrychta>
Component: DocumentationAssignee: Samantha Gidlow <sagidlow>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Jian Zhang <jiazha>
Severity: high Docs Contact: Vikram Goyal <vigoyal>
Priority: high    
Version: 4.4CC: aos-bugs, bluddy, dsover, ecordell, jiazha, jokerman, kalexand
Target Milestone: ---   
Target Release: 4.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1840370 (view as bug list) Environment:
Last Closed: 2021-03-03 18:40:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Dockerfile
none
manifests directory none

Description Filip Brychta 2020-05-07 16:15:29 UTC 
Created attachment 1686234 [details]
Dockerfile

Description of problem:
I created my custom registry based on manifests generated from following command:
oc adm catalog build --appregistry-org redhat-operators --from=registry.redhat.io/openshift4/ose-operator-registry:v4.4 --to=<mirror_registry_host>:55555/olm/redhat-operators:v1  -a auth.json --insecure --manifest-dir='/home/cloud-user'

oc adm catalog mirror fails with:
W0507 11:42:27.072825   21414 mirror.go:258] errors during mirroring. the full contents of the catalog may not have been mirrored: image does not exist

Version-Release number of selected component (if applicable):
OCP 4.4

How reproducible:
Always

Steps to Reproduce:
1. prepare mirror registry following [1]
2. podman login to both mirror registry and registry.redhat.io
3. build catalog (note that I will not actually use this catalog, I will build my own based only on some manifests generated by this command)
oc adm catalog build --appregistry-org redhat-operators --from=registry.redhat.io/openshift4/ose-operator-registry:v4.4 --to=<mirror_registry_host>:55555/olm/redhat-operators:v1  -a auth.json --insecure --manifest-dir='/home/cloud-user'
4. mkdir -p manifests/jaeger-product
5. copy generated manifests from step 3 for jaeger product e.g.:
cp -r manifests-843937294/jaeger-product/jaeger-product-4rcw0o35/* manifests/jaeger-product/
6. build custom catalog using attached Dockerfile:
podman build -f Dockerfile -t <mirror_registry_host>:55555/olm/custom-reg:v1
7. push catalog image
podman push <mirror_registry_host>:55555/olm/custom-reg:v1
8. mirror
oc adm catalog mirror <mirror_registry_host>:55555/olm/custom-reg:v1 <mirror_registry_host>:55555 -a auth.json


Actual results:
Fails with:
W0507 11:59:03.424681   21429 mirror.go:258] errors during mirroring. the full contents of the catalog may not have been mirrored: image does not exist
I0507 11:59:03.425613   21429 mirror.go:329] wrote mirroring manifests to custom-reg-manifests

No images are mirrored.

Expected results:
Images successfully mirrored.

Additional info:
Used Dockerfile attached.
Content of manifests dir attached.
[1] - https://docs.openshift.com/container-platform/4.4/installing/install_config/installing-restricted-networks-preparations.html

Mirroring of redhat-operators:v1 actually works fine:
oc adm catalog mirror <mirror_registry_host>:55555/olm/redhat-operators:v1 <mirror_registry_host>:55555 -a auth.json

Reason to do all mentioned steps is to create custom catalog which contains only a few manifests, not all from redhat-operators.
Comment 1 Filip Brychta 2020-05-07 16:20:45 UTC 
Created attachment 1686235 [details]
manifests directory
Comment 3 Evan Cordell 2020-05-07 19:35:25 UTC 
The attached Dockerfile is referencing latest, it should be referencing the target version of ocp (4.4 in this case). Do you see the same problem if you build with the 4.4 tag?
Comment 4 Filip Brychta 2020-05-11 08:39:56 UTC 
(In reply to Evan Cordell from comment #3)
> The attached Dockerfile is referencing latest, it should be referencing the
> target version of ocp (4.4 in this case). Do you see the same problem if you
> build with the 4.4 tag?

Yes, I tried also with:
registry.redhat.io/openshift4/ose-operator-registry:v4.4
quay.io/operator-framework/upstream-registry-builder

both with the same error.

It would be very helpful if anybody could try to reproduce and provide some workaround.
Comment 5 Filip Brychta 2020-05-11 18:33:05 UTC 
The issue is visible only when using podman.
I tried to build and push (steps 6 and 7) via docker and then the 'oc adm catalog mirror ...' (step 8) works fine.

Tested podman 1.9.1 and 1.6.4
Docker version 19.03.8, build afacb8b
Comment 6 Ben Luddy 2020-05-13 01:34:55 UTC 
Hi Filip,

I was able to reproduce the issue you're experiencing. A few behaviors seem to be involved:

- by default, "podman build" produces OCI images
- oc's requests to the registry manifests endpoint don't indicate that it can handle the OCI manifest media type via the Accept header
- quay.io will convert manifests on-the-fly to a media type understood by the client (https://github.com/quay/quay/blob/312717c7891b56ce5433dd573d06d3f579bce944/endpoints/v2/manifest.py#L126-L129)
- the docker registry implementation will only convert manifests from docker v2->v1 -- if it's an OCI manifest, and the request doesn't accept OCI manifests, an error is returned (https://github.com/docker/distribution/blob/244d5246c2f193d50b2f8fd220784958a5e1318b/registry/handlers/manifests.go#L168-L171)

The error being surfaced as "image not found" looks like this on the wire:

{
  "errors": [
    {
      "code": "MANIFEST_UNKNOWN",
      "message": "OCI manifest found, but accept header does not support OCI manifests"
    }
  ]
}

To work around this with podman, try repeating the process with the invocation of "podman build ..." changed to to "podman build --format docker ...".

There are at least two open questions:

1) Is there a reason oc can't or shouldn't support OCI image manifests?
2) Should the referenced documentation include instructions for deploying Quay in addition to or instead of the Docker registry?
Comment 7 Filip Brychta 2020-05-13 10:40:08 UTC 
Using "podman build --format docker ..." works fine.
Thank you very much for providing the workaround.
Comment 8 Evan Cordell 2020-05-26 19:27:08 UTC 
We should document that, when using podman directly, --format docker is required in order for `oc` commands to work.
Comment 9 Samantha Gidlow 2021-03-03 18:40:15 UTC 
Closing this ticket as there is not sufficient data on which documentation pages are impacted by this bug. Please reopen the bug with specific documentation pages that are impacted by this bug.