Bug 1840370

1, Use the default base image: quay.io/operator-framework/operator-registry-server:latest

[root@preserve-olm-env data]# oc adm catalog build --appregistry-org redhat-operators --to localhost:5000/olm/redhat-operators:v1
INFO[0024] loading Bundles                               dir=/tmp/manifests-138997843
INFO[0024] directory                                     dir=/tmp/manifests-138997843 file=manifests-138997843 load=bundles
INFO[0024] directory                                     dir=/tmp/manifests-138997843 file=3scale-operator load=bundles
...

2, 
[root@preserve-olm-env data]# mkdir -p manifests/jaeger-product
[root@preserve-olm-env data]# cp -r /tmp/manifests-138997843/jaeger-product/jaeger-product-4rcw0o35/* manifests/jaeger-product/

3, Create the image by using the Podman. Use the latest image(quay.io/operator-framework/upstream-registry-builder:latest) as the builder image.

[root@preserve-olm-env data]# cat Dockerfile 
FROM quay.io/operator-framework/upstream-registry-builder:latest AS builder

COPY manifests manifests

RUN /bin/initializer -o ./bundles.db

FROM registry.access.redhat.com/ubi7/ubi

COPY --from=builder /build/bundles.db /bundles.db
COPY --from=builder /bin/registry-server /registry-server
COPY --from=builder /bin/grpc_health_probe /bin/grpc_health_probe

EXPOSE 50051

ENTRYPOINT ["/registry-server"]

CMD ["--database", "bundles.db"]


[root@preserve-olm-env data]# podman version
Version:            1.6.4
RemoteAPI Version:  1
Go Version:         go1.12.12
OS/Arch:            linux/amd64


[root@preserve-olm-env data]# podman build -f Dockerfile -t localhost:5000/olm/custom-reg:v1
STEP 1: FROM quay.io/operator-framework/upstream-registry-builder:latest AS builder
STEP 2: COPY manifests manifests
--> Using cache f7a4ba0281c53d09f4ed5d1dd137710fefb433ef6ad9ef80a1a47e88356d4323
STEP 3: RUN /bin/initializer -o ./bundles.db
--> Using cache 85f9829dfca6736f50c2d3310ea710337584df2868a748786dc83223c144fe25
STEP 4: FROM registry.access.redhat.com/ubi7/ubi
STEP 5: COPY --from=builder /build/bundles.db /bundles.db
59ebab8782db29cf25b2ff3dd72a47c925dfeb2c0e11f2c94ea317425dd1fd7b
STEP 6: COPY --from=builder /bin/registry-server /registry-server
788bf5ec845d2830d1be3b501381013bfcde3d72b55e8c632e3d0a27101d4c21
STEP 7: COPY --from=builder /bin/grpc_health_probe /bin/grpc_health_probe
bd0cf2e94c5e173540f6b887aee5eda30545ee953e754fa747955036e21b7597
STEP 8: EXPOSE 50051
1cbc29c52bd83fe2a0e09ba3c1f9843ca320374759eb6aebd9737502c24c7381
STEP 9: ENTRYPOINT ["/registry-server"]
9632ead194210764ec44f4d5663e1701cdbe76d0e3f82c327755b24f02ac937d
STEP 10: CMD ["--database", "bundles.db"]
STEP 11: COMMIT localhost:5000/olm/custom-reg:v1
03be5180a534d6b3c5bbd63e83f2611c71491a1fa9d3dd52b1a6898b1fe4b059

4, push the image
[root@preserve-olm-env data]# podman push localhost:5000/olm/custom-reg:v1
Getting image source signatures
Copying blob 29fb9753ac06 done  
Copying blob 7d55429c05da done  
Copying blob 429c3acbc127 done  
Copying blob 933e707b3698 done  
Copying blob 895d16eabfa1 done  
Copying config 03be5180a5 done  
Writing manifest to image destination
Storing signatures

5, Mirror it, but still get the same error:
[root@preserve-olm-env data]# ./oc adm catalog mirror localhost:5000/olm/custom-reg:v1 localhost:5000
errors during mirroring. the full contents of the catalog may not have been mirrored: image does not exist
wrote mirroring manifests to custom-reg-manifests

Sorry, it was unclear what was changed due to the amount of context in the original report.

This is a change to the outputs produced by "opm index add" when the "--container-tool" option is set to "podman".

With the change, the media type of the index image's manifest is "application/vnd.docker.distribution.manifest.v2+json":

$ opm index add --container-tool podman --bundles '' -t xyz
INFO[0000] building the index                            bundles="[]"
INFO[0000] Generating dockerfile                         bundles="[]"
INFO[0000] writing dockerfile: index.Dockerfile432739094  bundles="[]"
INFO[0000] running podman build                          bundles="[]"
INFO[0000] [podman build --format docker -f index.Dockerfile432739094 -t xyz .]  bundles="[]"

$ podman image inspect -f '{{.ManifestType}}' xyz
application/vnd.docker.distribution.manifest.v2+json

And before the change, it is "application/vnd.oci.image.manifest.v1+json":

$ opm index add --container-tool podman --bundles '' -t abc
INFO[0000] building the index                            bundles="[]"
INFO[0000] Generating dockerfile                         bundles="[]"
INFO[0000] writing dockerfile: index.Dockerfile896731071  bundles="[]"
INFO[0000] running podman build                          bundles="[]"
INFO[0000] [podman build -f index.Dockerfile896731071 -t abc .]  bundles="[]"

$ podman image inspect -f '{{.ManifestType}}' abc
application/vnd.oci.image.manifest.v1+json

If you look at the output of opm index add after the change, you'll notice that it is passing "--format docker" to the underlying podman build command.

Hi Ben,

Thanks for your explanation! I see now.
Yes, I can see the manifest.v1+json before this change, as follows:
[root@preserve-olm-env opm]# ./opm index add --container-tool podman -b quay.io/olmqe/etcd-bundle:0.9.0 -t quay.io/olmqe/etcd-index:test
INFO[0000] building the index                            bundles="[quay.io/olmqe/etcd-bundle:0.9.0]"
INFO[0000] running /usr/bin/podman pull quay.io/olmqe/etcd-bundle:0.9.0  bundles="[quay.io/olmqe/etcd-bundle:0.9.0]"
INFO[0001] running /usr/bin/podman pull quay.io/olmqe/etcd-bundle:0.9.0  bundles="[quay.io/olmqe/etcd-bundle:0.9.0]"
...
INFO[0012] [podman build -f index.Dockerfile831554325 -t quay.io/olmqe/etcd-index:test .]  bundles="[quay.io/olmqe/etcd-bundle:0.9.0]"

[root@preserve-olm-env opm]# podman image inspect -f '{{.ManifestType}}' quay.io/olmqe/etcd-index:test
application/vnd.oci.image.manifest.v1+json

After this change, we can see the version changed to `manifest.v2+json`, and `--format docker` added in "podman build" process.

[root@preserve-olm-env operator-registry]# git log
commit 054cd90a84ecadc24d6b94e955c3437ac740a1a7
Merge: 7437af6 2c009be
Author: OpenShift Merge Robot <openshift-merge-robot.github.com>
Date:   Thu May 28 14:36:19 2020 -0400

    Merge pull request #340 from ecordell/ro
    
    Bug 1840727: fix(unpack): support unpacking readonly folders

..
[root@preserve-olm-env operator-registry]# make build
...

[root@preserve-olm-env operator-registry]# ./bin/opm index add --container-tool podman -b quay.io/olmqe/etcd-bundle:0.9.0 -t quay.io/olmqe/etcd-index:test3
INFO[0000] building the index                            bundles="[quay.io/olmqe/etcd-bundle:0.9.0]"
INFO[0000] running /usr/bin/podman pull quay.io/olmqe/etcd-bundle:0.9.0  bundles="[quay.io/olmqe/etcd-bundle:0.9.0]"
INFO[0010] running /usr/bin/podman pull quay.io/olmqe/etcd-bundle:0.9.0  bundles="[quay.io/olmqe/etcd-bundle:0.9.0]"
INFO[0011] running podman save                           bundles="[quay.io/olmqe/etcd-bundle:0.9.0]"
INFO[0011] Could not find optional dependencies file     dir=bundle_tmp894188428 file=bundle_tmp894188428/metadata load=annotations
INFO[0011] found csv, loading bundle                     dir=bundle_tmp894188428 file=bundle_tmp894188428/manifests load=bundle
INFO[0011] loading bundle file                           dir=bundle_tmp894188428/manifests file=etcdbackups.etcd.database.coreos.com.crd.yaml load=bundle
INFO[0011] loading bundle file                           dir=bundle_tmp894188428/manifests file=etcdclusters.etcd.database.coreos.com.crd.yaml load=bundle
INFO[0011] loading bundle file                           dir=bundle_tmp894188428/manifests file=etcdoperator.v0.9.0.clusterserviceversion.yaml load=bundle
INFO[0011] loading bundle file                           dir=bundle_tmp894188428/manifests file=etcdrestores.etcd.database.coreos.com.crd.yaml load=bundle
INFO[0011] Generating dockerfile                         bundles="[quay.io/olmqe/etcd-bundle:0.9.0]"
INFO[0011] writing dockerfile: index.Dockerfile067690954  bundles="[quay.io/olmqe/etcd-bundle:0.9.0]"
INFO[0011] running podman build                          bundles="[quay.io/olmqe/etcd-bundle:0.9.0]"
INFO[0011] [podman build --format docker -f index.Dockerfile067690954 -t quay.io/olmqe/etcd-index:test3 .]  bundles="[quay.io/olmqe/etcd-bundle:0.9.0]"

[root@preserve-olm-env operator-registry]# podman image inspect -f '{{.ManifestType}}' quay.io/olmqe/etcd-index:test3
application/vnd.docker.distribution.manifest.v2+json

LGTM, verify it.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409