Bug 1773821

Mirroring requires that the target repo have docker v2-2 support enabled in Quay. Very few namespaces currently have this enabled. 

Can you reproduce this if you mirror to a local file, or to a local docker registry that supports v2-2?

Evan,

I'm sorry, what's the `docker v2-2 support` mean? Could you help give me a link to its instructions?
I guess you mean that registry is an application registry, which stores the manifests, not the images, right?

> Can you reproduce this if you mirror to a local file, or to a local docker registry that supports v2-2?

Yes. I can reproduce it. 
I create an application registry in Quay, called "mirror-test". See the screenshot: https://user-images.githubusercontent.com/15416633/69316288-790eb200-0c73-11ea-9032-4e22c0f26be7.png

[root@dhcp-140-36 ~]# oc adm catalog mirror  quay.io/jiazha/learn-test:v4 quay.io/jiazha/mirror-test:v4
I1121 15:22:47.501411    4496 mirror.go:190] wrote database to /tmp/820442524/bundles.db
info: Planning completed in 0s
info: Mirroring completed in 0s (0B/s)
I1121 15:22:47.505642    4496 mirror.go:288] wrote mirroring manifests to learn-test-manifests

[root@dhcp-140-36 ~]# sqlite3 /tmp/820442524/bundles.db
SQLite version 3.22.0 2018-01-22 18:45:57
Enter ".help" for usage hints.
sqlite> .table
api                channel            package          
api_provider       channel_entry      related_image    
api_requirer       operatorbundle     schema_migrations
sqlite> select * from operatorbundle;
sqlite> select * from package;
sqlite> .quit
               
[root@dhcp-140-36 ~]# cat learn-test-manifests/imageContentSourcePolicy.yaml 
apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
  name: learn-test
spec:
  repositoryDigestMirrors: []

[root@dhcp-140-36 ~]# cat learn-test-manifests/mapping.txt 
[root@dhcp-140-36 ~]#

v2-2 is a version of the docker manifest spec: https://docs.docker.com/registry/spec/manifest-v2-2/

Very few namespaces in Quay can support this version. Your namespace, jiazha, does not. Any attempt to test `oc adm catalog build` that targets a registry which does not support v2-2 will fail in the way you're seeing here.

A simple way to get around this is to run the docker registry container:

docker run -it --rm -p 5000:5000 registry

and then mirror with that:

oc adm catalog build --appregistry-org=jiazha --to=localhost:5000/my/catalog:1
oc adm catalog mirror localhost:5000/my/catalog:1 localhost:5000

Hi, Evan

Thanks!

> Very few namespaces in Quay can support this version. Your namespace, jiazha, does not. Any attempt to test `oc adm catalog build` that targets a registry which does not support v2-2 will fail in the way you're seeing here.

I think we should highlight this in our 4.3 release doc, what do you think?

> A simple way to get around this is to run the docker registry container:

Yes, thanks! I run "docker run -it --rm -p 5000:5000 registry" as follows:
[root@dhcp-140-36 ~]# docker run -it --rm -p 5000:5000 registry
WARN[0000] No HTTP secret provided - generated random secret. This may cause problems with uploads if multiple registries are behind a load-balancer. To provide a shared secret, fill in http.secret in the configuration file or set the REGISTRY_HTTP_SECRET environment variable.  go.version=go1.11.2 instance.id=871badf0-b4d3-44fe-aca3-300b969ede4f service=registry version=v2.7.1
INFO[0000] redis not configured                          go.version=go1.11.2 instance.id=871badf0-b4d3-44fe-aca3-300b969ede4f service=registry version=v2.7.1
INFO[0000] Starting upload purge in 1m0s                 go.version=go1.11.2 instance.id=871badf0-b4d3-44fe-aca3-300b969ede4f service=registry version=v2.7.1
INFO[0000] using inmemory blob descriptor cache          go.version=go1.11.2 instance.id=871badf0-b4d3-44fe-aca3-300b969ede4f service=registry version=v2.7.1
INFO[0000] listening on [::]:5000                        go.version=go1.11.2 instance.id=871badf0-b4d3-44fe-aca3-300b969ede4f service=registry version=v2.7.1
INFO[0060] PurgeUploads starting: olderThan=2019-11-18 08:16:34.579628009 +0000 UTC m=-604739.979538633, actuallyDelete=true 
INFO[0060] Purge uploads finished.  Num deleted=0, num errors=1 
INFO[0060] Starting upload purge in 24h0m0s              go.version=go1.11.2 instance.id=871badf0-b4d3-44fe-aca3-300b969ede4f service=registry version=v2.7.1


And then, I run "oc adm catalog build --appregistry-org=jiazha --to=localhost:5000/jiazha/catalog:v1" in another terminal. But, got the errors below:
[root@dhcp-140-36 db-731491371]# oc adm catalog build --appregistry-org=jiazha --to=localhost:5000/jiazha/catalog:v1 
...
Unable to connect to the server: http: server gave HTTP response to HTTPS client

I already config the "--insecure-registry localhost:5000" in /etc/sysconfig/docker and "systemctl restart docker". You can see as follows, do you know how to solve it?
[root@dhcp-140-36 ~]# docker info
Containers: 5
 Running: 0
 Paused: 0
 Stopped: 5
Images: 40
Server Version: 1.13.1
...
Insecure Registries:
 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888
 localhost:5000
 127.0.0.0/8
Live Restore Enabled: true
Registries: docker.io (secure), registry.fedoraproject.org (secure), quay.io (secure), registry.access.redhat.com (secure), registry.centos.org (secure), docker.io (secure)


> oc adm catalog mirror localhost:5000/my/catalog:1 localhost:5000
> Very few namespaces in Quay can support this version.

I'm confused. I think we should mirror this image to the disconnected cluster, not the localhost:5000.
Does the "internal-registry.qe.devcluster.openshift.com:5000" support v2-2? If not, the users how to mirror the image to the disconnected cluster?

apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
  name: etcd-operator
spec:
  repositoryDigestMirrors:
  - mirrors:
    - internal-registry.qe.devcluster.openshift.com:5000/coreos/etcd-operato
    source: quay.io/coreos/etcd-operator

Hi, Evan

> I'm confused. I think we should mirror this image to the disconnected cluster, not the localhost:5000.
Does the "internal-registry.qe.devcluster.openshift.com:5000" support v2-2? If not, the users how to mirror the image to the disconnected cluster?

Please ignore this question. To be more clear, I list it below:

1, For example, I have a disconnected cluster and only can access this registry: internal-registry.qe.devcluster.openshift.com:5000(the customers create it by themselves)
So, the `oc adm catalog mirror` command can generate an ImageContentSourcePolicy object(as follows, correct me if I'm wrong).
So, this customized registry "internal-registry.qe.devcluster.openshift.com:5000" also needs to support v2-2, right? 

apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
  name: jiazha-operator
spec:
  repositoryDigestMirrors:
  - mirrors:
    - internal-registry.qe.devcluster.openshift.com:5000/jiazha/catalog:v1
    source: localhost:5000/jiazha/catalog:v1

And, the users still need to run this "oc image mirror" commands, right? 
Failed, the same errors. Could you help have a look? Thanks!
[root@dhcp-140-36 ~]# oc image mirror localhost:5000/jiazha/catalog:v1 internal-registry.qe.devcluster.openshift.com:5000/jiazha/catalog:v1

error: unable to connect to localhost:5000/jiazha/catalog: Get https://localhost:5000/v2/: http: server gave HTTP response to HTTPS client
error: an error occurred during planning

From comment #6, this is how you are exposing the registry:

# oc patch configs.imageregistry.operator.openshift.io/cluster -p='{"spec":{"defaultRoute":true}}' --type=merge

This seems to match the steps listed here: https://docs.openshift.com/container-platform/4.2/registry/securing-exposing-registry.html, which says: 

However, those docs also say:

--tls-verify=false is needed if the cluster’s default certificate for routes is untrusted.

If you add the cluster certs to your OS's trusted certs, it should work. 

The `oc adm catalog build` command does not support `--tls-verify=false`. I think that would be a reasonable thing to add for 4.4. But I don't think it's correct to say that there is a bug in the mirroring, which is what this BZ is about? 

Perhaps the docs should be updated to indicate the the internal registry must be exposed with a cert that is trusted by the host?

https://github.com/openshift/oc/pull/212

This PR adds options to `oc adm catalog` commands to configure talking to registries without trusted certs.

Change status to POST since the fixed PR hasn't been merged.

Hi, Evan, Jeff

Based on my understanding, this `--tls-verify=false` issue is an enhance fix. For now, I think there are two key problems:
1, The image created by the `oc adm catalog build` cannot work as expected. No packagemanifest are provided. Steps as follows:(Or am I missing something?)
Cluster version is 4.3.0-0.nightly-2019-12-24-221900
[jzhang@dhcp-140-36 ~]$ oc version
Client Version: 4.3.0-0.nightly-2019-12-24-053745
Server Version: 4.3.0-0.nightly-2019-12-24-221900
Kubernetes Version: v1.16.2

1) Builds a catalog container image and push it to the cluster internal registry. 
[jzhang@dhcp-140-36 ~]$ oc patch configs.imageregistry.operator.openshift.io/cluster -p='{"spec":{"defaultRoute":true}}' --type=merge
config.imageregistry.operator.openshift.io/cluster patched
[jzhang@dhcp-140-36 ~]$ oc get images.config.openshift.io/cluster  -o jsonpath={.status.externalRegistryHostnames[0]} 
default-route-openshift-image-registry.apps.qe-jiazha3-dis.qe.devcluster.openshift.com

[jzhang@dhcp-140-36 ~]$ oc get secret router-certs-default -n openshift-ingress -o json |jq -r '.data["tls.crt"]' | base64 -d | sudo tee /etc/pki/ca-trust/source/anchors/disconnected.crt
[jzhang@dhcp-140-36 ~]$ sudo update-ca-trust 

[jzhang@dhcp-140-36 ~]$ oc adm catalog build --appregistry-org=redhat-operator-art --to=default-route-openshift-image-registry.apps.qe-jiazha3-dis.qe.devcluster.openshift.com/openshift/olm-registry:test
INFO[0001] loading Bundles                               dir=/tmp/manifests-689646936
INFO[0001] directory                                     dir=/tmp/manifests-689646936 file=manifests-689646936 load=bundles
INFO[0001] loading Packages and Entries                  dir=/tmp/manifests-689646936
INFO[0001] directory                                     dir=/tmp/manifests-689646936 file=manifests-689646936 load=package
Uploading ... 468.6B/s
Uploading 3.405MB ...
Uploading 9.835MB ...
Pushed sha256:63af847e30cd6f16f81165c0e133f0cad6dd308b334cc96b6c9e75ef4b6c39a7 to default-route-openshift-image-registry.apps.qe-jiazha3-dis.qe.devcluster.openshift.com/openshift/olm-registry:test

[jzhang@dhcp-140-36 ~]$ oc get images |grep olm
sha256:63af847e30cd6f16f81165c0e133f0cad6dd308b334cc96b6c9e75ef4b6c39a7   image-registry.openshift-image-registry.svc:5000/openshift/olm-registry@sha256:63af847e30cd6f16f81165c0e133f0cad6dd308b334cc96b6c9e75ef4b6c39a7

2) Create a CatalogSource to consume this image: image-registry.openshift-image-registry.svc:5000/openshift/olm-registry:test

[jzhang@dhcp-140-36 ~]$ cat catalogsource.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: image-registry
  namespace: openshift-marketplace
spec:
  image: image-registry.openshift-image-registry.svc:5000/openshift/olm-registry:test
  sourceType: grpc
[jzhang@dhcp-140-36 ~]$ oc create -f catalogsource.yaml 
catalogsource.operators.coreos.com/image-registry created
[jzhang@dhcp-140-36 ~]$ oc get pods
NAME                                    READY   STATUS    RESTARTS   AGE
image-registry-fwnsh                    1/1     Running   0          113s
marketplace-operator-7bdc4c767b-m4tfk   1/1     Running   0          5h28m
[jzhang@dhcp-140-36 ~]$ oc get catalogsource
NAME             DISPLAY   TYPE   PUBLISHER   AGE
image-registry             grpc               2m14s

3) Check the packagemanifest.But, got nothing.
[jzhang@dhcp-140-36 ~]$ oc get packagemanifest
No resources found in openshift-marketplace namespace


2, The `oc adm catalog mirror` command cannot query the catalog database to find all of the images used in the manifests and failed to fill in the `ImageContentSourcePolicy` and mapping.txt. Steps as follows:

1) Create a registry, which supports v2-2.
[jzhang@dhcp-140-36 testdata]$ chcon -Rt svirt_sandbox_file_t /home/jzhang/goproject/src/github.com/operator-framework/operator-lifecycle-manager/pkg/lib/filemonitor/testdata
[jzhang@dhcp-140-36 testdata]$ 
[jzhang@dhcp-140-36 testdata]$ docker run -it --rm -p 5000:5000 --name=registry -v /home/jzhang/goproject/src/github.com/operator-framework/operator-lifecycle-manager/pkg/lib/filemonitor/testdata/:/certs -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/server-new.crt -e REGISTRY_HTTP_TLS_KEY=/certs/server-new.key registry
WARN[0000] No HTTP secret provided - generated random secret. This may cause problems with uploads if multiple registries are behind a load-balancer. To provide a shared secret, fill in http.secret in the configuration file or set the REGISTRY_HTTP_SECRET environment variable.  go.version=go1.11.2 instance.id=fe07c5a1-f25b-4c0e-b30c-50d80253cb6a service=registry version=v2.7.1
INFO[0000] Starting upload purge in 29m0s                go.version=go1.11.2 instance.id=fe07c5a1-f25b-4c0e-b30c-50d80253cb6a service=registry version=v2.7.1
INFO[0000] redis not configured                          go.version=go1.11.2 instance.id=fe07c5a1-f25b-4c0e-b30c-50d80253cb6a service=registry version=v2.7.1
INFO[0000] using inmemory blob descriptor cache          go.version=go1.11.2 instance.id=fe07c5a1-f25b-4c0e-b30c-50d80253cb6a service=registry version=v2.7.1
INFO[0000] listening on [::]:5000, tls                   go.version=go1.11.2 instance.id=fe07c5a1-f25b-4c0e-b30c-50d80253cb6a service=registry version=v2.7.1
INFO[0016] response completed                            go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=07b30663-4d9b-4696-aaec-1685db07a98e http.request.method=GET http.request.remoteaddr="172.17.0.1:35976" http.request.uri="/v2/" http.request.useragent="Go-http-client/2.0" http.response.contenttype="application/json; charset=utf-8" http.response.duration="605.159µs" http.response.status=200 http.response.written=2
172.17.0.1 - - [25/Dec/2019:09:08:11 +0000] "GET /v2/ HTTP/2.0" 200 2 "" "Go-http-client/2.0"
INFO[0019] response completed                            go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=01730582-a07b-4c85-9df7-f18839dfe9d5 http.request.method=POST http.request.remoteaddr="172.17.0.1:35976" http.request.uri="/v2/openshift/jiazha-registry/blobs/uploads/" http.request.useragent="Go-http-client/2.0" http.response.duration=12.696139ms http.response.status=202 http.response.written=0
...

2) Create the catalog images by using the "oc adm catalog build" command.
[jzhang@dhcp-140-36 ~]$ oc adm catalog build --appregistry-org=redhat-operator-art --to=localhost:5000/openshift/jiazha-registry:test
INFO[0001] loading Bundles                               dir=/tmp/manifests-097097322
INFO[0001] directory                                     dir=/tmp/manifests-097097322 file=manifests-097097322 load=bundles
INFO[0001] loading Packages and Entries                  dir=/tmp/manifests-097097322
INFO[0001] directory                                     dir=/tmp/manifests-097097322 file=manifests-097097322 load=package
Uploading ... 8.255kB/s
Uploading 9.835MB ...
Uploading 3.405MB ...
Pushed sha256:d30e41142a2c6f2a3c1fc7491ece07fe2b5ac70948bd02cb366eabeffd8d9535 to localhost:5000/openshift/jiazha-registry:test

3) Mirror the catalog image to the cluster internal registry.
[jzhang@dhcp-140-36 ~]$ oc adm catalog mirror localhost:5000/openshift/jiazha-registry:test default-route-openshift-image-registry.apps.qe-jiazha3-dis.qe.devcluster.openshift.com/openshift/jiazha-registry:test
I1225 17:10:56.464310    8291 mirror.go:190] wrote database to /tmp/419169569/bundles.db


info: Planning completed in 0s
info: Mirroring completed in 0s (0B/s)
I1225 17:10:56.465256    8291 mirror.go:288] wrote mirroring manifests to jiazha-registry-manifests

4) Check the database, no manifest extracted.
[jzhang@dhcp-140-36 ~]$ sqlite3 /tmp/419169569/bundles.db
SQLite version 3.22.0 2018-01-22 18:45:57
Enter ".help" for usage hints.
sqlite> .table
api                channel            package          
api_provider       channel_entry      related_image    
api_requirer       operatorbundle     schema_migrations
sqlite> select * from operatorbundle;
sqlite> 

5) Check the imageContentSourcePolicy.yaml and mapping.txt files. Nothing：
[jzhang@dhcp-140-36 ~]$ tree jiazha-registry-manifests/
jiazha-registry-manifests/
├── imageContentSourcePolicy.yaml
└── mapping.txt

0 directories, 2 files
[jzhang@dhcp-140-36 ~]$ cat jiazha-registry-manifests/imageContentSourcePolicy.yaml 
apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
  name: jiazha-registry
spec:
  repositoryDigestMirrors: []
[jzhang@dhcp-140-36 ~]$ cat jiazha-registry-manifests/mapping.txt 
[jzhang@dhcp-140-36 ~]$ 


In conclusion, the two commands： "catalog build" and "catalog mirror" don't help the customers to sync the images/manifest to the cluster.
Or am I missing something？Change status to ASSIGNED status based on the latest test.

Moving this to 4.4.  We will investigate and if issue is validated, consider for the first or early 4.3.z backport.

I think the primary issue is the appregistry source. redhat-operators-art is private and no auth token is provided via `--auth-token` for it.

I think it's a bug that `oc adm catalog build` doesn't immediately error out if it can't find any data in appregistry, though the logs you posted above indicate this (there are no files found via the logs walking through the downloaded manifests).

But we can still work through this with an appregistry repo that we know works, like redhat-operators:

Following along, I spun up a cluster:

$ oc version
Client Version: unknown
Server Version: 4.3.0-0.ci-2020-01-13-134436
Kubernetes Version: v1.16.2

(I used a locally built version of `oc` so that I could run it on macos)

$ ./oc patch configs.imageregistry.operator.openshift.io/cluster -p='{"spec":{"defaultRoute":true}}' --type=merge
config.imageregistry.operator.openshift.io/cluster patched

$ ./oc get images.config.openshift.io/cluster  -o json | jq '.status.externalRegistryHostnames[0]'
"default-route-openshift-image-registry.apps.ci-ln-hp9ckrt-d5d6b.origin-ci-int-aws.dev.rhcloud.com"

$ ./oc create serviceaccount registry
serviceaccount/registry created

$ ./oc adm policy add-cluster-role-to-user admin -z registry
clusterrole.rbac.authorization.k8s.io/admin added: "registry"

Added default-route-openshift-image-registry.apps.ci-ln-hp9ckrt-d5d6b.origin-ci-int-aws.dev.rhcloud.com to insecure registries in my local docker config.
 
$ docker login default-route-openshift-image-registry.apps.ci-ln-hp9ckrt-d5d6b.origin-ci-int-aws.dev.rhcloud.com -u registry -p `oc sa get-token registry`

$ oc adm catalog build --insecure --appregistry-org=redhat-operator-art --to=default-route-openshift-image-registry.apps.ci-ln-hp9ckrt-d5d6b.origin-ci-int-aws.dev.rhcloud.com/default/olm-registry:test
oad=package
< cut for space> 
INFO[0012] directory                                     dir=/var/folders/4m/pt431q9d2fsbk9zz2_vmg3tc0000gn/T/manifests-168009073 file=servicemeshoperator-hcujs8is load=package
INFO[0012] directory                                     dir=/var/folders/4m/pt431q9d2fsbk9zz2_vmg3tc0000gn/T/manifests-168009073 file=1.0.0 load=package
INFO[0012] directory                                     dir=/var/folders/4m/pt431q9d2fsbk9zz2_vmg3tc0000gn/T/manifests-168009073 file=1.0.1 load=package
INFO[0012] directory                                     dir=/var/folders/4m/pt431q9d2fsbk9zz2_vmg3tc0000gn/T/manifests-168009073 file=1.0.2 load=package
INFO[0012] directory                                     dir=/var/folders/4m/pt431q9d2fsbk9zz2_vmg3tc0000gn/T/manifests-168009073 file=1.0.3 load=package
INFO[0012] directory                                     dir=/var/folders/4m/pt431q9d2fsbk9zz2_vmg3tc0000gn/T/manifests-168009073 file=sriov-network-operator load=package
INFO[0012] directory                                     dir=/var/folders/4m/pt431q9d2fsbk9zz2_vmg3tc0000gn/T/manifests-168009073 file=sriov-network-operator-mvft_noa load=package
INFO[0012] directory                                     dir=/var/folders/4m/pt431q9d2fsbk9zz2_vmg3tc0000gn/T/manifests-168009073 file=4.2 load=package
INFO[0012] directory                                     dir=/var/folders/4m/pt431q9d2fsbk9zz2_vmg3tc0000gn/T/manifests-168009073 file=4.2-s390x load=package
Uploading ... 1.097MB/s
Pushed sha256:807378fe734367d62b91b2e36ccc31561858343ce5172c42439e62a588c99e7d to default-route-openshift-image-registry.apps.ci-ln-hp9ckrt-d5d6b.origin-ci-int-aws.dev.rhcloud.com/default/olm-registry:test

$ docker create default-route-openshift-image-registry.apps.ci-ln-hp9ckrt-d5d6b.origin-ci-int-aws.dev.rhcloud.com/default/olm-registry:test
835cef8cf9df66d8577ef489f9096c08f2c6b5d4cce98ddf7edb23baf7f7c3ce

$ docker cp 835cef8cf9df66d8577ef489f9096c08f2c6b5d4cce98ddf7edb23baf7f7c3ce:/bundles.db ./bundles.db

$  sqlite3 ./bundles.db
SQLite version 3.24.0 2018-06-04 14:10:15
Enter ".help" for usage hints.
sqlite> select * from package;
3scale-operator|threescale-2.7
amq-broker|alpha
amq-online|stable
amq-streams|stable
amq7-cert-manager|alpha
amq7-interconnect-operator|1.1.0
businessautomation-operator|stable
cam-operator|release-v1
cluster-logging|4.2
codeready-workspaces|latest
datagrid|alpha
dv-operator|alpha
eap|alpha
elasticsearch-operator|4.2
fuse-apicurito|alpha
fuse-camel-k|alpha
fuse-online|alpha
jaeger-product|stable
kiali-ossm|stable
kubevirt-hyperconverged|2.1
local-storage-operator|4.2
metering-ocp|4.2
nfd|4.2
openshiftansibleservicebroker|4.2
openshifttemplateservicebroker|4.2
serverless-operator|techpreview
servicemeshoperator|1.0
sriov-network-operator|4.2
sqlite> select count(*) from operatorbundle;
67


$ oc adm catalog mirror --insecure default-route-openshift-image-registry.apps.ci-ln-hp9ckrt-d5d6b.origin-ci-int-aws.dev.rhcloud.com/default/olm-
I0113 13:58:52.928111   97851 mirror.go:229] wrote database to /var/folders/4m/pt431q9d2fsbk9zz2_vmg3tc0000gn/T/098749310/bundles.db
W0113 13:58:52.931136   97851 mirror.go:161] couldn't parse registry.redhat.io/3scale-amp26/3scale-operator, skipping mirror
default-route-openshift-image-registry.apps.ci-ln-hp9ckrt-d5d6b.origin-ci-int-aws.dev.rhcloud.com/
  3scale-amp2/3scale-rhel7-operator
    blobs:
      registry.redhat.io/3scale-amp2/3scale-rhel7-operator sha256:4c98734f24339b059854b6f7ad77928ffb6b84756ecd4eeec4a15870b082d906 1.253KiB
      registry.redhat.io/3scale-amp2/3scale-rhel7-operator sha256:305d73a95c8fece2b53a34e040df1c97eb6b7f7cc4e0a7933465f0b7325e3d72 1.298KiB
      registry.redhat.io/3scale-amp2/3scale-rhel7-operator sha256:1c2551c529bf4998984f847f4842bfad68335f88d58c289be85d8abac3cc31f6 3.92KiB
      registry.redhat.io/3scale-amp2/3scale-rhel7-operator sha256:3b0e594e84ca967270f2dcb523d45b7af1cbe4ced54f7ba2d1df9030ce64d9a8 3.921KiB
      registry.redhat.io/3scale-amp2/3scale-rhel7-operator sha256:0ccf81cfc614dc1395d3ec222775c61a2e123c8b8fb0f4fd60d051a2673dc971 19.93MiB
      registry.redhat.io/3scale-amp2/3scale-rhel7-operator sha256:ddba65b93aa7fd747c75459e476c92c3362889f38d42537c6a7b9a936cbafea1 19.95MiB
      registry.redhat.io/3scale-amp2/3scale-rhel7-operator sha256:17942523bc4bb2db6eb9f7519db38bbb70e47356d3f0ae0f15b967c0628234c6 72.7MiB
      registry.redhat.io/3scale-amp2/3scale-rhel7-operator sha256:00f17e0b37b0515380a4aece3cb72086c0356fc780ef4526f75476bea36a2c8b 72.71MiB
    manifests:
      sha256:04c8347c30593f9e6945aff5d2da7a8ca5d2cc497f5200a4288ce56d3c8d36b3 -> 1.10-5
      sha256:9ab0e970ff905ca97bd3a6f038593ad71b48838035ea28f80eb013aa0d328662 -> 1.10-4
   <omitted for space>

error: unable to retrieve source image registry.redhat.io/rhcam/openshift-migration-rhel7-operator by tag v1.0: unknown: Not Found
W0113 13:59:36.958174   97851 mirror.go:256] errors during mirroring. the full contents of the catalog may not have been mirrored: mirroring failed: an error occurred during planning
I0113 13:59:36.962666   97851 mirror.go:327] wrote mirroring manifests to olm-registry-manifests

Note: the internal registry throws lots of 500s when mirroring, so we likely still need to advice the use of another registry. I will look into the 500s and may open another BZ to track it.


$ cat olm-registry-manifests/imageContentSourcePolicy.yaml
apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
  name: olm-registry
spec:
  repositoryDigestMirrors:
  - mirrors:
    - default-route-openshift-image-registry.apps.ci-ln-hp9ckrt-d5d6b.origin-ci-int-aws.dev.rhcloud.com/rhcam/openshift-migration-rhel7-operator
    source: registry.redhat.io/rhcam/openshift-migration-rhel7-operator
  - mirrors:
    - default-route-openshift-image-registry.apps.ci-ln-hp9ckrt-d5d6b.origin-ci-int-aws.dev.rhcloud.com/container-native-virtualization/virt-cdi-operator
    source: registry.redhat.io/container-native-virtualization/virt-cdi-operator
  - mirrors:
     <omitted for size>

The basic workflows seem to work here.

As a follow up to this, I will make some BZs to track issues I found:

- When an appregistry namespace is not found or no auth is provided to oc adm catalog build, no errors are thrown, but no manifests are downloaded.
- `oc adm catalog mirror` does not pass --inesecure to the underlying `oc extract` command, which means mirror doesn't work if mirroring from an untrusted registry (mirroring from a trusted registry or a file works).
- `oc image mirror` planning fails if any source tags are not found, which can fail the whole catalog mirroring process. The mapping file that is output after running can be edited to remove the offending tags, but oc adm catalog mirror should be updated to not treat the entire catalog as a single mirror plan, and should ignore individual mirroring errors. 

But even with the above improvements that we should follow up on, the basic `oc adm catalog build` and `oc adm catalog mirror` commands are working as intended.

Cluster version is: 4.3.0-0.nightly-2020-01-14-000626 
[jzhang@dhcp-140-36 ~]$ oc version
Client Version: 4.3.0-0.nightly-2020-01-14-000626
Server Version: 4.3.0-0.nightly-2020-01-14-000626
Kubernetes Version: v1.16.2

1, Enable the cluster registry.
[jzhang@dhcp-140-36 ~]$ oc patch configs.imageregistry.operator.openshift.io/cluster -p='{"spec":{"defaultRoute":true}}' --type=merge
config.imageregistry.operator.openshift.io/cluster patched
[jzhang@dhcp-140-36 ~]$ oc get images.config.openshift.io/cluster  -o jsonpath={.status.externalRegistryHostnames[0]} 
default-route-openshift-image-registry.apps.qe-jiazha-dis.qe.devcluster.openshift.com
[jzhang@dhcp-140-36 ~]$ oc get secret router-certs-default -n openshift-ingress -o json |jq -r '.data["tls.crt"]' | base64 -d | sudo tee /etc/pki/ca-trust/source/anchors/disconnected.crt
[jzhang@dhcp-140-36 ~]$ sudo update-ca-trust 


[jzhang@dhcp-140-36 ~]$ oc create serviceaccount registry
serviceaccount/registry created
[jzhang@dhcp-140-36 ~]$ oc adm policy add-cluster-role-to-user admin -z registry
clusterrole.rbac.authorization.k8s.io/admin added: "registry"
[jzhang@dhcp-140-36 ~]$ docker login default-route-openshift-image-registry.apps.qe-jiazha-dis.qe.devcluster.openshift.com -u reegistry -p `oc sa get-token registry`
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /home/jzhang/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

>  I think the primary issue is the appregistry source. redhat-operators-art is private and no auth token is provided via `--auth-token` for it.

Yes, I agree. We just provide this "--auth-token=" option, but we don't really use it. 
I didn't find any functions to call this "--auth-token" value in https://github.com/openshift/oc/blob/release-4.3/vendor/github.com/operator-framework/operator-registry/pkg/appregistry/manifest_downloader.go#L29
That means this "oc adm catalog build" command is unavailable for these private appregistry sources. Correct me if I'm wrong.

[jzhang@dhcp-140-36 ~]$ oc adm catalog build --help
Builds a catalog container image from a collection operator manifests.

 Extracts the contents of a collection of operator manifests to disk, and builds them into an operator registry catalog
image.

Usage:
  oc adm catalog build [flags]

Options:
      --appregistry-endpoint='https://quay.io/cnr': Endpoint for pulling from an application registry instance.
      --appregistry-org='': Organization (Namespace) to pull from an application registry instance
      --auth-token='': Auth token for communicating with an application registry.
      --dir='': The directory on disk that file:// images will be copied under.
      --from='quay.io/operator-framework/operator-registry-server:latest': The image to use as a base.
      --from-dir='': The directory on disk that file:// images will be read from. Overrides --dir
      --manifest-dir='': Local path to cache manifests when downloading.
      --to='': The image repository tag to apply to the built catalog image.
      --to-db='': Local path to save the database to.


Below are the related tests about the `--auth-token` for private appregistry resource:
[jzhang@dhcp-140-36 ~]$ oc adm catalog build --appregistry-org=redhat-operators-art --auth-token="xxx"(replace real token with xxx) --to=default-route-openshift-image-registry.apps.qe-jiazha-dis.qe.devcluster.openshift.com/openshift/olm:v1 
INFO[0001] loading Bundles                               dir=/tmp/manifests-445659376
INFO[0001] directory                                     dir=/tmp/manifests-445659376 file=manifests-445659376 load=bundles
INFO[0001] loading Packages and Entries                  dir=/tmp/manifests-445659376
INFO[0001] directory                                     dir=/tmp/manifests-445659376 file=manifests-445659376 load=package
Uploading ... 471.6B/s
Uploading 3.405MB ...
Uploading 9.833MB ...
Pushed sha256:18958ad670fc84e93e4149b933eaa6173adb22a4e2dfc70f0db84b36ee0d1bb2 to default-route-openshift-image-registry.apps.qe-jiazha-dis.qe.devcluster.openshift.com/openshift/olm:v1
[jzhang@dhcp-140-36 ~]$ ls /tmp/manifests-445659376
[jzhang@dhcp-140-36 ~]$ 

[jzhang@dhcp-140-36 ~]$ docker create default-route-openshift-image-registry.apps.qe-jiazha-dis.qe.devcluster.openshift.com/openshift/olm:v1 
Unable to find image 'default-route-openshift-image-registry.apps.qe-jiazha-dis.qe.devcluster.openshift.com/openshift/olm:v1' locally
v1: Pulling from openshift/olm
c62fc2a3e1b3: Pull complete 
792699522ed0: Pull complete 
0f2ed44dfe86: Pull complete 
Digest: sha256:18958ad670fc84e93e4149b933eaa6173adb22a4e2dfc70f0db84b36ee0d1bb2
Status: Downloaded newer image for default-route-openshift-image-registry.apps.qe-jiazha-dis.qe.devcluster.openshift.com/openshift/olm:v1
18288914cc0460d82eeda4319bee871a7cf9d776ca7ededc0610c072a16be022
[jzhang@dhcp-140-36 ~]$ docker cp 18288914cc0460d82eeda4319bee871a7cf9d776ca7ededc0610c072a16be022:/bundles.db ./bundles.db
[jzhang@dhcp-140-36 ~]$ 
[jzhang@dhcp-140-36 ~]$ sqlite3 ./bundles.db 
SQLite version 3.22.0 2018-01-22 18:45:57
Enter ".help" for usage hints.
sqlite> .table
api                channel            package          
api_provider       channel_entry      related_image    
api_requirer       operatorbundle     schema_migrations
sqlite> select * from package;
sqlite> 
sqlite> .quit


> I think it's a bug that `oc adm catalog build` doesn't immediately error out if it can't find any data in appregistry, though the logs you posted above indicate this (there are no files found via the logs walking through the downloaded manifests).

Totally agree. Thanks! I create a bug for this issue: https://bugzilla.redhat.com/show_bug.cgi?id=1790785


> But we can still work through this with an appregistry repo that we know works, like redhat-operators:

> $ oc adm catalog build --insecure --appregistry-org=redhat-operator-art --to=default-route-openshift-image-registry.apps.ci-ln-hp9ckrt-d5d6b.origin-ci-int-aws.dev.rhcloud.com/default/olm-registry:test

I used a public appregistry resource for a test(Sorry, no "--insecure" option for this version).
[jzhang@dhcp-140-36 ~]$ oc adm catalog build --insecure --appregistry-org=jiazha --to=default-route-openshift-image-registry.apps.qe-jiazha-dis.qe.devcluster.openshift.com/openshift/jiazha:v1
Error: unknown flag: --insecure
See 'oc adm catalog build --help' for usage.
[jzhang@dhcp-140-36 ~]$ oc adm catalog build --appregistry-org=jiazha --to=default-route-openshift-image-registry.apps.qe-jiazha-dis.qe.devcluster.openshift.com/openshift/jiazha:v1
INFO[0006] loading Bundles                               dir=/tmp/manifests-877027615
INFO[0006] directory                                     dir=/tmp/manifests-877027615 file=manifests-877027615 load=bundles
INFO[0006] directory                                     dir=/tmp/manifests-877027615 file=learn-operator load=bundles
INFO[0006] found csv, loading bundle                     dir=/tmp/manifests-877027615 file=learn-operator.v0.0.1.clusterserviceversion.yaml load=bundles
INFO[0006] loading bundle file                           dir=/tmp/manifests-877027615 file=Learn-v1.crd.yaml load=bundle
INFO[0006] loading bundle file                           dir=/tmp/manifests-877027615 file=learn-operator.package.yaml load=bundle
INFO[0006] loading bundle file                           dir=/tmp/manifests-877027615 file=learn-operator.v0.0.1.clusterserviceversion.yaml load=bundle
INFO[0006] loading bundle file                           dir=/tmp/manifests-877027615 file=learn-operator.v0.0.2.clusterserviceversion.yaml load=bundle
INFO[0006] found csv, loading bundle                     dir=/tmp/manifests-877027615 file=learn-operator.v0.0.2.clusterserviceversion.yaml load=bundles
INFO[0006] loading bundle file                           dir=/tmp/manifests-877027615 file=Learn-v1.crd.yaml load=bundle
INFO[0006] loading bundle file                           dir=/tmp/manifests-877027615 file=learn-operator.package.yaml load=bundle
INFO[0006] loading bundle file                           dir=/tmp/manifests-877027615 file=learn-operator.v0.0.1.clusterserviceversion.yaml load=bundle
INFO[0006] loading bundle file                           dir=/tmp/manifests-877027615 file=learn-operator.v0.0.2.clusterserviceversion.yaml load=bundle
INFO[0006] loading Packages and Entries                  dir=/tmp/manifests-877027615
INFO[0006] directory                                     dir=/tmp/manifests-877027615 file=manifests-877027615 load=package
INFO[0006] directory                                     dir=/tmp/manifests-877027615 file=learn-operator load=package
Uploading ... 102.5kB/s
Uploading 3.405MB ...
Uploading 9.833MB ...
Pushed sha256:cfb685af6e339da8dc33ade4b49b59f83ed9910c999967537ba2b2cd9628efc8 to default-route-openshift-image-registry.apps.qe-jiazha-dis.qe.devcluster.openshift.com/openshift/jiazha:v1
[jzhang@dhcp-140-36 ~]$ cd /tmp/manifests-877027615
[jzhang@dhcp-140-36 manifests-877027615]$ ls
learn-operator
[jzhang@dhcp-140-36 manifests-877027615]$ tree learn-operator/
learn-operator/
├── learn-operator.package.yaml
├── learn-operator.v0.0.1.clusterserviceversion.yaml
├── learn-operator.v0.0.2.clusterserviceversion.yaml
└── Learn-v1.crd.yaml

0 directories, 4 files

I create a CatalogSource to consume this image, like below:
[jzhang@dhcp-140-36 ~]$ cat catalogsource.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: image-registry
  namespace: openshift-marketplace
spec:
  image: image-registry.openshift-image-registry.svc:5000/openshift/jiazha:v1
  sourceType: grpc
[jzhang@dhcp-140-36 ~]$ oc create -f catalogsource.yaml 
catalogsource.operators.coreos.com/image-registry created
[jzhang@dhcp-140-36 ~]$ oc get pods -n openshift-marketplace
NAME                                    READY   STATUS    RESTARTS   AGE
image-registry-nzvp5                    1/1     Running   0          27s
marketplace-operator-78b9654f57-dh2gg   1/1     Running   0          4h53m
[jzhang@dhcp-140-36 ~]$ oc get catalogsource -n openshift-marketplace
NAME             DISPLAY   TYPE   PUBLISHER   AGE
image-registry             grpc               43s
[jzhang@dhcp-140-36 ~]$ oc get packagemanifest
NAME             CATALOG   AGE
learn-operator             97s

It works! So for now, it works for the public appregistry resource.

> - `oc image mirror` planning fails if any source tags are not found, which can fail the whole catalog mirroring process. The mapping file that is output after running can be edited to remove the offending tags, but oc adm catalog mirror should be updated to not treat the entire catalog as a single mirror plan, and should ignore individual mirroring errors. 


yes, it leads to the "oc adm catalog mirror " unavailable in real for the users. I open another bug for this issue: https://bugzilla.redhat.com/show_bug.cgi?id=1790798, the below is my trying:

[jzhang@dhcp-140-36 ~]$ oc adm catalog mirror default-route-openshift-image-registry.apps.qe-jiazha-dis.qe.devcluster.openshift.com/openshift/jiazha:v1 default-route-openshift-image-registry.apps.qe-jiazha-dis.qe.devcluster.openshift.com/openshift/jiazha:v2
I0114 16:19:57.712019    6848 mirror.go:190] wrote database to /tmp/004393117/bundles.db
W0114 16:19:57.712760    6848 mirror.go:139] couldn't parse quay.io/jiazha/learn-operator-image, skipping mirror


info: Planning completed in 0s
info: Mirroring completed in 0s (0B/s)
W0114 16:19:57.713254    6848 mirror.go:254] error parsing target reference for default-route-openshift-image-registry.apps.qe-jiazha-dis.qe.devcluster.openshift.com/openshift/jiazha:v2/jiazha/learn-operator-image
I0114 16:19:57.713570    6848 mirror.go:288] wrote mirroring manifests to jiazha-manifests
[jzhang@dhcp-140-36 ~]$ cat jiazha-manifests/imageContentSourcePolicy.yaml 
apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
  name: jiazha
spec:
  repositoryDigestMirrors: []
[jzhang@dhcp-140-36 ~]$ cat jiazha-manifests/mapping.txt 


> - `oc adm catalog mirror` does not pass --inesecure to the underlying `oc extract` command, which means mirror doesn't work if mirroring from an untrusted registry (mirroring from a trusted registry or a file works).

I create another bug for this issue: https://bugzilla.redhat.com/show_bug.cgi?id=1790802


To conclusion, 
1, The 'oc adm catalog build' can only work for the public appregistry resource. I modify this bug title and use this bug to trace.
2, The 'oc adm catalog mirror' cannot work since in real scenarios, source tags are not found is common. But, it will fail the whole catalog mirroring process

> Note: the internal registry throws lots of 500s when mirroring, so we likely still need to advice the use of another registry. I will look into the 500s and may open another BZ to track it.

I didn't encounter this issue. But, got some `404` errors, Or am I missing something?
[jzhang@dhcp-140-36 ~]$ oc adm catalog build --appregistry-org=redhat-operators --to=localhost:5000/openshift/jiazha2:v1
INFO[0112] loading Bundles                               dir=/tmp/manifests-074920506
INFO[0112] directory                                     dir=/tmp/manifests-074920506 file=manifests-074920506 load=bundles
INFO[0112] directory                                     dir=/tmp/manifests-074920506 file=3scale-operator load=bundles
INFO[0112] found csv, loading bundle                     dir=/tmp/manifests-074920506 file=3scale-operator.v0.3.0.clusterserviceversion.yaml load=bundles
...
Uploading 9.833MB ...
Pushed sha256:3ce4986e1a2e3a0fc560f06ffcb1859aa76cc0be0e5a6bb2e1904ed74c42eae0 to localhost:5000/openshift/jiazha2:v1

[jzhang@dhcp-140-36 ~]$ docker run -it --rm -p 5000:5000 --name=registry -v /home/jzhang/goproject/src/github.com/operator-framework/operator-lifecycle-manager/pkg/lib/filemonitor/testdata/:/certs -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/server-new.crt -e REGISTRY_HTTP_TLS_KEY=/certs/server-new.key registry
WARN[0000] No HTTP secret provided - generated random secret. This may cause problems with uploads if multiple registries are behind a load-balancer. To provide a shared secret, fill in http.secret in the configuration file or set the REGISTRY_HTTP_SECRET environment variable.  go.version=go1.11.2 instance.id=5455c471-0177-4982-a19a-b10400548c09 service=registry version=v2.7.1
INFO[0000] redis not configured                          go.version=go1.11.2 instance.id=5455c471-0177-4982-a19a-b10400548c09 service=registry version=v2.7.1
INFO[0000] Starting upload purge in 1m0s                 go.version=go1.11.2 instance.id=5455c471-0177-4982-a19a-b10400548c09 service=registry version=v2.7.1
INFO[0000] using inmemory blob descriptor cache          go.version=go1.11.2 instance.id=5455c471-0177-4982-a19a-b10400548c09 service=registry version=v2.7.1
INFO[0000] listening on [::]:5000, tls                   go.version=go1.11.2 instance.id=5455c471-0177-4982-a19a-b10400548c09 service=registry version=v2.7.1
INFO[0060] PurgeUploads starting: olderThan=2020-01-07 09:21:22.589908199 +0000 UTC m=-604739.980899188, actuallyDelete=true 
...

172.17.0.1 - - [14/Jan/2020:09:22:47 +0000] "HEAD /v2/openshift/jiazha2/blobs/sha256:1032e09fec0a562f9716c459721147a414ee81bba51b067024e7ec0c7d693543 HTTP/2.0" 200 0 "" "Go-http-client/2.0"
ERRO[0144] response completed with error                 err.code="blob unknown" err.detail=sha256:c62fc2a3e1b346545f74c6c17ad029eaef6406f79bca4852fc5afe45f4a3e769 err.message="blob unknown to registry" go.version=go1.11.2 http.request.host="localhost:5000" http.request.id=1ab16bfd-295a-40e3-ac49-20e8065d29ea http.request.method=HEAD http.request.remoteaddr="172.17.0.1:59398" http.request.uri="/v2/openshift/jiazha2/blobs/sha256:c62fc2a3e1b346545f74c6c17ad029eaef6406f79bca4852fc5afe45f4a3e769" http.request.useragent="Go-http-client/2.0" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.653286ms http.response.status=404 http.response.written=157 vars.digest="sha256:c62fc2a3e1b346545f74c6c17ad029eaef6406f79bca4852fc5afe45f4a3e769" vars.name="openshift/jiazha2"
...

Thank you for making the follow up BZs! There are PRs out for them now.


> Yes, I agree. We just provide this "--auth-token=" option, but we don't really use it. 
> I didn't find any functions to call this "--auth-token" value in https://github.com/openshift/oc/blob/release-4.3/vendor/github.com/operator-framework/operator-registry/pkg/appregistry/manifest_downloader.go#L29
> That means this "oc adm catalog build" command is unavailable for these private appregistry sources. Correct me if I'm wrong.

Private registries are available for use, the token is passed down to the underlying appregistry client.

Token is passed from `oc` to here: 
https://github.com/openshift/oc/blob/release-4.3/vendor/github.com/operator-framework/operator-registry/pkg/appregistry/builder.go#L64

to here: 
https://github.com/openshift/oc/blob/release-4.3/vendor/github.com/operator-framework/operator-registry/pkg/appregistry/builder.go#L90

and then included in all client calls from here:
https://github.com/openshift/oc/blob/release-4.3/vendor/github.com/operator-framework/operator-registry/pkg/apprclient/apprclient.go#L33


I have also tested that this works with the private repo redhat-operators-art:

AUTH_TOKEN=$(curl -sH "Content-Type: application/json" -XPOST https://quay.io/cnr/api/v1/users/login -d '
{
    "user": {
        "username": "'"${QUAY_USERNAME}"'",
        "password": "'"${QUAY_PASSWORD}"'"
    }
}' | jq -r '.token')

$ oc adm catalog build --insecure --appregistry-org=redhat-operators-art --auth-token="$AUTH_TOKEN" --to=localhost:5000/me:1

INFO[0014] loading Bundles                               dir=/var/folders/4m/pt431q9d2fsbk9zz2_vmg3tc0000gn/T/cache-510524847/manifests-416355780
INFO[0014] directory                                     dir=/var/folders/4m/pt431q9d2fsbk9zz2_vmg3tc0000gn/T/cache-510524847/manifests-416355780 file=manifests-416355780 load=bundles
INFO[0014] directory                                     dir=/var/folders/4m/pt431q9d2fsbk9zz2_vmg3tc0000gn/T/cache-510524847/manifests-416355780 file=cluster-logging load=bundles
INFO[0014] directory                                     dir=/var/folders/4m/pt431q9d2fsbk9zz2_vmg3tc0000gn/T/cache-510524847/manifests-416355780 file=cluster-logging-or9q_au2 load=bundles
INFO[0014] directory                                     dir=/var/folders/4m/pt431q9d2fsbk9zz2_vmg3tc0000gn/T/cache-510524847/manifests-416355780 file=4.1 load=bundles
INFO[0014] found csv, loading bundle                     dir=/var/folders/4m/pt431q9d2fsbk9zz2_vmg3tc0000gn/T/cache-510524847/manifests-416355780 file=cluster-logging.v4.1.0.clusterserviceversion.yaml load=bundles
<cut for space>
Uploading ... 10.69MB/s
Pushed sha256:d4cb895c007ee5f4f18c4ff3cd9727402ef137b89567638046fb2e1dee43a277 to localhost:5000/me:1

So I don't think that the current title of this BZ applies.

> 2, The 'oc adm catalog mirror' cannot work since in real scenarios, source tags are not found is common. But, it will fail the whole catalog mirroring process

There is a PR for the other BZ that fixes this, however, there is a workaround that is available to users today with the released `oc`. 

In addition to the `imageContentSourcePolicy.yaml` there is a `mapping.txt` file which contains all of the source images and where to map them to in the target registry. This file is in the format that `oc image mirror` (note that this is not `oc adm catalog`) understands as input. 

When invalid source tags are found, the offending mappings can be removed from mapping.txt and the file can be passed to `oc image mirror`. 

> I didn't encounter this issue. But, got some `404` errors, Or am I missing something?

The 500s I referred to occurred when trying to run the `mirror` step on an internal registry, not the `build` step. For example, something like:

 $ oc adm catalog mirror --insecure default-route-openshift-image-registry.apps.ci-ln-hp9ckrt-d5d6b.origin-ci-int-aws.dev.rhcloud.com/default/olm-registry:test default-route-openshift-image-registry.apps.ci-ln-hp9ckrt-d5d6b.origin-ci-int-aws.dev.rhcloud.com

Would result in logs in the image-registry pod in the cluster:

time="2020-01-13T19:31:25.259634105Z" level=error msg="client disconnected during blob PATCH" contentLength=-1 copied=30557536 error="http: unexpected EOF reading trailer" go.version=go1.11.13 http.request.host=default-route-openshift-image-registry.apps.ci-ln-hp9ckrt-d5d6b.origin-ci-int-aws.dev.rhcloud.com http.request.id=19799512-8a9b-48c5-86dd-467ed75c92e7 http.request.method=PATCH http.request.remoteaddr=207.251.102.227 http.request.uri="/v2/openshift4/ose-template-service-broker-operator/blobs/uploads/39067d5a-0541-4c29-868d-f14c3df14877?_state=XyU1RYP28wV3m5jyBYf0mJcbPlWZJ8EWvdP4yPuV1G17Ik5hbWUiOiJvcGVuc2hpZnQ0L29zZS10ZW1wbGF0ZS1zZXJ2aWNlLWJyb2tlci1vcGVyYXRvciIsIlVVSUQiOiIzOTA2N2Q1YS0wNTQxLTRjMjktODY4ZC1mMTRjM2RmMTQ4NzciLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjAtMDEtMTNUMTk6MzE6MTAuOTc1OTc3NTY3WiJ9" http.request.useragent=Go-http-client/1.1 openshift.auth.user="system:serviceaccount:default:registry" vars.name=openshift4/ose-template-service-broker-operator vars.uuid=39067d5a-0541-4c29-868d-f14c3df14877
time="2020-01-13T19:31:25.261569863Z" level=info msg=response go.version=go1.11.13 http.request.host="10.128.2.6:5000" http.request.id=79047e5d-67fb-47e9-9d80-37a91d683a18 http.request.method=GET http.request.remoteaddr="10.128.2.1:58510" http.request.uri=/healthz http.request.useragent=kube-probe/1.16+ http.response.duration="51.048µs" http.response.status=200 http.response.written=0
2020/01/13 19:31:25 http: multiple response.WriteHeader calls
time="2020-01-13T19:31:25.52581407Z" level=error msg="response completed with error" err.code=unknown err.detail="client disconnected" err.message="unknown error" go.version=go1.11.13 http.request.host=default-route-openshift-image-registry.apps.ci-ln-hp9ckrt-d5d6b.origin-ci-int-aws.dev.rhcloud.com http.request.id=a06939fd-edf2-4e27-b2b4-351a22a3d7c6 http.request.method=PATCH http.request.remoteaddr=207.251.102.227 http.request.uri="/v2/openshift4/ose-template-service-broker-operator/blobs/uploads/d53f7686-62fa-483c-a1c3-ff160b100124?_state=bTpKaBpP0BGZkXQF7XHxH837W5MFzuv_uvWuGg2WDaR7Ik5hbWUiOiJvcGVuc2hpZnQ0L29zZS10ZW1wbGF0ZS1zZXJ2aWNlLWJyb2tlci1vcGVyYXRvciIsIlVVSUQiOiJkNTNmNzY4Ni02MmZhLTQ4M2MtYTFjMy1mZjE2MGIxMDAxMjQiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjAtMDEtMTNUMTk6MzE6MTQuNDI0NDM3Mzc5WiJ9" http.request.useragent=Go-http-client/1.1 http.response.contenttype="application/json; charset=utf-8" http.response.duration=10.959116567s http.response.status=500 http.response.written=89 openshift.auth.user="system:serviceaccount:default:registry" vars.name=openshift4/ose-template-service-broker-operator vars.uuid=d53f7686-62fa-483c-a1c3-ff160b100124

The internal registry cannot be used as the target of mirroring, see this BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1746635 

This explains the 500s I was seeing, but it looks like there are no plans to fix it.

I think everything here is now resolved or tracked elsewhere. Please review and close if you agree.

> I have also tested that this works with the private repo redhat-operators-art:

Yes, sorry for that, my mistake. I forget to add the "basic" string in my last trying. It works well in the latest test.

[jzhang@dhcp-140-36 ~]$ oc adm catalog build --appregistry-org=redhat-operators-art --auth-token="basic username:password(based64)" --to=default-route-openshift-image-registry.apps.qe-jiazha2-dis.qe.devcluster.openshift.com/openshift/olm:v1 
INFO[0022] loading Bundles                               dir=/tmp/manifests-256129201
INFO[0022] directory                                     dir=/tmp/manifests-256129201 file=manifests-256129201 load=bundles
...
Pushed sha256:e1aafbe713996b89b363bf444a55fe58afad6f746319139c4b6d1a445dcd3b68 to default-route-openshift-image-registry.apps.qe-jiazha2-dis.qe.devcluster.openshift.com/openshift/olm:v1
[jzhang@dhcp-140-36 ~]$ docker create default-route-openshift-image-registry.apps.qe-jiazha2-dis.qe.devcluster.openshift.com/openshift/olm:v1
...
Status: Downloaded newer image for default-route-openshift-image-registry.apps.qe-jiazha2-dis.qe.devcluster.openshift.com/openshift/olm:v1
a6839ab585451c6b1299b7a41bf704e67562442c9c563a60e7138161f46648e1
[jzhang@dhcp-140-36 ~]$ docker cp a6839ab585451c6b1299b7a41bf704e67562442c9c563a60e7138161f46648e1:bundles.db ./bundles.db
[jzhang@dhcp-140-36 ~]$ sqlite3 bundles.db 
SQLite version 3.22.0 2018-01-22 18:45:57
Enter ".help" for usage hints.
sqlite> .table
api                channel            package          
api_provider       channel_entry      related_image    
api_requirer       operatorbundle     schema_migrations
sqlite> select * from package;
cluster-logging|4.4
clusterresourceoverride|4.4
elasticsearch-operator|4.4
local-storage-operator|4.4
metering-ocp|4.4
nfd|4.4
openshiftansibleservicebroker|4.3
openshifttemplateservicebroker|4.3
ptp-operator|4.4
sriov-network-operator|4.4
sqlite>

> there is a workaround that is available to users today with the released `oc`.

What's the workaround? In my trying, the `imageContentSourcePolicy.yaml` and `mapping.txt` are empty. I think we can trace it in bug 1790798

> The internal registry cannot be used as the target of mirroring, see this BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1746635 

Ok, got it, thanks!

> I think everything here is now resolved or tracked elsewhere. Please review and close if you agree.

Yes, thank you very much! verify it.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581