Description of problem: `oc image mirror` planning fails if any source tags are not found, which can fail the whole catalog mirroring process. The mapping file that is output after running can be edited to remove the offending tags, but oc adm catalog mirror should be updated to not treat the entire catalog as a single mirror plan, and should ignore individual mirroring errors. Version-Release number of selected component (if applicable): Cluster version is: 4.3.0-0.nightly-2020-01-14-000626 [jzhang@dhcp-140-36 ~]$ oc version Client Version: 4.3.0-0.nightly-2020-01-14-000626 Server Version: 4.3.0-0.nightly-2020-01-14-000626 Kubernetes Version: v1.16.2 How reproducible: always Steps to Reproduce: 1. Create an OCP 4.3. 2. Enable the cluster registry. [jzhang@dhcp-140-36 ~]$ oc patch configs.imageregistry.operator.openshift.io/cluster -p='{"spec":{"defaultRoute":true}}' --type=merge config.imageregistry.operator.openshift.io/cluster patched [jzhang@dhcp-140-36 ~]$ oc get images.config.openshift.io/cluster -o jsonpath={.status.externalRegistryHostnames[0]} default-route-openshift-image-registry.apps.qe-jiazha-dis.qe.devcluster.openshift.com [jzhang@dhcp-140-36 ~]$ oc get secret router-certs-default -n openshift-ingress -o json |jq -r '.data["tls.crt"]' | base64 -d | sudo tee /etc/pki/ca-trust/source/anchors/disconnected.crt [jzhang@dhcp-140-36 ~]$ sudo update-ca-trust [jzhang@dhcp-140-36 ~]$ oc create serviceaccount registry serviceaccount/registry created [jzhang@dhcp-140-36 ~]$ oc adm policy add-cluster-role-to-user admin -z registry clusterrole.rbac.authorization.k8s.io/admin added: "registry" [jzhang@dhcp-140-36 ~]$ docker login default-route-openshift-image-registry.apps.qe-jiazha-dis.qe.devcluster.openshift.com -u reegistry -p `oc sa get-token registry` 3. Build a catalog image by running the "oc adm catalog build" command: [jzhang@dhcp-140-36 ~]$ oc adm catalog build --appregistry-org=redhat-operators --to=default-route-openshift-image-registry.apps.qe-jiazha-dis.qe.devcluster.openshift.com/openshift/redhat:v1 INFO[0035] loading Bundles dir=/tmp/manifests-848288140 INFO[0035] directory dir=/tmp/manifests-848288140 file=manifests-848288140 load=bundles INFO[0035] directory dir=/tmp/manifests-848288140 file=3scale-operator load=bundles INFO[0035] found csv, loading bundle dir=/tmp/manifests-848288140 file=3scale-operator.v0.3.0.clusterserviceversion.yaml load=bundles INFO[0035] loading bundle file dir=/tmp/manifests-848288140 file=3scale-operator.package.yaml load=bundle INFO[0035] loading bundle file dir=/tmp/manifests-848288140 file=3scale-operator.v0.3.0.clusterserviceversion.yaml load=bundle ... Uploading 9.833MB ... Pushed sha256:b18f3fde5ac9856f96dd22a7fe5c2a639e751e8568657e7122eef14781cd98ee to default-route-openshift-image-registry.apps.qe-jiazha-dis.qe.devcluster.openshift.com/openshift/redhat:v1 4, Mirror this image to this cluster registry. [jzhang@dhcp-140-36 ~]$ oc adm catalog mirror default-route-openshift-image-registry.apps.qe-jiazha-dis.qe.devcluster.openshift.com/openshift/redhat:v1 default-route-openshift-image-registry.apps.qe-jiazha-dis.qe.devcluster.openshift.com/openshift/redhat:v2 ... Actual results: [jzhang@dhcp-140-36 ~]$ oc adm catalog mirror default-route-openshift-image-registry.apps.qe-jiazha-dis.qe.devcluster.openshift.com/openshift/redhat:v1 default-route-openshift-image-registry.apps.qe-jiazha-dis.qe.devcluster.openshift.com/openshift/redhat:v2 I0114 16:38:21.061474 6929 mirror.go:190] wrote database to /tmp/092943701/bundles.db W0114 16:38:21.064088 6929 mirror.go:144] couldn't parse default-route-openshift-image-registry.apps.qe-jiazha-dis.qe.devcluster.openshift.com/openshift/redhat:v2/openshift4/ose-cluster-nfd-operator@sha256:5ab9eb8f352e7e4adad8c541a1423e16972593e127448628f311af8f7881f69e, skipping mirror ... info: Planning completed in 0s info: Mirroring completed in 0s (0B/s) W0114 16:38:21.076013 6929 mirror.go:254] error parsing target reference for default-route-openshift-image-registry.apps.qe-jiazha-dis.qe.devcluster.openshift.com/openshift/redhat:v2/openshift4/ose-sriov-network-operator@sha256:0a405702046a26f268590d53ec963ddd770c656d1c1a8d4f9f1e5cdfc61817dd W0114 16:38:21.076149 6929 mirror.go:254] error parsing target reference for default-route-openshift-image-registry.apps.qe-jiazha-dis.qe.devcluster.openshift.com/openshift/redhat:v2/amq7/amq-broker-rhel7-operator:0.9 W0114 16:38:21.076282 6929 mirror.go:254] error parsing target reference for default-route-openshift-image-registry.apps.qe-jiazha-dis.qe.devcluster.openshift.com/openshift/redhat:v2/amq7/amq-online-1-controller-manager:1.2 ... [jzhang@dhcp-140-36 ~]$ tree redhat-manifests/ redhat-manifests/ ├── imageContentSourcePolicy.yaml └── mapping.txt 0 directories, 2 files [jzhang@dhcp-140-36 ~]$ cat redhat-manifests/imageContentSourcePolicy.yaml apiVersion: operator.openshift.io/v1alpha1 kind: ImageContentSourcePolicy metadata: name: redhat spec: repositoryDigestMirrors: [] Expected results: Should parse the source image successfully. Additional info:
*** Bug 1796096 has been marked as a duplicate of this bug. ***
Cluster version is 4.4.0-0.nightly-2020-02-02-225006 [jzhang@dhcp-140-36 ~]$ oc version Client Version: 4.4.0-0.nightly-2020-02-02-225006 Server Version: 4.4.0-0.nightly-2020-02-02-225006 Kubernetes Version: v1.17.1 1, Following the above steps to create a CatalogSource image: [jzhang@dhcp-140-36 ~]$ oc adm catalog build --appregistry-org=redhat-operators --to=default-route-openshift-image-registry.apps.yinzhou.qe.devcluster.openshift.com/openshift/redhat:v1 INFO[0027] loading Bundles dir=/tmp/cache-435723435/manifests-012173424 INFO[0027] directory ... 2, Mirror this image to the cluster internal registry too. [jzhang@dhcp-140-36 ~]$ oc adm catalog mirror default-route-openshift-image-registry.apps.yinzhou.qe.devcluster.openshift.com/openshift/redhat:v1 default-route-openshift-image-registry.apps.yinzhou.qe.devcluster.openshift.com I0203 16:45:32.489715 19982 mirror.go:230] wrote database to /tmp/408262702/bundles.db default-route-openshift-image-registry.apps.yinzhou.qe.devcluster.openshift.com/ openshift4/ose-logging-curator5 blobs: registry.redhat.io/openshift4/ose-logging-curator5 sha256:ebdfbc48e7aad008a7c3c63bae79ca43b033efa91eb309355de278a7fee945a4 1.604KiB registry.redhat.io/openshift4/ose-logging-curator5 sha256:ef59a3e36b15752f81502187d80ca9bb9fe991d32b27ce0141371bb062f9b4a1 5.293KiB registry.redhat.io/openshift4/ose-logging-curator5 sha256:46b71c0b675d0ea8766a1c41ce930c5281c3d993dbfb6870355cbf313005d193 4.166MiB registry.redhat.io/openshift4/ose-logging-curator5 sha256:078444edb1ee8d4e1a401fba6e157918e7056657e6c750cd612d7e8a80b345f5 8.653MiB registry.redhat.io/openshift4/ose-logging-curator5 sha256:ba308e6b72af78ca699abbe6cc76b7eb583afb0930eadca3999d61eb575ede86 73.04MiB manifests: sha256:25920618104acbc6d0fe0bc85f2731d32629eebd595b8b961b90a26cc25134c0 stats: shared=0 unique=5 size=85.87MiB ratio=1.00 phase 0: default-route-openshift-image-registry.apps.yinzhou.qe.devcluster.openshift.com openshift4/ose-logging-curator5 blobs=5 mounts=0 manifests=1 shared=0 info: Planning completed in 4.9s uploading: default-route-openshift-image-registry.apps.yinzhou.qe.devcluster.openshift.com/openshift4/ose-logging-curator5 sha256:46b71c0b675d0ea8766a1c41ce930c5281c3d993dbfb6870355cbf313005d193 4.166MiB uploading: default-route-openshift-image-registry.apps.yinzhou.qe.devcluster.openshift.com/openshift4/ose-logging-curator5 sha256:ba308e6b72af78ca699abbe6cc76b7eb583afb0930eadca3999d61eb575ede86 73.04MiB uploading: default-route-openshift-image-registry.apps.yinzhou.qe.devcluster.openshift.com/openshift4/ose-logging-curator5 sha256:078444edb1ee8d4e1a401fba6e157918e7056657e6c750cd612d7e8a80b345f5 8.653MiB ... LGTM, but I tried to use another source image which no tag and digest, it failed. [jzhang@dhcp-140-36 ~]$ oc adm catalog build --auth-token='basic xxx' --appregistry-org=learn-operator --to=default-route-openshift-image-registry.apps.yinzhou.qe.devcluster.openshift.com/openshift/learn:v1 [jzhang@dhcp-140-36 ~]$ oc adm catalog mirror default-route-openshift-image-registry.apps.yinzhou.qe.devcluster.openshift.com/openshift/learn:v1 default-route-openshift-image-registry.apps.yinzhou.qe.devcluster.openshift.com I0203 17:01:31.485081 20034 mirror.go:230] wrote database to /tmp/282905947/bundles.db W0203 17:01:31.485668 20034 mirror.go:159] couldn't parse quay.io/jiazha/learn-operator-image, skipping mirror: source references must have a tag or digest specified I0203 17:01:31.486006 20034 mirror.go:328] wrote mirroring manifests to learn-manifests [jzhang@dhcp-140-36 ~]$ cat learn-manifests/ imageContentSourcePolicy.yaml mapping.txt [jzhang@dhcp-140-36 ~]$ cat learn-manifests/imageContentSourcePolicy.yaml apiVersion: operator.openshift.io/v1alpha1 kind: ImageContentSourcePolicy metadata: name: learn spec: repositoryDigestMirrors: - mirrors: - default-route-openshift-image-registry.apps.yinzhou.qe.devcluster.openshift.com/jiazha/learn-operator-image source: quay.io/jiazha/learn-operator-image [jzhang@dhcp-140-36 ~]$ cat learn-manifests/mapping.txt quay.io/jiazha/learn-operator-image=default-route-openshift-image-registry.apps.yinzhou.qe.devcluster.openshift.com/jiazha/learn-operator-image @evan if no tag and digest set in the image, can we mirror it with the "latest" tag for it automatically? Change the status to ASSIGNED first.
> @evan if no tag and digest set in the image, can we mirror it with the "latest" tag for it automatically? Change the status to ASSIGNED first. No, we can't reasonably do that at the moment. ImageContentSourcePolicy will only remap images using digests (technically it's possible to enable tag mirroring in cri-o but it is disabled in openshift) which means that even though we could put the images into the target registry, it would fail to pull when you try to install the operator on openshift. I think it's worth discussing a feature which converts the catalog itself (i.e. modifies the catalog content to point to whatever digests the current tags point to) so that all images in a catalog are valid for mirroring by digest. I think we should track that via an RFE though, not this BZ.
Hi, Evan Yes, I agree. Thanks! Create a feature request here: https://issues.redhat.com/browse/RFE-560 Verify this bug based on comment 3, 4.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0581