Bug 1790798 - "oc adm catalog mirror" fails to parse the source image
Summary: "oc adm catalog mirror" fails to parse the source image
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.3.z
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.4.0
Assignee: Evan Cordell
QA Contact: Jian Zhang
URL:
Whiteboard:
Depends On:
Blocks: 1797730
TreeView+ depends on / blocked
 
Reported: 2020-01-14 08:51 UTC by Jian Zhang
Modified: 2020-05-04 11:24 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1797730 (view as bug list)
Environment:
Last Closed: 2020-05-04 11:24:06 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift oc pull 253 None closed Bug 1790798: Mirror images one by one for oc adm catalog mirror 2020-07-08 02:05:48 UTC
Red Hat Product Errata RHBA-2020:0581 None None None 2020-05-04 11:24:32 UTC

Description Jian Zhang 2020-01-14 08:51:58 UTC
Description of problem:
`oc image mirror` planning fails if any source tags are not found, which can fail the whole catalog mirroring process. The mapping file that is output after running can be edited to remove the offending tags, but oc adm catalog mirror should be updated to not treat the entire catalog as a single mirror plan, and should ignore individual mirroring errors. 

Version-Release number of selected component (if applicable):
Cluster version is: 4.3.0-0.nightly-2020-01-14-000626 
[jzhang@dhcp-140-36 ~]$ oc version
Client Version: 4.3.0-0.nightly-2020-01-14-000626
Server Version: 4.3.0-0.nightly-2020-01-14-000626
Kubernetes Version: v1.16.2

How reproducible:
always

Steps to Reproduce:
1. Create an OCP 4.3.

2. Enable the cluster registry.
[jzhang@dhcp-140-36 ~]$ oc patch configs.imageregistry.operator.openshift.io/cluster -p='{"spec":{"defaultRoute":true}}' --type=merge
config.imageregistry.operator.openshift.io/cluster patched
[jzhang@dhcp-140-36 ~]$ oc get images.config.openshift.io/cluster  -o jsonpath={.status.externalRegistryHostnames[0]} 
default-route-openshift-image-registry.apps.qe-jiazha-dis.qe.devcluster.openshift.com
[jzhang@dhcp-140-36 ~]$ oc get secret router-certs-default -n openshift-ingress -o json |jq -r '.data["tls.crt"]' | base64 -d | sudo tee /etc/pki/ca-trust/source/anchors/disconnected.crt
[jzhang@dhcp-140-36 ~]$ sudo update-ca-trust 
[jzhang@dhcp-140-36 ~]$ oc create serviceaccount registry
serviceaccount/registry created
[jzhang@dhcp-140-36 ~]$ oc adm policy add-cluster-role-to-user admin -z registry
clusterrole.rbac.authorization.k8s.io/admin added: "registry"
[jzhang@dhcp-140-36 ~]$ docker login default-route-openshift-image-registry.apps.qe-jiazha-dis.qe.devcluster.openshift.com -u reegistry -p `oc sa get-token registry`

3. Build a catalog image by running the "oc adm catalog build" command:
[jzhang@dhcp-140-36 ~]$ oc adm catalog build --appregistry-org=redhat-operators --to=default-route-openshift-image-registry.apps.qe-jiazha-dis.qe.devcluster.openshift.com/openshift/redhat:v1
INFO[0035] loading Bundles                               dir=/tmp/manifests-848288140
INFO[0035] directory                                     dir=/tmp/manifests-848288140 file=manifests-848288140 load=bundles
INFO[0035] directory                                     dir=/tmp/manifests-848288140 file=3scale-operator load=bundles
INFO[0035] found csv, loading bundle                     dir=/tmp/manifests-848288140 file=3scale-operator.v0.3.0.clusterserviceversion.yaml load=bundles
INFO[0035] loading bundle file                           dir=/tmp/manifests-848288140 file=3scale-operator.package.yaml load=bundle
INFO[0035] loading bundle file                           dir=/tmp/manifests-848288140 file=3scale-operator.v0.3.0.clusterserviceversion.yaml load=bundle
...
Uploading 9.833MB ...
Pushed sha256:b18f3fde5ac9856f96dd22a7fe5c2a639e751e8568657e7122eef14781cd98ee to default-route-openshift-image-registry.apps.qe-jiazha-dis.qe.devcluster.openshift.com/openshift/redhat:v1

4, Mirror this image to this cluster registry.
[jzhang@dhcp-140-36 ~]$ oc adm catalog mirror default-route-openshift-image-registry.apps.qe-jiazha-dis.qe.devcluster.openshift.com/openshift/redhat:v1  default-route-openshift-image-registry.apps.qe-jiazha-dis.qe.devcluster.openshift.com/openshift/redhat:v2 
...


Actual results:

[jzhang@dhcp-140-36 ~]$ oc adm catalog mirror default-route-openshift-image-registry.apps.qe-jiazha-dis.qe.devcluster.openshift.com/openshift/redhat:v1  default-route-openshift-image-registry.apps.qe-jiazha-dis.qe.devcluster.openshift.com/openshift/redhat:v2 
I0114 16:38:21.061474    6929 mirror.go:190] wrote database to /tmp/092943701/bundles.db
W0114 16:38:21.064088    6929 mirror.go:144] couldn't parse default-route-openshift-image-registry.apps.qe-jiazha-dis.qe.devcluster.openshift.com/openshift/redhat:v2/openshift4/ose-cluster-nfd-operator@sha256:5ab9eb8f352e7e4adad8c541a1423e16972593e127448628f311af8f7881f69e, skipping mirror
...
info: Planning completed in 0s
info: Mirroring completed in 0s (0B/s)
W0114 16:38:21.076013    6929 mirror.go:254] error parsing target reference for default-route-openshift-image-registry.apps.qe-jiazha-dis.qe.devcluster.openshift.com/openshift/redhat:v2/openshift4/ose-sriov-network-operator@sha256:0a405702046a26f268590d53ec963ddd770c656d1c1a8d4f9f1e5cdfc61817dd
W0114 16:38:21.076149    6929 mirror.go:254] error parsing target reference for default-route-openshift-image-registry.apps.qe-jiazha-dis.qe.devcluster.openshift.com/openshift/redhat:v2/amq7/amq-broker-rhel7-operator:0.9
W0114 16:38:21.076282    6929 mirror.go:254] error parsing target reference for default-route-openshift-image-registry.apps.qe-jiazha-dis.qe.devcluster.openshift.com/openshift/redhat:v2/amq7/amq-online-1-controller-manager:1.2
...
[jzhang@dhcp-140-36 ~]$ tree redhat-manifests/
redhat-manifests/
├── imageContentSourcePolicy.yaml
└── mapping.txt

0 directories, 2 files
[jzhang@dhcp-140-36 ~]$ cat redhat-manifests/imageContentSourcePolicy.yaml 
apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
  name: redhat
spec:
  repositoryDigestMirrors: []


Expected results:
Should parse the source image successfully.

Additional info:

Comment 1 Kevin Rizza 2020-01-29 19:15:48 UTC
*** Bug 1796096 has been marked as a duplicate of this bug. ***

Comment 3 Jian Zhang 2020-02-03 09:10:49 UTC
Cluster version is 4.4.0-0.nightly-2020-02-02-225006

[jzhang@dhcp-140-36 ~]$ oc version
Client Version: 4.4.0-0.nightly-2020-02-02-225006
Server Version: 4.4.0-0.nightly-2020-02-02-225006
Kubernetes Version: v1.17.1

1, Following the above steps to create a CatalogSource image:
[jzhang@dhcp-140-36 ~]$ oc adm catalog build --appregistry-org=redhat-operators --to=default-route-openshift-image-registry.apps.yinzhou.qe.devcluster.openshift.com/openshift/redhat:v1
INFO[0027] loading Bundles                               dir=/tmp/cache-435723435/manifests-012173424
INFO[0027] directory   
...

2, Mirror this image to the cluster internal registry too.
[jzhang@dhcp-140-36 ~]$ oc adm catalog mirror default-route-openshift-image-registry.apps.yinzhou.qe.devcluster.openshift.com/openshift/redhat:v1 default-route-openshift-image-registry.apps.yinzhou.qe.devcluster.openshift.com
I0203 16:45:32.489715   19982 mirror.go:230] wrote database to /tmp/408262702/bundles.db
default-route-openshift-image-registry.apps.yinzhou.qe.devcluster.openshift.com/
  openshift4/ose-logging-curator5
    blobs:
      registry.redhat.io/openshift4/ose-logging-curator5 sha256:ebdfbc48e7aad008a7c3c63bae79ca43b033efa91eb309355de278a7fee945a4 1.604KiB
      registry.redhat.io/openshift4/ose-logging-curator5 sha256:ef59a3e36b15752f81502187d80ca9bb9fe991d32b27ce0141371bb062f9b4a1 5.293KiB
      registry.redhat.io/openshift4/ose-logging-curator5 sha256:46b71c0b675d0ea8766a1c41ce930c5281c3d993dbfb6870355cbf313005d193 4.166MiB
      registry.redhat.io/openshift4/ose-logging-curator5 sha256:078444edb1ee8d4e1a401fba6e157918e7056657e6c750cd612d7e8a80b345f5 8.653MiB
      registry.redhat.io/openshift4/ose-logging-curator5 sha256:ba308e6b72af78ca699abbe6cc76b7eb583afb0930eadca3999d61eb575ede86 73.04MiB
    manifests:
      sha256:25920618104acbc6d0fe0bc85f2731d32629eebd595b8b961b90a26cc25134c0
  stats: shared=0 unique=5 size=85.87MiB ratio=1.00

phase 0:
  default-route-openshift-image-registry.apps.yinzhou.qe.devcluster.openshift.com openshift4/ose-logging-curator5 blobs=5 mounts=0 manifests=1 shared=0

info: Planning completed in 4.9s
uploading: default-route-openshift-image-registry.apps.yinzhou.qe.devcluster.openshift.com/openshift4/ose-logging-curator5 sha256:46b71c0b675d0ea8766a1c41ce930c5281c3d993dbfb6870355cbf313005d193 4.166MiB
uploading: default-route-openshift-image-registry.apps.yinzhou.qe.devcluster.openshift.com/openshift4/ose-logging-curator5 sha256:ba308e6b72af78ca699abbe6cc76b7eb583afb0930eadca3999d61eb575ede86 73.04MiB
uploading: default-route-openshift-image-registry.apps.yinzhou.qe.devcluster.openshift.com/openshift4/ose-logging-curator5 sha256:078444edb1ee8d4e1a401fba6e157918e7056657e6c750cd612d7e8a80b345f5 8.653MiB
...

LGTM, but I tried to use another source image which no tag and digest, it failed.

[jzhang@dhcp-140-36 ~]$ oc adm catalog build --auth-token='basic xxx' --appregistry-org=learn-operator --to=default-route-openshift-image-registry.apps.yinzhou.qe.devcluster.openshift.com/openshift/learn:v1

[jzhang@dhcp-140-36 ~]$ oc adm catalog mirror default-route-openshift-image-registry.apps.yinzhou.qe.devcluster.openshift.com/openshift/learn:v1 default-route-openshift-image-registry.apps.yinzhou.qe.devcluster.openshift.com
I0203 17:01:31.485081   20034 mirror.go:230] wrote database to /tmp/282905947/bundles.db
W0203 17:01:31.485668   20034 mirror.go:159] couldn't parse quay.io/jiazha/learn-operator-image, skipping mirror: source references must have a tag or digest specified
I0203 17:01:31.486006   20034 mirror.go:328] wrote mirroring manifests to learn-manifests
[jzhang@dhcp-140-36 ~]$ cat learn-manifests/
imageContentSourcePolicy.yaml  mapping.txt                    
[jzhang@dhcp-140-36 ~]$ cat learn-manifests/imageContentSourcePolicy.yaml 
apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
  name: learn
spec:
  repositoryDigestMirrors:
  - mirrors:
    - default-route-openshift-image-registry.apps.yinzhou.qe.devcluster.openshift.com/jiazha/learn-operator-image
    source: quay.io/jiazha/learn-operator-image
[jzhang@dhcp-140-36 ~]$ cat learn-manifests/mapping.txt 
quay.io/jiazha/learn-operator-image=default-route-openshift-image-registry.apps.yinzhou.qe.devcluster.openshift.com/jiazha/learn-operator-image

@evan if no tag and digest set in the image, can we mirror it with the "latest" tag for it automatically? Change the status to ASSIGNED first.

Comment 4 Evan Cordell 2020-02-03 19:08:25 UTC
> @evan if no tag and digest set in the image, can we mirror it with the "latest" tag for it automatically? Change the status to ASSIGNED first.

No, we can't reasonably do that at the moment. ImageContentSourcePolicy will only remap images using digests (technically it's possible to enable tag mirroring in cri-o but it is disabled in openshift) which means that even though we could put the images into the target registry, it would fail to pull when you try to install the operator on openshift.

I think it's worth discussing a feature which converts the catalog itself (i.e. modifies the catalog content to point to whatever digests the current tags point to) so that all images in a catalog are valid for mirroring by digest. I think we should track that via an RFE though, not this BZ.

Comment 6 Jian Zhang 2020-02-04 01:52:40 UTC
Hi, Evan

Yes, I agree. Thanks! Create a feature request here: https://issues.redhat.com/browse/RFE-560
Verify this bug based on comment 3, 4.

Comment 8 errata-xmlrpc 2020-05-04 11:24:06 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581


Note You need to log in before you can comment on or make changes to this bug.