Bug 1797730

Summary: "oc adm catalog mirror" fails to parse the source image
Product: OpenShift Container Platform Reporter: Ben Luddy <bluddy>
Component: OLMAssignee: Ben Luddy <bluddy>
OLM sub component: OLM QA Contact: Jian Zhang <jiazha>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: bandrade, ecordell, fdeutsch, jiazha, jmontleo, krizza, stirabos, xjiang
Version: 4.3.z   
Target Milestone: ---   
Target Release: 4.3.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1790798 Environment:
Last Closed: 2020-02-25 06:18:00 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 1790798    
Bug Blocks:    

Comment 1 Jian Zhang 2020-02-11 04:48:39 UTC
*** Bug 1800658 has been marked as a duplicate of this bug. ***

Comment 2 Jian Zhang 2020-02-12 10:13:55 UTC
[jzhang@dhcp-140-36 ~]$ ./oc version -o yaml
clientVersion:
  buildDate: "2020-02-11T21:43:17Z"
  compiler: gc
  gitCommit: 2f6b9293684dd18c4fc36ece9803a86d294a153e
  gitTreeState: clean
  gitVersion: openshift-clients-4.3-6-g2f6b9293
  goVersion: go1.12.12
  major: ""
  minor: ""
  platform: linux/amd64
releaseClientVersion: 4.3.0-0.nightly-2020-02-11-224725

This version(git commits) of the `oc` contains fixed PR.

1, Create a sample Operator which lack some images in its manifests. For example, https://github.com/jianzhangbjz/learn-operator/tree/multi-versions/manifests/learn

2, Push it to your Quay app registry. For example, jiazha

3, Build CatalogSource image for redhat-operators-stage registry:
[jzhang@dhcp-140-36 ~]$ docker run -it --rm -p 5000:5000 --name=registry -v "$(pwd)"/auth:/auth -e "REGISTRY_AUTH=htpasswd" -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd -v /home/jzhang/goproject/src/github.com/operator-framework/operator-lifecycle-manager/pkg/lib/filemonitor/testdata/:/certs -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/server-new.crt -e REGISTRY_HTTP_TLS_KEY=/certs/server-new.key registry

4, Build it to the CatalogSource image, see below:
[jzhang@dhcp-140-36 ~]$ ./oc adm catalog build --appregistry-org jiazha --to=localhost:5000/jiazha/catalog:v1797730 --auth-token="xxx"
INFO[0003] loading Bundles                               dir=/tmp/manifests-944481905
INFO[0003] directory                                     dir=/tmp/manifests-944481905 file=manifests-944481905 load=bundles
INFO[0003] directory                                     dir=/tmp/manifests-944481905 file=learn-operator load=bundles
INFO[0003] directory                                     dir=/tmp/manifests-944481905 file=learn-operator-kjaud_1y load=bundles
INFO[0003] directory                                     dir=/tmp/manifests-944481905 file=0.0.1 load=bundles
INFO[0003] found csv, loading bundle                     dir=/tmp/manifests-944481905 file=learn-operator.v0.0.1.clusterserviceversion.yaml load=bundles
INFO[0003] loading bundle file                           dir=/tmp/manifests-944481905 file=learn-operator.v0.0.1.clusterserviceversion.yaml load=bundle
INFO[0003] loading bundle file                           dir=/tmp/manifests-944481905 file=learn.crd.yaml load=bundle
INFO[0003] directory                                     dir=/tmp/manifests-944481905 file=0.0.2 load=bundles
INFO[0003] found csv, loading bundle                     dir=/tmp/manifests-944481905 file=learn-operator.v0.0.1.clusterserviceversion.yaml load=bundles
INFO[0003] loading bundle file                           dir=/tmp/manifests-944481905 file=learn-operator.v0.0.1.clusterserviceversion.yaml load=bundle
INFO[0003] loading bundle file                           dir=/tmp/manifests-944481905 file=learn-operator.v0.0.2.clusterserviceversion.yaml load=bundle
INFO[0003] loading bundle file                           dir=/tmp/manifests-944481905 file=learn.crd.yaml load=bundle
INFO[0003] found csv, loading bundle                     dir=/tmp/manifests-944481905 file=learn-operator.v0.0.2.clusterserviceversion.yaml load=bundles
INFO[0003] loading bundle file                           dir=/tmp/manifests-944481905 file=learn-operator.v0.0.1.clusterserviceversion.yaml load=bundle
INFO[0003] loading bundle file                           dir=/tmp/manifests-944481905 file=learn-operator.v0.0.2.clusterserviceversion.yaml load=bundle
INFO[0003] loading bundle file                           dir=/tmp/manifests-944481905 file=learn.crd.yaml load=bundle
INFO[0003] directory                                     dir=/tmp/manifests-944481905 file=0.0.3 load=bundles
INFO[0003] found csv, loading bundle                     dir=/tmp/manifests-944481905 file=learn-operator.v0.0.3.clusterserviceversion.yaml load=bundles
INFO[0003] loading bundle file                           dir=/tmp/manifests-944481905 file=learn-operator.v0.0.3.clusterserviceversion.yaml load=bundle
INFO[0003] loading bundle file                           dir=/tmp/manifests-944481905 file=learn.crd.yaml load=bundle
INFO[0003] loading Packages and Entries                  dir=/tmp/manifests-944481905
INFO[0003] directory                                     dir=/tmp/manifests-944481905 file=manifests-944481905 load=package
INFO[0003] directory                                     dir=/tmp/manifests-944481905 file=learn-operator load=package
INFO[0003] directory                                     dir=/tmp/manifests-944481905 file=learn-operator-kjaud_1y load=package
INFO[0003] directory                                     dir=/tmp/manifests-944481905 file=0.0.1 load=package
INFO[0003] directory                                     dir=/tmp/manifests-944481905 file=0.0.2 load=package
INFO[0003] directory                                     dir=/tmp/manifests-944481905 file=0.0.3 load=package
W0212 18:00:09.736106     738 builder.go:141] error building database: error adding operator bundle : UNIQUE constraint failed: operatorbundle.name
Uploading ... 5.198MB/s
Uploading 3.405MB ...
Uploading 9.484MB ...
Pushed sha256:3486e022aea23a9f220d12fa87e08ef21dbad0d51919d162f24393f67895ae94 to localhost:5000/jiazha/catalog:v1797730

5, Mirror it to your local registry.
[jzhang@dhcp-140-36 ~]$ ./oc adm catalog mirror --to-manifests manifests-jiazha2 localhost:5000/jiazha/catalog:v1797730 localhost:5000
I0212 18:02:12.678277     816 mirror.go:201] wrote database to /tmp/429975449/bundles.db
warning: Image quay.io/jiazha/learn-operator-image:v1 does not exist and will not be mirrored


info: Planning completed in 2.17s
info: Mirroring completed in 0s (0B/s)
warning: Image quay.io/jiazha/learn-operator-image:v2 does not exist and will not be mirrored


info: Planning completed in 1.36s
info: Mirroring completed in 0s (0B/s)
localhost:5000/
  jiazha/learn-operator-image
    blobs:
      quay.io/jiazha/learn-operator-image sha256:79e893d7d56d047ec40855620de65986932983d0834d77898e067f1882ba1502 262B
      quay.io/jiazha/learn-operator-image sha256:e7d5172bc71b8e39403ac482bdc5c2a9cc40d00b55d562f29ce0803047bdc774 1.633KiB
      quay.io/jiazha/learn-operator-image sha256:63c1fd1ccb7daeff42df6ad24e275cb5bc6e35e64d2933bb9ba4a40cf29bac2b 3.29KiB
    manifests:
      sha256:5ff42112a3faedabd88823b78e53917c8a2cbf6255d7eb4fc7e21e0debe0dfb5 -> v3
  stats: shared=0 unique=3 size=5.179KiB ratio=1.00

phase 0:
  localhost:5000 jiazha/learn-operator-image blobs=3 mounts=0 manifests=1 shared=0

info: Planning completed in 1.42s
sha256:5ff42112a3faedabd88823b78e53917c8a2cbf6255d7eb4fc7e21e0debe0dfb5 localhost:5000/jiazha/learn-operator-image:v3
info: Mirroring completed in 1.69s (3.132kB/s)
I0212 18:02:19.327904     816 mirror.go:299] wrote mirroring manifests to manifests-jiazha2

As we can see from above, although there are 2 images(v1, v2) not found, it didn't fail the whole catalog mirroring process. LGTM, verify it.

Additional info:
[jzhang@dhcp-140-36 ~]$ cat manifests-jiazha2/imageContentSourcePolicy.yaml 
apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
  name: catalog
spec:
  repositoryDigestMirrors:
  - mirrors:
    - localhost:5000/jiazha/learn-operator-image
    source: quay.io/jiazha/learn-operator-image
  - mirrors:
    - localhost:5000/jiazha/learn-operator-image
    source: quay.io/jiazha/learn-operator-image
  - mirrors:
    - localhost:5000/jiazha/learn-operator-image
    source: quay.io/jiazha/learn-operator-image
[jzhang@dhcp-140-36 ~]$ cat manifests-jiazha2/mapping.txt 
quay.io/jiazha/learn-operator-image:v3=localhost:5000/jiazha/learn-operator-image:v3
quay.io/jiazha/learn-operator-image:v1=localhost:5000/jiazha/learn-operator-image:v1
quay.io/jiazha/learn-operator-image:v2=localhost:5000/jiazha/learn-operator-image:v2

Comment 4 errata-xmlrpc 2020-02-25 06:18:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0528