Bug 1797730 - "oc adm catalog mirror" fails to parse the source image
Summary: "oc adm catalog mirror" fails to parse the source image
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.3.z
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.3.z
Assignee: Ben Luddy
QA Contact: Jian Zhang
URL:
Whiteboard:
: 1800658 (view as bug list)
Depends On: 1790798
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-02-03 18:43 UTC by Ben Luddy
Modified: 2020-02-25 06:18 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1790798
Environment:
Last Closed: 2020-02-25 06:18:00 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift oc pull 278 0 None closed [release-4.3] Bug 1797730: Mirror images one by one for oc adm catalog mirror 2020-11-19 11:41:22 UTC
Red Hat Product Errata RHBA-2020:0528 0 None None None 2020-02-25 06:18:18 UTC

Comment 1 Jian Zhang 2020-02-11 04:48:39 UTC
*** Bug 1800658 has been marked as a duplicate of this bug. ***

Comment 2 Jian Zhang 2020-02-12 10:13:55 UTC
[jzhang@dhcp-140-36 ~]$ ./oc version -o yaml
clientVersion:
  buildDate: "2020-02-11T21:43:17Z"
  compiler: gc
  gitCommit: 2f6b9293684dd18c4fc36ece9803a86d294a153e
  gitTreeState: clean
  gitVersion: openshift-clients-4.3-6-g2f6b9293
  goVersion: go1.12.12
  major: ""
  minor: ""
  platform: linux/amd64
releaseClientVersion: 4.3.0-0.nightly-2020-02-11-224725

This version(git commits) of the `oc` contains fixed PR.

1, Create a sample Operator which lack some images in its manifests. For example, https://github.com/jianzhangbjz/learn-operator/tree/multi-versions/manifests/learn

2, Push it to your Quay app registry. For example, jiazha

3, Build CatalogSource image for redhat-operators-stage registry:
[jzhang@dhcp-140-36 ~]$ docker run -it --rm -p 5000:5000 --name=registry -v "$(pwd)"/auth:/auth -e "REGISTRY_AUTH=htpasswd" -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd -v /home/jzhang/goproject/src/github.com/operator-framework/operator-lifecycle-manager/pkg/lib/filemonitor/testdata/:/certs -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/server-new.crt -e REGISTRY_HTTP_TLS_KEY=/certs/server-new.key registry

4, Build it to the CatalogSource image, see below:
[jzhang@dhcp-140-36 ~]$ ./oc adm catalog build --appregistry-org jiazha --to=localhost:5000/jiazha/catalog:v1797730 --auth-token="xxx"
INFO[0003] loading Bundles                               dir=/tmp/manifests-944481905
INFO[0003] directory                                     dir=/tmp/manifests-944481905 file=manifests-944481905 load=bundles
INFO[0003] directory                                     dir=/tmp/manifests-944481905 file=learn-operator load=bundles
INFO[0003] directory                                     dir=/tmp/manifests-944481905 file=learn-operator-kjaud_1y load=bundles
INFO[0003] directory                                     dir=/tmp/manifests-944481905 file=0.0.1 load=bundles
INFO[0003] found csv, loading bundle                     dir=/tmp/manifests-944481905 file=learn-operator.v0.0.1.clusterserviceversion.yaml load=bundles
INFO[0003] loading bundle file                           dir=/tmp/manifests-944481905 file=learn-operator.v0.0.1.clusterserviceversion.yaml load=bundle
INFO[0003] loading bundle file                           dir=/tmp/manifests-944481905 file=learn.crd.yaml load=bundle
INFO[0003] directory                                     dir=/tmp/manifests-944481905 file=0.0.2 load=bundles
INFO[0003] found csv, loading bundle                     dir=/tmp/manifests-944481905 file=learn-operator.v0.0.1.clusterserviceversion.yaml load=bundles
INFO[0003] loading bundle file                           dir=/tmp/manifests-944481905 file=learn-operator.v0.0.1.clusterserviceversion.yaml load=bundle
INFO[0003] loading bundle file                           dir=/tmp/manifests-944481905 file=learn-operator.v0.0.2.clusterserviceversion.yaml load=bundle
INFO[0003] loading bundle file                           dir=/tmp/manifests-944481905 file=learn.crd.yaml load=bundle
INFO[0003] found csv, loading bundle                     dir=/tmp/manifests-944481905 file=learn-operator.v0.0.2.clusterserviceversion.yaml load=bundles
INFO[0003] loading bundle file                           dir=/tmp/manifests-944481905 file=learn-operator.v0.0.1.clusterserviceversion.yaml load=bundle
INFO[0003] loading bundle file                           dir=/tmp/manifests-944481905 file=learn-operator.v0.0.2.clusterserviceversion.yaml load=bundle
INFO[0003] loading bundle file                           dir=/tmp/manifests-944481905 file=learn.crd.yaml load=bundle
INFO[0003] directory                                     dir=/tmp/manifests-944481905 file=0.0.3 load=bundles
INFO[0003] found csv, loading bundle                     dir=/tmp/manifests-944481905 file=learn-operator.v0.0.3.clusterserviceversion.yaml load=bundles
INFO[0003] loading bundle file                           dir=/tmp/manifests-944481905 file=learn-operator.v0.0.3.clusterserviceversion.yaml load=bundle
INFO[0003] loading bundle file                           dir=/tmp/manifests-944481905 file=learn.crd.yaml load=bundle
INFO[0003] loading Packages and Entries                  dir=/tmp/manifests-944481905
INFO[0003] directory                                     dir=/tmp/manifests-944481905 file=manifests-944481905 load=package
INFO[0003] directory                                     dir=/tmp/manifests-944481905 file=learn-operator load=package
INFO[0003] directory                                     dir=/tmp/manifests-944481905 file=learn-operator-kjaud_1y load=package
INFO[0003] directory                                     dir=/tmp/manifests-944481905 file=0.0.1 load=package
INFO[0003] directory                                     dir=/tmp/manifests-944481905 file=0.0.2 load=package
INFO[0003] directory                                     dir=/tmp/manifests-944481905 file=0.0.3 load=package
W0212 18:00:09.736106     738 builder.go:141] error building database: error adding operator bundle : UNIQUE constraint failed: operatorbundle.name
Uploading ... 5.198MB/s
Uploading 3.405MB ...
Uploading 9.484MB ...
Pushed sha256:3486e022aea23a9f220d12fa87e08ef21dbad0d51919d162f24393f67895ae94 to localhost:5000/jiazha/catalog:v1797730

5, Mirror it to your local registry.
[jzhang@dhcp-140-36 ~]$ ./oc adm catalog mirror --to-manifests manifests-jiazha2 localhost:5000/jiazha/catalog:v1797730 localhost:5000
I0212 18:02:12.678277     816 mirror.go:201] wrote database to /tmp/429975449/bundles.db
warning: Image quay.io/jiazha/learn-operator-image:v1 does not exist and will not be mirrored


info: Planning completed in 2.17s
info: Mirroring completed in 0s (0B/s)
warning: Image quay.io/jiazha/learn-operator-image:v2 does not exist and will not be mirrored


info: Planning completed in 1.36s
info: Mirroring completed in 0s (0B/s)
localhost:5000/
  jiazha/learn-operator-image
    blobs:
      quay.io/jiazha/learn-operator-image sha256:79e893d7d56d047ec40855620de65986932983d0834d77898e067f1882ba1502 262B
      quay.io/jiazha/learn-operator-image sha256:e7d5172bc71b8e39403ac482bdc5c2a9cc40d00b55d562f29ce0803047bdc774 1.633KiB
      quay.io/jiazha/learn-operator-image sha256:63c1fd1ccb7daeff42df6ad24e275cb5bc6e35e64d2933bb9ba4a40cf29bac2b 3.29KiB
    manifests:
      sha256:5ff42112a3faedabd88823b78e53917c8a2cbf6255d7eb4fc7e21e0debe0dfb5 -> v3
  stats: shared=0 unique=3 size=5.179KiB ratio=1.00

phase 0:
  localhost:5000 jiazha/learn-operator-image blobs=3 mounts=0 manifests=1 shared=0

info: Planning completed in 1.42s
sha256:5ff42112a3faedabd88823b78e53917c8a2cbf6255d7eb4fc7e21e0debe0dfb5 localhost:5000/jiazha/learn-operator-image:v3
info: Mirroring completed in 1.69s (3.132kB/s)
I0212 18:02:19.327904     816 mirror.go:299] wrote mirroring manifests to manifests-jiazha2

As we can see from above, although there are 2 images(v1, v2) not found, it didn't fail the whole catalog mirroring process. LGTM, verify it.

Additional info:
[jzhang@dhcp-140-36 ~]$ cat manifests-jiazha2/imageContentSourcePolicy.yaml 
apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
  name: catalog
spec:
  repositoryDigestMirrors:
  - mirrors:
    - localhost:5000/jiazha/learn-operator-image
    source: quay.io/jiazha/learn-operator-image
  - mirrors:
    - localhost:5000/jiazha/learn-operator-image
    source: quay.io/jiazha/learn-operator-image
  - mirrors:
    - localhost:5000/jiazha/learn-operator-image
    source: quay.io/jiazha/learn-operator-image
[jzhang@dhcp-140-36 ~]$ cat manifests-jiazha2/mapping.txt 
quay.io/jiazha/learn-operator-image:v3=localhost:5000/jiazha/learn-operator-image:v3
quay.io/jiazha/learn-operator-image:v1=localhost:5000/jiazha/learn-operator-image:v1
quay.io/jiazha/learn-operator-image:v2=localhost:5000/jiazha/learn-operator-image:v2

Comment 4 errata-xmlrpc 2020-02-25 06:18:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0528


Note You need to log in before you can comment on or make changes to this bug.