Bug 1851817 - oc adm catalog mirror --filter-by-os does not work
Summary: oc adm catalog mirror --filter-by-os does not work
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.5
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.6.0
Assignee: Evan Cordell
QA Contact: Jian Zhang
URL:
Whiteboard:
Depends On:
Blocks: 1851511 1853852
TreeView+ depends on / blocked
 
Reported: 2020-06-29 06:00 UTC by Jian Zhang
Modified: 2020-10-27 16:10 UTC (History)
0 users

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-27 16:09:46 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift oc pull 480 0 None closed Bug 1851817: fix filter-by-os parsing for oc adm catalog mirror 2021-02-14 12:17:35 UTC
Red Hat Product Errata RHBA-2020:4196 0 None None None 2020-10-27 16:10:06 UTC

Description Jian Zhang 2020-06-29 06:00:50 UTC
-----This is a clone of bug 1851511 -----
Description of problem:
Unable to run oc adm catalog mirror with --filter-by-os="linux/amd64"

Version-Release number of selected component (if applicable):
oc version
Client Version: openshift-clients-4.6.0-202005210021-28-g711c56a65
Server Version: 4.5.0-0.nightly-2020-06-25-033011
Kubernetes Version: v1.18.3+77c9a6a


How reproducible:
always

Steps to Reproduce:
1.oc adm catalog mirror registry.redhat.io/openshift4/ose-operator-registry:v4.4 quay.io/tbuskey/xyzzy:v4.4 --filter-by-os="linux/amd64"

Actual results:
error: the image is a manifest list and contains multiple images - use --filter-by-os to select from:

  OS            DIGEST
  linux/amd64   sha256:59ebe755481e26b340f0c02784bab6ff43d7dbc1be1ca56f5056ffc70daed271
  linux/ppc64le sha256:46aeaa82a28507fa0c44dadda6b6d6ecd65fbf8cd20ad4f0bb222b86cabb8bcd
  linux/s390x   sha256:d86dad0f8925370c4a4ea45c3856ea58eac49f8733b31fe6f76be1839f5f6750

Expected results:
To have it mirror

Additional info:

Comment 3 Jian Zhang 2020-07-06 01:45:47 UTC
1, Use the latest OC client, which contains this fixed PR.
[root@preserve-olm-env data]# ./oc version -o yaml
clientVersion:
  buildDate: "2020-07-05T15:35:18Z"
  compiler: gc
  gitCommit: fdc10d05ca16e71b8b1ae8b99fc4e257600787c5
  gitTreeState: clean
  gitVersion: openshift-clients-4.6.0-202006250705.p0-6-gfdc10d05c
  goVersion: go1.14.4
  major: ""
  minor: ""
  platform: linux/amd64
releaseClientVersion: 4.6.0-0.nightly-2020-07-05-214050
...

2, Running the mirror, but failed.
[root@preserve-olm-env data]# ./oc adm catalog mirror registry.redhat.io/openshift4/ose-operator-registry:v4.4 quay.io/olmqe --filter-by-os="linux/amd64"
using database path mapping: /:/tmp/502875084
wrote database to /tmp/502875084
errors during mirroring. the full contents of the catalog may not have been mirrored: no database file found in /tmp/502875084
wrote mirroring manifests to ose-operator-registry-manifests

No bundle.db generated.
[root@preserve-olm-env data]# ls -l /tmp/502875084
total 4
lrwxrwxrwx.  1 root root    7 May 11 13:16 bin -> usr/bin
drwxr-xr-x.  2 root root    6 Dec 14  2017 boot
drwxr-xr-x.  2 root root    6 May 11 13:16 dev
drwxr-xr-x. 50 root root 4096 Jun 21 14:17 etc
drwxr-xr-x.  2 root root    6 May 11 13:18 home
lrwxrwxrwx.  1 root root    7 May 11 13:16 lib -> usr/lib
lrwxrwxrwx.  1 root root    9 May 11 13:16 lib64 -> usr/lib64
drwxr-xr-x.  2 root root    6 Dec 14  2017 media
drwxr-xr-x.  2 root root    6 Dec 14  2017 mnt
drwxr-xr-x.  2 root root    6 Dec 14  2017 opt
drwxr-xr-x.  2 root root    6 May 11 13:16 proc
drwxr-xr-x.  2 root root    6 Jun 21 14:17 registry
drwxr-xr-x.  3 root root  154 May 11 13:22 root
drwxr-xr-x. 13 root root  163 Jun 21 13:32 run
lrwxrwxrwx.  1 root root    8 May 11 13:16 sbin -> usr/sbin
drwxr-xr-x.  2 root root    6 Dec 14  2017 srv
drwxr-xr-x.  2 root root    6 May 11 13:16 sys
drwxr-xr-x.  7 root root  132 Jun 21 14:17 tmp
drwxr-xr-x. 13 root root  155 May 11 13:16 usr
drwxr-xr-x. 18 root root  238 May 11 13:16 var

The same error without the --filter-by-os="linux/amd64" flag.
[root@preserve-olm-env data]# ./oc adm catalog mirror registry.redhat.io/openshift4/ose-operator-registry:v4.4 quay.io/olmqe
using database path mapping: /:/tmp/833747218
wrote database to /tmp/833747218
errors during mirroring. the full contents of the catalog may not have been mirrored: no database file found in /tmp/833747218
wrote mirroring manifests to ose-operator-registry-manifests

Comment 4 Evan Cordell 2020-07-06 15:13:15 UTC
`registry.redhat.io/openshift4/ose-operator-registry:v4.4` is not an index image - please use an index image created with OPM or via `oc adm catalog build` to test.

Comment 6 Jian Zhang 2020-07-07 05:51:28 UTC
oc version:
[root@preserve-olm-env catalogsource]# ./oc version -o yaml
clientVersion:
  buildDate: "2020-07-05T15:35:18Z"
  compiler: gc
  gitCommit: fdc10d05ca16e71b8b1ae8b99fc4e257600787c5
  gitTreeState: clean
  gitVersion: openshift-clients-4.6.0-202006250705.p0-6-gfdc10d05c
  goVersion: go1.14.4
  major: ""
  minor: ""
  platform: linux/amd64
releaseClientVersion: 4.6.0-0.nightly-2020-07-06-202123
...

1, Build a CatalogSource image that contains cluster-logging images with multi-arch.
[root@preserve-olm-env catalogsource]# ./oc image info registry.redhat.io/openshift4/ose-cluster-logging-operator@sha256:46b9600c1fbced28f00fa57980973e2f3c1df553c7852447a699e240a19b6fc2
error: the image is a manifest list and contains multiple images - use --filter-by-os to select from:

  OS            DIGEST
  linux/amd64   sha256:41c0c49830a73e3b4feb0cfbd77fd0e9d7d6b6f592eff5d61d3b62fdc8644d5c
  linux/ppc64le sha256:6abbc06ebc0f1a94f3a3f155a048ee4e0a3ac95f80969b7b04e46521bf6f2253
  linux/s390x   sha256:6bd490fd61de8c9dbc083a14f713ef4f127001cbc2ac24bcce041d32cf4c4923

[root@preserve-olm-env catalogsource]# ./oc image info registry.redhat.io/openshift4/ose-cluster-logging-operator@sha256:46b9600c1fbced28f00fa57980973e2f3c1df553c7852447a699e240a19b6fc2 --filter-by-os="linux/amd64"
Name:          registry.redhat.io/openshift4/ose-cluster-logging-operator@sha256:46b9600c1fbced28f00fa57980973e2f3c1df553c7852447a699e240a19b6fc2
Digest:        sha256:41c0c49830a73e3b4feb0cfbd77fd0e9d7d6b6f592eff5d61d3b62fdc8644d5c
Manifest List: sha256:46b9600c1fbced28f00fa57980973e2f3c1df553c7852447a699e240a19b6fc2
Media Type:    application/vnd.docker.distribution.manifest.v2+json
...

[root@preserve-olm-env catalogsource]# cat Dockerfile 
FROM quay.io/operator-framework/upstream-registry-builder:latest AS builder

COPY cluster-logging manifests

RUN /bin/initializer -o ./bundles.db

FROM registry.access.redhat.com/ubi7/ubi

COPY --from=builder /build/bundles.db /bundles.db
COPY --from=builder /bin/registry-server /registry-server
COPY --from=builder /bin/grpc_health_probe /bin/grpc_health_probe

EXPOSE 50051

ENTRYPOINT ["/registry-server"]

CMD ["--database", "bundles.db"]

[root@preserve-olm-env catalogsource]# docker build . -t quay.io/olmqe/catalogsource:cluster-logging
Sending build context to Docker daemon  180.2kB
...

2, Push it to Quay.io
[root@preserve-olm-env catalogsource]# docker push quay.io/olmqe/catalogsource:cluster-logging
The push refers to repository [quay.io/olmqe/catalogsource]
4e91c98fc2ae: Pushed 
...


3, Mirror it to the localhost image registry.
[root@preserve-olm-env catalogsource]# ./oc adm catalog mirror quay.io/olmqe/catalogsource:cluster-logging localhost:5000/olmqe
using database path mapping: /:/tmp/748123651
wrote database to /tmp/748123651
using database at: /tmp/748123651/bundles.db

error: unable to retrieve source image registry.redhat.io/openshift4/ose-promtail manifest sha256:9e2daf10101a4e482bf65a1bc2f5f1472555b3f2fa3cc33157624363eff6676a: unknown: Not Found
...

[root@preserve-olm-env catalogsource]# ./oc adm catalog mirror quay.io/olmqe/catalogsource:cluster-logging localhost:5000/arch --filter-by-os to="linux/amd64"
using database path mapping: /:/tmp/986875192
wrote database to /tmp/986875192
using database at: /tmp/986875192/bundles.db
localhost:5000/
  arch/openshift4-ose-logging-curator5
...

Both work well.

4, Check images in the localhost registry.
[root@preserve-olm-env catalogsource]# curl -k --user test:xxx https://localhost:5000/v2/_catalog 
{"repositories":["arch/openshift4-ose-cluster-logging-operator","arch/openshift4-ose-logging-curator5","arch/openshift4-ose-logging-elasticsearch5","arch/openshift4-ose-logging-fluentd","arch/openshift4-ose-logging-kibana5","arch/openshift4-ose-oauth-proxy","olmqe/openshift4-ose-cluster-logging-operator","olmqe/openshift4-ose-logging-curator5","olmqe/openshift4-ose-logging-elasticsearch5","olmqe/openshift4-ose-logging-fluentd","olmqe/openshift4-ose-logging-kibana5","olmqe/openshift4-ose-oauth-proxy"]}

LGTM, verify it.

Comment 8 errata-xmlrpc 2020-10-27 16:09:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196


Note You need to log in before you can comment on or make changes to this bug.