Bug 1904297 - The index image generated by "opm index prune" leaves unrelated images
Summary: The index image generated by "opm index prune" leaves unrelated images
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.6
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.7.0
Assignee: Evan Cordell
QA Contact: Jian Zhang
URL:
Whiteboard:
Depends On:
Blocks: 1918012
TreeView+ depends on / blocked
 
Reported: 2020-12-04 02:49 UTC by checheng
Modified: 2021-02-24 15:39 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Stranded Operator image bundles that were not referenced by any channel entries remained after an `opm index prune` operation. Consequence: This lead to unexpected index images being mirrored. Fix: Stranded image bundles are now removed when an index is pruned. Result: The unexpected images are not included when the Operator catalog is later mirrored.
Clone Of:
Environment:
Last Closed: 2021-02-24 15:37:53 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github operator-framework operator-registry pull 556 0 None closed Bug 1904297: Unexpected images left in `related_images` after pruning 2021-02-20 00:10:37 UTC
Red Hat Product Errata RHSA-2020:5633 0 None None None 2021-02-24 15:39:59 UTC

Description checheng 2020-12-04 02:49:03 UTC
Description of problem:

The need image index is limit at the `opm index prune` step. But `oc adm catalog mirror` ignore index image push all images into image mirror.

When run 

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Get operator indexes of redhat-operator.
$ podman run -p50051:50051 -it registry.redhat.io/redhat/redhat-operator-index:v4.6

2. Run the following command to prune the source index of all but the specified packages:
$ opm index prune \
    -f registry.redhat.io/redhat/redhat-operator-index:v4.6 \
    -p cluster-logging \
    -t <target_registry>:<port>/<namespace>/redhat-operator-index:v4.6

3. Run the following command to push the new index image to your target registry:
$ podman push <target_registry>:<port>/<namespace>/redhat-operator-index:v4.6

4. Running oc adm catalog mirror command extracts the contents of an index image to generate the manifests required for mirroring. But all indexes is pushed that the index image is ignored.

$ oc adm catalog mirror \
    <index_image> \
    <mirror_registry>:<port> \
    -a ${REG_CREDS} \
    --filter-by-os="<os>/<arch>"

Actual results:


Expected results:


Additional info:

The reference document 

https://access.redhat.com/documentation/en-us/openshift_container_platform/4.6/html/operators/administrator-tasks#olm-restricted-networks

Comment 1 Kevin Rizza 2020-12-04 16:47:32 UTC
Can you supply the output logs of the commands in question as well as the actual commands used? The resulting index from running `opm index prune` should only have a single package in it, so if it's still mirroring all of the operators in the index instead of just one, it means your `oc adm catalog mirror` command ran against an index that had more than one package.

Can you confirm that the index image referenced in `oc adm catalog mirror` was the pruned index and not the public default index?

Comment 3 checheng 2020-12-10 07:35:04 UTC
@Kevin Rizza

Thank you for responsing.

> Can you supply the output logs of the commands in question as well as the actual commands used? 

I will ready the log for you.

> The resulting index from running `opm index prune` should only have a single package in it, so if it's still mirroring all of the operators in the index instead of just one, it means your `oc adm catalog mirror` command ran against an index that had more than one package.

Yes, I think the result of `opm index prune` is right. But `oc adm catalog mirror` command mirror all images.

> Can you confirm that the index image referenced in `oc adm catalog mirror` was the pruned index and not the public default index?

Yes, the customer and my teamer confirm it. `oc adm catalog mirror` will mirror all images.

Best regards.

Comment 14 Evan Cordell 2021-01-18 23:20:11 UTC
With the information in https://github.com/operator-framework/operator-registry/issues/542, I was able to reproduce this by adding a bundle to the database that was no longer referenced by a channel entry.

Comment 16 Jian Zhang 2021-01-21 07:25:07 UTC
1, Build the opm with the fixed PR.
[root@preserve-olm-env operator-registry]# make build
GOFLAGS="-mod=vendor" go build  -tags "json1" -o bin/appregistry-server ./cmd/appregistry-server
GOFLAGS="-mod=vendor" go build  -tags "json1" -o bin/configmap-server ./cmd/configmap-server
GOFLAGS="-mod=vendor" go build  -tags "json1" -o bin/initializer ./cmd/initializer
GOFLAGS="-mod=vendor" go build  -tags "json1" -o bin/registry-server ./cmd/registry-server
GOFLAGS="-mod=vendor" go build -ldflags "-X 'github.com/operator-framework/operator-registry/cmd/opm/version.gitCommit=064f85a' -X 'github.com/operator-framework/operator-registry/cmd/opm/version.opmVersion=v1.14.3-90-g064f85a' -X 'github.com/operator-framework/operator-registry/cmd/opm/version.buildDate=2021-01-21T06:31:53Z'"  -tags "json1" -o bin/opm ./cmd/opm
[root@preserve-olm-env operator-registry]# 


[root@preserve-olm-env operator-registry]# ./bin/opm version
Version: version.Version{OpmVersion:"v1.14.3-90-g064f85a", GitCommit:"064f85a", BuildDate:"2021-01-21T06:31:53Z", GoOs:"linux", GoArch:"amd64"}

2, Prune the registry.redhat.io/redhat/redhat-operator-index:v4.6 to keep cluster-logging only. And, push it.
[root@preserve-olm-env operator-registry]# ./bin/opm index prune -f registry.redhat.io/redhat/redhat-operator-index:v4.6 -p cluster-logging -t quay.io/olmqe/redhat-operator-index:v4.6-2
INFO[0000] pruning the index                             packages="[cluster-logging]"
INFO[0000] Pulling previous image registry.redhat.io/redhat/redhat-operator-index:v4.6 to get metadata  packages="[cluster-logging]"
INFO[0000] running /usr/bin/podman pull registry.redhat.io/redhat/redhat-operator-index:v4.6  packages="[cluster-logging]"
...
INFO[0013] [podman build --format docker -f index.Dockerfile070313030 -t quay.io/olmqe/redhat-operator-index:v4.6-2 .]  packages="[cluster-logging]"

[root@preserve-olm-env operator-registry]# podman push quay.io/olmqe/redhat-operator-index:v4.6-2
Getting image source signatures

3, Check the database. There is no related images, looks good.
[root@preserve-olm-env operator-registry]# podman create quay.io/olmqe/redhat-operator-index:v4.6-2
46ea17a69db19e57bd5e8e0c44126bbcac074c32ffa851c3f64f01b83bc275e2
[root@preserve-olm-env operator-registry]# podman cp 46ea17a69db19e57bd5e8e0c44126bbcac074c32ffa851c3f64f01b83bc275e2:/database/index.db logging.db
[root@preserve-olm-env operator-registry]# sqlite3  logging.db "select distinct(operatorbundle_name) from related_image where operatorbundle_name not in (select operatorbundle_name from channel_entry) order by 1;"

4, check the image mirroring mapping.

[root@preserve-olm-env operator-registry]# oc adm catalog mirror quay.io/olmqe/redhat-operator-index:v4.6-2 localhost:5000 --manifests-only
src image has index label for database path: /database/index.db
using database path mapping: /database/index.db:/tmp/354694744
wrote database to /tmp/354694744
using database at: /tmp/354694744/index.db
no digest mapping available for quay.io/olmqe/redhat-operator-index:v4.6-2, skip writing to ImageContentSourcePolicy
wrote mirroring manifests to manifests-redhat-operator-index-1611212950
[root@preserve-olm-env operator-registry]# tree manifests-redhat-operator-index-1611212950/catalogSource.yaml 
manifests-redhat-operator-index-1611212950/catalogSource.yaml [error opening dir]

0 directories, 0 files
[root@preserve-olm-env operator-registry]# tree manifests-redhat-operator-index-1611212950
manifests-redhat-operator-index-1611212950
├── catalogSource.yaml
├── imageContentSourcePolicy.yaml
└── mapping.txt

0 directories, 3 files
[root@preserve-olm-env operator-registry]# cat manifests-redhat-operator-index-1611212950/catalogSource.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: redhat-operator-index
  namespace: openshift-marketplace
spec:
  image: localhost:5000/olmqe/redhat-operator-index:v4.6-2
  sourceType: grpc
[root@preserve-olm-env operator-registry]# 
[root@preserve-olm-env operator-registry]# cat manifests-redhat-operator-index-1611212950/imageContentSourcePolicy.yaml 
apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
  name: redhat-operator-index
spec:
  repositoryDigestMirrors:
  - mirrors:
    - localhost:5000/openshift4/ose-logging-curator5
    source: registry.redhat.io/openshift4/ose-logging-curator5
  - mirrors:
    - localhost:5000/openshift4/ose-logging-fluentd
    source: registry.redhat.io/openshift4/ose-logging-fluentd
  - mirrors:
    - localhost:5000/openshift4/ose-cluster-logging-operator-bundle
    source: registry.redhat.io/openshift4/ose-cluster-logging-operator-bundle
  - mirrors:
    - localhost:5000/openshift4/ose-cluster-logging-operator
    source: registry.redhat.io/openshift4/ose-cluster-logging-operator


[root@preserve-olm-env operator-registry]# cat manifests-redhat-operator-index-1611212950/mapping.txt 
registry.redhat.io/openshift4/ose-logging-curator5@sha256:73884604ac4506bcfb2a3c112eb621f40e0cd53fede8118e0f7a1b292ac8f924=localhost:5000/openshift4/ose-logging-curator5:f6789123
registry.redhat.io/openshift4/ose-logging-fluentd@sha256:11ccb42f3d96b065f7d94879611a7aefabbe509b522c11ac36be7a1c959a34d6=localhost:5000/openshift4/ose-logging-fluentd:1f0b20f7
registry.redhat.io/openshift4/ose-cluster-logging-operator-bundle@sha256:61aca61840dcf1d50f4a17fc9b2e10b7855c563bd6680b4dc77e3e9283c81369=localhost:5000/openshift4/ose-cluster-logging-operator-bundle:8f6d6578
registry.redhat.io/openshift4/ose-cluster-logging-operator@sha256:540b0d087c5e5529bab555030310478630249a9339a4a4c3fac6d0d7037d5eac=localhost:5000/openshift4/ose-cluster-logging-operator:5f9e513e
quay.io/olmqe/redhat-operator-index:v4.6-2=localhost:5000/olmqe/redhat-operator-index:v4.6-2


Only "cluster-logging" related images list, looks good to me, verify it.

Comment 21 errata-xmlrpc 2021-02-24 15:37:53 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633


Note You need to log in before you can comment on or make changes to this bug.