Bug 1868004 - opm index add appears to produce image with wrong registry server binary
Summary: opm index add appears to produce image with wrong registry server binary
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.6
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.7.0
Assignee: Haseeb Tariq
QA Contact: Jian Zhang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-08-11 11:49 UTC by Jian Zhang
Modified: 2021-02-24 15:15 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-02-24 15:15:27 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2020:5633 0 None None None 2021-02-24 15:15:56 UTC

Description Jian Zhang 2020-08-11 11:49:37 UTC
Description of problem:
Get the below error when subscribe to operator:
[root@preserve-olm-env etcd]# oc logs catalog-operator-784d66f4fc-fvwsx -n openshift-operator-lifecycle-manager |grep etcd-sub
...
time="2020-08-11T03:49:20Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subscriptions.operators.coreos.com \"etcd-sub\": the object has been modified; please apply your changes to the latest version and try again" event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/default/subscriptions/etcd-sub
E0811 03:49:20.698379       1 queueinformer_operator.go:290] sync {"update" "default/etcd-sub"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "etcd-sub": the object has been modified; please apply your changes to the latest version and try again
time="2020-08-11T03:49:20Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/default/subscriptions/etcd-sub


Version-Release number of selected component (if applicable):
# opm version
Version: version.Version{OpmVersion:"v1.12.5-104-g018a040", GitCommit:"018a040", BuildDate:"2020-08-10T02:53:22Z", GoOs:"linux", GoArch:"amd64"}


How reproducible:
always

Steps to Reproduce:
1. Install the OPM binary.
2. Get the etcd operator manifest. I used https://github.com/operator-framework/community-operators/tree/master/community-operators/etcd

3. opm build the 0.9.0 bundle.
[root@preserve-olm-env etcd]# opm alpha bundle build  -c 4.6 -e 4.6 -d ./0.9.0/ -p etcd -t quay.io/olmqe/etcd-bundle:0.9.0-test -o=true
INFO[0000] Building annotations.yaml                    
...
Successfully tagged quay.io/olmqe/etcd-bundle:0.9.0-test
[root@preserve-olm-env etcd]# docker push quay.io/olmqe/etcd-bundle:0.9.0-test
The push refers to repository [quay.io/olmqe/etcd-bundle]
f48764a200ff: Layer already exists
c65a27adc000: Layer already exists
0.9.0-test: digest: sha256:2b3990cfb0917e62b76d0ceec4f3a3ebe4ef7f48da5622a028b8c6acc0fb31aa size: 732

4.opm build the 0.9.2 bundle.
[root@preserve-olm-env etcd]# opm alpha bundle build  -c 4.6 -e 4.6 -d ./0.9.2/ -p etcd -t quay.io/olmqe/etcd-bundle:0.9.2-test -o=true
INFO[0000] Building annotations.yaml                    
INFO[0000] Writing annotations.yaml in /data/goproject/src/github.com/operator-framework/community-operators/community-operators/etcd/metadata
...
Successfully tagged quay.io/olmqe/etcd-bundle:0.9.2-test
 
[root@preserve-olm-env etcd]# docker push quay.io/olmqe/etcd-bundle:0.9.2-test
The push refers to repository [quay.io/olmqe/etcd-bundle]
daff5414f7a8: Pushed
e2901b8220f0: Pushed
0.9.2-test: digest: sha256:12a16909ad6829dd48d31fc3a6cc3559c249ababe5b8efb9730ff6f61a194e5d size: 732

5, Add the 0.9.0 bundle to the index image.
[root@preserve-olm-env etcd]# opm index add -b quay.io/olmqe/etcd-bundle:0.9.0-test -t quay.io/olmqe/etcd-index:test -c docker
INFO[0000] building the index                            bundles="[quay.io/olmqe/etcd-bundle:0.9.0-test]"
INFO[0000] running /usr/bin/docker pull quay.io/olmqe/etcd-bundle:0.9.0-test  bundles="[quay.io/olmqe/etcd-bundle:0.9.0-test]"
...
INFO[0001] [docker build -f index.Dockerfile144352666 -t quay.io/olmqe/etcd-index:test .]  bundles="[quay.io/olmqe/etcd-bundle:0.9.0-test]"
[root@preserve-olm-env etcd]# docker push quay.io/olmqe/etcd-index:test
The push refers to repository [quay.io/olmqe/etcd-index]
97d2b375dfce: Pushed
9e3f512ced9c: Layer already exists
5658e440f799: Layer already exists
d8a53ddb879b: Layer already exists
3e207b409db3: Layer already exists
test: digest: sha256:2209900d6c18974ea8734f6e70d20897a234a03ed093b2e1ccdc9c1611acda40 size: 1371

6, Add the 0.9.2 bundle to the same index image.
[root@preserve-olm-env etcd]# opm index add -b quay.io/olmqe/etcd-bundle:0.9.2-test -f quay.io/olmqe/etcd-index:test -t quay.io/olmqe/etcd-index:test -c docker
INFO[0000] building the index                            bundles="[quay.io/olmqe/etcd-bundle:0.9.2-test]"
INFO[0000] Pulling previous image quay.io/olmqe/etcd-index:test to get metadata  bundles="[quay.io/olmqe/etcd-bundle:0.9.2-test]"
...
INFO[0003] [docker build -f index.Dockerfile481017038 -t quay.io/olmqe/etcd-index:test .]  bundles="[quay.io/olmqe/etcd-bundle:0.9.2-test]"
 
[root@preserve-olm-env etcd]# docker push quay.io/olmqe/etcd-index:test
The push refers to repository [quay.io/olmqe/etcd-index]
1653ae90c782: Pushed
9e3f512ced9c: Layer already exists
5658e440f799: Layer already exists
d8a53ddb879b: Layer already exists
3e207b409db3: Layer already exists
test: digest: sha256:c74e0421f2eec763e96c3d4e4b84e3f8b6c6b2b3bb2b7c80b9fad95293582856 size: 1371

7, Create a CatalogSource to consume this index image.
[root@preserve-olm-env etcd]# cat /data/cs-share.yaml
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: share-test
  namespace: openshift-marketplace
spec:
  displayName: Share Test
  publisher: Jian
  sourceType: grpc
  image: quay.io/olmqe/etcd-index:test

[root@preserve-olm-env etcd]# oc create -f /data/cs-share.yaml
catalogsource.operators.coreos.com/share-test created
[root@preserve-olm-env etcd]# oc get catalogsource
NAME              DISPLAY          TYPE   PUBLISHER   AGE
...
share-test        Share Test       grpc   Jian        5s

[root@preserve-olm-env etcd]# oc get packagemanifest|grep etcd
etcd                                Priority2 Test   4h51m
etcd                                Priority1 Test   156m
etcd                                Share Test       2m8s

8.Subscribe to etcd operator, 0.9.0 version.
[root@preserve-olm-env etcd]# oc create -f /data/sub-etcd-share.yaml
subscription.operators.coreos.com/etcd-sub created
[root@preserve-olm-env etcd]# cat /data/sub-etcd-share.yaml
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: etcd-sub
  namespace: default
spec:
  channel: "4.6"
  name: etcd
  source: share-test
  sourceNamespace: openshift-marketplace
  startingCSV: etcdoperator.v0.9.0



Actual results:
Failed to subscribe the 0.9.0 etcd operator. Get above errors.

[root@preserve-olm-env etcd]# oc get sub -n default
NAME       PACKAGE   SOURCE       CHANNEL
etcd-sub   etcd      share-test   4.6
[root@preserve-olm-env etcd]# oc get ip -n default
No resources found in default namespace.
[root@preserve-olm-env etcd]# oc get csv -n default
No resources found in default namespace.

Expected results:
Subscribe to the 0.9.0 etcd operator successfully.

Additional info:
1) It works well when subscribing to 0.9.2 version etcd operator.

2) Check the database, no found the bundle of the 0.9.0. As follows:
[root@preserve-olm-env etcd]# docker create quay.io/olmqe/etcd-index:test
[root@preserve-olm-env etcd]# docker cp 397531ba522df5ede9e624a18f3ebcc978810e040bdc41376ad47d3f662fb4a1:/database/index.db .
[root@preserve-olm-env etcd]#
[root@preserve-olm-env etcd]# sqlite3 index.db
SQLite version 3.7.17 2013-05-20 00:56:22
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> .header on
sqlite> .mode column
sqlite> .table
api                channel            operatorbundle     related_image    
api_provider       channel_entry      package            schema_migrations
api_requirer       dependencies       properties      
sqlite> select * from channel_entry;
entry_id    channel_name  package_name  operatorbundle_name  replaces    depth    
----------  ------------  ------------  -------------------  ----------  ----------
1           4.6           etcd          etcdoperator.v0.9.2  2           0        
2           4.6           etcd          etcdoperator.v0.9.0              1        
sqlite> select * from operatorbundle;
name                 csv         bundle      bundlepath                            skiprange   version     replaces    skips    
-------------------  ----------  ----------  ------------------------------------  ----------  ----------  ----------  ----------
etcdoperator.v0.9.0                          quay.io/olmqe/etcd-bundle:0.9.0-test              0.9.0                            
etcdoperator.v0.9.2  {"apiVersi  {"apiVersi  quay.io/olmqe/etcd-bundle:0.9.2-test              0.9.2       etcdoperat            
sqlite>
sqlite> .quit

3) I also removed the `replace` field of the etcdoperator.v0.9.2 csv. And, recreate the bundle/index image. Get the same error.

Comment 1 Kevin Rizza 2020-08-20 20:02:36 UTC
Jian,

Can you pass along the logs from the catalog operator as well as if any install plans were created?

Marking for upcoming sprint as we are still actively triaging.

Comment 2 Ben Luddy 2020-08-20 20:23:04 UTC
I believe this has recently been fixed. I was able to reproduce with quay.io/olmqe/etcd-index:test:

> found more than one head for channel: [etcdoperator.v0.9.2 etcdoperator.v0.9.0]

As noted, the database does have the appropriate replaces, but it doesn't appear in the registry GRPC endpoint:

$ grpcurl --plaintext localhost:32772 api.Registry/ListBundles | jq -s '.[]|select(.packageName="etcd")|{csvName,channelName,replaces}'
{
  "csvName": "etcdoperator.v0.9.2",
  "channelName": "4.6",
  "replaces": null
}
{
  "csvName": "etcdoperator.v0.9.0",
  "channelName": "4.6",
  "replaces": null
}

After rebuilding with the latest registry image:

$ grpcurl --plaintext localhost:32775 api.Registry/ListBundles | jq -s '.[]|select(.packageName="etcd")|{csvName,channelName,replaces}'
{
  "csvName": "etcdoperator.v0.9.2",
  "channelName": "4.6",
  "replaces": "etcdoperator.v0.9.0"
}
{
  "csvName": "etcdoperator.v0.9.0",
  "channelName": "4.6",
  "replaces": null
}

Using the updated registry image, the subscription resolves to etcdoperator.v0.9.0 as expected.

Jian, can you confirm that this is no longer reproducible if the index is rebuilt with the latest index base image?

Comment 3 Jian Zhang 2020-08-28 07:30:48 UTC
Hi Kevin, Ben, 

> Jian, can you confirm that this is no longer reproducible if the index is rebuilt with the latest index base image?

I guess no, still no `replaces` in the registry GRPC endpoint.

> Can you pass along the logs from the catalog operator as well as if any install plans were created?

No InstallPlan generated. Get the "failed: found more than one head for channel" error.

[root@preserve-olm-env data]# opm version
Version: version.Version{OpmVersion:"v1.12.5-121-gfa10df6", GitCommit:"fa10df6", BuildDate:"2020-08-28T05:48:21Z", GoOs:"linux", GoArch:"amd64"}

1, Pull the latest index base image.
[root@preserve-olm-env etcd]# docker pull quay.io/operator-framework/upstream-opm-builder
Using default tag: latest
latest: Pulling from operator-framework/upstream-opm-builder
df20fa9351a1: Already exists 
03260e9c3f62: Already exists 
e6601cc232b2: Pull complete 
6be2a1160d9d: Pull complete 
Digest: sha256:8da9d072233bc2aabf62891b39e6670b57e6975fb05f466d00d6709ff44c798c
Status: Downloaded newer image for quay.io/operator-framework/upstream-opm-builder:latest
quay.io/operator-framework/upstream-opm-builder:latest

[root@preserve-olm-env data]# docker inspect quay.io/operator-framework/upstream-opm-builder:latest |grep Create
        "Created": "2020-08-20T15:39:30.246135683Z",

I'm not sure why the latest index image is so old. Could you help add me(jiazha) to the "quay.io/operator-framework/" repo so that I can check the build history? Thanks!

2, Create the index image.
[root@preserve-olm-env etcd]# opm index add -b quay.io/olmqe/etcd-bundle:0.9.0-test -t quay.io/olmqe/etcd-index:bug2-1868004
INFO[0000] building the index                            bundles="[quay.io/olmqe/etcd-bundle:0.9.0-test]"
...
INFO[0000] [podman build --format docker -f index.Dockerfile154608027 -t quay.io/olmqe/etcd-index:bug2-1868004 .]  bundles="[quay.io/olmqe/etcd-bundle:0.9.0-test]"
[root@preserve-olm-env etcd]# podman push quay.io/olmqe/etcd-index:bug2-1868004

[root@preserve-olm-env etcd]# opm index add -b quay.io/olmqe/etcd-bundle:0.9.2-test -f quay.io/olmqe/etcd-index:bug2-1868004 -t quay.io/olmqe/etcd-index:bug2-1868004 -c podman
INFO[0000] building the index                            bundles="[quay.io/olmqe/etcd-bundle:0.9.2-test]"
...
INFO[0005] [podman build --format docker -f index.Dockerfile075245023 -t quay.io/olmqe/etcd-index:bug2-1868004 .]  bundles="[quay.io/olmqe/etcd-bundle:0.9.2-test]"
[root@preserve-olm-env etcd]# podman push quay.io/olmqe/etcd-index:bug2-1868004
..

3, Create the CatalogSource to consume this index image:
[root@preserve-olm-env data]# cat cs-share.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: share-test
  namespace: openshift-marketplace
spec:
  displayName: Share Test
  publisher: Jian
  sourceType: grpc
  image: quay.io/olmqe/etcd-index:bug2-1868004
[root@preserve-olm-env data]# oc create -f cs-share.yaml 
catalogsource.operators.coreos.com/share-test created

[root@preserve-olm-env data]# oc get packagemanifest|grep etcd
etcd                                Share Test             23s

[root@preserve-olm-env data]# oc port-forward share-test-56dhr 50051
Forwarding from 127.0.0.1:50051 -> 50051
Forwarding from [::1]:50051 -> 50051
Handling connection for 50051
...

[root@preserve-olm-env data]# grpcurl --plaintext localhost:50051 api.Registry/ListBundles | jq -s '.[]|select(.packageName="etcd")|{csvName,channelName,replaces}'
{
  "csvName": "etcdoperator.v0.9.2",
  "channelName": "4.6",
  "replaces": null
}
{
  "csvName": "etcdoperator.v0.9.0",
  "channelName": "4.6",
  "replaces": null
}

4, Subscribe to the etcdoperator.v0.9.0

[root@preserve-olm-env data]# cat sub-etcd-share.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: etcd-sub
  namespace: default
spec:
  channel: "4.6"
  name: etcd
  source: share-test
  sourceNamespace: openshift-marketplace
  startingCSV: etcdoperator.v0.9.0
[root@preserve-olm-env data]# oc create -f sub-etcd-share.yaml 
subscription.operators.coreos.com/etcd-sub created

[root@preserve-olm-env data]# oc get sub -n default
NAME       PACKAGE   SOURCE       CHANNEL
etcd-sub   etcd      share-test   4.6
[root@preserve-olm-env data]# oc get ip -n default
No resources found in default namespace.

The Catalog operator logs:
833 time="2020-08-28T07:21:27Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subs    criptions.operators.coreos.com \"etcd-sub\": the object has been modified; please apply your changes to the latest version and try again" e    vent=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1alpha1/namespaces/default/subscriptions/etcd-sub
834 E0828 07:21:27.517975       1 queueinformer_operator.go:290] sync {"update" "default/etcd-sub"} failed: Operation cannot be fulfilled on su    bscriptions.operators.coreos.com "etcd-sub": the object has been modified; please apply your changes to the latest version and try again
835 time="2020-08-28T07:21:27Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1    alpha1/namespaces/default/subscriptions/etcd-sub
836 time="2020-08-28T07:21:27Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=/apis/operators.coreos.com/v1    alpha1/namespaces/default/subscriptions/etcd-sub
837 E0828 07:21:27.533557       1 queueinformer_operator.go:290] sync "default" failed: found more than one head for channel
838 I0828 07:21:27.534170       1 event.go:278] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"", Name:"default", UID:"2f2e1801-64dc-4ac    6-977b-51afff62aa8e", APIVersion:"v1", ResourceVersion:"233534", FieldPath:""}): type: 'Warning' reason: 'ResolutionFailed' found more than     one head for channel

Comment 4 Ben Luddy 2020-08-31 14:21:13 UTC
Hi Jian,

Have you tried pulling upstream-opm-builder with podman before running opm index add? In the above steps, the base image is pulled with docker, and the index build would have used podman by default, which in this case seems to have a stale copy of that image.

The following index build works reliably for me:

$ podman pull quay.io/operator-framework/upstream-registry-builder:latest
...

$ opm index add -c podman -b quay.io/olmqe/etcd-bundle:0.9.0-test,quay.io/olmqe/etcd-bundle:0.9.2-test -t foo:bar
...

$ podman run -d -P foo:bar
....a6814d43e87331dec3845bd72e09eaacfdcba55bd65be3caab7d14c8....

$ podman port ....a6814d43e87331dec3845bd72e09eaacfdcba55bd65be3caab7d14c8.... 50051
0.0.0.0:46351

$ grpcurl --plaintext localhost:46351 api.Registry/ListBundles | jq -s '.[]|select(.packageName="etcd")|{csvName,channelName,replaces}'
{
  "csvName": "etcdoperator.v0.9.2",
  "channelName": "4.6",
  "replaces": "etcdoperator.v0.9.0"
}
{
  "csvName": "etcdoperator.v0.9.0",
  "channelName": "4.6",
  "replaces": null
}

Comment 5 Jian Zhang 2020-09-01 08:47:51 UTC
Hi Ben,

Thanks! But, that's streange, as follows, I used the latest upstream-registry-builder image, but still cannot the "replaces" field. Details:

[root@preserve-olm-env /]# podman pull quay.io/operator-framework/upstream-registry-builder:latest
Trying to pull quay.io/operator-framework/upstream-registry-builder:latest...
Getting image source signatures
...

[root@preserve-olm-env /]# podman images |grep registry-builder
quay.io/operator-framework/upstream-registry-builder                                    latest        c09ddd455210  2 days ago    1.32 GB


[root@preserve-olm-env data]# opm index add -b quay.io/olmqe/etcd-bundle:0.9.0-test -t quay.io/olmqe/etcd-index:1868004
INFO[0000] building the index                            bundles="[quay.io/olmqe/etcd-bundle:0.9.0-test]"
...
INFO[0000] [podman build --format docker -f index.Dockerfile203771347 -t quay.io/olmqe/etcd-index:1868004 .]  bundles="[quay.io/olmqe/etcd-bundle:0.9.0-test]"

[root@preserve-olm-env data]# podman push quay.io/olmqe/etcd-index:1868004
Getting image source signatures
...
Storing signatures


[root@preserve-olm-env data]# opm index add -b quay.io/olmqe/etcd-bundle:0.9.2-test -f quay.io/olmqe/etcd-index:1868004 -t quay.io/olmqe/etcd-index:1868004 -c podman
INFO[0000] building the index                            bundles="[quay.io/olmqe/etcd-bundle:0.9.2-test]"
...

INFO[0005] [podman build --format docker -f index.Dockerfile028277275 -t quay.io/olmqe/etcd-index:1868004 .]  bundles="[quay.io/olmqe/etcd-bundle:0.9.2-test]"
[root@preserve-olm-env data]# podman push quay.io/olmqe/etcd-index:1868004
Getting image source signatures
...
Storing signatures

[root@preserve-olm-env data]# cat cs-share.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: share-test
  namespace: openshift-marketplace
spec:
  displayName: Share Test
  publisher: Jian
  sourceType: grpc
  image: quay.io/olmqe/etcd-index:1868004

[root@preserve-olm-env data]# oc create -f cs-share.yaml 
catalogsource.operators.coreos.com/share-test created
[root@preserve-olm-env data]# oc get packagemanifest|grep etcd
etcd                                         Community Operators   30h
etcd                                         Share Test            85m

[root@preserve-olm-env data]# oc port-forward share-test-4d7q2 50051
Forwarding from 127.0.0.1:50051 -> 50051
Forwarding from [::1]:50051 -> 50051
Handling connection for 50051

[root@preserve-olm-env data]# oc get pods
NAME                                    READY   STATUS    RESTARTS   AGE
...
share-test-4d7q2                        1/1     Running   0          86m

[root@preserve-olm-env data]# grpcurl --plaintext localhost:50051 api.Registry/ListBundles | jq -s '.[]|select(.packageName="etcd")|{csvName,channelName,replaces}'
{
  "csvName": "etcdoperator.v0.9.2",
  "channelName": "4.6",
  "replaces": null
}
{
  "csvName": "etcdoperator.v0.9.0",
  "channelName": "4.6",
  "replaces": null
}

It did exist in the database:
[root@preserve-olm-env data]# oc cp share-test-4d7q2:/database/index.db in.db
tar: removing leading '/' from member names
[root@preserve-olm-env data]# sqlite3 in.db 
SQLite version 3.7.17 2013-05-20 00:56:22
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> .header on
sqlite> .mode column
sqlite> 
sqlite> .table
api                channel            operatorbundle     related_image    
api_provider       channel_entry      package            schema_migrations
api_requirer       dependencies       properties       
sqlite> 
sqlite> select * from channel;
name        package_name  head_operatorbundle_name
----------  ------------  ------------------------
4.6         etcd          etcdoperator.v0.9.2     
sqlite> select * from channel_entry;
entry_id    channel_name  package_name  operatorbundle_name  replaces    depth     
----------  ------------  ------------  -------------------  ----------  ----------
1           4.6           etcd          etcdoperator.v0.9.2  2           0         
2           4.6           etcd          etcdoperator.v0.9.0              1         
sqlite> 
sqlite> select replaces from operatorbundle;
replaces            
--------------------
                    
etcdoperator.v0.9.0 


I also tried it with podman, but get the same error:
[root@preserve-olm-env data]# podman run -d -P quay.io/olmqe/etcd-index:1868004
f4b91430940f32b8802c4144831d8eaf9acc17a7c63989dfbf4b6c87ecc88680
[root@preserve-olm-env data]# podman port f4b91430940f32b8802c4144831d8eaf9acc17a7c63989dfbf4b6c87ecc88680 50051
0.0.0.0:36436
[root@preserve-olm-env data]# grpcurl --plaintext localhost:36436 api.Registry/ListBundles | jq -s '.[]|select(.packageName="etcd")|{csvName,channelName,replaces}'
{
  "csvName": "etcdoperator.v0.9.2",
  "channelName": "4.6",
  "replaces": null
}
{
  "csvName": "etcdoperator.v0.9.0",
  "channelName": "4.6",
  "replaces": null
}

[root@preserve-olm-env data]# opm index add -c podman -b quay.io/olmqe/etcd-bundle:0.9.0-test,quay.io/olmqe/etcd-bundle:0.9.2-test -t foo:bar
INFO[0000] building the index        
...

[root@preserve-olm-env data]# podman run -d -P foo:bar
7151de8e017f798fd0aca826cddf5c2b05c323f61066f524b05c5b0396087c96
[root@preserve-olm-env data]# podman port 7151de8e017f798fd0aca826cddf5c2b05c323f61066f524b05c5b0396087c96 50051
0.0.0.0:39237
[root@preserve-olm-env data]# grpcurl --plaintext localhost:39237 api.Registry/ListBundles | jq -s '.[]|select(.packageName="etcd")|{csvName,channelName,replaces}'
{
  "csvName": "etcdoperator.v0.9.2",
  "channelName": "4.6",
  "replaces": null
}
{
  "csvName": "etcdoperator.v0.9.0",
  "channelName": "4.6",
  "replaces": null
}

[root@preserve-olm-env data]# podman version
Version:      2.0.2
API Version:  1
Go Version:   go1.13.11
Built:        Wed Dec 31 19:00:00 1969
OS/Arch:      linux/amd64

Comment 6 Ben Luddy 2020-09-01 15:44:29 UTC
There's some important difference between the single call to "opm index add" that I have been using and the two separate calls that you used in your repro steps, but I'm not sure what it is yet.

Comment 9 Haseeb Tariq 2020-10-29 01:19:06 UTC
There doesn't seem to be a difference between the single vs multi step call to "opm index add" based on my testing with the bundle images that Jian provided.
I've tested with opm v1.15.0 (and an older version v1.12.3). Although I do have the latest upstream-registry-builder image that is different from what Jian tested with.


1. Pull latest image of quay.io/operator-framework/upstream-registry-builder

$ docker pull quay.io/operator-framework/upstream-registry-builder:latest
latest: Pulling from operator-framework/upstream-registry-builder
Digest: sha256:fae640fcf0a740db6131c7a66e3cc38799d42521b8ffc446d523e90c112e4157
Status: Image is up to date for quay.io/operator-framework/upstream-registry-builder:latest
quay.io/operator-framework/upstream-registry-builder:latest

$ docker inspect quay.io/operator-framework/upstream-registry-builder:latest
[
    {
        "Id": "sha256:bd16cf7045f0a7992f2589aa71e816cf199606e34150d30dccfdb6e7100c54a8",
        "RepoTags": [
            "quay.io/operator-framework/upstream-registry-builder:latest"
        ],
        "RepoDigests": [
            "quay.io/operator-framework/upstream-registry-builder@sha256:fae640fcf0a740db6131c7a66e3cc38799d42521b8ffc446d523e90c112e4157"
        ],
        "Created": "2020-10-27T19:57:22.346920979Z",
        ...


2. Build a new index image by adding bundles individually

$ opm version
Version: version.Version{OpmVersion:"1.12.3", GitCommit:"04e4e88", BuildDate:"2020-05-26T15:41:40Z", GoOs:"darwin", GoArch:"amd64"}

$ opm index add -b quay.io/olmqe/etcd-bundle:0.9.0-test -t quay.io/haseeb_tariq/index-image:etcd-index -c docker
$ docker push quay.io/haseeb_tariq/index-image:etcd-index

$ opm index add -b quay.io/olmqe/etcd-bundle:0.9.2-test -f quay.io/haseeb_tariq/index-image:etcd-index -t quay.io/haseeb_tariq/index-image:etcd-index -c docker
$ docker push quay.io/haseeb_tariq/index-image:etcd-index


3. Run the index locally

$ docker run -d -P quay.io/haseeb_tariq/index-image:etcd-index
e5433ea5bb64a7da5fa98ebb110c622dfec600c81eeb6339ebfd89076fabd7db

$ docker port e5433ea5bb64a7da5fa98ebb110c622dfec600c81eeb6339ebfd89076fabd7db
50051/tcp -> 0.0.0.0:32768


4. Check the bundles in the registry

$ grpcurl --plaintext localhost:32768  api.Registry/ListBundles | jq -s '.[]|select(.packageName="etcd")|{csvName,channelName,replaces}'
{
  "csvName": "etcdoperator.v0.9.0",
  "channelName": "4.6",
  "replaces": null
}
{
  "csvName": "etcdoperator.v0.9.2",
  "channelName": "4.6",
  "replaces": "etcdoperator.v0.9.0"
}


@jiazha Can you please try pulling the latest registry-builder image again with `docker pull quay.io/operator-framework/upstream-registry-builder:latest`. 
And share the output of `docker pull quay.io/operator-framework/upstream-registry-builder:latest`.

If you do get an updated image can you try rebuilding the index and checking again.

Comment 10 Haseeb Tariq 2020-10-29 01:21:57 UTC
Typo in the last comment.
Meant to ask for the output of `docker inspect quay.io/operator-framework/upstream-registry-builder:latest`

Comment 11 Haseeb Tariq 2020-11-09 20:53:18 UTC
@jiazha Closing this for now. 
Please reopen if you are still seeing the same issue after pulling the latest registry builder image or can confirm that you don't have the latest image pulled down.

Comment 12 Jian Zhang 2020-11-12 07:12:47 UTC
The latest image "quay.io/operator-framework/upstream-registry-builder:latest" works well. LGTM, verify it.

1, Build the index image with the latest registry image.
[root@preserve-olm-env data]# podman image ls quay.io/operator-framework/upstream-registry-builder:latest
REPOSITORY                                            TAG     IMAGE ID      CREATED      SIZE
quay.io/operator-framework/upstream-registry-builder  latest  1e9ffaf0ffc0  2 hours ago  208 MB
[root@preserve-olm-env data]# opm version
Version: version.Version{OpmVersion:"v1.14.3-49-gbfe56a7", GitCommit:"bfe56a7", BuildDate:"2020-11-05T01:16:40Z", GoOs:"linux", GoArch:"amd64"}

[root@preserve-olm-env data]# opm index add -b quay.io/olmqe/etcd-bundle:0.9.0-test -t quay.io/olmqe/etcd-index:v1 -c podman
INFO[0000] building the index                            bundles="[quay.io/olmqe/etcd-bundle:0.9.0-test]"
...
[root@preserve-olm-env data]# podman push quay.io/olmqe/etcd-index:v1
...

[root@preserve-olm-env data]# opm index add -b quay.io/olmqe/etcd-bundle:0.9.2-test -f quay.io/olmqe/etcd-index:v1 -t quay.io/olmqe/etcd-index:v1 -c podman
INFO[0000] building the index                            bundles="[quay.io/olmqe/etcd-bundle:0.9.2-test]"
...
[root@preserve-olm-env data]# podman push quay.io/olmqe/etcd-index:v1


2, Check if the index servers well.
[root@preserve-olm-env data]# docker pull quay.io/olmqe/etcd-index:v1
v1: Pulling from olmqe/etcd-index
...
[root@preserve-olm-env data]# docker run -d -P  quay.io/olmqe/etcd-index:v1
8f01ccc24072922bcf99ff0e72d2421dc641a6c20a50a67805e368dc2c0c1fbf
[root@preserve-olm-env data]# docker port 8f01ccc24072922bcf99ff0e72d2421dc641a6c20a50a67805e368dc2c0c1fbf
50051/tcp -> 0.0.0.0:32771
[root@preserve-olm-env data]# 
[root@preserve-olm-env data]# 
[root@preserve-olm-env data]# grpcurl --plaintext localhost:32771  api.Registry/ListBundles | jq -s '.[]|select(.packageName="etcd")|{csvName,channelName,replaces}'
{
  "csvName": "etcdoperator.v0.9.0",
  "channelName": "4.6",
  "replaces": null
}
{
  "csvName": "etcdoperator.v0.9.2",
  "channelName": "4.6",
  "replaces": "etcdoperator.v0.9.0"
}

3, Create a CatalogSource to consume this index image: quay.io/olmqe/etcd-index:v1
[root@preserve-olm-env data]# cat cs-share.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: share-test
  namespace: openshift-marketplace
spec:
  displayName: Share Test
  publisher: Jian
  sourceType: grpc
  image: quay.io/olmqe/etcd-index:v1

[root@preserve-olm-env data]# oc create -f cs-share.yaml 
catalogsource.operators.coreos.com/share-test created

Seems like it servers well:
[root@preserve-olm-env data]# oc port-forward share-test-kjbr7 50051
Forwarding from 127.0.0.1:50051 -> 50051
Forwarding from [::1]:50051 -> 50051
Handling connection for 50051
...

[root@preserve-olm-env data]# grpcurl --plaintext localhost:50051 api.Registry/ListBundles | jq -s '.[]|select(.packageName="etcd")|{csvName,channelName,replaces}'
{
  "csvName": "etcdoperator.v0.9.2",
  "channelName": "4.6",
  "replaces": "etcdoperator.v0.9.0"
}
{
  "csvName": "etcdoperator.v0.9.0",
  "channelName": "4.6",
  "replaces": null
}

4, Subscribe the etcd 0.9.0, the CSV generated successfully. LGTM.
[root@preserve-olm-env data]# cat sub-0.9.0.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: etcd-sub
  namespace: default
spec:
  channel: "4.6"
  name: etcd
  source: share-test
  sourceNamespace: openshift-marketplace
  startingCSV: etcdoperator.v0.9.0
[root@preserve-olm-env data]# oc create -f sub-0.9.0.yaml 
subscription.operators.coreos.com/etcd-sub created
[root@preserve-olm-env data]# oc create -f sub-0.9.0.yaml 
subscription.operators.coreos.com/etcd-test created
[root@preserve-olm-env data]# oc get ip -n default
NAME            CSV                   APPROVAL    APPROVED
install-phmzg   etcdoperator.v0.9.0   Automatic   true
[root@preserve-olm-env data]# oc get csv -n default
NAME                  DISPLAY   VERSION   REPLACES   PHASE
etcdoperator.v0.9.0   etcd      0.9.0                Installing

[root@preserve-olm-env data]#  oc get csv -n default
NAME                  DISPLAY   VERSION   REPLACES              PHASE
etcdoperator.v0.9.0   etcd      0.9.0                           Installing
etcdoperator.v0.9.2   etcd      0.9.2     etcdoperator.v0.9.0   Pending

Comment 15 errata-xmlrpc 2021-02-24 15:15:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633


Note You need to log in before you can comment on or make changes to this bug.