Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1820419

Summary: Package server fail to read the custom CatalogSource due to "no such column: api_provider.operatorbundle_name"
Product: OpenShift Container Platform Reporter: Jian Zhang <jiazha>
Component: OLMAssignee: Evan Cordell <ecordell>
OLM sub component: OLM QA Contact: Jian Zhang <jiazha>
Status: CLOSED DUPLICATE Docs Contact:
Severity: high    
Priority: high CC: adellape, agreene, anli, ashoshan, baptiste.millemathias, jritter, mr.anson.wang, qitang
Version: 4.4   
Target Milestone: ---   
Target Release: 4.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: Docs should indicate to set `--from` field when building a disconnected index Consequence: Not setting this can cause a mismatch between the built catalog and the base image, where some database fields expected to exist do not. Fix: Set `--from` to `quay.io/openshift/origin-operator-registry:<openshift version>` where <openshift version> is e.g. "4.3" for 4.3.z for `oc adm catalog build` Result: Image will be built with a base image for the target ocp version, rather than the latest ocp version.
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-04-12 18:02:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Transscript of Catalog Image build none

Description Jian Zhang 2020-04-03 03:08:42 UTC
Description of problem:
No packagemanifest generated when using a custom CatalogSource image. Error logs: 
time="2020-04-03T02:25:01Z" level=warning msg="error getting bundle, eliding channel" action="refresh cache" channel=4.4 error="rpc error: code = Unknown desc = no such column: api_provider.operatorbundle_name" source="{qe-app-registry openshift-marketplace}"


Version-Release number of selected component (if applicable):
Cluster version is 4.4.0-0.nightly-2020-04-02-130551
mac:~ jianzhang$ oc exec catalog-operator-f575d4c6b-swvjh -- olm --version
OLM version: 0.14.2
git commit: 3f7675c6867e4d43c99eb8138cfb5fc686a37b3b

How reproducible:
always

Steps to Reproduce:
1. Create a CatalogSource, as follow:
mac:~ jianzhang$ cat cs-qe.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: qe-app-registry
  namespace: openshift-marketplace
spec:
  displayName: QE Operators
  image: quay.io/openshift-qe-optional-operators/qe-operators-registry:latest-20200402
  publisher: QE
  sourceType: grpc

mac:~ jianzhang$ oc get catalogsource -n openshift-marketplace
NAME                  DISPLAY               TYPE   PUBLISHER   AGE
certified-operators   Certified Operators   grpc   Red Hat     8h
community-operators   Community Operators   grpc   Red Hat     8h
qe-app-registry       QE Operators          grpc   QE          7h57m
redhat-marketplace    Red Hat Marketplace   grpc   Red Hat     8h
redhat-operators      Red Hat Operators     grpc   Red Hat     7h53m
mac:~ jianzhang$ oc get pods
NAME                                   READY   STATUS    RESTARTS   AGE
certified-operators-857cd5cd46-n54cs   1/1     Running   0          6h57m
community-operators-7c59bb6dc9-bhbxq   1/1     Running   0          7h58m
marketplace-operator-9c57fd757-l6bv5   1/1     Running   0          7h59m
qe-app-registry-sbx4z                  1/1     Running   0          60m
redhat-marketplace-656978459-fxqfr     1/1     Running   0          6h57m
redhat-operators-68cdc45d46-664qg      1/1     Running   0          7h54m

2. Check if the c generated.

Actual results:
No responding Packamnigest generated.
mac:~ jianzhang$ oc get packagemanifest |grep -i qe
mac:~ jianzhang$ 


Expected results:
The customize Packmanifest should be displayed successfully.

Additional info:
Debug progress:

1, check the logs of the Packageserver, got:
mac:~ jianzhang$ oc logs deployment/packageserver 
Found 2 pods, using pod/packageserver-845f456869-btsb4
...
I0403 00:04:17.720147       1 balancer_conn_wrappers.go:127] pickfirstBalancer: HandleSubConnStateChange: 0xc00035a000, READY
time="2020-04-03T00:04:17Z" level=warning msg="error getting bundle, eliding channel" action="refresh cache" channel=4.2 error="rpc error: code = Unknown desc = no such column: api_provider.operatorbundle_name" source="{qe-app-registry openshift-marketplace}"
time="2020-04-03T00:04:17Z" level=warning msg="error getting bundle, eliding channel" action="refresh cache" channel=4.2 error="rpc error: code = Unknown desc = no such column: api_provider.operatorbundle_name" source="{qe-app-registry openshift-marketplace}"
...

2, Check if this customizes CatalogSource serves well. It serves well.
mac:~ jianzhang$ oc port-forward qe-app-registry-sbx4z  50051
Forwarding from 127.0.0.1:50051 -> 50051
Forwarding from [::1]:50051 -> 50051
Handling connection for 50051
...
mac:~ jianzhang$ grpcurl -plaintext  localhost:50051 api.Registry/ListPackages
{
  "name": "cluster-kube-descheduler-operator"
}
{
  "name": "cluster-logging"
}
{
  "name": "clusterresourceoverride"
}
{
  "name": "elasticsearch-operator"
}
{
  "name": "local-storage-operator"
}
{
  "name": "metering-ocp"
}
{
  "name": "nfd"
}
{
  "name": "openshiftansibleservicebroker"
}
{
  "name": "openshifttemplateservicebroker"
}
{
  "name": "ptp-operator"
}
{
  "name": "sriov-network-operator"
}

3, Check its database. It doesn't have the "api_provider.operatorbundle_name" column.
mac:~ jianzhang$ oc rsh qe-app-registry-sbx4z   
sh-4.2$ sqlite3 /bundles.db
SQLite version 3.7.17 2013-05-20 00:56:22
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> .header on
sqlite> .mode column
sqlite> .table
api                channel            package          
api_provider       channel_entry      related_image    
api_requirer       operatorbundle     schema_migrations
sqlite> select * from api_provider;
group_name           version     kind        channel_entry_id
-------------------  ----------  ----------  ----------------
automationbroker.io  v1alpha1    Bundle      28              
automationbroker.io  v1alpha1    Bundle      29 

Seems like something wrong in the provider of the Packageserver.

Comment 4 Juerg Ritter 2020-04-06 12:52:24 UTC
Created attachment 1676580 [details]
Transscript of Catalog Image build

Comment 5 Evan Cordell 2020-04-06 13:51:24 UTC
Could you please try to create the catalog image with `oc adm catalog build`, but this time, please set the `--from` flag to the correct base image for the version of ocp.

For example, for 4.4 the command would be `oc adm catalog build --from=quay.io/openshift/origin-operator-registry:4.4 <other options>`. 

Please update here if this works - I have also added a doc task to this BZ to update the docs to show that this is a required part of the disconnected flow.

Comment 6 Evan Cordell 2020-04-06 13:53:15 UTC
*** Bug 1820653 has been marked as a duplicate of this bug. ***

Comment 7 Juerg Ritter 2020-04-06 16:19:54 UTC
Thanks Evan, the following setup worked for OCP 4.3.8:

oc adm catalog build --from=quay.io/openshift/origin-operator-registry:4.3 --appregistry-org redhat-operators --to=uxlpbastion01.centrisag.ch:5000/olm/redhat-operators:v9 -a ~/pull-secret.text

Documentation should be updated accordingly.
https://docs.openshift.com/container-platform/4.3/operators/olm-restricted-networks.html

Comment 9 Evan Cordell 2020-04-08 12:23:02 UTC
*** Bug 1821406 has been marked as a duplicate of this bug. ***

Comment 10 Alex Dellapenta 2020-04-08 20:41:27 UTC
Docs PRs:

* Add --from usage: https://github.com/openshift/openshift-docs/pull/21025
* Follow-up to switch to downstream image usage: https://github.com/openshift/openshift-docs/pull/21032

Comment 11 Asher Shoshan 2020-04-09 13:40:47 UTC
(In reply to Alex Dellapenta from comment #10)
> Docs PRs:
> 
> * Add --from usage: https://github.com/openshift/openshift-docs/pull/21025
> * Follow-up to switch to downstream image usage:
> https://github.com/openshift/openshift-docs/pull/21032

I can't understand this PR: is it downloading the same catalog "redhat-operators" from osbs?
But I'm looking to build the catalog rh-verified-operators or redhat-operators-stage from anywhere  i.e. --appregistry-org redhat-operators-stage
Which is not working U/S or D/S..
Please advise

Comment 12 Baptiste Mille-Mathias 2020-04-10 07:43:08 UTC
Question from someone affected also by this bug and dig into this issue.

using command "strings" or looking in the source code of "oc" the image used is not on the same project you mentionned, it uses quay.io/operator-framework/operator-registry-server:latest
https://github.com/openshift/oc/blob/f2b01c4e4ae8c4ca11caabf8cb8e76b7a28b7009/vendor/github.com/operator-framework/operator-registry/pkg/appregistry/builder_options.go#L159-L166


I eventually make the catalog image working using quay.io/operator-framework/operator-registry-server:v1.6.1 images.

Comment 13 Feng Wang 2020-04-11 07:18:18 UTC
I'm using OCP 4.3.12, the same issue

Comment 14 Evan Cordell 2020-04-12 18:02:39 UTC
This is tracked in the linked docs BZ, which clarifies the issue.

> I can't understand this PR: is it downloading the same catalog "redhat-operators" from osbs?
But I'm looking to build the catalog rh-verified-operators or redhat-operators-stage from anywhere  i.e. --appregistry-org redhat-operators-stage
Which is not working U/S or D/S..
Please advise

The issue is not which catalog you are pulling from, the issue is the base image used to build the catalog. It should match the version of ocp you are targeting. For example, if you are targeting ocp 4.3, you should use `quay.io/openshift/origin-operator-registry:4.3` (or, more accurately, the registry.redhat image which is mentioned in the docs).

> using command "strings" or looking in the source code of "oc" the image used is not on the same project you mentionned, it uses quay.io/operator-framework/operator-registry-server:latest
https://github.com/openshift/oc/blob/f2b01c4e4ae8c4ca11caabf8cb8e76b7a28b7009/vendor/github.com/operator-framework/operator-registry/pkg/appregistry/builder_options.go#L159-L166

The image in oc defaults to the latest, the docs should now indicate that you should specify the version of ocp you are targeting instead.

> I eventually make the catalog image working using quay.io/operator-framework/operator-registry-server:v1.6.1 images.

I would advise against using the upstream images (i.e. in quay.io/operator-framework) for anything targeting ocp.

*** This bug has been marked as a duplicate of bug 1822157 ***