Bug 1921953 - ClusterServiceVersion property inference does not infer package and version
Summary: ClusterServiceVersion property inference does not infer package and version
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.6
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.8.0
Assignee: Ben Luddy
QA Contact: kuiwang
URL:
Whiteboard:
Depends On:
Blocks: 1929904
TreeView+ depends on / blocked
 
Reported: 2021-01-28 23:20 UTC by Ben Luddy
Modified: 2021-07-27 22:37 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Operators that were installed on pre-4.6 clusters are not identified as coming from a given operator package for the purposes of dependency resolution and upgrade selection. Consequence: Existing operator installations can conflict with the criteria of their own Subscription, blocking upgrades and dependency resolution within the namespace. Fix: Infer package name and version for operators that are referenced by a Subscription. Result: Upgrades and dependency resolution proceed as expected in the presence of operators installed pre-4.6.
Clone Of:
: 1929904 (view as bug list)
Environment:
Last Closed: 2021-07-27 22:37:34 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github operator-framework operator-lifecycle-manager pull 1993 0 None open Bug 1921953: Infer package name property for unannotated CSVs, if possible. 2021-02-22 19:25:00 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 22:37:54 UTC

Internal Links: 1913822

Description Ben Luddy 2021-01-28 23:20:22 UTC 
All ClusterServiceVersions installed on OCP 4.6+ are automatically annotated with a snapshot of relevant information (properties) from the catalog they were installed from. During dependency resolution, this annotation determines whether or not a CSV is able to satisfy the dependencies of other operators. If a ClusterServiceVersion is missing this annotation (which is the case if it was installed prior to an upgrade to OCP 4.6+), some of this information is inferred by inspecting the ClusterServiceVersion's spec.

Because "package" is not part of the CSV spec, it is not inferred. If a user later tries to install an operator that declares a direct package dependency (introduced in 4.6+), an unannotated (pre-4.6) CSV can never be chosen to satisfy the dependency.

It is sometimes possible to infer package by finding a Subscription that relates an unannotated CSV to a CatalogSource. It's not possible in cases where there is no associated Subscription or if the operator in question is no longer present in the catalog.

Steps to Reproduce:

$ cat << EOF | kubectl create -f-
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: ibm-common-service-catalog
  namespace: default
spec:
  image: docker.io/ibmcom/ibm-common-service-catalog:latest
  sourceType: grpc
---
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: ibm-operator-catalog
  namespace: default
spec:
  image: docker.io/ibmcom/ibm-operator-catalog:latest
  sourceType: grpc
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: test-operator-group
  namespace: default
spec: {}
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: test-ibm-common-service-operator
  namespace: default
spec:
  channel: stable-v1
  installPlanApproval: Automatic
  name: ibm-common-service-operator
  source: ibm-common-service-catalog
  sourceNamespace: default
EOF

catalogsource.operators.coreos.com/ibm-common-service-catalog created
catalogsource.operators.coreos.com/ibm-operator-catalog created
operatorgroup.operators.coreos.com/test-operator-group created
subscription.operators.coreos.com/test-ibm-common-service-operator created

$ kubectl get csv -w # Wait until this CSV is installed!

NAME                                 DISPLAY                              VERSION   REPLACES                             PHASE
ibm-common-service-operator.v3.6.0   IBM Cloud Platform Common Services   3.6.0     ibm-common-service-operator.v3.5.6   Succeeded

$ kubectl patch -n default csv ibm-common-service-operator.v3.6.3 --type=json -p='[{"op":"remove","path":"/metadata/annotations/operatorframework.io~1properties"}]'

clusterserviceversion.operators.coreos.com/ibm-common-service-operator.v3.6.3 patched

$ cat << EOF | kubectl create -f-
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: test-ibm-integration-platform-navigator
  namespace: default
spec:
  channel: v4.1-eus
  installPlanApproval: Automatic
  name: ibm-integration-platform-navigator
  source: ibm-operator-catalog
  sourceNamespace: default
EOF

subscription.operators.coreos.com/test-ibm-integration-platform-navigator created

$ kubectl get csv -w # Observe that ibm-integration-platform-navigator is never installed.

NAME                                 DISPLAY                              VERSION   REPLACES                             PHASE
ibm-common-service-operator.v3.6.3   IBM Cloud Platform Common Services   3.6.3     ibm-common-service-operator.v3.6.0   Succeeded

Expected Results:

ibm-integration-platform-navigator is installed after a few moments.

Actual Results:

ibm-integration-platform-navigator is never installed. You will probably also see a big event explaining why it thinks the installation can't succeed, like this one:

$ kubectl get -n default event --sort-by=.metadata.creationTimestamp | grep ResolutionFailed | tail -1
2m55s       Warning   ResolutionFailed          namespace/default                                                     constraints not satisfiable: gvkunique/operator.ibm.com/v3/CommonService permits at most 1 of cs-catalog/olm/dev/ibm-common-service-operator.v3.5.0, cs-catalog/olm/beta/ibm-common-service-operator.v3.4.1, @existing/default//ibm-common-service-operator.v3.6.3, cs-catalog/olm/dev/ibm-common-service-operator.v3.5.3, ibm-common-service-catalog/default/stable-v1/ibm-common-service-operator.v3.4.3, cs-catalog/olm/stable-v1/ibm-common-service-operator.v3.6.3, ibm-common-service-catalog/default/stable-v1/ibm-common-service-operator.v3.5.5, cs-catalog/olm/dev/ibm-common-service-operator.v3.6.2, cs-catalog/olm/beta/ibm-common-service-operator.v3.4.0, ibm-common-service-catalog/default/dev/ibm-common-service-operator.v3.5.4, ibm-common-service-catalog/default/beta/ibm-common-service-operator.v3.6.0, ibm-common-service-catalog/default/dev/ibm-common-service-operator.v3.4.1, cs-catalog/olm/dev/ibm-common-service-operator.v3.4.2, ibm-common-service-catalog/default/stable-v1/ibm-common-service-operator.v3.5.1, ibm-common-service-catalog/default/stable-v1/ibm-common-service-operator.v3.6.3, ibm-common-service-catalog/default/stable-v1/ibm-common-service-operator.v3.3.0, cs-catalog/olm/beta/ibm-common-service-operator.v3.4.3, cs-catalog/olm/beta/ibm-common-service-operator.v3.6.0, cs-catalog/olm/dev/ibm-common-service-operator.v3.5.6, ibm-common-service-catalog/default/stable-v1/ibm-common-service-operator.v3.5.2, ibm-common-service-catalog/default/stable-v1/ibm-common-service-operator.v3.6.1, ibm-common-service-catalog/default/dev/ibm-common-service-operator.v3.4.0, ibm-common-service-catalog/default/dev/ibm-common-service-operator.v3.6.2, cs-catalog/olm/stable-v1/ibm-common-service-operator.v3.6.1, cs-catalog/olm/dev/ibm-common-service-operator.v3.5.4, ibm-common-service-catalog/default/beta/ibm-common-service-operator.v3.5.0, ibm-common-service-catalog/default/stable-v1/ibm-common-service-operator.v3.4.2, ibm-common-service-catalog/default/stable-v1/ibm-common-service-operator.v3.5.3, cs-catalog/olm/dev/ibm-common-service-operator.v3.5.1, cs-catalog/olm/stable-v1/ibm-common-service-operator.v3.5.5, cs-catalog/olm/stable-v1/ibm-common-service-operator.v3.3.0, ibm-common-service-catalog/default/beta/ibm-common-service-operator.v3.5.6, cs-catalog/olm/beta/ibm-common-service-operator.v3.5.2, the subscription "ibm-common-service-operator" exists, ibm-common-service-operator requires at least one of @existing/default//ibm-common-service-operator.v3.6.3, ibm-operator-catalog/default/v4.1-eus/ibm-integration-platform-navigator.v4.1.0 requires at least one of ibm-common-service-catalog/default/stable-v1/ibm-common-service-operator.v3.6.3, ibm-common-service-catalog/default/dev/ibm-common-service-operator.v3.6.2, ibm-common-service-catalog/default/stable-v1/ibm-common-service-operator.v3.6.1, ibm-common-service-catalog/default/beta/ibm-common-service-operator.v3.6.0, ibm-common-service-catalog/default/dev/ibm-common-service-operator.v3.6.2, ibm-common-service-catalog/default/stable-v1/ibm-common-service-operator.v3.6.1, ibm-common-service-catalog/default/beta/ibm-common-service-operator.v3.6.0, ibm-common-service-catalog/default/dev/ibm-common-service-operator.v3.6.2, ibm-common-service-catalog/default/stable-v1/ibm-common-service-operator.v3.6.1, ibm-common-service-catalog/default/beta/ibm-common-service-operator.v3.6.0, cs-catalog/olm/stable-v1/ibm-common-service-operator.v3.6.3, cs-catalog/olm/dev/ibm-common-service-operator.v3.6.2, cs-catalog/olm/stable-v1/ibm-common-service-operator.v3.6.1, cs-catalog/olm/beta/ibm-common-service-operator.v3.6.0, cs-catalog/olm/dev/ibm-common-service-operator.v3.6.2, cs-catalog/olm/stable-v1/ibm-common-service-operator.v3.6.1, cs-catalog/olm/beta/ibm-common-service-operator.v3.6.0, cs-catalog/olm/dev/ibm-common-service-operator.v3.6.2, cs-catalog/olm/stable-v1/ibm-common-service-operator.v3.6.1, cs-catalog/olm/beta/ibm-common-service-operator.v3.6.0, ibm-integration-platform-navigator requires at least one of ibm-operator-catalog/default/v4.1-eus/ibm-integration-platform-navigator.v4.1.0, the subscription "ibm-integration-platform-navigator" exists
Comment 1 Kevin Rizza 2021-02-01 15:29:25 UTC 
Marking for upcoming sprint and removing from the 4.7.0 release. This is not a blocker for 4.7 and will be addressed in a future sprint.
Comment 4 kuiwang 2021-02-20 08:29:05 UTC 
verify it on 4.8. Fail

--
[root@preserve-olm-env 1921953]# oc get pod -n openshift-operator-lifecycle-manager
NAME                                READY   STATUS    RESTARTS   AGE
catalog-operator-679db8fbc4-qnqv2   1/1     Running   0          44m
olm-operator-79cb7777cc-klkdx       1/1     Running   0          44m
packageserver-f467b6887-58cfs       1/1     Running   0          37m
packageserver-f467b6887-6m7qn       1/1     Running   0          37m
[root@preserve-olm-env 1921953]# oc exec catalog-operator-679db8fbc4-qnqv2 -n openshift-operator-lifecycle-manager -- olm --version
OLM version: 0.17.0
git commit: 034b4001f752c23aa5a35744f7a8536afab0cd63
[root@preserve-olm-env 1921953]# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-02-19-182344   True        False         19m     Cluster version is 4.8.0-0.nightly-2021-02-19-182344
[root@preserve-olm-env 1921953]# 


[root@preserve-olm-env 1921953]# cat res.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: ibm-common-service-catalog
  namespace: default
spec:
  image: docker.io/ibmcom/ibm-common-service-catalog:latest
  sourceType: grpc
---
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: ibm-operator-catalog
  namespace: default
spec:
  image: docker.io/ibmcom/ibm-operator-catalog:latest
  sourceType: grpc
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: test-operator-group
  namespace: default
spec: {}
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: test-ibm-common-service-operator
  namespace: default
spec:
  channel: stable-v1
  installPlanApproval: Automatic
  name: ibm-common-service-operator
  source: ibm-common-service-catalog
  sourceNamespace: default

[root@preserve-olm-env 1921953]# 
[root@preserve-olm-env 1921953]# kubectl create -f res.yaml 
catalogsource.operators.coreos.com/ibm-common-service-catalog created
catalogsource.operators.coreos.com/ibm-operator-catalog created
operatorgroup.operators.coreos.com/test-operator-group created
subscription.operators.coreos.com/test-ibm-common-service-operator created

[root@preserve-olm-env 1921953]# kubectl get csv
NAME                                 DISPLAY                              VERSION   REPLACES                             PHASE
ibm-common-service-operator.v3.6.3   IBM Cloud Platform Common Services   3.6.3     ibm-common-service-operator.v3.6.2   Succeeded
[root@preserve-olm-env 1921953]# 

[root@preserve-olm-env 1921953]# kubectl get csv ibm-common-service-operator.v3.6.3 -o yaml|grep "operatorframework.io/properties"
    operatorframework.io/properties: '{"properties":[{"type":"olm.gvk","value":{"group":"operator.ibm.com","kind":"CommonService","version":"v3"}},{"type":"olm.package","value":{"packageName":"ibm-common-service-operator","version":"3.6.3"}}]}'
          f:operatorframework.io/properties: {}
[root@preserve-olm-env 1921953]# 

[root@preserve-olm-env 1921953]# kubectl patch -n default csv ibm-common-service-operator.v3.6.3 --type=json -p='[{"op":"remove","path":"/metadata/annotations/operatorframework.io~1properties"}]'
clusterserviceversion.operators.coreos.com/ibm-common-service-operator.v3.6.3 patched
[root@preserve-olm-env 1921953]# kubectl get csv ibm-common-service-operator.v3.6.3 -o yaml|grep "operatorframework.io/properties"
[root@preserve-olm-env 1921953]# 

[root@preserve-olm-env 1921953]# cat sub.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: test-ibm-integration-platform-navigator
  namespace: default
spec:
  channel: v4.1-eus
  installPlanApproval: Automatic
  name: ibm-integration-platform-navigator
  source: ibm-operator-catalog
  sourceNamespace: default
[root@preserve-olm-env 1921953]# kubectl create -f sub.yaml 
subscription.operators.coreos.com/test-ibm-integration-platform-navigator created
[root@preserve-olm-env 1921953]# 

[root@preserve-olm-env 1921953]# kubectl get csv
NAME                                 DISPLAY                              VERSION   REPLACES                             PHASE
ibm-common-service-operator.v3.6.3   IBM Cloud Platform Common Services   3.6.3     ibm-common-service-operator.v3.6.2   Succeeded
[root@preserve-olm-env 1921953]# 

[root@preserve-olm-env 1921953]# kubectl get -n default event --sort-by=.metadata.creationTimestamp | grep ResolutionFailed | tail -1
5m31s       Warning   ResolutionFailed                             namespace/default                                                     (combined from similar events): constraints not satisfiable: ibm-operator-catalog/default/v4.1-eus/ibm-integration-platform-navigator.v4.1.0 requires at least one of ibm-common-service-catalog/default/stable-v1/ibm-common-service-operator.v3.6.3, ibm-common-service-catalog/default/beta/ibm-common-service-operator.v3.6.2, ibm-common-service-catalog/default/dev/ibm-common-service-operator.v3.6.1, ibm-common-service-catalog/default/dev/ibm-common-service-operator.v3.6.0, ibm-common-service-catalog/default/beta/ibm-common-service-operator.v3.6.2, ibm-common-service-catalog/default/dev/ibm-common-service-operator.v3.6.1, ibm-common-service-catalog/default/dev/ibm-common-service-operator.v3.6.0, ibm-common-service-catalog/default/beta/ibm-common-service-operator.v3.6.2, ibm-common-service-catalog/default/dev/ibm-common-service-operator.v3.6.1, ibm-common-service-catalog/default/dev/ibm-common-service-operator.v3.6.0, subscription test-ibm-common-service-operator requires @existing/default//ibm-common-service-operator.v3.6.3, subscription test-ibm-common-service-operator exists, ibm-common-service-catalog/default/beta/ibm-common-service-operator.v3.4.0, ibm-common-service-catalog/default/dev/ibm-common-service-operator.v3.5.1, @existing/default//ibm-common-service-operator.v3.6.3, ibm-common-service-catalog/default/dev/ibm-common-service-operator.v3.4.3, ibm-common-service-catalog/default/dev/ibm-common-service-operator.v3.6.1, ibm-common-service-catalog/default/beta/ibm-common-service-operator.v3.6.2, ibm-common-service-catalog/default/stable-v1/ibm-common-service-operator.v3.5.2, ibm-common-service-catalog/default/dev/ibm-common-service-operator.v3.6.0, ibm-common-service-catalog/default/beta/ibm-common-service-operator.v3.4.1, ibm-common-service-catalog/default/dev/ibm-common-service-operator.v3.5.0, ibm-common-service-catalog/default/stable-v1/ibm-common-service-operator.v3.6.3, ibm-common-service-catalog/default/dev/ibm-common-service-operator.v3.4.2, ibm-common-service-catalog/default/dev/ibm-common-service-operator.v3.5.6, ibm-common-service-catalog/default/stable-v1/ibm-common-service-operator.v3.5.5, ibm-common-service-catalog/default/dev/ibm-common-service-operator.v3.3.0, ibm-common-service-catalog/default/beta/ibm-common-service-operator.v3.5.4 and ibm-common-service-catalog/default/dev/ibm-common-service-operator.v3.5.3 provide CommonService (operator.ibm.com/v3), subscription test-ibm-integration-platform-navigator requires ibm-operator-catalog/default/v4.1-eus/ibm-integration-platform-navigator.v4.1.0, subscription test-ibm-integration-platform-navigator exists
[root@preserve-olm-env 1921953]# 


--
Comment 6 Ben Luddy 2021-02-22 17:43:12 UTC 
This is linked to the wrong PR. No fix for this BZ has merged yet.
Comment 8 kuiwang 2021-02-24 01:48:00 UTC 
verify it on 4.8. LGTM

--
[root@preserve-olm-env 1921953]# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-02-23-160229   True        False         5m5s    Cluster version is 4.8.0-0.nightly-2021-02-23-160229
[root@preserve-olm-env 1921953]# oc get pod -n openshift-operator-lifecycle-manager
NAME                                READY   STATUS    RESTARTS   AGE
catalog-operator-5bcc4c469b-pmlfr   1/1     Running   0          35m
olm-operator-9f454ff4d-xxp7q        1/1     Running   0          35m
packageserver-5558bbd88f-45xs8      1/1     Running   0          27m
packageserver-5558bbd88f-j2clv      1/1     Running   0          27m
[root@preserve-olm-env 1921953]# oc exec catalog-operator-5bcc4c469b-pmlfr -n openshift-operator-lifecycle-manager -- olm --version
OLM version: 0.17.0
git commit: fd0ce87079d83588ad4c6e74f3175edf340e28a8
[root@preserve-olm-env 1921953]# 

[root@preserve-olm-env 1921953]# cat res.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: ibm-common-service-catalog
  namespace: default
spec:
  image: docker.io/ibmcom/ibm-common-service-catalog:latest
  sourceType: grpc
---
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: ibm-operator-catalog
  namespace: default
spec:
  image: docker.io/ibmcom/ibm-operator-catalog:latest
  sourceType: grpc
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: test-operator-group
  namespace: default
spec: {}
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: test-ibm-common-service-operator
  namespace: default
spec:
  channel: stable-v1
  installPlanApproval: Automatic
  name: ibm-common-service-operator
  source: ibm-common-service-catalog
  sourceNamespace: default

[root@preserve-olm-env 1921953]# kubectl create -f res.yaml
catalogsource.operators.coreos.com/ibm-common-service-catalog created
catalogsource.operators.coreos.com/ibm-operator-catalog created
operatorgroup.operators.coreos.com/test-operator-group created
subscription.operators.coreos.com/test-ibm-common-service-operator created

[root@preserve-olm-env 1921953]# kubectl get csv
NAME                                 DISPLAY                              VERSION   REPLACES                             PHASE
ibm-common-service-operator.v3.6.3   IBM Cloud Platform Common Services   3.6.3     ibm-common-service-operator.v3.6.2   Succeeded
[root@preserve-olm-env 1921953]# 

[root@preserve-olm-env 1921953]# kubectl get csv ibm-common-service-operator.v3.6.3 -o yaml|grep "operatorframework.io/properties"
    operatorframework.io/properties: '{"properties":[{"type":"olm.gvk","value":{"group":"operator.ibm.com","kind":"CommonService","version":"v3"}},{"type":"olm.package","value":{"packageName":"ibm-common-service-operator","version":"3.6.3"}}]}'
          f:operatorframework.io/properties: {}
[root@preserve-olm-env 1921953]# 


[root@preserve-olm-env 1921953]# kubectl patch -n default csv ibm-common-service-operator.v3.6.3 --type=json -p='[{"op":"remove","path":"/metadata/annotations/operatorframework.io~1properties"}]'
clusterserviceversion.operators.coreos.com/ibm-common-service-operator.v3.6.3 patched
[root@preserve-olm-env 1921953]# kubectl get csv ibm-common-service-operator.v3.6.3 -o yaml|grep "operatorframework.io/properties"
[root@preserve-olm-env 1921953]# 


[root@preserve-olm-env 1921953]# cat sub.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: test-ibm-integration-platform-navigator
  namespace: default
spec:
  channel: v4.1-eus
  installPlanApproval: Automatic
  name: ibm-integration-platform-navigator
  source: ibm-operator-catalog
  sourceNamespace: default
[root@preserve-olm-env 1921953]# kubectl create -f sub.yaml 
subscription.operators.coreos.com/test-ibm-integration-platform-navigator created
[root@preserve-olm-env 1921953]# kubectl get csv
NAME                                        DISPLAY                                            VERSION   REPLACES                             PHASE
ibm-common-service-operator.v3.6.3          IBM Cloud Platform Common Services                 3.6.3     ibm-common-service-operator.v3.6.2   Succeeded
ibm-integration-platform-navigator.v4.1.0   IBM Cloud Pak for Integration Platform Navigator   4.1.0                                          Succeeded
[root@preserve-olm-env 1921953]# 

--
Comment 9 Jiaming Hu 2021-03-01 16:06:17 UTC 
Will this fix be picked to OCP 4.6?
Comment 10 Ben Luddy 2021-03-01 17:25:36 UTC 
Yes, it's working its way back to 4.6 now. It's currently waiting to be accepted into 4.7 (see https://bugzilla.redhat.com/show_bug.cgi?id=1929904).
Comment 13 errata-xmlrpc 2021-07-27 22:37:34 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438
Note You need to log in before you can comment on or make changes to this bug.