Bug 1907381

Summary: OLM fails to deploy an operator if its deployment template contains a description annotation that doesn't match the CSV one
Product: OpenShift Container Platform Reporter: Simone Tiraboschi <stirabos>
Component: OLMAssignee: Alexander Greene <agreene>
OLM sub component: OLM QA Contact: kuiwang
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: medium CC: agreene, nunnatsa, vdinh
Version: 4.6   
Target Milestone: ---   
Target Release: 4.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: OLM expects annotations on the CSV to be present on pods deployed as part of a CSV. Consequence: If a pod template in a CSV's StrategyDetailsDeployment includes an annotation present on the CSV with a different value OLM will fail to install the operator because the pod's annotations do not match those defined in the CSV. Fix: Overwrite Pod Template annotations with those defined by the CSV. Result: OLM no longer fails to rollout CSVs whose annotations conflicts with those defined in the Pod Template.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2021-02-24 15:43:41 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Simone Tiraboschi 2020-12-14 11:48:54 UTC 
Description of problem:
If one of the deploymentSpec contained in the CSV contains
something like:

          template:
            metadata:
              annotations:
                description: custom description

and that value doesn't match what set on
  metadata:
    annotations:
      description: parent description

on the CSV object,

OLM fails to deploy the operator with something like:

ClusterServiceVersion kubevirt-hyperconverged-operator.v1.3.0 status message: installing: unexpected annotation on deployment. Expected description <parent description> , found description: <custom description>
ClusterServiceVersion kubevirt-hyperconverged-operator.v1.3.0 status reason: InstallWaiting

And the operator will be stuck in InstallWaiting

Version-Release number of selected component (if applicable):
4.6.0

How reproducible:
100%

Steps to Reproduce:
1. define a CSV that includes a deployment with a template that contains an annotation with a custom description that doesn't match the one at CSV level
2.
3.

Actual results:
ClusterServiceVersion kubevirt-hyperconverged-operator.v1.3.0 status message: installing: unexpected annotation on deployment. Expected description <parent description> , found description: <custom description>
ClusterServiceVersion kubevirt-hyperconverged-operator.v1.3.0 status reason: InstallWaiting

Expected results:
operator developes are allowed to define deployment templates with custom description annotation

Additional info:
it's probably something in this area:
https://github.com/operator-framework/operator-lifecycle-manager/blob/master/pkg/controller/install/deployment.go#L263-L269
Comment 1 Vu Dinh 2020-12-14 19:14:07 UTC 
*** Bug 1907417 has been marked as a duplicate of this bug. ***
Comment 2 Alexander Greene 2020-12-17 20:45:25 UTC 
Hello Simone,

Annotations on CSVs are meant to overwrite mathcing annotations present on pods defined in the CSV. This is part of an existing mechanism that informs operators which namespaces they should start and watch. This feature also allows cluster admins to override existing annotations on pods.

This should not stop the operator from installing correctly though, the annotations on the pod should simply be overwritten by the values defined in the CSV, the PR associated with the BZ addresses this bug.

Thanks,

Alex
Comment 4 kuiwang 2020-12-25 08:19:56 UTC 
There is no new build to include the fix till now. So, change it back to MODIFIED
but I reproduced it on cluster without the fix.
--
[root@preserve-olm-env operator-registry]# grep -A5 template manifests/cockroachdb-1907381/2.1.11/cockroachdb.v2.1.11.clusterserviceversion.yaml 
          template:
            metadata:
              annotations:
                description: custom description
              labels:
                name: cockroachdb
[root@preserve-olm-env operator-registry]# grep description  manifests/cockroachdb-1907381/2.1.11/cockroachdb.v2.1.11.clusterserviceversion.yaml |grep -v "-"
    description: CockroachDB Operator based on the CockroachDB helm chart
                description: custom description

[root@preserve-olm-env operator-registry]# ./bin/opm alpha bundle build -d manifests/cockroachdb-1907381/2.1.11 -t quay.io/kuiwang/cockroachdb-operator:2.1.11-1907381 -c alpha -p cockroachdb -e alpha 
INFO[0000] Building annotations.yaml                    
INFO[0000] Writing annotations.yaml in /root/kuiwang/operator-registry/manifests/cockroachdb-1907381/metadata 
INFO[0000] Building Dockerfile                          
INFO[0000] Writing bundle.Dockerfile in /root/kuiwang/operator-registry 
INFO[0000] Building bundle image                        
Sending build context to Docker daemon  129.4MB
Step 1/9 : FROM scratch
 ---> 
Step 2/9 : LABEL operators.operatorframework.io.bundle.mediatype.v1=registry+v1
 ---> Using cache
 ---> 9883d48285fd
Step 3/9 : LABEL operators.operatorframework.io.bundle.manifests.v1=manifests/
 ---> Using cache
 ---> 6a774e9c2bdf
Step 4/9 : LABEL operators.operatorframework.io.bundle.metadata.v1=metadata/
 ---> Using cache
 ---> 8844d39ae36a
Step 5/9 : LABEL operators.operatorframework.io.bundle.package.v1=cockroachdb
 ---> Using cache
 ---> e752bbd7ed88
Step 6/9 : LABEL operators.operatorframework.io.bundle.channels.v1=alpha
 ---> Using cache
 ---> 515be74d3a48
Step 7/9 : LABEL operators.operatorframework.io.bundle.channel.default.v1=alpha
 ---> Using cache
 ---> 156c2a5263fe
Step 8/9 : COPY manifests/cockroachdb-1907381/2.1.11 /manifests/
 ---> f78956744274
Step 9/9 : COPY manifests/cockroachdb-1907381/metadata /metadata/
 ---> 10622c51eece
Successfully built 10622c51eece
Successfully tagged quay.io/kuiwang/cockroachdb-operator:2.1.11-1907381
[root@preserve-olm-env operator-registry]# docker push quay.io/kuiwang/cockroachdb-operator:2.1.11-1907381
The push refers to repository [quay.io/kuiwang/cockroachdb-operator]
0f7352959086: Pushed 
9b8a39f2d597: Pushed 
2.1.11-1907381: digest: sha256:3b575351142d2e981f3824615bd9ca1d524638cb724c0185043ecad22abd0465 size: 732
[root@preserve-olm-env operator-registry]# rm -fr bundle.Dockerfile manifests/cockroachdb-1907381/metadata/
[root@preserve-olm-env operator-registry]# 

[root@preserve-olm-env operator-registry]# ./bin/opm index add -b quay.io/kuiwang/cockroachdb-operator:2.1.11-1907381 -t quay.io/kuiwang/cockroachdb-index:2.1.11-1907381 -c docker
INFO[0000] building the index                            bundles="[quay.io/kuiwang/cockroachdb-operator:2.1.11-1907381]"
INFO[0000] running /usr/bin/docker pull quay.io/kuiwang/cockroachdb-operator:2.1.11-1907381  bundles="[quay.io/kuiwang/cockroachdb-operator:2.1.11-1907381]"
INFO[0000] running docker create                         bundles="[quay.io/kuiwang/cockroachdb-operator:2.1.11-1907381]"
INFO[0000] running docker cp                             bundles="[quay.io/kuiwang/cockroachdb-operator:2.1.11-1907381]"
INFO[0000] running docker rm                             bundles="[quay.io/kuiwang/cockroachdb-operator:2.1.11-1907381]"
INFO[0000] Could not find optional dependencies file     dir=bundle_tmp417622570 file=bundle_tmp417622570/metadata load=annotations
INFO[0000] found csv, loading bundle                     dir=bundle_tmp417622570 file=bundle_tmp417622570/manifests load=bundle
INFO[0000] loading bundle file                           dir=bundle_tmp417622570/manifests file=cockroachdb.v2.1.11.clusterserviceversion.yaml load=bundle
INFO[0000] loading bundle file                           dir=bundle_tmp417622570/manifests file=cockroachdbs.charts.helm.k8s.io.crd.yaml load=bundle
INFO[0000] Generating dockerfile                         bundles="[quay.io/kuiwang/cockroachdb-operator:2.1.11-1907381]"
INFO[0000] writing dockerfile: index.Dockerfile139542072  bundles="[quay.io/kuiwang/cockroachdb-operator:2.1.11-1907381]"
INFO[0000] running docker build                          bundles="[quay.io/kuiwang/cockroachdb-operator:2.1.11-1907381]"
INFO[0000] [docker build -f index.Dockerfile139542072 -t quay.io/kuiwang/cockroachdb-index:2.1.11-1907381 .]  bundles="[quay.io/kuiwang/cockroachdb-operator:2.1.11-1907381]"
[root@preserve-olm-env operator-registry]# docker push quay.io/kuiwang/cockroachdb-index:2.1.11-1907381
The push refers to repository [quay.io/kuiwang/cockroachdb-index]
eaa607771554: Pushed 
90ac2e9b7039: Layer already exists 
6d7e94c0241a: Layer already exists 
13a27d679967: Layer already exists 
6160650cb81f: Layer already exists 
f4666769fca7: Layer already exists 
2.1.11-1907381: digest: sha256:098e75b7d93bc5dd395cedde9da2a7b9a32c1ff713c4e69f00467f5be2f06f2f size: 1577
[root@preserve-olm-env operator-registry]# 
[root@preserve-olm-env operator-registry]# cd /root/kuiwang/cases/1907381
[root@preserve-olm-env 1907381]# cat og-single.yaml 
kind: OperatorGroup
apiVersion: operators.coreos.com/v1
metadata:
  name: og-single1
  namespace: default
spec:
  targetNamespaces:
  - default
[root@preserve-olm-env 1907381]# oc apply -f og-single.yaml 
operatorgroup.operators.coreos.com/og-single1 created
[root@preserve-olm-env 1907381]# cat catsrc.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: cockroachdb-catalog
  namespace: default
spec:
  displayName: cockroachdb Operator Catalog
  image: quay.io/kuiwang/cockroachdb-index:2.1.11-1907381
  publisher: QE
  sourceType: grpc
[root@preserve-olm-env 1907381]# oc apply -f catsrc.yaml 
catalogsource.operators.coreos.com/cockroachdb-catalog created
[root@preserve-olm-env 1907381]# cat sub.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: cockroachdb
  namespace: default
spec:
  channel: alpha
  installPlanApproval: Automatic
  name: cockroachdb
  source: cockroachdb-catalog
  sourceNamespace: default
[root@preserve-olm-env 1907381]# oc apply -f sub.yaml 
subscription.operators.coreos.com/cockroachdb created
[root@preserve-olm-env 1907381]# 
[root@preserve-olm-env 1907381]# oc get csv
NAME                  DISPLAY       VERSION   REPLACES   PHASE
cockroachdb.v2.1.11   CockroachDB   2.1.11               Installing
[root@preserve-olm-env 1907381]# oc get csv cockroachdb.v2.1.11 -o yaml
apiVersion: operators.coreos.com/v1alpha1
kind: ClusterServiceVersion
metadata:
  annotations:
    alm-examples: '[{ "apiVersion": "charts.helm.k8s.io/v1alpha1", "kind": "Cockroachdb", "metadata": { "name": "example" }, "spec": { "Name": "cdb", "Image": "cockroachdb/cockroach", "ImageTag": "v19.1.3", "ImagePullPolicy": "Always", "Replicas": 3, "MaxUnavailable": 1, "Component": "cockroachdb", "InternalGrpcPort": 26257, "ExternalGrpcPort": 26257, "InternalGrpcName": "grpc", "ExternalGrpcName": "grpc", "InternalHttpPort": 8080, "ExternalHttpPort": 8080, "HttpName": "http", "Resources": { "requests": { "cpu": "500m", "memory": "512Mi" } }, "InitPodResources": { }, "Storage": "10Gi", "StorageClass": null, "CacheSize": "25%", "MaxSQLMemory": "25%", "ClusterDomain": "cluster.local", "NetworkPolicy": { "Enabled": false, "AllowExternal": true }, "Service": { "type": "ClusterIP", "annotations": { } }, "PodManagementPolicy": "Parallel", "UpdateStrategy": { "type": "RollingUpdate" }, "NodeSelector": { }, "Tolerations": { }, "Secure": { "Enabled": false, "RequestCertsImage": "cockroachdb/cockroach-k8s-request-cert", "RequestCertsImageTag": "0.4", "ServiceAccount": { "Create": true } } } }]'
    capabilities: Basic Install
    categories: Database
    certified: "false"
    containerImage: quay.io/helmoperators/cockroachdb:2.1.1
    createdAt: 2019-01-24T15-33-43Z
    description: CockroachDB Operator based on the CockroachDB helm chart
    olm.operatorGroup: og-single1
...
  install:
    spec:
      deployments:
      - name: cockroachdb
        spec:
          replicas: 1
          selector:
            matchLabels:
              name: cockroachdb
          strategy: {}
          template:
            metadata:
              annotations:
                description: custom description
              creationTimestamp: null
              labels:
                name: cockroachdb
...
  - lastTransitionTime: "2020-12-25T08:10:49Z"
    lastUpdateTime: "2020-12-25T08:10:49Z"
    message: 'installing: unexpected annotation on deployment. Expected description:CockroachDB Operator based on the CockroachDB helm chart, found description:custom description'
    phase: InstallReady
    reason: InstallWaiting
  lastTransitionTime: "2020-12-25T08:10:49Z"
  lastUpdateTime: "2020-12-25T08:10:49Z"
  message: 'installing: unexpected annotation on deployment. Expected description:CockroachDB Operator based on the CockroachDB helm chart, found description:custom description'
  phase: InstallReady
  reason: InstallWaiting
...
    group: ""
    kind: ServiceAccount
    message: ""
    name: cockroachdb-operator
    status: Present
    version: v1
[root@preserve-olm-env 1907381]# 

--
Comment 6 kuiwang 2021-01-05 02:04:36 UTC 
verify it on 4.7. LGTM
--
[root@preserve-olm-env operator-registry]# grep -A5 template manifests/cockroachdb-1907381/2.1.11/cockroachdb.v2.1.11.clusterserviceversion.yaml 
          template:
            metadata:
              annotations:
                description: custom description
              labels:
                name: cockroachdb
[root@preserve-olm-env operator-registry]# grep description  manifests/cockroachdb-1907381/2.1.11/cockroachdb.v2.1.11.clusterserviceversion.yaml |grep -v "-"
    description: CockroachDB Operator based on the CockroachDB helm chart
                description: custom description

[root@preserve-olm-env operator-registry]# ./bin/opm alpha bundle build -d manifests/cockroachdb-1907381/2.1.11 -t quay.io/kuiwang/cockroachdb-operator:2.1.11-1907381 -c alpha -p cockroachdb -e alpha 
INFO[0000] Building annotations.yaml                    
INFO[0000] Writing annotations.yaml in /root/kuiwang/operator-registry/manifests/cockroachdb-1907381/metadata 
INFO[0000] Building Dockerfile                          
INFO[0000] Writing bundle.Dockerfile in /root/kuiwang/operator-registry 
INFO[0000] Building bundle image                        
Sending build context to Docker daemon  129.4MB
Step 1/9 : FROM scratch
 ---> 
Step 2/9 : LABEL operators.operatorframework.io.bundle.mediatype.v1=registry+v1
 ---> Using cache
 ---> 9883d48285fd
Step 3/9 : LABEL operators.operatorframework.io.bundle.manifests.v1=manifests/
 ---> Using cache
 ---> 6a774e9c2bdf
Step 4/9 : LABEL operators.operatorframework.io.bundle.metadata.v1=metadata/
 ---> Using cache
 ---> 8844d39ae36a
Step 5/9 : LABEL operators.operatorframework.io.bundle.package.v1=cockroachdb
 ---> Using cache
 ---> e752bbd7ed88
Step 6/9 : LABEL operators.operatorframework.io.bundle.channels.v1=alpha
 ---> Using cache
 ---> 515be74d3a48
Step 7/9 : LABEL operators.operatorframework.io.bundle.channel.default.v1=alpha
 ---> Using cache
 ---> 156c2a5263fe
Step 8/9 : COPY manifests/cockroachdb-1907381/2.1.11 /manifests/
 ---> f78956744274
Step 9/9 : COPY manifests/cockroachdb-1907381/metadata /metadata/
 ---> 10622c51eece
Successfully built 10622c51eece
Successfully tagged quay.io/kuiwang/cockroachdb-operator:2.1.11-1907381
[root@preserve-olm-env operator-registry]# docker push quay.io/kuiwang/cockroachdb-operator:2.1.11-1907381
The push refers to repository [quay.io/kuiwang/cockroachdb-operator]
0f7352959086: Pushed 
9b8a39f2d597: Pushed 
2.1.11-1907381: digest: sha256:3b575351142d2e981f3824615bd9ca1d524638cb724c0185043ecad22abd0465 size: 732
[root@preserve-olm-env operator-registry]# rm -fr bundle.Dockerfile manifests/cockroachdb-1907381/metadata/
[root@preserve-olm-env operator-registry]# 

[root@preserve-olm-env operator-registry]# ./bin/opm index add -b quay.io/kuiwang/cockroachdb-operator:2.1.11-1907381 -t quay.io/kuiwang/cockroachdb-index:2.1.11-1907381 -c docker
INFO[0000] building the index                            bundles="[quay.io/kuiwang/cockroachdb-operator:2.1.11-1907381]"
INFO[0000] running /usr/bin/docker pull quay.io/kuiwang/cockroachdb-operator:2.1.11-1907381  bundles="[quay.io/kuiwang/cockroachdb-operator:2.1.11-1907381]"
INFO[0000] running docker create                         bundles="[quay.io/kuiwang/cockroachdb-operator:2.1.11-1907381]"
INFO[0000] running docker cp                             bundles="[quay.io/kuiwang/cockroachdb-operator:2.1.11-1907381]"
INFO[0000] running docker rm                             bundles="[quay.io/kuiwang/cockroachdb-operator:2.1.11-1907381]"
INFO[0000] Could not find optional dependencies file     dir=bundle_tmp417622570 file=bundle_tmp417622570/metadata load=annotations
INFO[0000] found csv, loading bundle                     dir=bundle_tmp417622570 file=bundle_tmp417622570/manifests load=bundle
INFO[0000] loading bundle file                           dir=bundle_tmp417622570/manifests file=cockroachdb.v2.1.11.clusterserviceversion.yaml load=bundle
INFO[0000] loading bundle file                           dir=bundle_tmp417622570/manifests file=cockroachdbs.charts.helm.k8s.io.crd.yaml load=bundle
INFO[0000] Generating dockerfile                         bundles="[quay.io/kuiwang/cockroachdb-operator:2.1.11-1907381]"
INFO[0000] writing dockerfile: index.Dockerfile139542072  bundles="[quay.io/kuiwang/cockroachdb-operator:2.1.11-1907381]"
INFO[0000] running docker build                          bundles="[quay.io/kuiwang/cockroachdb-operator:2.1.11-1907381]"
INFO[0000] [docker build -f index.Dockerfile139542072 -t quay.io/kuiwang/cockroachdb-index:2.1.11-1907381 .]  bundles="[quay.io/kuiwang/cockroachdb-operator:2.1.11-1907381]"
[root@preserve-olm-env operator-registry]# docker push quay.io/kuiwang/cockroachdb-index:2.1.11-1907381
The push refers to repository [quay.io/kuiwang/cockroachdb-index]
eaa607771554: Pushed 
90ac2e9b7039: Layer already exists 
6d7e94c0241a: Layer already exists 
13a27d679967: Layer already exists 
6160650cb81f: Layer already exists 
f4666769fca7: Layer already exists 
2.1.11-1907381: digest: sha256:098e75b7d93bc5dd395cedde9da2a7b9a32c1ff713c4e69f00467f5be2f06f2f size: 1577
[root@preserve-olm-env operator-registry]# 
[root@preserve-olm-env operator-registry]# cd /root/kuiwang/cases/1907381
[root@preserve-olm-env 1907381]# oc get pod -n openshift-operator-lifecycle-manager
NAME                                READY   STATUS    RESTARTS   AGE
catalog-operator-5b4b6587c4-9vdb4   1/1     Running   0          23m
olm-operator-7655fbdf8b-twnhn       1/1     Running   0          21m
packageserver-c684b6d54-lfwd5       1/1     Running   0          23m
packageserver-c684b6d54-vkkw4       1/1     Running   0          21m
[root@preserve-olm-env 1907381]# oc exec catalog-operator-5b4b6587c4-9vdb4 -n openshift-operator-lifecycle-manager -- olm --version
OLM version: 0.17.0
git commit: 69dc076d610b033d3964ed40c4ac27ece3424851
[root@preserve-olm-env 1907381]# 

[root@preserve-olm-env 1907381]# cat og-single.yaml
kind: OperatorGroup
apiVersion: operators.coreos.com/v1
metadata:
  name: og-single1
  namespace: default
spec:
  targetNamespaces:
  - default
[root@preserve-olm-env 1907381]# oc apply -f og-single.yaml 
operatorgroup.operators.coreos.com/og-single1 created
[root@preserve-olm-env 1907381]# cat catsrc.yaml
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: cockroachdb-catalog
  namespace: default
spec:
  displayName: cockroachdb Operator Catalog
  image: quay.io/kuiwang/cockroachdb-index:2.1.11-1907381
  publisher: QE
  sourceType: grpc
[root@preserve-olm-env 1907381]# oc apply -f catsrc.yaml 
catalogsource.operators.coreos.com/cockroachdb-catalog created
[root@preserve-olm-env 1907381]# cat sub.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: cockroachdb
  namespace: default
spec:
  channel: alpha
  installPlanApproval: Automatic
  name: cockroachdb
  source: cockroachdb-catalog
  sourceNamespace: default
[root@preserve-olm-env 1907381]# oc apply -f sub.yaml 
subscription.operators.coreos.com/cockroachdb created
[root@preserve-olm-env 1907381]# oc get csv
NAME                  DISPLAY       VERSION   REPLACES   PHASE
cockroachdb.v2.1.11   CockroachDB   2.1.11               Installing
[root@preserve-olm-env 1907381]# oc get csv
NAME                  DISPLAY       VERSION   REPLACES   PHASE
cockroachdb.v2.1.11   CockroachDB   2.1.11               Succeeded
[root@preserve-olm-env 1907381]# 

--
Comment 7 Kevin Rizza 2021-01-11 19:25:21 UTC 
*** Bug 1906437 has been marked as a duplicate of this bug. ***
Comment 10 errata-xmlrpc 2021-02-24 15:43:41 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633