Bug 1950047

Summary: CSV deployment template custom annotations are not propagated to deployments
Product: OpenShift Container Platform Reporter: Jim Ramsay <jramsay>
Component: OLMAssignee: Alexander Greene <agreene>
OLM sub component: OLM QA Contact: Bruno Andrade <bandrade>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: dhellmann, dsover, imiller, jlanford, keyoung, krizza, tflannag, vgrinber
Version: 4.8   
Target Milestone: ---   
Target Release: 4.8.0   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-07-27 23:01:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jim Ramsay 2021-04-15 16:06:41 UTC
Description of problem:

Adding a custom annotation to the deployment template spec of a CSV should be propagated through to the associated deployment, but they are not. The only annotations propagated to the deployments are those copied from the CSV's own metadata. 

Version-Release number of selected component (if applicable):

OCP 4.8 (and probably as early as 4.6?)

How reproducible:

Every time

Steps to Reproduce:

Add a custom annotation to both the CSV's metadata and the CSV's deployment's template's metadata, and check which of those get propagated to the associated deployment object:
> #!/usr/bin/bash
> set +e
> oc patch -n openshift-operator-lifecycle-manager csv packageserver --type=json -p '[
>         { "op": "add", "path": "/metadata/annotations", "value": {"custom.csv": "custom csv value"} },
>         { "op": "add", "path": "/spec/install/spec/deployments/0/spec/template/metadata/annotations", "value": {"custom.template": "custom template value"} }
> ]'
> oc get -n openshift-operator-lifecycle-manager csv packageserver -o yaml 
> echo "---"
> oc get -n openshift-operator-lifecycle-manager deployment packageserver -o yaml

Actual results:

The actual deployment object's template has the "custom.csv" annotation copied from the CSV's metadata, but not the "custom.template" which should have been copied from the CSV's deployment's template:
> apiVersion: apps/v1
> kind: Deployment
> metadata:
>   name: packageserver
>   namespace: openshift-operator-lifecycle-manager
> ...
> spec:
>   template:
>     metadata:
>       annotations:
>         custom.csv: custom csv value
>         olm.operatorGroup: olm-operators
>         olm.operatorNamespace: openshift-operator-lifecycle-manager
>         olm.targetNamespaces: openshift-operator-lifecycle-manager
>         olmcahash: c6bad68fb2a477cbf096b18afc837237f27488f814b4f81c79a59bf79e9b7206
>       labels:
>         app: packageserver
> ...

Expected results:

The actual deployment object's template should have the "custom.template" in its annotations:
> apiVersion: apps/v1
> kind: Deployment
> metadata:
>   name: packageserver
>   namespace: openshift-operator-lifecycle-manager
> ...
> spec:
>   template:
>     metadata:
>       annotations:
>         custom.csv: custom csv value
>         custom.template: custom template value
>         olm.operatorGroup: olm-operators
>         olm.operatorNamespace: openshift-operator-lifecycle-manager
>         olm.targetNamespaces: openshift-operator-lifecycle-manager
>         olmcahash: c6bad68fb2a477cbf096b18afc837237f27488f814b4f81c79a59bf79e9b7206
>       labels:
>         app: packageserver
> ...

Additional info:

After a browse through the code, I'm pretty sure that https://github.com/operator-framework/operator-lifecycle-manager/blob/master/pkg/controller/install/deployment.go does the right thing in explicitly merging the DeploymentSpec.Template annotations, but the updateDeploymentSpecsWithApiServiceData function in https://github.com/operator-framework/operator-lifecycle-manager/blob/master/pkg/controller/operators/olm/apiservices.go is suspect in that it calls 'SetAnnotations()' on the DeploymentSpec.Template without doing any merge of pre-existing annotations.

Comment 2 Joe Lanford 2021-05-19 16:42:47 UTC
*** Bug 1961747 has been marked as a duplicate of this bug. ***

Comment 6 Kevin Rizza 2021-05-24 13:00:01 UTC
@Ken

I can certainly appreciate that this issue causes extra work and toil in order to supply those annotations, but Blocker+ should be exclusively used for issues that prevent clusters from running successfully and is used as a way for OpenShift to track whether a release needs to be delayed. It sounds like there is a valid workaround in this case, so I'm setting the severity and priority to High/High and Blocker-. That being said, we (The OLM dev team) still track any High/High bug as an interrupt and will still actively try to get a fix in for this bug for 4.8.0, we just aren't going to use the OpenShift process to delay an OCP release for this BZ.

Comment 10 Bruno Andrade 2021-06-14 05:05:45 UTC
Tested with https://openshift-release.apps.ci.l2s4.p1.openshiftapps.com/releasestream/4.8.0-0.nightly/release/4.8.0-0.nightly-2021-06-11-175723,
oc exec catalog-operator-769d8b9bbf-h5mhb  -n openshift-operator-lifecycle-manager -- olm --version
OLM version: 0.17.0
git commit: f25f670c03e849ba0fd53a56daa0d8a697f68d16

1) Applied annotations as oriented
oc patch -n openshift-operator-lifecycle-manager csv packageserver --type=json -p '[
         { "op": "add", "path": "/metadata/annotations", "value": {"custom.csv": "custom csv value"} },
         { "op": "add", "path": "/spec/install/spec/deployments/0/spec/template/metadata/annotations", "value": {"custom.template": "custom template value"} }

2) Check if the csv is properly changed
oc get -n openshift-operator-lifecycle-manager csv packageserver -o yaml 
         apiVersion: operators.coreos.com/v1alpha1
         kind: ClusterServiceVersion
         metadata:
           annotations:
             custom.csv: custom csv value
             olm.operatorGroup: olm-operators
             olm.operatorNamespace: openshift-operator-lifecycle-manager
             olm.targetNamespaces: openshift-operator-lifecycle-manager
           creationTimestamp: "2021-06-14T01:33:46Z"
           generation: 2
           labels:
             olm.api.4bca9f23e412d79d: provided
             olm.clusteroperator.name: operator-lifecycle-manager-packageserver
             olm.version: 0.17.0
           name: packageserver
           namespace: openshift-operator-lifecycle-manager
           resourceVersion: "96308"
           uid: 32a1d0c8-b4e8-4936-9d22-40b79e2dbbe0
         spec:
           apiservicedefinitions:
             owned:
             - containerPort: 5443
               deploymentName: packageserver
               description: A PackageManifest is a resource generated from existing CatalogSources
                 and their ConfigMaps
               displayName: PackageManifest
               group: packages.operators.coreos.com
               kind: PackageManifest
               name: packagemanifests
               version: v1
           cleanup:
             enabled: false
           customresourcedefinitions: {}
           description: Represents an Operator package that is available from a given CatalogSource
             which will resolve to a ClusterServiceVersion.
           displayName: Package Server
           install:
             spec:
               clusterPermissions:
               - rules:
                 - apiGroups:
                   - authorization.k8s.io
                   resources:
                   - subjectaccessreviews
                   verbs:
                   - create
                   - get
                 - apiGroups:
                   - ""
                   resources:
                   - configmaps
                   verbs:
                   - get
                   - list
                   - watch
                 - apiGroups:
                   - operators.coreos.com
                   resources:
                   - catalogsources
                   verbs:
                   - get
                   - list
                   - watch
                 - apiGroups:
                   - packages.operators.coreos.com
                   resources:
                   - packagemanifests
                   verbs:
                   - get
                   - list
                 serviceAccountName: olm-operator-serviceaccount
               deployments:
               - name: packageserver
                 spec:
                   replicas: 2
                   selector:
                     matchLabels:
                       app: packageserver
                   strategy:
                     rollingUpdate:
                       maxSurge: 1
                       maxUnavailable: 1
                     type: RollingUpdate
                   template:
                     metadata:
                       annotations:
                         custom.template: custom template value
                       creationTimestamp: null
                       labels:
                         app: packageserver
                     spec:
                       containers:
                       - command:
                         - /bin/package-server
                         - -v=4
                         - --secure-port
                         - "5443"
                         - --global-namespace
                         - openshift-marketplace
                         image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff7049f95c8e0764d7c8908d5f447007185a6eb183946fe737edac8a2d0ce7ca
                         imagePullPolicy: IfNotPresent
                         livenessProbe:
                           httpGet:
                             path: /healthz
                             port: 5443
                             scheme: HTTPS
                         name: packageserver
                         ports:
                         - containerPort: 5443
                           protocol: TCP
                         readinessProbe:
                           httpGet:
                             path: /healthz
                             port: 5443
                             scheme: HTTPS
                         resources:
                           requests:
                             cpu: 10m
                             memory: 50Mi
                         terminationMessagePolicy: FallbackToLogsOnError
                         volumeMounts:
                         - mountPath: /tmp
                           name: tmpfs
                       nodeSelector:
                         kubernetes.io/os: linux
                         node-role.kubernetes.io/master: ""
                       priorityClassName: system-cluster-critical
                       serviceAccountName: olm-operator-serviceaccount
                       tolerations:
                       - effect: NoSchedule
                         key: node-role.kubernetes.io/master
                         operator: Exists
                       - effect: NoExecute
                         key: node.kubernetes.io/unreachable
                         operator: Exists
                         tolerationSeconds: 120
                       - effect: NoExecute
                         key: node.kubernetes.io/not-ready
                         operator: Exists
                         tolerationSeconds: 120
                       volumes:
                       - emptyDir: {}
                         name: tmpfs
             strategy: deployment
           installModes:
           - supported: true
             type: OwnNamespace
           - supported: true
             type: SingleNamespace
           - supported: true
             type: MultiNamespace
           - supported: true
             type: AllNamespaces
           keywords:
           - packagemanifests
           - olm
           - packages
           links:
           - name: Package Server
             url: https://github.com/operator-framework/operator-lifecycle-manager/tree/master/pkg/package-server
           maintainers:
           - email: openshift-operators
             name: Red Hat
           maturity: alpha
           minKubeVersion: 1.11.0
           provider:
             name: Red Hat
           version: 0.17.0
         status:
           certsLastUpdated: "2021-06-14T04:42:28Z"
           certsRotateAt: "2023-06-13T04:42:28Z"
           cleanup: {}
           conditions:
           - lastTransitionTime: "2021-06-14T01:41:25Z"
             lastUpdateTime: "2021-06-14T01:41:25Z"
             message: requirements not yet checked
             phase: Pending
             reason: RequirementsUnknown
           - lastTransitionTime: "2021-06-14T01:41:25Z"
             lastUpdateTime: "2021-06-14T01:41:25Z"
             message: all requirements found, attempting install
             phase: InstallReady
             reason: AllRequirementsMet
           - lastTransitionTime: "2021-06-14T01:41:25Z"
             lastUpdateTime: "2021-06-14T01:41:25Z"
             message: waiting for install components to report healthy
             phase: Installing
             reason: InstallSucceeded
           - lastTransitionTime: "2021-06-14T01:41:25Z"
             lastUpdateTime: "2021-06-14T01:41:26Z"
             message: APIServices not installed
             phase: Installing
             reason: InstallWaiting
           - lastTransitionTime: "2021-06-14T01:45:16Z"
             lastUpdateTime: "2021-06-14T01:45:16Z"
             message: install strategy completed with no errors
             phase: Succeeded
             reason: InstallSucceeded
           - lastTransitionTime: "2021-06-14T04:42:28Z"
             lastUpdateTime: "2021-06-14T04:42:28Z"
             message: 'installing: annotations on deployment does not contain expected key:
               custom.csv'
             phase: InstallReady
             reason: ComponentUnhealthy
           - lastTransitionTime: "2021-06-14T04:42:28Z"
             lastUpdateTime: "2021-06-14T04:42:28Z"
             message: waiting for install components to report healthy
             phase: Installing
             reason: InstallSucceeded
           - lastTransitionTime: "2021-06-14T04:42:28Z"
             lastUpdateTime: "2021-06-14T04:42:28Z"
             message: 'installing: waiting for deployment packageserver to become ready: waiting
               for spec update of deployment "packageserver" to be observed...'
             phase: Installing
             reason: InstallWaiting
           - lastTransitionTime: "2021-06-14T04:42:31Z"
             lastUpdateTime: "2021-06-14T04:42:31Z"
             message: install strategy completed with no errors
             phase: Succeeded
             reason: InstallSucceeded
           lastTransitionTime: "2021-06-14T04:42:31Z"
           lastUpdateTime: "2021-06-14T04:42:31Z"
           message: install strategy completed with no errors
           phase: Succeeded
           reason: InstallSucceeded
           requirementStatus:
           - group: operators.coreos.com
             kind: ClusterServiceVersion
             message: CSV minKubeVersion (1.11.0) less than server version (v1.21.0-rc.0+186ed72)
             name: packageserver
             status: Present
             version: v1alpha1
           - group: apiregistration.k8s.io
             kind: APIService
             message: ""
             name: v1.packages.operators.coreos.com
             status: DeploymentFound
             version: v1
           - dependents:
             - group: rbac.authorization.k8s.io
               kind: PolicyRule
               message: cluster rule:{"verbs":["create","get"],"apiGroups":["authorization.k8s.io"],"resources":["subjectaccessreviews"]}
               status: Satisfied
               version: v1
             - group: rbac.authorization.k8s.io
               kind: PolicyRule
               message: cluster rule:{"verbs":["get","list","watch"],"apiGroups":[""],"resources":["configmaps"]}
               status: Satisfied
               version: v1
             - group: rbac.authorization.k8s.io
               kind: PolicyRule
               message: cluster rule:{"verbs":["get","list","watch"],"apiGroups":["operators.coreos.com"],"resources":["catalogsources"]}
               status: Satisfied
               version: v1
             - group: rbac.authorization.k8s.io
               kind: PolicyRule
               message: cluster rule:{"verbs":["get","list"],"apiGroups":["packages.operators.coreos.com"],"resources":["packagemanifests"]}
               status: Satisfied
               version: v1
             group: ""
             kind: ServiceAccount
             message: ""
             name: olm-operator-serviceaccount
             status: Present
             version: v1

3) The actual deployment object's template should have the "custom.template" in its annotations:
oc get -n openshift-operator-lifecycle-manager deployment packageserver -o yaml
             apiVersion: apps/v1
             kind: Deployment
             metadata:
               annotations:
                 deployment.kubernetes.io/revision: "3"
               creationTimestamp: "2021-06-14T01:41:26Z"
               generation: 4
               labels:
                 olm.deployment-spec-hash: 5f99dcbf55
                 olm.owner: packageserver
                 olm.owner.kind: ClusterServiceVersion
                 olm.owner.namespace: openshift-operator-lifecycle-manager
               name: packageserver
               namespace: openshift-operator-lifecycle-manager
               ownerReferences:
               - apiVersion: operators.coreos.com/v1alpha1
                 blockOwnerDeletion: false
                 controller: false
                 kind: ClusterServiceVersion
                 name: packageserver
                 uid: 32a1d0c8-b4e8-4936-9d22-40b79e2dbbe0
               resourceVersion: "97307"
               uid: 07a95b4b-10a2-4a9f-b83f-9b60c4651b86
             spec:
               progressDeadlineSeconds: 600
               replicas: 2
               revisionHistoryLimit: 1
               selector:
                 matchLabels:
                   app: packageserver
               strategy:
                 rollingUpdate:
                   maxSurge: 1
                   maxUnavailable: 1
                 type: RollingUpdate
               template:
                 metadata:
                   annotations:
                     include.release.openshift.io/ibm-cloud-managed: "true"
                     include.release.openshift.io/self-managed-high-availability: "true"
                     include.release.openshift.io/single-node-developer: "true"
                     olm.operatorGroup: olm-operators
                     olm.operatorNamespace: openshift-operator-lifecycle-manager
                     olm.targetNamespaces: openshift-operator-lifecycle-manager
                     olmcahash: a11114aa1a87ab6b4b05a178019b46b200915454e689912f17b55f794ef8c0f9
                     target.workload.openshift.io/management: '{"effect": "PreferredDuringScheduling"}'
                   creationTimestamp: null
                   labels:
                     app: packageserver
                 spec:
                   containers:
                   - command:
                     - /bin/package-server
                     - -v=4
                     - --secure-port
                     - "5443"
                     - --global-namespace
                     - openshift-marketplace
                     env:
                     - name: OPERATOR_CONDITION_NAME
                       value: packageserver
                     image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff7049f95c8e0764d7c8908d5f447007185a6eb183946fe737edac8a2d0ce7ca
                     imagePullPolicy: IfNotPresent
                     livenessProbe:
                       failureThreshold: 3
                       httpGet:
                         path: /healthz
                         port: 5443
                         scheme: HTTPS
                       periodSeconds: 10
                       successThreshold: 1
                       timeoutSeconds: 1
                     name: packageserver
                     ports:
                     - containerPort: 5443
                       protocol: TCP
                     readinessProbe:
                       failureThreshold: 3
                       httpGet:
                         path: /healthz
                         port: 5443
                         scheme: HTTPS
                       periodSeconds: 10
                       successThreshold: 1
                       timeoutSeconds: 1
                     resources:
                       requests:
                         cpu: 10m
                         memory: 50Mi
                     terminationMessagePath: /dev/termination-log
                     terminationMessagePolicy: FallbackToLogsOnError
                     volumeMounts:
                     - mountPath: /tmp
                       name: tmpfs
                     - mountPath: /apiserver.local.config/certificates
                       name: apiservice-cert
                     - mountPath: /tmp/k8s-webhook-server/serving-certs
                       name: webhook-cert
                   dnsPolicy: ClusterFirst
                   nodeSelector:
                     kubernetes.io/os: linux
                     node-role.kubernetes.io/master: ""
                   priorityClassName: system-cluster-critical
                   restartPolicy: Always
                   schedulerName: default-scheduler
                   securityContext: {}
                   serviceAccount: olm-operator-serviceaccount
                   serviceAccountName: olm-operator-serviceaccount
                   terminationGracePeriodSeconds: 30
                   tolerations:
                   - effect: NoSchedule
                     key: node-role.kubernetes.io/master
                     operator: Exists
                   - effect: NoExecute
                     key: node.kubernetes.io/unreachable
                     operator: Exists
                     tolerationSeconds: 120
                   - effect: NoExecute
                     key: node.kubernetes.io/not-ready
                     operator: Exists
                     tolerationSeconds: 120
                   volumes:
                   - emptyDir: {}
                     name: tmpfs
                   - name: apiservice-cert
                     secret:
                       defaultMode: 420
                       items:
                       - key: tls.crt
                         path: apiserver.crt
                       - key: tls.key
                         path: apiserver.key
                       secretName: packageserver-service-cert
                   - name: webhook-cert
                     secret:
                       defaultMode: 420
                       items:
                       - key: tls.crt
                         path: tls.crt
                       - key: tls.key
                         path: tls.key
                       secretName: packageserver-service-cert
             status:
               availableReplicas: 2
               conditions:
               - lastTransitionTime: "2021-06-14T01:41:30Z"
                 lastUpdateTime: "2021-06-14T01:41:30Z"
                 message: Deployment has minimum availability.
                 reason: MinimumReplicasAvailable
                 status: "True"
                 type: Available
               - lastTransitionTime: "2021-06-14T01:41:26Z"
                 lastUpdateTime: "2021-06-14T04:44:44Z"
                 message: ReplicaSet "packageserver-5fd4c6f6b5" has successfully progressed.
                 reason: NewReplicaSetAvailable
                 status: "True"
                 type: Progressing
               observedGeneration: 4
               readyReplicas: 2
               replicas: 2
               updatedReplicas: 2

               
LGTM, marking as VERIFIED.

Comment 13 errata-xmlrpc 2021-07-27 23:01:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438