Description of problem: Adding a custom annotation to the deployment template spec of a CSV should be propagated through to the associated deployment, but they are not. The only annotations propagated to the deployments are those copied from the CSV's own metadata. Version-Release number of selected component (if applicable): OCP 4.8 (and probably as early as 4.6?) How reproducible: Every time Steps to Reproduce: Add a custom annotation to both the CSV's metadata and the CSV's deployment's template's metadata, and check which of those get propagated to the associated deployment object: > #!/usr/bin/bash > set +e > oc patch -n openshift-operator-lifecycle-manager csv packageserver --type=json -p '[ > { "op": "add", "path": "/metadata/annotations", "value": {"custom.csv": "custom csv value"} }, > { "op": "add", "path": "/spec/install/spec/deployments/0/spec/template/metadata/annotations", "value": {"custom.template": "custom template value"} } > ]' > oc get -n openshift-operator-lifecycle-manager csv packageserver -o yaml > echo "---" > oc get -n openshift-operator-lifecycle-manager deployment packageserver -o yaml Actual results: The actual deployment object's template has the "custom.csv" annotation copied from the CSV's metadata, but not the "custom.template" which should have been copied from the CSV's deployment's template: > apiVersion: apps/v1 > kind: Deployment > metadata: > name: packageserver > namespace: openshift-operator-lifecycle-manager > ... > spec: > template: > metadata: > annotations: > custom.csv: custom csv value > olm.operatorGroup: olm-operators > olm.operatorNamespace: openshift-operator-lifecycle-manager > olm.targetNamespaces: openshift-operator-lifecycle-manager > olmcahash: c6bad68fb2a477cbf096b18afc837237f27488f814b4f81c79a59bf79e9b7206 > labels: > app: packageserver > ... Expected results: The actual deployment object's template should have the "custom.template" in its annotations: > apiVersion: apps/v1 > kind: Deployment > metadata: > name: packageserver > namespace: openshift-operator-lifecycle-manager > ... > spec: > template: > metadata: > annotations: > custom.csv: custom csv value > custom.template: custom template value > olm.operatorGroup: olm-operators > olm.operatorNamespace: openshift-operator-lifecycle-manager > olm.targetNamespaces: openshift-operator-lifecycle-manager > olmcahash: c6bad68fb2a477cbf096b18afc837237f27488f814b4f81c79a59bf79e9b7206 > labels: > app: packageserver > ... Additional info: After a browse through the code, I'm pretty sure that https://github.com/operator-framework/operator-lifecycle-manager/blob/master/pkg/controller/install/deployment.go does the right thing in explicitly merging the DeploymentSpec.Template annotations, but the updateDeploymentSpecsWithApiServiceData function in https://github.com/operator-framework/operator-lifecycle-manager/blob/master/pkg/controller/operators/olm/apiservices.go is suspect in that it calls 'SetAnnotations()' on the DeploymentSpec.Template without doing any merge of pre-existing annotations.
*** Bug 1961747 has been marked as a duplicate of this bug. ***
@Ken I can certainly appreciate that this issue causes extra work and toil in order to supply those annotations, but Blocker+ should be exclusively used for issues that prevent clusters from running successfully and is used as a way for OpenShift to track whether a release needs to be delayed. It sounds like there is a valid workaround in this case, so I'm setting the severity and priority to High/High and Blocker-. That being said, we (The OLM dev team) still track any High/High bug as an interrupt and will still actively try to get a fix in for this bug for 4.8.0, we just aren't going to use the OpenShift process to delay an OCP release for this BZ.
Tested with https://openshift-release.apps.ci.l2s4.p1.openshiftapps.com/releasestream/4.8.0-0.nightly/release/4.8.0-0.nightly-2021-06-11-175723, oc exec catalog-operator-769d8b9bbf-h5mhb -n openshift-operator-lifecycle-manager -- olm --version OLM version: 0.17.0 git commit: f25f670c03e849ba0fd53a56daa0d8a697f68d16 1) Applied annotations as oriented oc patch -n openshift-operator-lifecycle-manager csv packageserver --type=json -p '[ { "op": "add", "path": "/metadata/annotations", "value": {"custom.csv": "custom csv value"} }, { "op": "add", "path": "/spec/install/spec/deployments/0/spec/template/metadata/annotations", "value": {"custom.template": "custom template value"} } 2) Check if the csv is properly changed oc get -n openshift-operator-lifecycle-manager csv packageserver -o yaml apiVersion: operators.coreos.com/v1alpha1 kind: ClusterServiceVersion metadata: annotations: custom.csv: custom csv value olm.operatorGroup: olm-operators olm.operatorNamespace: openshift-operator-lifecycle-manager olm.targetNamespaces: openshift-operator-lifecycle-manager creationTimestamp: "2021-06-14T01:33:46Z" generation: 2 labels: olm.api.4bca9f23e412d79d: provided olm.clusteroperator.name: operator-lifecycle-manager-packageserver olm.version: 0.17.0 name: packageserver namespace: openshift-operator-lifecycle-manager resourceVersion: "96308" uid: 32a1d0c8-b4e8-4936-9d22-40b79e2dbbe0 spec: apiservicedefinitions: owned: - containerPort: 5443 deploymentName: packageserver description: A PackageManifest is a resource generated from existing CatalogSources and their ConfigMaps displayName: PackageManifest group: packages.operators.coreos.com kind: PackageManifest name: packagemanifests version: v1 cleanup: enabled: false customresourcedefinitions: {} description: Represents an Operator package that is available from a given CatalogSource which will resolve to a ClusterServiceVersion. displayName: Package Server install: spec: clusterPermissions: - rules: - apiGroups: - authorization.k8s.io resources: - subjectaccessreviews verbs: - create - get - apiGroups: - "" resources: - configmaps verbs: - get - list - watch - apiGroups: - operators.coreos.com resources: - catalogsources verbs: - get - list - watch - apiGroups: - packages.operators.coreos.com resources: - packagemanifests verbs: - get - list serviceAccountName: olm-operator-serviceaccount deployments: - name: packageserver spec: replicas: 2 selector: matchLabels: app: packageserver strategy: rollingUpdate: maxSurge: 1 maxUnavailable: 1 type: RollingUpdate template: metadata: annotations: custom.template: custom template value creationTimestamp: null labels: app: packageserver spec: containers: - command: - /bin/package-server - -v=4 - --secure-port - "5443" - --global-namespace - openshift-marketplace image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff7049f95c8e0764d7c8908d5f447007185a6eb183946fe737edac8a2d0ce7ca imagePullPolicy: IfNotPresent livenessProbe: httpGet: path: /healthz port: 5443 scheme: HTTPS name: packageserver ports: - containerPort: 5443 protocol: TCP readinessProbe: httpGet: path: /healthz port: 5443 scheme: HTTPS resources: requests: cpu: 10m memory: 50Mi terminationMessagePolicy: FallbackToLogsOnError volumeMounts: - mountPath: /tmp name: tmpfs nodeSelector: kubernetes.io/os: linux node-role.kubernetes.io/master: "" priorityClassName: system-cluster-critical serviceAccountName: olm-operator-serviceaccount tolerations: - effect: NoSchedule key: node-role.kubernetes.io/master operator: Exists - effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists tolerationSeconds: 120 - effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists tolerationSeconds: 120 volumes: - emptyDir: {} name: tmpfs strategy: deployment installModes: - supported: true type: OwnNamespace - supported: true type: SingleNamespace - supported: true type: MultiNamespace - supported: true type: AllNamespaces keywords: - packagemanifests - olm - packages links: - name: Package Server url: https://github.com/operator-framework/operator-lifecycle-manager/tree/master/pkg/package-server maintainers: - email: openshift-operators name: Red Hat maturity: alpha minKubeVersion: 1.11.0 provider: name: Red Hat version: 0.17.0 status: certsLastUpdated: "2021-06-14T04:42:28Z" certsRotateAt: "2023-06-13T04:42:28Z" cleanup: {} conditions: - lastTransitionTime: "2021-06-14T01:41:25Z" lastUpdateTime: "2021-06-14T01:41:25Z" message: requirements not yet checked phase: Pending reason: RequirementsUnknown - lastTransitionTime: "2021-06-14T01:41:25Z" lastUpdateTime: "2021-06-14T01:41:25Z" message: all requirements found, attempting install phase: InstallReady reason: AllRequirementsMet - lastTransitionTime: "2021-06-14T01:41:25Z" lastUpdateTime: "2021-06-14T01:41:25Z" message: waiting for install components to report healthy phase: Installing reason: InstallSucceeded - lastTransitionTime: "2021-06-14T01:41:25Z" lastUpdateTime: "2021-06-14T01:41:26Z" message: APIServices not installed phase: Installing reason: InstallWaiting - lastTransitionTime: "2021-06-14T01:45:16Z" lastUpdateTime: "2021-06-14T01:45:16Z" message: install strategy completed with no errors phase: Succeeded reason: InstallSucceeded - lastTransitionTime: "2021-06-14T04:42:28Z" lastUpdateTime: "2021-06-14T04:42:28Z" message: 'installing: annotations on deployment does not contain expected key: custom.csv' phase: InstallReady reason: ComponentUnhealthy - lastTransitionTime: "2021-06-14T04:42:28Z" lastUpdateTime: "2021-06-14T04:42:28Z" message: waiting for install components to report healthy phase: Installing reason: InstallSucceeded - lastTransitionTime: "2021-06-14T04:42:28Z" lastUpdateTime: "2021-06-14T04:42:28Z" message: 'installing: waiting for deployment packageserver to become ready: waiting for spec update of deployment "packageserver" to be observed...' phase: Installing reason: InstallWaiting - lastTransitionTime: "2021-06-14T04:42:31Z" lastUpdateTime: "2021-06-14T04:42:31Z" message: install strategy completed with no errors phase: Succeeded reason: InstallSucceeded lastTransitionTime: "2021-06-14T04:42:31Z" lastUpdateTime: "2021-06-14T04:42:31Z" message: install strategy completed with no errors phase: Succeeded reason: InstallSucceeded requirementStatus: - group: operators.coreos.com kind: ClusterServiceVersion message: CSV minKubeVersion (1.11.0) less than server version (v1.21.0-rc.0+186ed72) name: packageserver status: Present version: v1alpha1 - group: apiregistration.k8s.io kind: APIService message: "" name: v1.packages.operators.coreos.com status: DeploymentFound version: v1 - dependents: - group: rbac.authorization.k8s.io kind: PolicyRule message: cluster rule:{"verbs":["create","get"],"apiGroups":["authorization.k8s.io"],"resources":["subjectaccessreviews"]} status: Satisfied version: v1 - group: rbac.authorization.k8s.io kind: PolicyRule message: cluster rule:{"verbs":["get","list","watch"],"apiGroups":[""],"resources":["configmaps"]} status: Satisfied version: v1 - group: rbac.authorization.k8s.io kind: PolicyRule message: cluster rule:{"verbs":["get","list","watch"],"apiGroups":["operators.coreos.com"],"resources":["catalogsources"]} status: Satisfied version: v1 - group: rbac.authorization.k8s.io kind: PolicyRule message: cluster rule:{"verbs":["get","list"],"apiGroups":["packages.operators.coreos.com"],"resources":["packagemanifests"]} status: Satisfied version: v1 group: "" kind: ServiceAccount message: "" name: olm-operator-serviceaccount status: Present version: v1 3) The actual deployment object's template should have the "custom.template" in its annotations: oc get -n openshift-operator-lifecycle-manager deployment packageserver -o yaml apiVersion: apps/v1 kind: Deployment metadata: annotations: deployment.kubernetes.io/revision: "3" creationTimestamp: "2021-06-14T01:41:26Z" generation: 4 labels: olm.deployment-spec-hash: 5f99dcbf55 olm.owner: packageserver olm.owner.kind: ClusterServiceVersion olm.owner.namespace: openshift-operator-lifecycle-manager name: packageserver namespace: openshift-operator-lifecycle-manager ownerReferences: - apiVersion: operators.coreos.com/v1alpha1 blockOwnerDeletion: false controller: false kind: ClusterServiceVersion name: packageserver uid: 32a1d0c8-b4e8-4936-9d22-40b79e2dbbe0 resourceVersion: "97307" uid: 07a95b4b-10a2-4a9f-b83f-9b60c4651b86 spec: progressDeadlineSeconds: 600 replicas: 2 revisionHistoryLimit: 1 selector: matchLabels: app: packageserver strategy: rollingUpdate: maxSurge: 1 maxUnavailable: 1 type: RollingUpdate template: metadata: annotations: include.release.openshift.io/ibm-cloud-managed: "true" include.release.openshift.io/self-managed-high-availability: "true" include.release.openshift.io/single-node-developer: "true" olm.operatorGroup: olm-operators olm.operatorNamespace: openshift-operator-lifecycle-manager olm.targetNamespaces: openshift-operator-lifecycle-manager olmcahash: a11114aa1a87ab6b4b05a178019b46b200915454e689912f17b55f794ef8c0f9 target.workload.openshift.io/management: '{"effect": "PreferredDuringScheduling"}' creationTimestamp: null labels: app: packageserver spec: containers: - command: - /bin/package-server - -v=4 - --secure-port - "5443" - --global-namespace - openshift-marketplace env: - name: OPERATOR_CONDITION_NAME value: packageserver image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff7049f95c8e0764d7c8908d5f447007185a6eb183946fe737edac8a2d0ce7ca imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 3 httpGet: path: /healthz port: 5443 scheme: HTTPS periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 name: packageserver ports: - containerPort: 5443 protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /healthz port: 5443 scheme: HTTPS periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 resources: requests: cpu: 10m memory: 50Mi terminationMessagePath: /dev/termination-log terminationMessagePolicy: FallbackToLogsOnError volumeMounts: - mountPath: /tmp name: tmpfs - mountPath: /apiserver.local.config/certificates name: apiservice-cert - mountPath: /tmp/k8s-webhook-server/serving-certs name: webhook-cert dnsPolicy: ClusterFirst nodeSelector: kubernetes.io/os: linux node-role.kubernetes.io/master: "" priorityClassName: system-cluster-critical restartPolicy: Always schedulerName: default-scheduler securityContext: {} serviceAccount: olm-operator-serviceaccount serviceAccountName: olm-operator-serviceaccount terminationGracePeriodSeconds: 30 tolerations: - effect: NoSchedule key: node-role.kubernetes.io/master operator: Exists - effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists tolerationSeconds: 120 - effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists tolerationSeconds: 120 volumes: - emptyDir: {} name: tmpfs - name: apiservice-cert secret: defaultMode: 420 items: - key: tls.crt path: apiserver.crt - key: tls.key path: apiserver.key secretName: packageserver-service-cert - name: webhook-cert secret: defaultMode: 420 items: - key: tls.crt path: tls.crt - key: tls.key path: tls.key secretName: packageserver-service-cert status: availableReplicas: 2 conditions: - lastTransitionTime: "2021-06-14T01:41:30Z" lastUpdateTime: "2021-06-14T01:41:30Z" message: Deployment has minimum availability. reason: MinimumReplicasAvailable status: "True" type: Available - lastTransitionTime: "2021-06-14T01:41:26Z" lastUpdateTime: "2021-06-14T04:44:44Z" message: ReplicaSet "packageserver-5fd4c6f6b5" has successfully progressed. reason: NewReplicaSetAvailable status: "True" type: Progressing observedGeneration: 4 readyReplicas: 2 replicas: 2 updatedReplicas: 2 LGTM, marking as VERIFIED.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438