Description of problem: catalog resource update failed of spec.secrets set to "" message: 'couldn''t ensure registry server - error ensuring updated catalog source pod: : detected imageID change: error during update: failed to create manager for existing fields: failed to convert new object (/v1, Kind=Pod) to smd typed: .spec.imagePullSecrets: element 0: associative list with keys has an element that omits key field "name" (and doesn''t have default value)' Version-Release number of selected component (if applicable): zhaoxia@xia-MacBook-Pro ditto-operator % oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.8.0-0.nightly-2021-03-31-211319 True False 6h6m Cluster version is 4.8.0-0.nightly-2021-03-31-211319 How reproducible: easy Steps to Reproduce: 1. prepare operator bundle image clone community-operators repo #cd community-operators/community-operators/ditto-operator #opm alpha bundle build -c alpha -e alpha -d 0.1.0 -p ditto-operator -t quay.io/olmqe/ditto-operator:0.1.0 -o=true #docker push quay.io/olmqe/ditto-operator:0.1.0 #opm alpha bundle build -c alpha -e alpha -d 0.1.1 -p ditto-operator -t quay.io/olmqe/ditto-operator:0.1.1 -o=true #docker push quay.io/olmqe/ditto-operator:0.1. 2.generate catalog resource index image #opm index add -b quay.io/olmqe/ditto-operator:0.1.0 -t quay.io/olmqe/ditto-index:0.1 -c docker #docker push quay.io/olmqe/ditto-operator:0.1.0 3. create catsrc and sub zhaoxia@xia-MacBook-Pro ditto-operator % cat catsrc.yaml apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource metadata: name: ditto-index namespace: openshift-marketplace spec: displayName: test-ditto publisher: testqe sourceType: grpc image: quay.io/olmqe/ditto-index:0.1 secrets: - "" updateStrategy: registryPoll: interval: 2m zhaoxia@xia-MacBook-Pro ditto-operator % cat sub.yaml apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: ditto-test-operator namespace: test-1 spec: channel: "4.8" installPlanApproval: Automatic name: ditto-operator source: ditto-index sourceNamespace: openshift-marketplace #oc apply -f catsrc.yaml #oc apply -f sub.yaml zhaoxia@xia-MacBook-Pro ditto-operator % oc get csv NAME DISPLAY VERSION REPLACES PHASE ditto-operator.v0.1.0 Eclipse Ditto 0.1.0 Succeeded zhaoxia@xia-MacBook-Pro ditto-operator % oc get pod ditto-index-pmntw -n openshift-marketplace -o yaml apiVersion: v1 kind: Pod ... spec: containers: - image: quay.io/olmqe/ditto-index:0.1 imagePullPolicy: Always livenessProbe: exec: command: - grpc_health_probe - -addr=:50051 failureThreshold: 3 initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 name: registry-server ports: - containerPort: 50051 name: grpc protocol: TCP readinessProbe: exec: command: - grpc_health_probe - -addr=:50051 failureThreshold: 3 initialDelaySeconds: 5 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 5 resources: requests: cpu: 10m memory: 50Mi securityContext: capabilities: drop: - MKNOD terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: ditto-index-token-xvsnw readOnly: true dnsPolicy: ClusterFirst enableServiceLinks: true imagePullSecrets: - {} - name: ditto-index-dockercfg-5x8dq nodeName: ip-10-0-171-218.us-east-2.compute.internal .... status: conditions: - lastProbeTime: null lastTransitionTime: "2021-04-01T08:25:30Z" status: "True" type: Initialized - lastProbeTime: null lastTransitionTime: "2021-04-01T08:25:42Z" status: "True" type: Ready - lastProbeTime: null lastTransitionTime: "2021-04-01T08:25:42Z" status: "True" type: ContainersReady - lastProbeTime: null lastTransitionTime: "2021-04-01T08:25:30Z" status: "True" type: PodScheduled containerStatuses: - containerID: cri-o://c690d614688de038f49629cfe12f1db05de61a7ccab11659441e9cbda377af97 image: quay.io/olmqe/ditto-index:0.1 imageID: quay.io/olmqe/ditto-index@sha256:6192818822dc055a42f990e59df4fd03fa9b4749a1398bd8aec889fb4e5f970c lastState: {} name: registry-server ready: true restartCount: 0 started: true state: running: startedAt: "2021-04-01T08:25:34Z" hostIP: 10.0.171.218 phase: Running podIP: 10.129.2.177 podIPs: - ip: 10.129.2.177 qosClass: Burstable startTime: "2021-04-01T08:25:30Z" 4. update catalog resource index image zhaoxia@xia-MacBook-Pro ditto-operator % opm index add -b quay.io/olmqe/ditto-operator:0.1.1 -f quay.io/olmqe/ditto-index:0.1 -t quay.io/olmqe/ditto-index:0.1 -c docker zhaoxia@xia-MacBook-Pro ditto-operator % docker push quay.io/olmqe/ditto-index:0.1 5. check catalog status zhaoxia@xia-MacBook-Pro ditto-operator % oc get catsrc ditto-index -n openshift-marketplace -o yaml status: connectionState: address: ditto-index.openshift-marketplace.svc:50051 lastConnect: "2021-04-01T08:31:36Z" lastObservedState: READY latestImageRegistryPoll: "2021-04-01T08:31:36Z" message: 'couldn''t ensure registry server - error ensuring updated catalog source pod: : detected imageID change: error during update: failed to create manager for existing fields: failed to convert new object (/v1, Kind=Pod) to smd typed: .spec.imagePullSecrets: element 0: associative list with keys has an element that omits key field "name" (and doesn''t have default value)' reason: RegistryServerError registryService: createdAt: "2021-04-01T08:25:30Z" port: "50051" protocol: grpc serviceName: ditto-index serviceNamespace: openshift-marketplace zhaoxia@xia-MacBook-Pro ditto-operator % oc get pod -n openshift-marketplace NAME READY STATUS RESTARTS AGE ditto-index-pmntw 1/1 Running 0 48m ditto-index-qz4dq 1/1 Running 0 42m Actual results: catalog resource update failed Expected results: catalog resource update success Additional info: If OLM does not support spec.secrets set to be "", should raise error when create catalog resource. If support spec.secrets set to be "", catalog resource update should be success.
I was able to easily reproduce this. Setting spec.secrets=[""] results in that being propagated to the pod spec (via the serviceaccount) so the pod spec ends up looking like ``` spec: imagePullSecrets: - {} - name: ditto-index-dockercfg-djmz7 ``` Seems like having an empty string in a list breaks server side apply since there are no `metadata.managedFields` on the pod spec well. It's similar to https://github.com/kubernetes-sigs/structured-merge-diff/issues/130 except that was fixed by having a default for the ContainerPort protocol field whereas you can't really default the secret name in this case. Still trying to make sense of what's going on with serverside apply (https://github.com/kubernetes-sigs/structured-merge-diff/blob/ea1021dbc0f242313159d5dd4801ff29304712fe/typed/helpers.go#L216) Although a simple fix for this might be to sanitize spec.secrets=[""] on the CatalogSource so we don't pass that on to the created serviceaccount.
verify: zhaoxia@xzha-mac bug-1945548 % oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.8.0-0.nightly-2021-05-21-233425 True False 7h34m Cluster version is 4.8.0-0.nightly-2021-05-21-233425 zhaoxia@xzha-mac bug-1945548 % oc exec catalog-operator-79bf8bbd6b-kqmpb -- olm --version OLM version: 0.17.0 git commit: ca1f0b69c3e2eb06ab4e62517fe5bd11e59a3239 1. prepare operator bundle image clone community-operators repo #cd community-operators/community-operators/ditto-operator #opm alpha bundle build -c alpha -e alpha -d 0.1.0 -p ditto-operator -t quay.io/olmqe/ditto-operator:0.1.0 -o=true #docker push quay.io/olmqe/ditto-operator:0.1.0 #opm alpha bundle build -c alpha -e alpha -d 0.1.1 -p ditto-operator -t quay.io/olmqe/ditto-operator:0.1.1 -o=true #docker push quay.io/olmqe/ditto-operator:0.1. 2.generate catalog resource index image #opm index add -b quay.io/olmqe/ditto-operator:0.1.0 -t quay.io/olmqe/ditto-index:0.1 -c docker #docker push quay.io/olmqe/ditto-operator:0.1.0 3. create catsrc and sub zhaoxia@xia-MacBook-Pro ditto-operator % cat catsrc.yaml apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource 4. update catalog resource index image zhaoxia@xia-MacBook-Pro ditto-operator % opm index add -b quay.io/olmqe/ditto-operator:0.1.1 -f quay.io/olmqe/ditto-index:0.1 -t quay.io/olmqe/ditto-index:0.1 -c docker metadata: name: ditto-index namespace: openshift-marketplace spec: displayName: test-ditto publisher: testqe sourceType: grpc image: quay.io/olmqe/ditto-index:0.1 secrets: - "" updateStrategy: registryPoll: interval: 2m zhaoxia@xia-MacBook-Pro ditto-operator % cat sub.yaml apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: ditto-test-operator namespace: test-1 spec: channel: "alpha" installPlanApproval: Automatic name: ditto-operator source: ditto-index sourceNamespace: openshift-marketplace #oc apply -f catsrc.yaml #oc apply -f sub.yaml zhaoxia@xia-MacBook-Pro ditto-operator % oc get csv NAME DISPLAY VERSION REPLACES PHASE ditto-operator.v0.1.0 Eclipse Ditto 0.1.0 Succeeded 5. check catalog status, pod updated zhaoxia@xzha-mac bug-1945548 % oc get pod NAME READY STATUS RESTARTS AGE ditto-index-85d8f 1/1 Running 0 2m17s zhaoxia@xzha-mac bug-1945548 % oc get pod NAME READY STATUS RESTARTS AGE ditto-index-pwv42 1/1 Running 0 31s zhaoxia@xzha-mac bug-1945548 % oc get catsrc ditto-index -n openshift-marketplace -o yaml apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"operators.coreos.com/v1alpha1","kind":"CatalogSource","metadata":{"annotations":{},"name":"ditto-index","namespace":"openshift-marketplace"},"spec":{"displayName":"test-ditto","image":"quay.io/olmqe/ditto-index:0.1","publisher":"testqe","secrets":[""],"sourceType":"grpc","updateStrategy":{"registryPoll":{"interval":"2m"}}}} creationTimestamp: "2021-05-24T08:55:18Z" generation: 1 name: ditto-index namespace: openshift-marketplace resourceVersion: "215514" uid: 4114ab88-6bad-4a78-8c83-9a461e5c2e07 spec: displayName: test-ditto image: quay.io/olmqe/ditto-index:0.1 publisher: testqe secrets: - "" sourceType: grpc updateStrategy: registryPoll: interval: 2m status: connectionState: address: ditto-index.openshift-marketplace.svc:50051 lastConnect: "2021-05-24T09:12:03Z" lastObservedState: READY latestImageRegistryPoll: "2021-05-24T09:12:03Z" registryService: createdAt: "2021-05-24T08:55:19Z" port: "50051" protocol: grpc serviceName: ditto-index serviceNamespace: openshift-marketplace 6. check sub is upgrade to verison 0.1.1 zhaoxia@xzha-mac bug-1945548 % oc get csv -n test-1 NAME DISPLAY VERSION REPLACES PHASE ditto-operator.v0.1.1 Eclipse Ditto 0.1.1 ditto-operator.v0.1.0 Succeeded LGTM, verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438