Bug 1945548
| Summary: | catalog resource update failed if spec.secrets set to "" | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | xzha |
| Component: | OLM | Assignee: | Haseeb Tariq <htariq> |
| OLM sub component: | OLM | QA Contact: | xzha |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | medium | ||
| Priority: | medium | CC: | davegord, htariq, xzha |
| Version: | 4.8 | ||
| Target Milestone: | --- | ||
| Target Release: | 4.8.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
Cause: During the registry server sync the image pull secrets from the catalogsource's spec.secrets are passed
unfiltered to the serviceaccount for the registry pod.
Consequence: This breaks server-side apply for the registry pod which gets created without the metadata.managedFields when it has an empty element in the pod's spec.imagePullSecrets list. As a result that pod can't be updated by the catalog operator's server-side apply client.
Fix: Filter empty strings in spec.secrets from the CatalogSource when creating the registry pod's service account.
Result: The registry pod is created with a valid metadata.managedFields block and can be updated by the catalog-operator in the event of an index image update for the CatalogSource.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-07-27 22:57:00 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
I was able to easily reproduce this.
Setting spec.secrets=[""] results in that being propagated to the pod spec (via the serviceaccount) so the pod spec ends up looking like
```
spec:
imagePullSecrets:
- {}
- name: ditto-index-dockercfg-djmz7
```
Seems like having an empty string in a list breaks server side apply since there are no `metadata.managedFields` on the pod spec well.
It's similar to https://github.com/kubernetes-sigs/structured-merge-diff/issues/130 except that was fixed by having a default for the ContainerPort protocol field whereas you can't really default the secret name in this case.
Still trying to make sense of what's going on with serverside apply (https://github.com/kubernetes-sigs/structured-merge-diff/blob/ea1021dbc0f242313159d5dd4801ff29304712fe/typed/helpers.go#L216)
Although a simple fix for this might be to sanitize spec.secrets=[""] on the CatalogSource so we don't pass that on to the created serviceaccount.
verify:
zhaoxia@xzha-mac bug-1945548 % oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.8.0-0.nightly-2021-05-21-233425 True False 7h34m Cluster version is 4.8.0-0.nightly-2021-05-21-233425
zhaoxia@xzha-mac bug-1945548 % oc exec catalog-operator-79bf8bbd6b-kqmpb -- olm --version
OLM version: 0.17.0
git commit: ca1f0b69c3e2eb06ab4e62517fe5bd11e59a3239
1. prepare operator bundle image
clone community-operators repo
#cd community-operators/community-operators/ditto-operator
#opm alpha bundle build -c alpha -e alpha -d 0.1.0 -p ditto-operator -t quay.io/olmqe/ditto-operator:0.1.0 -o=true
#docker push quay.io/olmqe/ditto-operator:0.1.0
#opm alpha bundle build -c alpha -e alpha -d 0.1.1 -p ditto-operator -t quay.io/olmqe/ditto-operator:0.1.1 -o=true
#docker push quay.io/olmqe/ditto-operator:0.1.
2.generate catalog resource index image
#opm index add -b quay.io/olmqe/ditto-operator:0.1.0 -t quay.io/olmqe/ditto-index:0.1 -c docker
#docker push quay.io/olmqe/ditto-operator:0.1.0
3. create catsrc and sub
zhaoxia@xia-MacBook-Pro ditto-operator % cat catsrc.yaml
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
4. update catalog resource index image
zhaoxia@xia-MacBook-Pro ditto-operator % opm index add -b quay.io/olmqe/ditto-operator:0.1.1 -f quay.io/olmqe/ditto-index:0.1 -t quay.io/olmqe/ditto-index:0.1 -c docker
metadata:
name: ditto-index
namespace: openshift-marketplace
spec:
displayName: test-ditto
publisher: testqe
sourceType: grpc
image: quay.io/olmqe/ditto-index:0.1
secrets:
- ""
updateStrategy:
registryPoll:
interval: 2m
zhaoxia@xia-MacBook-Pro ditto-operator % cat sub.yaml
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: ditto-test-operator
namespace: test-1
spec:
channel: "alpha"
installPlanApproval: Automatic
name: ditto-operator
source: ditto-index
sourceNamespace: openshift-marketplace
#oc apply -f catsrc.yaml
#oc apply -f sub.yaml
zhaoxia@xia-MacBook-Pro ditto-operator % oc get csv
NAME DISPLAY VERSION REPLACES PHASE
ditto-operator.v0.1.0 Eclipse Ditto 0.1.0 Succeeded
5. check catalog status, pod updated
zhaoxia@xzha-mac bug-1945548 % oc get pod
NAME READY STATUS RESTARTS AGE
ditto-index-85d8f 1/1 Running 0 2m17s
zhaoxia@xzha-mac bug-1945548 % oc get pod
NAME READY STATUS RESTARTS AGE
ditto-index-pwv42 1/1 Running 0 31s
zhaoxia@xzha-mac bug-1945548 % oc get catsrc ditto-index -n openshift-marketplace -o yaml
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"operators.coreos.com/v1alpha1","kind":"CatalogSource","metadata":{"annotations":{},"name":"ditto-index","namespace":"openshift-marketplace"},"spec":{"displayName":"test-ditto","image":"quay.io/olmqe/ditto-index:0.1","publisher":"testqe","secrets":[""],"sourceType":"grpc","updateStrategy":{"registryPoll":{"interval":"2m"}}}}
creationTimestamp: "2021-05-24T08:55:18Z"
generation: 1
name: ditto-index
namespace: openshift-marketplace
resourceVersion: "215514"
uid: 4114ab88-6bad-4a78-8c83-9a461e5c2e07
spec:
displayName: test-ditto
image: quay.io/olmqe/ditto-index:0.1
publisher: testqe
secrets:
- ""
sourceType: grpc
updateStrategy:
registryPoll:
interval: 2m
status:
connectionState:
address: ditto-index.openshift-marketplace.svc:50051
lastConnect: "2021-05-24T09:12:03Z"
lastObservedState: READY
latestImageRegistryPoll: "2021-05-24T09:12:03Z"
registryService:
createdAt: "2021-05-24T08:55:19Z"
port: "50051"
protocol: grpc
serviceName: ditto-index
serviceNamespace: openshift-marketplace
6. check sub is upgrade to verison 0.1.1
zhaoxia@xzha-mac bug-1945548 % oc get csv -n test-1
NAME DISPLAY VERSION REPLACES PHASE
ditto-operator.v0.1.1 Eclipse Ditto 0.1.1 ditto-operator.v0.1.0 Succeeded
LGTM, verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438 |
Description of problem: catalog resource update failed of spec.secrets set to "" message: 'couldn''t ensure registry server - error ensuring updated catalog source pod: : detected imageID change: error during update: failed to create manager for existing fields: failed to convert new object (/v1, Kind=Pod) to smd typed: .spec.imagePullSecrets: element 0: associative list with keys has an element that omits key field "name" (and doesn''t have default value)' Version-Release number of selected component (if applicable): zhaoxia@xia-MacBook-Pro ditto-operator % oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.8.0-0.nightly-2021-03-31-211319 True False 6h6m Cluster version is 4.8.0-0.nightly-2021-03-31-211319 How reproducible: easy Steps to Reproduce: 1. prepare operator bundle image clone community-operators repo #cd community-operators/community-operators/ditto-operator #opm alpha bundle build -c alpha -e alpha -d 0.1.0 -p ditto-operator -t quay.io/olmqe/ditto-operator:0.1.0 -o=true #docker push quay.io/olmqe/ditto-operator:0.1.0 #opm alpha bundle build -c alpha -e alpha -d 0.1.1 -p ditto-operator -t quay.io/olmqe/ditto-operator:0.1.1 -o=true #docker push quay.io/olmqe/ditto-operator:0.1. 2.generate catalog resource index image #opm index add -b quay.io/olmqe/ditto-operator:0.1.0 -t quay.io/olmqe/ditto-index:0.1 -c docker #docker push quay.io/olmqe/ditto-operator:0.1.0 3. create catsrc and sub zhaoxia@xia-MacBook-Pro ditto-operator % cat catsrc.yaml apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource metadata: name: ditto-index namespace: openshift-marketplace spec: displayName: test-ditto publisher: testqe sourceType: grpc image: quay.io/olmqe/ditto-index:0.1 secrets: - "" updateStrategy: registryPoll: interval: 2m zhaoxia@xia-MacBook-Pro ditto-operator % cat sub.yaml apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: ditto-test-operator namespace: test-1 spec: channel: "4.8" installPlanApproval: Automatic name: ditto-operator source: ditto-index sourceNamespace: openshift-marketplace #oc apply -f catsrc.yaml #oc apply -f sub.yaml zhaoxia@xia-MacBook-Pro ditto-operator % oc get csv NAME DISPLAY VERSION REPLACES PHASE ditto-operator.v0.1.0 Eclipse Ditto 0.1.0 Succeeded zhaoxia@xia-MacBook-Pro ditto-operator % oc get pod ditto-index-pmntw -n openshift-marketplace -o yaml apiVersion: v1 kind: Pod ... spec: containers: - image: quay.io/olmqe/ditto-index:0.1 imagePullPolicy: Always livenessProbe: exec: command: - grpc_health_probe - -addr=:50051 failureThreshold: 3 initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 name: registry-server ports: - containerPort: 50051 name: grpc protocol: TCP readinessProbe: exec: command: - grpc_health_probe - -addr=:50051 failureThreshold: 3 initialDelaySeconds: 5 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 5 resources: requests: cpu: 10m memory: 50Mi securityContext: capabilities: drop: - MKNOD terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: ditto-index-token-xvsnw readOnly: true dnsPolicy: ClusterFirst enableServiceLinks: true imagePullSecrets: - {} - name: ditto-index-dockercfg-5x8dq nodeName: ip-10-0-171-218.us-east-2.compute.internal .... status: conditions: - lastProbeTime: null lastTransitionTime: "2021-04-01T08:25:30Z" status: "True" type: Initialized - lastProbeTime: null lastTransitionTime: "2021-04-01T08:25:42Z" status: "True" type: Ready - lastProbeTime: null lastTransitionTime: "2021-04-01T08:25:42Z" status: "True" type: ContainersReady - lastProbeTime: null lastTransitionTime: "2021-04-01T08:25:30Z" status: "True" type: PodScheduled containerStatuses: - containerID: cri-o://c690d614688de038f49629cfe12f1db05de61a7ccab11659441e9cbda377af97 image: quay.io/olmqe/ditto-index:0.1 imageID: quay.io/olmqe/ditto-index@sha256:6192818822dc055a42f990e59df4fd03fa9b4749a1398bd8aec889fb4e5f970c lastState: {} name: registry-server ready: true restartCount: 0 started: true state: running: startedAt: "2021-04-01T08:25:34Z" hostIP: 10.0.171.218 phase: Running podIP: 10.129.2.177 podIPs: - ip: 10.129.2.177 qosClass: Burstable startTime: "2021-04-01T08:25:30Z" 4. update catalog resource index image zhaoxia@xia-MacBook-Pro ditto-operator % opm index add -b quay.io/olmqe/ditto-operator:0.1.1 -f quay.io/olmqe/ditto-index:0.1 -t quay.io/olmqe/ditto-index:0.1 -c docker zhaoxia@xia-MacBook-Pro ditto-operator % docker push quay.io/olmqe/ditto-index:0.1 5. check catalog status zhaoxia@xia-MacBook-Pro ditto-operator % oc get catsrc ditto-index -n openshift-marketplace -o yaml status: connectionState: address: ditto-index.openshift-marketplace.svc:50051 lastConnect: "2021-04-01T08:31:36Z" lastObservedState: READY latestImageRegistryPoll: "2021-04-01T08:31:36Z" message: 'couldn''t ensure registry server - error ensuring updated catalog source pod: : detected imageID change: error during update: failed to create manager for existing fields: failed to convert new object (/v1, Kind=Pod) to smd typed: .spec.imagePullSecrets: element 0: associative list with keys has an element that omits key field "name" (and doesn''t have default value)' reason: RegistryServerError registryService: createdAt: "2021-04-01T08:25:30Z" port: "50051" protocol: grpc serviceName: ditto-index serviceNamespace: openshift-marketplace zhaoxia@xia-MacBook-Pro ditto-operator % oc get pod -n openshift-marketplace NAME READY STATUS RESTARTS AGE ditto-index-pmntw 1/1 Running 0 48m ditto-index-qz4dq 1/1 Running 0 42m Actual results: catalog resource update failed Expected results: catalog resource update success Additional info: If OLM does not support spec.secrets set to be "", should raise error when create catalog resource. If support spec.secrets set to be "", catalog resource update should be success.