Bug 1945548 - catalog resource update failed if spec.secrets set to ""
Summary: catalog resource update failed if spec.secrets set to ""
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.8
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.8.0
Assignee: Haseeb Tariq
QA Contact: xzha
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-04-01 09:20 UTC by xzha
Modified: 2021-07-27 22:57 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: During the registry server sync the image pull secrets from the catalogsource's spec.secrets are passed unfiltered to the serviceaccount for the registry pod. Consequence: This breaks server-side apply for the registry pod which gets created without the metadata.managedFields when it has an empty element in the pod's spec.imagePullSecrets list. As a result that pod can't be updated by the catalog operator's server-side apply client. Fix: Filter empty strings in spec.secrets from the CatalogSource when creating the registry pod's service account. Result: The registry pod is created with a valid metadata.managedFields block and can be updated by the catalog-operator in the event of an index image update for the CatalogSource.
Clone Of:
Environment:
Last Closed: 2021-07-27 22:57:00 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift operator-framework-olm pull 83 0 None open Bug 1945548: Filter secret names for registry pod's sa 2021-05-19 17:48:45 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 22:57:28 UTC

Description xzha 2021-04-01 09:20:46 UTC
Description of problem:
catalog resource update failed of spec.secrets set to ""
message: 'couldn''t ensure registry server - error ensuring updated catalog source
    pod: : detected imageID change: error during update: failed to create manager
    for existing fields: failed to convert new object (/v1, Kind=Pod) to smd typed:
    .spec.imagePullSecrets: element 0: associative list with keys has an element that
    omits key field "name" (and doesn''t have default value)'


Version-Release number of selected component (if applicable):
zhaoxia@xia-MacBook-Pro ditto-operator % oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-03-31-211319   True        False         6h6m    Cluster version is 4.8.0-0.nightly-2021-03-31-211319

How reproducible:
easy

Steps to Reproduce:
1. prepare operator bundle image
clone community-operators repo
#cd community-operators/community-operators/ditto-operator
#opm alpha bundle build  -c alpha -e alpha -d 0.1.0 -p ditto-operator -t quay.io/olmqe/ditto-operator:0.1.0  -o=true
#docker push quay.io/olmqe/ditto-operator:0.1.0
#opm alpha bundle build  -c alpha -e alpha -d 0.1.1 -p ditto-operator -t quay.io/olmqe/ditto-operator:0.1.1  -o=true
#docker push quay.io/olmqe/ditto-operator:0.1.

2.generate catalog resource index image
#opm index add -b quay.io/olmqe/ditto-operator:0.1.0 -t quay.io/olmqe/ditto-index:0.1 -c docker
#docker push quay.io/olmqe/ditto-operator:0.1.0

3. create catsrc and sub
zhaoxia@xia-MacBook-Pro ditto-operator % cat catsrc.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: ditto-index
  namespace: openshift-marketplace
spec:
  displayName: test-ditto
  publisher: testqe
  sourceType: grpc
  image: quay.io/olmqe/ditto-index:0.1
  secrets:
    - ""  
  updateStrategy:
    registryPoll:
      interval: 2m

zhaoxia@xia-MacBook-Pro ditto-operator % cat sub.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: ditto-test-operator
  namespace: test-1
spec:
  channel: "4.8"
  installPlanApproval: Automatic
  name: ditto-operator
  source: ditto-index
  sourceNamespace: openshift-marketplace

#oc apply -f catsrc.yaml
#oc apply -f sub.yaml
zhaoxia@xia-MacBook-Pro ditto-operator % oc get csv
NAME                    DISPLAY         VERSION   REPLACES   PHASE
ditto-operator.v0.1.0   Eclipse Ditto   0.1.0                Succeeded

zhaoxia@xia-MacBook-Pro ditto-operator % oc get pod ditto-index-pmntw -n openshift-marketplace -o yaml
apiVersion: v1
kind: Pod
...
spec:
  containers:
  - image: quay.io/olmqe/ditto-index:0.1
    imagePullPolicy: Always
    livenessProbe:
      exec:
        command:
        - grpc_health_probe
        - -addr=:50051
      failureThreshold: 3
      initialDelaySeconds: 10
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    name: registry-server
    ports:
    - containerPort: 50051
      name: grpc
      protocol: TCP
    readinessProbe:
      exec:
        command:
        - grpc_health_probe
        - -addr=:50051
      failureThreshold: 3
      initialDelaySeconds: 5
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 5
    resources:
      requests:
        cpu: 10m
        memory: 50Mi
    securityContext:
      capabilities:
        drop:
        - MKNOD
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: ditto-index-token-xvsnw
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  imagePullSecrets:
  - {}
  - name: ditto-index-dockercfg-5x8dq
  nodeName: ip-10-0-171-218.us-east-2.compute.internal

....
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2021-04-01T08:25:30Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2021-04-01T08:25:42Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2021-04-01T08:25:42Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2021-04-01T08:25:30Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: cri-o://c690d614688de038f49629cfe12f1db05de61a7ccab11659441e9cbda377af97
    image: quay.io/olmqe/ditto-index:0.1
    imageID: quay.io/olmqe/ditto-index@sha256:6192818822dc055a42f990e59df4fd03fa9b4749a1398bd8aec889fb4e5f970c
    lastState: {}
    name: registry-server
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2021-04-01T08:25:34Z"
  hostIP: 10.0.171.218
  phase: Running
  podIP: 10.129.2.177
  podIPs:
  - ip: 10.129.2.177
  qosClass: Burstable
  startTime: "2021-04-01T08:25:30Z"


4. update catalog resource index image
zhaoxia@xia-MacBook-Pro ditto-operator % opm index add -b quay.io/olmqe/ditto-operator:0.1.1 -f quay.io/olmqe/ditto-index:0.1 -t quay.io/olmqe/ditto-index:0.1 -c docker

zhaoxia@xia-MacBook-Pro ditto-operator % docker push quay.io/olmqe/ditto-index:0.1                                                  

5. check catalog status
zhaoxia@xia-MacBook-Pro ditto-operator % oc get catsrc ditto-index -n openshift-marketplace -o yaml
status:
  connectionState:
    address: ditto-index.openshift-marketplace.svc:50051
    lastConnect: "2021-04-01T08:31:36Z"
    lastObservedState: READY
  latestImageRegistryPoll: "2021-04-01T08:31:36Z"
  message: 'couldn''t ensure registry server - error ensuring updated catalog source
    pod: : detected imageID change: error during update: failed to create manager
    for existing fields: failed to convert new object (/v1, Kind=Pod) to smd typed:
    .spec.imagePullSecrets: element 0: associative list with keys has an element that
    omits key field "name" (and doesn''t have default value)'
  reason: RegistryServerError
  registryService:
    createdAt: "2021-04-01T08:25:30Z"
    port: "50051"
    protocol: grpc
    serviceName: ditto-index
    serviceNamespace: openshift-marketplace

zhaoxia@xia-MacBook-Pro ditto-operator % oc get pod -n openshift-marketplace                       
NAME                                                              READY   STATUS             RESTARTS   AGE
ditto-index-pmntw                                                 1/1     Running            0          48m
ditto-index-qz4dq                                                 1/1     Running            0          42m

Actual results:

catalog resource update failed


Expected results:
catalog resource update success


Additional info:
If OLM does not support spec.secrets set to be "", should raise error when create catalog resource.
If support spec.secrets set to be "", catalog resource update should be success.

Comment 1 Haseeb Tariq 2021-05-18 15:33:16 UTC
I was able to easily reproduce this.

Setting spec.secrets=[""] results in that being propagated to the pod spec (via the serviceaccount) so the pod spec ends up looking like

```
spec:
  imagePullSecrets:
    - {}
    - name: ditto-index-dockercfg-djmz7
```

Seems like having an empty string in a list breaks server side apply since there are no `metadata.managedFields` on the pod spec well.
It's similar to https://github.com/kubernetes-sigs/structured-merge-diff/issues/130 except that was fixed by having a default for the ContainerPort protocol field whereas you can't really default the secret name in this case.

Still trying to make sense of what's going on with serverside apply (https://github.com/kubernetes-sigs/structured-merge-diff/blob/ea1021dbc0f242313159d5dd4801ff29304712fe/typed/helpers.go#L216)

Although a simple fix for this might be to sanitize spec.secrets=[""] on the CatalogSource so we don't pass that on to the created serviceaccount.

Comment 3 xzha 2021-05-24 09:13:23 UTC
verify:
zhaoxia@xzha-mac bug-1945548 % oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-05-21-233425   True        False         7h34m   Cluster version is 4.8.0-0.nightly-2021-05-21-233425

zhaoxia@xzha-mac bug-1945548 % oc exec catalog-operator-79bf8bbd6b-kqmpb -- olm --version
OLM version: 0.17.0
git commit: ca1f0b69c3e2eb06ab4e62517fe5bd11e59a3239

1. prepare operator bundle image
clone community-operators repo
#cd community-operators/community-operators/ditto-operator
#opm alpha bundle build  -c alpha -e alpha -d 0.1.0 -p ditto-operator -t quay.io/olmqe/ditto-operator:0.1.0  -o=true
#docker push quay.io/olmqe/ditto-operator:0.1.0
#opm alpha bundle build  -c alpha -e alpha -d 0.1.1 -p ditto-operator -t quay.io/olmqe/ditto-operator:0.1.1  -o=true
#docker push quay.io/olmqe/ditto-operator:0.1.

2.generate catalog resource index image
#opm index add -b quay.io/olmqe/ditto-operator:0.1.0 -t quay.io/olmqe/ditto-index:0.1 -c docker
#docker push quay.io/olmqe/ditto-operator:0.1.0

3. create catsrc and sub
zhaoxia@xia-MacBook-Pro ditto-operator % cat catsrc.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource

4. update catalog resource index image
zhaoxia@xia-MacBook-Pro ditto-operator % opm index add -b quay.io/olmqe/ditto-operator:0.1.1 -f quay.io/olmqe/ditto-index:0.1 -t quay.io/olmqe/ditto-index:0.1 -c docker


metadata:
  name: ditto-index
  namespace: openshift-marketplace
spec:
  displayName: test-ditto
  publisher: testqe
  sourceType: grpc
  image: quay.io/olmqe/ditto-index:0.1
  secrets:
    - ""  
  updateStrategy:
    registryPoll:
      interval: 2m

zhaoxia@xia-MacBook-Pro ditto-operator % cat sub.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: ditto-test-operator
  namespace: test-1
spec:
  channel: "alpha"
  installPlanApproval: Automatic
  name: ditto-operator
  source: ditto-index
  sourceNamespace: openshift-marketplace

#oc apply -f catsrc.yaml
#oc apply -f sub.yaml
zhaoxia@xia-MacBook-Pro ditto-operator % oc get csv
NAME                    DISPLAY         VERSION   REPLACES   PHASE
ditto-operator.v0.1.0   Eclipse Ditto   0.1.0                Succeeded

5. check catalog status, pod updated
zhaoxia@xzha-mac bug-1945548 % oc get pod
NAME                                                              READY   STATUS         RESTARTS   AGE
ditto-index-85d8f                                                 1/1     Running        0          2m17s

zhaoxia@xzha-mac bug-1945548 % oc get pod
NAME                                                              READY   STATUS             RESTARTS   AGE
ditto-index-pwv42                                                 1/1     Running            0          31s

zhaoxia@xzha-mac bug-1945548 % oc get catsrc ditto-index -n openshift-marketplace -o yaml
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"operators.coreos.com/v1alpha1","kind":"CatalogSource","metadata":{"annotations":{},"name":"ditto-index","namespace":"openshift-marketplace"},"spec":{"displayName":"test-ditto","image":"quay.io/olmqe/ditto-index:0.1","publisher":"testqe","secrets":[""],"sourceType":"grpc","updateStrategy":{"registryPoll":{"interval":"2m"}}}}
  creationTimestamp: "2021-05-24T08:55:18Z"
  generation: 1
  name: ditto-index
  namespace: openshift-marketplace
  resourceVersion: "215514"
  uid: 4114ab88-6bad-4a78-8c83-9a461e5c2e07
spec:
  displayName: test-ditto
  image: quay.io/olmqe/ditto-index:0.1
  publisher: testqe
  secrets:
  - ""
  sourceType: grpc
  updateStrategy:
    registryPoll:
      interval: 2m
status:
  connectionState:
    address: ditto-index.openshift-marketplace.svc:50051
    lastConnect: "2021-05-24T09:12:03Z"
    lastObservedState: READY
  latestImageRegistryPoll: "2021-05-24T09:12:03Z"
  registryService:
    createdAt: "2021-05-24T08:55:19Z"
    port: "50051"
    protocol: grpc
    serviceName: ditto-index
    serviceNamespace: openshift-marketplace

6. check sub is upgrade to verison 0.1.1
zhaoxia@xzha-mac bug-1945548 % oc get csv -n test-1
NAME                    DISPLAY         VERSION   REPLACES                PHASE
ditto-operator.v0.1.1   Eclipse Ditto   0.1.1     ditto-operator.v0.1.0   Succeeded

LGTM, verified.

Comment 6 errata-xmlrpc 2021-07-27 22:57:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438


Note You need to log in before you can comment on or make changes to this bug.