Hide Forgot
Description of problem: Operator deployed on OCP 4.1 cluster slowly accumulates extra secrets of type kubernetes.io/dockercfg and kubernetes.io/service-account-token for the Operator's ServiceAccount. Version-Release number of selected component (if applicable): 4.1.20 How reproducible: 100% Steps to Reproduce: 1. install version A of operator 2. update source to include version B of operator Actual results: SA for operator has extra secrets: $ oc -n example-operator get secrets | grep example-operator example-operator-dockercfg-mtpbs kubernetes.io/dockercfg 1 3m40s example-operator-dockercfg-vkg5h kubernetes.io/dockercfg 1 2m34s example-operator-token-7shtj kubernetes.io/service-account-token 4 2m34s example-operator-token-hpv7k kubernetes.io/service-account-token 4 2m34s example-operator-token-mc7rt kubernetes.io/service-account-token 4 3m40s example-operator-token-mgsh9 kubernetes.io/service-account-token 4 3m40s Expected results: SA for operator has only expected secrets, extras are not created: $ oc -n example-operator get secrets | grep example-operator example-operator-dockercfg-vkg5h kubernetes.io/dockercfg 1 2m34s example-operator-token-7shtj kubernetes.io/service-account-token 4 2m34s example-operator-token-hpv7k kubernetes.io/service-account-token 4 2m34s Additional info: I have seen this for operators installed via cli and UI and from OperatorHub (backed by OperatorSource) and custom catalog (backed by CatalogSource). Reproducer: # install version 0.0.1 git clone https://github.com/jewzaam/example-operator.git cd example-operator oc create -R -f install/openshift-4.1/ # wait for it to start the operator pod oc -n example-operator get pods -l name=example-operator --no-headers # check secrets for the SA oc -n example-operator get secrets | grep example-operator # update to 0.0.2 oc -n example-operator get catalogsource example-operator-registry -o json | jq -r '.spec.image = "quay.io/nmalik/example-operator-registry:0.0.2"' | oc replace -f - # wait for it upgrade the operator pod oc -n example-operator get pods -l name=example-operator --no-headers # check secrets for the SA oc -n example-operator get secrets | grep example-operator
I should note this is a potential problem with systems that have many operator updates such as development environments. In our non-prod environments we have seen in the order of 1000's of extra secrets for individual operators. Over time this will have an impact on etcd performance. Possibly related: https://bugzilla.redhat.com/show_bug.cgi?id=1666991
Change the version to 4.4 since we have already bug 1776521 for 4.3.
Hi follow this step: Step 1: # install version 0.0.1 git clone https://github.com/jewzaam/example-operator.git Step2: #change the apiVersion on file 08_operatorgroup.yaml vi install/openshift-4.1/08_operatorgroup.yaml --> from apiVersion: operators.coreos.com/v1alpha2 to --> to: apiVersion: operators.coreos.com/v1 Step3 cd example-operator oc create -R -f install/openshift-4.1/ [scolange@scolange example-operator]$ oc create -R -f install/openshift-4.1/ namespace/example-operator created catalogsource.operators.coreos.com/example-operator-registry created operatorgroup.operators.coreos.com/example-operators created subscription.operators.coreos.com/example-operator created Step 4: # wait for it to start the operator pod oc -n example-operator get pods -l name=example-operator --no-headers Step 3: # check secrets for the SA oc -n example-operator get secrets | grep example-operator [scolange@scolange example-operator]$ oc -n example-operator get secrets | grep example-operator example-operator-dockercfg-8f57b kubernetes.io/dockercfg 1 4m4s example-operator-token-5mtln kubernetes.io/service-account-token 4 4m5s example-operator-token-dqpgc kubernetes.io/service-account-token 4 4m5s # update to 0.0.2 oc -n example-operator get catalogsource example-operator-registry -o json | jq -r '.spec.image = "quay.io/nmalik/example-operator-registry:0.0.2"' | oc replace -f - [scolange@scolange example-operator]$ oc -n example-operator get catalogsource example-operator-registry -o json | jq -r '.spec.image = "quay.io/nmalik/example-operator-registry:0.0.2"' | oc replace -f - catalogsource.operators.coreos.com/example-operator-registry replaced # wait for it upgrade the operator pod oc -n example-operator get pods -l name=example-operator --no-headers [scolange@scolange example-operator]$ oc -n example-operator get pods -l name=example-operator --no-headers example-operator-66dc694b6c-wwnk4 1/1 Running 0 71s # check secrets for the SA oc -n example-operator get secrets | grep example-operator [scolange@scolange example-operator]$ oc -n example-operator get secrets | grep example-operator example-operator-dockercfg-8f57b kubernetes.io/dockercfg 1 4m57s example-operator-token-5mtln kubernetes.io/service-account-token 4 4m58s example-operator-token-dqpgc kubernetes.io/service-account-token 4 4m58s The secret are not duplicated
Just for add the version of cluster: [scolange@scolange example-operator]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.4.0-0.nightly-2020-01-29-012724 True False 93m Cluster version is 4.4.0-0.nightly-2020-01-29-012724
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0581