Bug 1769030 - Replacing (updating) operator creates duplicate secrets for the operator's ServiceAccount
Summary: Replacing (updating) operator creates duplicate secrets for the operator's Se...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.4
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.4.0
Assignee: Bowen Song
QA Contact: Salvatore Colangelo
URL:
Whiteboard:
Depends On:
Blocks: 1776521
TreeView+ depends on / blocked
 
Reported: 2019-11-05 18:50 UTC by Naveen Malik
Modified: 2020-05-13 21:52 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1769561 1769562 1776521 (view as bug list)
Environment:
Last Closed: 2020-05-13 21:52:18 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github operator-framework operator-lifecycle-manager pull 1123 'None' closed Bug 1769030: Replacing operator creates duplicate secrets 2020-05-11 07:22:35 UTC
Red Hat Product Errata RHBA-2020:0581 None None None 2020-05-13 21:52:20 UTC

Description Naveen Malik 2019-11-05 18:50:36 UTC
Description of problem:
Operator deployed on OCP 4.1 cluster slowly accumulates extra secrets of type kubernetes.io/dockercfg and kubernetes.io/service-account-token for the Operator's ServiceAccount.


Version-Release number of selected component (if applicable):
4.1.20

How reproducible:
100%

Steps to Reproduce:
1. install version A of operator
2. update source to include version B of operator


Actual results:
SA for operator has extra secrets:

$ oc -n example-operator get secrets | grep example-operator
example-operator-dockercfg-mtpbs   kubernetes.io/dockercfg               1      3m40s
example-operator-dockercfg-vkg5h   kubernetes.io/dockercfg               1      2m34s
example-operator-token-7shtj       kubernetes.io/service-account-token   4      2m34s
example-operator-token-hpv7k       kubernetes.io/service-account-token   4      2m34s
example-operator-token-mc7rt       kubernetes.io/service-account-token   4      3m40s
example-operator-token-mgsh9       kubernetes.io/service-account-token   4      3m40s


Expected results:
SA for operator has only expected secrets, extras are not created:

$ oc -n example-operator get secrets | grep example-operator
example-operator-dockercfg-vkg5h   kubernetes.io/dockercfg               1      2m34s
example-operator-token-7shtj       kubernetes.io/service-account-token   4      2m34s
example-operator-token-hpv7k       kubernetes.io/service-account-token   4      2m34s


Additional info:
I have seen this for operators installed via cli and UI and from OperatorHub (backed by OperatorSource) and custom catalog (backed by CatalogSource).

Reproducer:

# install version 0.0.1
git clone https://github.com/jewzaam/example-operator.git
cd example-operator
oc create -R -f install/openshift-4.1/

# wait for it to start the operator pod
oc -n example-operator get pods -l name=example-operator --no-headers

# check secrets for the SA
oc -n example-operator get secrets | grep example-operator

# update to 0.0.2
oc -n example-operator get catalogsource example-operator-registry -o json | jq -r '.spec.image = "quay.io/nmalik/example-operator-registry:0.0.2"' | oc replace -f -

# wait for it upgrade the operator pod
oc -n example-operator get pods -l name=example-operator --no-headers

# check secrets for the SA
oc -n example-operator get secrets | grep example-operator

Comment 1 Naveen Malik 2019-11-05 18:53:25 UTC
I should note this is a potential problem with systems that have many operator updates such as development environments.  In our non-prod environments we have seen in the order of 1000's of extra secrets for individual operators.  Over time this will have an impact on etcd performance.

Possibly related:  https://bugzilla.redhat.com/show_bug.cgi?id=1666991

Comment 3 Jian Zhang 2019-12-05 06:18:51 UTC
Change the version to 4.4 since we have already bug 1776521 for 4.3.

Comment 7 Salvatore Colangelo 2020-01-29 23:49:28 UTC
Hi 

follow this step:

Step 1:
# install version 0.0.1
git clone https://github.com/jewzaam/example-operator.git

Step2:
#change the apiVersion on file 08_operatorgroup.yaml

vi install/openshift-4.1/08_operatorgroup.yaml -->  from apiVersion: operators.coreos.com/v1alpha2 to --> to: apiVersion: operators.coreos.com/v1

Step3

cd example-operator
oc create -R -f install/openshift-4.1/


[scolange@scolange example-operator]$ oc create -R -f install/openshift-4.1/
namespace/example-operator created
catalogsource.operators.coreos.com/example-operator-registry created
operatorgroup.operators.coreos.com/example-operators created
subscription.operators.coreos.com/example-operator created


Step 4:
# wait for it to start the operator pod
oc -n example-operator get pods -l name=example-operator --no-headers


Step 3:
# check secrets for the SA
oc -n example-operator get secrets | grep example-operator

[scolange@scolange example-operator]$ oc -n example-operator get secrets | grep example-operator
example-operator-dockercfg-8f57b   kubernetes.io/dockercfg               1      4m4s
example-operator-token-5mtln       kubernetes.io/service-account-token   4      4m5s
example-operator-token-dqpgc       kubernetes.io/service-account-token   4      4m5s


# update to 0.0.2
oc -n example-operator get catalogsource example-operator-registry -o json | jq -r '.spec.image = "quay.io/nmalik/example-operator-registry:0.0.2"' | oc replace -f -

[scolange@scolange example-operator]$ oc -n example-operator get catalogsource example-operator-registry -o json | jq -r '.spec.image = "quay.io/nmalik/example-operator-registry:0.0.2"' | oc replace -f -
catalogsource.operators.coreos.com/example-operator-registry replaced


# wait for it upgrade the operator pod
oc -n example-operator get pods -l name=example-operator --no-headers

[scolange@scolange example-operator]$ oc -n example-operator get pods -l name=example-operator --no-headers
example-operator-66dc694b6c-wwnk4   1/1   Running   0     71s


# check secrets for the SA
oc -n example-operator get secrets | grep example-operator


[scolange@scolange example-operator]$ oc -n example-operator get secrets | grep example-operator
example-operator-dockercfg-8f57b   kubernetes.io/dockercfg               1      4m57s
example-operator-token-5mtln       kubernetes.io/service-account-token   4      4m58s
example-operator-token-dqpgc       kubernetes.io/service-account-token   4      4m58s


The secret are not duplicated

Comment 8 Salvatore Colangelo 2020-01-29 23:50:37 UTC
Just for add the version of cluster:


[scolange@scolange example-operator]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.0-0.nightly-2020-01-29-012724   True        False         93m     Cluster version is 4.4.0-0.nightly-2020-01-29-012724

Comment 10 errata-xmlrpc 2020-05-13 21:52:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581


Note You need to log in before you can comment on or make changes to this bug.