Bug 1722604

Summary:	tuning operator creating thousands of secrets over time
Product:	OpenShift Container Platform	Reporter:	Justin Pierce <jupierce>
Component:	Node Tuning Operator	Assignee:	Jiří Mencák <jmencak>
Status:	CLOSED DUPLICATE	QA Contact:	Mike Fiedler <mifiedle>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	4.1.z	CC:	sejug
Target Milestone:	---	Keywords:	DeliveryBlocker, OpsBlocker
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-06-21 13:29:43 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Justin Pierce 2019-06-20 18:00:43 UTC

Description of problem:
Over time, the node-tuning operator is creating thousands of secrets. Over 50 days, >26K have been created on a one of our starter clusters.

[ec2-user us-west-1 ~]$ oc get secrets -A | grep node-tuning | wc
  26361  131805 4590991

This includes 17K service-account-tokens. e.g:
openshift-cluster-node-tuning-operator                  tuned-token-j8hvb                                                    kubernetes.io/service-account-token

8.7K dockercfg secrets. e.g:
openshift-cluster-node-tuning-operator                  tuned-dockercfg-nnwh7                                                kubernetes.io/dockercfg

Version-Release number of selected component (if applicable):
4.1.2

How reproducible:
Affects all starter clusters.

Steps to Reproduce:
1. Install a cluster and allow it to run for multiple days
2.
3.

Additional info:
Possibly related: https://bugzilla.redhat.com/show_bug.cgi?id=1719967

Comment 1 Justin Pierce 2019-06-20 19:26:33 UTC

*** Bug 1717244 has been marked as a duplicate of this bug. ***

Comment 2 Jiří Mencák 2019-06-20 21:18:53 UTC

Are you sure that the secrets are still being accumulated/leaked on a 4.1.2 cluster or are the secrets just leftovers from a 4.1.0 installation?  See: https://bugzilla.redhat.com/show_bug.cgi?id=1714484 and https://bugzilla.redhat.com/show_bug.cgi?id=1718842 

The problem was fixed in 4.1.1 (the operator should not leak any more secrets), but the automated removal of leaked secrets is still only upstream and I'd say it needs to be performed manually on large clusters like the starter clusters as described in both BZs.

I believe this bug is now a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1718842

Comment 3 Jiří Mencák 2019-06-21 13:29:43 UTC

Cannot reproduce on 4.1.2.  I'm quite certain the secrets were leaked in 4.1.0 and left uncleaned at operator upgrade.

$ oc get clusterversion    
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.1.2     False       True          3h47m   Unable to apply 4.1.2: the cluster operator marketplace has not yet successfully rolled out
$ oc get secrets -n openshift-cluster-node-tuning-operator | grep ^tuned
tuned-dockercfg-w8fkv                          kubernetes.io/dockercfg               1      3h41m
tuned-token-7chnb                              kubernetes.io/service-account-token   4      3h41m
tuned-token-zv88j                              kubernetes.io/service-account-token   4      3h41m

Marking as a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1718842

If this doesn't work for you, please re-open.

*** This bug has been marked as a duplicate of bug 1718842 ***

Comment 4 Red Hat Bugzilla 2023-09-14 05:30:42 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days