Bug 1821666

Summary:	some resources are not cleaned up after disabling techPreviewUserWorkload
Product:	OpenShift Container Platform	Reporter:	Junqi Zhao <juzhao>
Component:	Monitoring	Assignee:	Paul Gier <pgier>
Status:	CLOSED ERRATA	QA Contact:	Junqi Zhao <juzhao>
Severity:	low	Docs Contact:
Priority:	low
Version:	4.5	CC:	alegrand, anpicker, erooth, kakkoyun, lcosic, mloibl, pkrupa, surbania
Target Milestone:	---
Target Release:	4.5.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-07-13 17:25:52 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Junqi Zhao 2020-04-07 11:18:58 UTC

Description of problem:
enabled techPreviewUserWorkload, check with the following commands, then set enabled: false for techPreviewUserWorkload in cluster-monitoring-config configmap,
thanos-ruler-trusted-ca-bundle/some thanos-ruler secrets/prometheusrule file are not deleted
 
oc -n openshift-user-workload-monitoring get pod
oc -n openshift-user-workload-monitoring get statefulset
oc -n openshift-user-workload-monitoring get ThanosRuler
oc -n openshift-user-workload-monitoring get configmaps
oc -n openshift-user-workload-monitoring get services
oc -n openshift-user-workload-monitoring get endpoints
oc -n openshift-user-workload-monitoring get prometheusrule
oc -n openshift-user-workload-monitoring get secret


enabled techPreviewUserWorkload
**************************************
# oc -n openshift-user-workload-monitoring get pod
NAME                                   READY   STATUS    RESTARTS   AGE
prometheus-operator-5dd94b66d7-hbd8g   2/2     Running   0          8h
prometheus-user-workload-0             5/5     Running   1          8h
prometheus-user-workload-1             5/5     Running   1          8h
thanos-ruler-user-workload-0           3/3     Running   0          8h
thanos-ruler-user-workload-1           3/3     Running   0          8h

# oc -n openshift-user-workload-monitoring get statefulset
NAME                         READY   AGE
prometheus-user-workload     2/2     8h
thanos-ruler-user-workload   2/2     8h

# oc -n openshift-user-workload-monitoring get ThanosRuler
NAME            AGE
user-workload   8h

# oc -n openshift-user-workload-monitoring get configmaps
NAME                                           DATA   AGE
prometheus-user-workload-rulefiles-0           0      8h
serving-certs-ca-bundle                        1      8h
thanos-ruler-trusted-ca-bundle                 1      8h
thanos-ruler-trusted-ca-bundle-39man1pbaa8jq   1      8h
thanos-ruler-user-workload-rulefiles-0         0      8h

# oc -n openshift-user-workload-monitoring get services
NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)               AGE
prometheus-operated        ClusterIP   None             <none>        9090/TCP,10901/TCP    8h
prometheus-operator        ClusterIP   None             <none>        8443/TCP              8h
prometheus-user-workload   ClusterIP   172.30.102.167   <none>        9091/TCP              8h
thanos-ruler               ClusterIP   172.30.135.27    <none>        9091/TCP,10901/TCP    8h
thanos-ruler-operated      ClusterIP   None             <none>        10902/TCP,10901/TCP   8h

# oc -n openshift-user-workload-monitoring get endpoints
NAME                       ENDPOINTS                                                           AGE
prometheus-operated        10.129.2.20:10901,10.131.0.19:10901                                 8h
prometheus-operator        10.128.0.51:8443                                                    8h
prometheus-user-workload   10.129.2.20:9091,10.131.0.19:9091                                   8h
thanos-ruler               10.129.2.24:10901,10.131.0.24:10901,10.129.2.24:9091 + 1 more...    8h
thanos-ruler-operated      10.129.2.24:10902,10.131.0.24:10902,10.129.2.24:10901 + 1 more...   8h

# oc -n openshift-user-workload-monitoring get prometheusrule
NAME           AGE
thanos-ruler   8h

# oc -n openshift-user-workload-monitoring get secret
NAME                                              TYPE                                  DATA   AGE
builder-dockercfg-zrqng                           kubernetes.io/dockercfg               1      11h
builder-token-dtbvp                               kubernetes.io/service-account-token   4      11h
builder-token-m6vp5                               kubernetes.io/service-account-token   4      11h
default-dockercfg-h7wk2                           kubernetes.io/dockercfg               1      11h
default-token-rgdlb                               kubernetes.io/service-account-token   4      11h
default-token-t5whh                               kubernetes.io/service-account-token   4      11h
deployer-dockercfg-r7ckc                          kubernetes.io/dockercfg               1      11h
deployer-token-ggzjx                              kubernetes.io/service-account-token   4      11h
deployer-token-jd5zn                              kubernetes.io/service-account-token   4      11h
prometheus-operator-dockercfg-m2r4j               kubernetes.io/dockercfg               1      8h
prometheus-operator-token-t42kt                   kubernetes.io/service-account-token   4      8h
prometheus-operator-token-wldh4                   kubernetes.io/service-account-token   4      8h
prometheus-operator-user-workload-tls             kubernetes.io/tls                     2      8h
prometheus-user-workload                          Opaque                                1      8h
prometheus-user-workload-dockercfg-m99kr          kubernetes.io/dockercfg               1      8h
prometheus-user-workload-grpc-tls-72bp07neiijjl   Opaque                                3      8h
prometheus-user-workload-tls                      kubernetes.io/tls                     2      8h
prometheus-user-workload-tls-assets               Opaque                                0      8h
prometheus-user-workload-token-fksrn              kubernetes.io/service-account-token   4      8h
prometheus-user-workload-token-nrjxq              kubernetes.io/service-account-token   4      8h
thanos-ruler-alertmanagers-config                 Opaque                                1      8h
thanos-ruler-dockercfg-5sl27                      kubernetes.io/dockercfg               1      8h
thanos-ruler-grpc-tls-72bp07neiijjl               Opaque                                3      8h
thanos-ruler-oauth-cookie                         Opaque                                1      8h
thanos-ruler-oauth-htpasswd                       Opaque                                1      8h
thanos-ruler-query-config                         Opaque                                1      8h
thanos-ruler-tls                                  kubernetes.io/tls                     2      8h
thanos-ruler-token-h9v7w                          kubernetes.io/service-account-token   4      8h
thanos-ruler-token-k8zqh                          kubernetes.io/service-account-token   4      8h
**************************************

disabled techPreviewUserWorkload
**************************************
# oc -n openshift-user-workload-monitoring get configmaps
NAME                                           DATA   AGE
thanos-ruler-trusted-ca-bundle-39man1pbaa8jq   1      8h
# oc -n openshift-monitoring edit cm cluster-monitoring-config
Edit cancelled, no changes made.

# oc -n openshift-user-workload-monitoring get pod
No resources found in openshift-user-workload-monitoring namespace.

# oc -n openshift-user-workload-monitoring get statefulset
No resources found in openshift-user-workload-monitoring namespace.

# oc -n openshift-user-workload-monitoring get ThanosRuler
No resources found in openshift-user-workload-monitoring namespace.

# oc -n openshift-user-workload-monitoring get configmaps
NAME                                           DATA   AGE
thanos-ruler-trusted-ca-bundle-39man1pbaa8jq   1      8h

# oc -n openshift-user-workload-monitoring get services
No resources found in openshift-user-workload-monitoring namespace.

# oc -n openshift-user-workload-monitoring get endpoints
No resources found in openshift-user-workload-monitoring namespace.

# oc -n openshift-user-workload-monitoring get prometheusrule
NAME           AGE
thanos-ruler   8h
# oc -n openshift-user-workload-monitoring get secret
NAME                                  TYPE                                  DATA   AGE
builder-dockercfg-zrqng               kubernetes.io/dockercfg               1      11h
builder-token-dtbvp                   kubernetes.io/service-account-token   4      11h
builder-token-m6vp5                   kubernetes.io/service-account-token   4      11h
default-dockercfg-h7wk2               kubernetes.io/dockercfg               1      11h
default-token-rgdlb                   kubernetes.io/service-account-token   4      11h
default-token-t5whh                   kubernetes.io/service-account-token   4      11h
deployer-dockercfg-r7ckc              kubernetes.io/dockercfg               1      11h
deployer-token-ggzjx                  kubernetes.io/service-account-token   4      11h
deployer-token-jd5zn                  kubernetes.io/service-account-token   4      11h
thanos-ruler-alertmanagers-config     Opaque                                1      8h
thanos-ruler-grpc-tls-72bp07neiijjl   Opaque                                3      8h
thanos-ruler-query-config             Opaque                                1      8h


Version-Release number of selected component (if applicable):
4.5.0-0.nightly-2020-04-06-184201

How reproducible:
always

Steps to Reproduce:
1. see the description
2.
3.

Actual results:


Expected results:


Additional info:

Comment 5 Junqi Zhao 2020-05-06 02:09:01 UTC

much better now, tested with 4.5.0-0.nightly-2020-05-04-113741, the thanos-ruler-trusted-ca-bundle-*** configmap is not deleted.

enabled techPreviewUserWorkload
# oc -n openshift-user-workload-monitoring get configmaps
NAME                                           DATA   AGE
prometheus-user-workload-rulefiles-0           0      35m
serving-certs-ca-bundle                        1      35m
thanos-ruler-trusted-ca-bundle                 1      35m
thanos-ruler-trusted-ca-bundle-39man1pbaa8jq   1      31m
thanos-ruler-user-workload-rulefiles-0         1      6m4s

disabled techPreviewUserWorkload
# oc -n openshift-user-workload-monitoring get configmaps
NAME                                           DATA   AGE
thanos-ruler-trusted-ca-bundle-39man1pbaa8jq   1      63m

Comment 7 Junqi Zhao 2020-05-21 02:29:38 UTC

Issue is fixed with 4.5.0-0.nightly-2020-05-20-183547, hashed ca bundle configmaps are cleaned
enabled techPreviewUserWorkload
# oc -n openshift-user-workload-monitoring get configmaps
prometheus-user-workload-rulefiles-0           0      111s
serving-certs-ca-bundle                        1      2m17s
thanos-ruler-trusted-ca-bundle                 1      116s
thanos-ruler-trusted-ca-bundle-39man1pbaa8jq   1      113s
thanos-ruler-user-workload-rulefiles-0         0      108s

disabled techPreviewUserWorkload
# oc -n openshift-user-workload-monitoring get configmaps
No resources found in openshift-user-workload-monitoring namespace.

Comment 8 errata-xmlrpc 2020-07-13 17:25:52 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409