Description of problem: enabled techPreviewUserWorkload, check with the following commands, then set enabled: false for techPreviewUserWorkload in cluster-monitoring-config configmap, thanos-ruler-trusted-ca-bundle/some thanos-ruler secrets/prometheusrule file are not deleted oc -n openshift-user-workload-monitoring get pod oc -n openshift-user-workload-monitoring get statefulset oc -n openshift-user-workload-monitoring get ThanosRuler oc -n openshift-user-workload-monitoring get configmaps oc -n openshift-user-workload-monitoring get services oc -n openshift-user-workload-monitoring get endpoints oc -n openshift-user-workload-monitoring get prometheusrule oc -n openshift-user-workload-monitoring get secret enabled techPreviewUserWorkload ************************************** # oc -n openshift-user-workload-monitoring get pod NAME READY STATUS RESTARTS AGE prometheus-operator-5dd94b66d7-hbd8g 2/2 Running 0 8h prometheus-user-workload-0 5/5 Running 1 8h prometheus-user-workload-1 5/5 Running 1 8h thanos-ruler-user-workload-0 3/3 Running 0 8h thanos-ruler-user-workload-1 3/3 Running 0 8h # oc -n openshift-user-workload-monitoring get statefulset NAME READY AGE prometheus-user-workload 2/2 8h thanos-ruler-user-workload 2/2 8h # oc -n openshift-user-workload-monitoring get ThanosRuler NAME AGE user-workload 8h # oc -n openshift-user-workload-monitoring get configmaps NAME DATA AGE prometheus-user-workload-rulefiles-0 0 8h serving-certs-ca-bundle 1 8h thanos-ruler-trusted-ca-bundle 1 8h thanos-ruler-trusted-ca-bundle-39man1pbaa8jq 1 8h thanos-ruler-user-workload-rulefiles-0 0 8h # oc -n openshift-user-workload-monitoring get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE prometheus-operated ClusterIP None <none> 9090/TCP,10901/TCP 8h prometheus-operator ClusterIP None <none> 8443/TCP 8h prometheus-user-workload ClusterIP 172.30.102.167 <none> 9091/TCP 8h thanos-ruler ClusterIP 172.30.135.27 <none> 9091/TCP,10901/TCP 8h thanos-ruler-operated ClusterIP None <none> 10902/TCP,10901/TCP 8h # oc -n openshift-user-workload-monitoring get endpoints NAME ENDPOINTS AGE prometheus-operated 10.129.2.20:10901,10.131.0.19:10901 8h prometheus-operator 10.128.0.51:8443 8h prometheus-user-workload 10.129.2.20:9091,10.131.0.19:9091 8h thanos-ruler 10.129.2.24:10901,10.131.0.24:10901,10.129.2.24:9091 + 1 more... 8h thanos-ruler-operated 10.129.2.24:10902,10.131.0.24:10902,10.129.2.24:10901 + 1 more... 8h # oc -n openshift-user-workload-monitoring get prometheusrule NAME AGE thanos-ruler 8h # oc -n openshift-user-workload-monitoring get secret NAME TYPE DATA AGE builder-dockercfg-zrqng kubernetes.io/dockercfg 1 11h builder-token-dtbvp kubernetes.io/service-account-token 4 11h builder-token-m6vp5 kubernetes.io/service-account-token 4 11h default-dockercfg-h7wk2 kubernetes.io/dockercfg 1 11h default-token-rgdlb kubernetes.io/service-account-token 4 11h default-token-t5whh kubernetes.io/service-account-token 4 11h deployer-dockercfg-r7ckc kubernetes.io/dockercfg 1 11h deployer-token-ggzjx kubernetes.io/service-account-token 4 11h deployer-token-jd5zn kubernetes.io/service-account-token 4 11h prometheus-operator-dockercfg-m2r4j kubernetes.io/dockercfg 1 8h prometheus-operator-token-t42kt kubernetes.io/service-account-token 4 8h prometheus-operator-token-wldh4 kubernetes.io/service-account-token 4 8h prometheus-operator-user-workload-tls kubernetes.io/tls 2 8h prometheus-user-workload Opaque 1 8h prometheus-user-workload-dockercfg-m99kr kubernetes.io/dockercfg 1 8h prometheus-user-workload-grpc-tls-72bp07neiijjl Opaque 3 8h prometheus-user-workload-tls kubernetes.io/tls 2 8h prometheus-user-workload-tls-assets Opaque 0 8h prometheus-user-workload-token-fksrn kubernetes.io/service-account-token 4 8h prometheus-user-workload-token-nrjxq kubernetes.io/service-account-token 4 8h thanos-ruler-alertmanagers-config Opaque 1 8h thanos-ruler-dockercfg-5sl27 kubernetes.io/dockercfg 1 8h thanos-ruler-grpc-tls-72bp07neiijjl Opaque 3 8h thanos-ruler-oauth-cookie Opaque 1 8h thanos-ruler-oauth-htpasswd Opaque 1 8h thanos-ruler-query-config Opaque 1 8h thanos-ruler-tls kubernetes.io/tls 2 8h thanos-ruler-token-h9v7w kubernetes.io/service-account-token 4 8h thanos-ruler-token-k8zqh kubernetes.io/service-account-token 4 8h ************************************** disabled techPreviewUserWorkload ************************************** # oc -n openshift-user-workload-monitoring get configmaps NAME DATA AGE thanos-ruler-trusted-ca-bundle-39man1pbaa8jq 1 8h # oc -n openshift-monitoring edit cm cluster-monitoring-config Edit cancelled, no changes made. # oc -n openshift-user-workload-monitoring get pod No resources found in openshift-user-workload-monitoring namespace. # oc -n openshift-user-workload-monitoring get statefulset No resources found in openshift-user-workload-monitoring namespace. # oc -n openshift-user-workload-monitoring get ThanosRuler No resources found in openshift-user-workload-monitoring namespace. # oc -n openshift-user-workload-monitoring get configmaps NAME DATA AGE thanos-ruler-trusted-ca-bundle-39man1pbaa8jq 1 8h # oc -n openshift-user-workload-monitoring get services No resources found in openshift-user-workload-monitoring namespace. # oc -n openshift-user-workload-monitoring get endpoints No resources found in openshift-user-workload-monitoring namespace. # oc -n openshift-user-workload-monitoring get prometheusrule NAME AGE thanos-ruler 8h # oc -n openshift-user-workload-monitoring get secret NAME TYPE DATA AGE builder-dockercfg-zrqng kubernetes.io/dockercfg 1 11h builder-token-dtbvp kubernetes.io/service-account-token 4 11h builder-token-m6vp5 kubernetes.io/service-account-token 4 11h default-dockercfg-h7wk2 kubernetes.io/dockercfg 1 11h default-token-rgdlb kubernetes.io/service-account-token 4 11h default-token-t5whh kubernetes.io/service-account-token 4 11h deployer-dockercfg-r7ckc kubernetes.io/dockercfg 1 11h deployer-token-ggzjx kubernetes.io/service-account-token 4 11h deployer-token-jd5zn kubernetes.io/service-account-token 4 11h thanos-ruler-alertmanagers-config Opaque 1 8h thanos-ruler-grpc-tls-72bp07neiijjl Opaque 3 8h thanos-ruler-query-config Opaque 1 8h Version-Release number of selected component (if applicable): 4.5.0-0.nightly-2020-04-06-184201 How reproducible: always Steps to Reproduce: 1. see the description 2. 3. Actual results: Expected results: Additional info:
much better now, tested with 4.5.0-0.nightly-2020-05-04-113741, the thanos-ruler-trusted-ca-bundle-*** configmap is not deleted. enabled techPreviewUserWorkload # oc -n openshift-user-workload-monitoring get configmaps NAME DATA AGE prometheus-user-workload-rulefiles-0 0 35m serving-certs-ca-bundle 1 35m thanos-ruler-trusted-ca-bundle 1 35m thanos-ruler-trusted-ca-bundle-39man1pbaa8jq 1 31m thanos-ruler-user-workload-rulefiles-0 1 6m4s disabled techPreviewUserWorkload # oc -n openshift-user-workload-monitoring get configmaps NAME DATA AGE thanos-ruler-trusted-ca-bundle-39man1pbaa8jq 1 63m
Issue is fixed with 4.5.0-0.nightly-2020-05-20-183547, hashed ca bundle configmaps are cleaned enabled techPreviewUserWorkload # oc -n openshift-user-workload-monitoring get configmaps prometheus-user-workload-rulefiles-0 0 111s serving-certs-ca-bundle 1 2m17s thanos-ruler-trusted-ca-bundle 1 116s thanos-ruler-trusted-ca-bundle-39man1pbaa8jq 1 113s thanos-ruler-user-workload-rulefiles-0 0 108s disabled techPreviewUserWorkload # oc -n openshift-user-workload-monitoring get configmaps No resources found in openshift-user-workload-monitoring namespace.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2409