Hide Forgot
Created attachment 1888623 [details] sa from 4.11.0-0.nightly-2022-06-06-201913 cluster Description of problem: upgrade from 4.10.16 to 4.11.0-0.nightly-2022-06-06-201913, find the serviceaccounts are not updated, but they don't affect the function, example: # oc -n openshift-monitoring get sa prometheus-k8s -oyaml apiVersion: v1 imagePullSecrets: - name: prometheus-k8s-dockercfg-tj2k7 kind: ServiceAccount metadata: annotations: serviceaccounts.openshift.io/oauth-redirectreference.prometheus-k8s: '{"kind":"OAuthRedirectReference","apiVersion":"v1","reference":{"kind":"Route","name":"prometheus-k8s"}}' creationTimestamp: "2022-06-08T17:39:21Z" labels: app.kubernetes.io/component: prometheus app.kubernetes.io/instance: k8s app.kubernetes.io/name: prometheus app.kubernetes.io/part-of: openshift-monitoring app.kubernetes.io/version: 2.32.1 name: prometheus-k8s namespace: openshift-monitoring resourceVersion: "25206" uid: e59e4e8c-0a15-4fdd-96ff-a132b26e4620 secrets: - name: prometheus-k8s-token-5p8ls - name: prometheus-k8s-dockercfg-tj2k7 ******************************* there is not automountServiceAccountToken: false, app.kubernetes.io/version: 2.32.1, which should be 2.35.0, should not have prometheus-k8s-token secret based on bug 2093780 # oc -n openshift-monitoring logs -c prometheus prometheus-k8s-0 | head ts=2022-06-10T00:52:24.392Z caller=main.go:542 level=info msg="Starting Prometheus" version="(version=2.35.0, branch=rhaos-4.11-rhel-8, revision=023408b0362e6fe738a42e5820ba3f4073039666)" in a 4.11.0-0.nightly-2022-06-06-201913 cluster, it is # oc -n openshift-monitoring get sa prometheus-k8s -oyaml apiVersion: v1 automountServiceAccountToken: false imagePullSecrets: - name: prometheus-k8s-dockercfg-8znp7 kind: ServiceAccount metadata: annotations: serviceaccounts.openshift.io/oauth-redirectreference.prometheus-k8s: '{"kind":"OAuthRedirectReference","apiVersion":"v1","reference":{"kind":"Route","name":"prometheus-k8s"}}' creationTimestamp: "2022-06-09T23:26:08Z" labels: app.kubernetes.io/component: prometheus app.kubernetes.io/instance: k8s app.kubernetes.io/name: prometheus app.kubernetes.io/part-of: openshift-monitoring app.kubernetes.io/version: 2.35.0 name: prometheus-k8s namespace: openshift-monitoring resourceVersion: "21302" uid: 6e652fad-76b4-43a1-9fb2-636e61e09a96 secrets: - name: prometheus-k8s-dockercfg-8znp7 same for other monitoring serviceaccounts, for example kube-state-metrics/prometheus-operator and etc, their version label are kept the same as 4.10.16 # for i in $(oc -n openshift-monitoring get sa | grep -v NAME| awk '{print $1}'); do echo $i; oc -n openshift-monitoring get sa $i -oyaml | grep "app.kubernetes.io/version";echo -e "\n";done alertmanager-main app.kubernetes.io/version: 0.23.0 builder cluster-monitoring-operator default deployer kube-state-metrics app.kubernetes.io/version: 2.3.0 node-exporter app.kubernetes.io/version: 1.3.1 openshift-state-metrics prometheus-adapter app.kubernetes.io/version: 0.9.1 prometheus-k8s app.kubernetes.io/version: 2.32.1 prometheus-operator app.kubernetes.io/version: 0.53.1 prometheus-operator-admission-webhook app.kubernetes.io/version: 0.55.1 telemeter-client thanos-querier app.kubernetes.io/version: 0.23.1 Version-Release number of selected component (if applicable): upgrade from 4.10.16 to 4.11.0-0.nightly-2022-06-06-201913 How reproducible: always Steps to Reproduce: 1. upgrade from 4.10.16 to 4.11.0-0.nightly-2022-06-06-201913, and compare the sa with a 4.11.0-0.nightly-2022-06-06-201913 2. 3. Actual results: Expected results: Additional info: since must-gather does not have serviceaccounts info, only provide serviceaccounts file here
4.11.0-0.nightly-2022-06-06-201913 sa version labels # for i in $(oc -n openshift-monitoring get sa | grep -v NAME| awk '{print $1}'); do echo $i; oc -n openshift-monitoring get sa $i -oyaml | grep "app.kubernetes.io/version";echo -e "\n";done alertmanager-main app.kubernetes.io/version: 0.24.0 builder cluster-monitoring-operator default deployer kube-state-metrics app.kubernetes.io/version: 2.4.2 node-exporter app.kubernetes.io/version: 1.3.1 openshift-state-metrics prometheus-adapter app.kubernetes.io/version: 0.9.1 prometheus-k8s app.kubernetes.io/version: 2.35.0 prometheus-operator app.kubernetes.io/version: 0.55.1 prometheus-operator-admission-webhook app.kubernetes.io/version: 0.55.1 telemeter-client thanos-querier app.kubernetes.io/version: 0.25.2
Good catch! This is even documented in the CMO code base with a potential fix. https://github.com/openshift/cluster-monitoring-operator/blob/03e8db6a55d7ecd35a8c82909a94252441b363f6/pkg/client/client.go#L1451-L1466
upgrade from 4.10.0-0.nightly-2022-06-08-150219 to 4.11.0-0.nightly-2022-07-06-145812, note down the sa, then upgrade to 4.12.0-0.nightly-2022-07-08-015358, compare the sa files, the sa files are updated for 4.12 cluster, see the attached file
upgrade from 4.12.0-0.nightly-2022-11-28-145121 to 4.13.0-0.nightly-2022-11-28-190649, serviceaccounts are updated after upgrade, see the differences for prometheus-operator/prometheus-operator-admission-webhook before and after upgrade $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.12.0-0.nightly-2022-11-28-145121 True False 64m Cluster version is 4.12.0-0.nightly-2022-11-28-145121 $ for i in $(oc -n openshift-monitoring get sa | grep -v NAME| awk '{print $1}'); do echo $i; oc -n openshift-monitoring get sa $i -oyaml | grep "app.kubernetes.io/version";done alertmanager-main app.kubernetes.io/version: 0.24.0 kube-state-metrics app.kubernetes.io/version: 2.6.0 node-exporter app.kubernetes.io/version: 1.4.0 prometheus-adapter app.kubernetes.io/version: 0.10.0 prometheus-k8s app.kubernetes.io/version: 2.39.1 prometheus-operator app.kubernetes.io/version: 0.60.1 prometheus-operator-admission-webhook app.kubernetes.io/version: 0.60.1 thanos-querier app.kubernetes.io/version: 0.28.1 $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.13.0-0.nightly-2022-11-28-190649 True False 3h3m Cluster version is 4.13.0-0.nightly-2022-11-28-190649 $ for i in $(oc -n openshift-monitoring get sa | grep -v NAME| awk '{print $1}'); do echo $i; oc -n openshift-monitoring get sa $i -oyaml | grep "app.kubernetes.io/version";done alertmanager-main app.kubernetes.io/version: 0.24.0 kube-state-metrics app.kubernetes.io/version: 2.6.0 node-exporter app.kubernetes.io/version: 1.4.0 prometheus-adapter app.kubernetes.io/version: 0.10.0 prometheus-k8s app.kubernetes.io/version: 2.39.1 prometheus-operator app.kubernetes.io/version: 0.61.1 prometheus-operator-admission-webhook app.kubernetes.io/version: 0.61.1 thanos-querier app.kubernetes.io/version: 0.28.1