Bug 2095719

Summary: serviceaccounts are not updated after upgrade from 4.10 to 4.11
Product: OpenShift Container Platform Reporter: Junqi Zhao <juzhao>
Component: MonitoringAssignee: Joao Marcal <jmarcal>
Status: CLOSED ERRATA QA Contact: Junqi Zhao <juzhao>
Severity: low Docs Contact:
Priority: low    
Version: 4.11CC: anpicker, bleanhar
Target Milestone: ---   
Target Release: 4.13.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2115527 (view as bug list) Environment:
Last Closed: 2023-05-17 22:46:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
sa from 4.11.0-0.nightly-2022-06-06-201913 cluster none

Description Junqi Zhao 2022-06-10 11:10:06 UTC
Created attachment 1888623 [details]
sa from 4.11.0-0.nightly-2022-06-06-201913 cluster

Description of problem:
upgrade from 4.10.16 to 4.11.0-0.nightly-2022-06-06-201913, find the serviceaccounts are not updated, but they don't affect the function, example:
# oc -n openshift-monitoring get sa prometheus-k8s -oyaml
apiVersion: v1
imagePullSecrets:
- name: prometheus-k8s-dockercfg-tj2k7
kind: ServiceAccount
metadata:
  annotations:
    serviceaccounts.openshift.io/oauth-redirectreference.prometheus-k8s: '{"kind":"OAuthRedirectReference","apiVersion":"v1","reference":{"kind":"Route","name":"prometheus-k8s"}}'
  creationTimestamp: "2022-06-08T17:39:21Z"
  labels:
    app.kubernetes.io/component: prometheus
    app.kubernetes.io/instance: k8s
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/part-of: openshift-monitoring
    app.kubernetes.io/version: 2.32.1
  name: prometheus-k8s
  namespace: openshift-monitoring
  resourceVersion: "25206"
  uid: e59e4e8c-0a15-4fdd-96ff-a132b26e4620
secrets:
- name: prometheus-k8s-token-5p8ls
- name: prometheus-k8s-dockercfg-tj2k7
*******************************
there is not automountServiceAccountToken: false, app.kubernetes.io/version: 2.32.1, which should be 2.35.0, should not have prometheus-k8s-token secret based on bug 2093780
# oc -n openshift-monitoring logs -c prometheus prometheus-k8s-0 | head
ts=2022-06-10T00:52:24.392Z caller=main.go:542 level=info msg="Starting Prometheus" version="(version=2.35.0, branch=rhaos-4.11-rhel-8, revision=023408b0362e6fe738a42e5820ba3f4073039666)"


in a 4.11.0-0.nightly-2022-06-06-201913 cluster, it is
# oc -n openshift-monitoring get sa prometheus-k8s -oyaml
apiVersion: v1
automountServiceAccountToken: false
imagePullSecrets:
- name: prometheus-k8s-dockercfg-8znp7
kind: ServiceAccount
metadata:
  annotations:
    serviceaccounts.openshift.io/oauth-redirectreference.prometheus-k8s: '{"kind":"OAuthRedirectReference","apiVersion":"v1","reference":{"kind":"Route","name":"prometheus-k8s"}}'
  creationTimestamp: "2022-06-09T23:26:08Z"
  labels:
    app.kubernetes.io/component: prometheus
    app.kubernetes.io/instance: k8s
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/part-of: openshift-monitoring
    app.kubernetes.io/version: 2.35.0
  name: prometheus-k8s
  namespace: openshift-monitoring
  resourceVersion: "21302"
  uid: 6e652fad-76b4-43a1-9fb2-636e61e09a96
secrets:
- name: prometheus-k8s-dockercfg-8znp7


same for other monitoring serviceaccounts, for example kube-state-metrics/prometheus-operator and etc, their version label are kept the same as 4.10.16
# for i in $(oc -n openshift-monitoring get sa | grep -v NAME| awk '{print $1}'); do echo $i; oc -n openshift-monitoring get sa $i -oyaml | grep "app.kubernetes.io/version";echo -e "\n";done

alertmanager-main
    app.kubernetes.io/version: 0.23.0


builder


cluster-monitoring-operator


default


deployer


kube-state-metrics
    app.kubernetes.io/version: 2.3.0


node-exporter
    app.kubernetes.io/version: 1.3.1


openshift-state-metrics


prometheus-adapter
    app.kubernetes.io/version: 0.9.1


prometheus-k8s
    app.kubernetes.io/version: 2.32.1


prometheus-operator
    app.kubernetes.io/version: 0.53.1


prometheus-operator-admission-webhook
    app.kubernetes.io/version: 0.55.1


telemeter-client


thanos-querier
    app.kubernetes.io/version: 0.23.1

Version-Release number of selected component (if applicable):
upgrade from 4.10.16 to 4.11.0-0.nightly-2022-06-06-201913

How reproducible:
always

Steps to Reproduce:
1. upgrade from 4.10.16 to 4.11.0-0.nightly-2022-06-06-201913, and compare the sa with a 4.11.0-0.nightly-2022-06-06-201913
2.
3.

Actual results:


Expected results:


Additional info:
since must-gather does not have serviceaccounts info, only provide serviceaccounts file here

Comment 2 Junqi Zhao 2022-06-10 11:12:56 UTC
4.11.0-0.nightly-2022-06-06-201913 sa version labels
# for i in $(oc -n openshift-monitoring get sa | grep -v NAME| awk '{print $1}'); do echo $i; oc -n openshift-monitoring get sa $i -oyaml | grep "app.kubernetes.io/version";echo -e "\n";done

alertmanager-main
    app.kubernetes.io/version: 0.24.0


builder


cluster-monitoring-operator


default


deployer


kube-state-metrics
    app.kubernetes.io/version: 2.4.2


node-exporter
    app.kubernetes.io/version: 1.3.1


openshift-state-metrics


prometheus-adapter
    app.kubernetes.io/version: 0.9.1


prometheus-k8s
    app.kubernetes.io/version: 2.35.0


prometheus-operator
    app.kubernetes.io/version: 0.55.1


prometheus-operator-admission-webhook
    app.kubernetes.io/version: 0.55.1


telemeter-client


thanos-querier
    app.kubernetes.io/version: 0.25.2

Comment 3 Simon Pasquier 2022-06-10 12:46:36 UTC
Good catch! This is even documented in the CMO code base with a potential fix.

https://github.com/openshift/cluster-monitoring-operator/blob/03e8db6a55d7ecd35a8c82909a94252441b363f6/pkg/client/client.go#L1451-L1466

Comment 9 Junqi Zhao 2022-07-08 13:56:28 UTC
upgrade from 4.10.0-0.nightly-2022-06-08-150219 to 4.11.0-0.nightly-2022-07-06-145812, note down the sa, then upgrade to 4.12.0-0.nightly-2022-07-08-015358, compare the sa files, the sa files are updated for 4.12 cluster, see the attached file

Comment 18 Junqi Zhao 2022-11-29 07:05:39 UTC
upgrade from 4.12.0-0.nightly-2022-11-28-145121 to 4.13.0-0.nightly-2022-11-28-190649, serviceaccounts are updated after upgrade, see the differences for prometheus-operator/prometheus-operator-admission-webhook before and after upgrade
$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.12.0-0.nightly-2022-11-28-145121   True        False         64m     Cluster version is 4.12.0-0.nightly-2022-11-28-145121
$ for i in $(oc -n openshift-monitoring get sa | grep -v NAME| awk '{print $1}'); do echo $i; oc -n openshift-monitoring get sa $i -oyaml | grep "app.kubernetes.io/version";done
alertmanager-main
    app.kubernetes.io/version: 0.24.0
kube-state-metrics
    app.kubernetes.io/version: 2.6.0
node-exporter
    app.kubernetes.io/version: 1.4.0
prometheus-adapter
    app.kubernetes.io/version: 0.10.0
prometheus-k8s
    app.kubernetes.io/version: 2.39.1
prometheus-operator
    app.kubernetes.io/version: 0.60.1
prometheus-operator-admission-webhook
    app.kubernetes.io/version: 0.60.1
thanos-querier
    app.kubernetes.io/version: 0.28.1

$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.13.0-0.nightly-2022-11-28-190649   True        False         3h3m    Cluster version is 4.13.0-0.nightly-2022-11-28-190649

$ for i in $(oc -n openshift-monitoring get sa | grep -v NAME| awk '{print $1}'); do echo $i; oc -n openshift-monitoring get sa $i -oyaml | grep "app.kubernetes.io/version";done
alertmanager-main
    app.kubernetes.io/version: 0.24.0
kube-state-metrics
    app.kubernetes.io/version: 2.6.0
node-exporter
    app.kubernetes.io/version: 1.4.0
prometheus-adapter
    app.kubernetes.io/version: 0.10.0
prometheus-k8s
    app.kubernetes.io/version: 2.39.1
prometheus-operator
    app.kubernetes.io/version: 0.61.1
prometheus-operator-admission-webhook
    app.kubernetes.io/version: 0.61.1
thanos-querier
    app.kubernetes.io/version: 0.28.1

Comment 21 errata-xmlrpc 2023-05-17 22:46:53 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.13.0 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:1326