Bug 2095719 - serviceaccounts are not updated after upgrade from 4.10 to 4.11
Summary: serviceaccounts are not updated after upgrade from 4.10 to 4.11
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Monitoring
Version: 4.11
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: 4.13.0
Assignee: Joao Marcal
QA Contact: Junqi Zhao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-06-10 11:10 UTC by Junqi Zhao
Modified: 2023-05-17 22:47 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2115527 (view as bug list)
Environment:
Last Closed: 2023-05-17 22:46:53 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
sa from 4.11.0-0.nightly-2022-06-06-201913 cluster (2.01 KB, application/gzip)
2022-06-10 11:10 UTC, Junqi Zhao
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-monitoring-operator pull 1704 0 None Merged Bug 2095719: Updates CreateOrUpdateServiceAccounts 2022-08-04 20:38:26 UTC
Github openshift cluster-monitoring-operator pull 1745 0 None Merged Bug 2095719: Updates CreateOrUpdateServiceAccounts 2022-11-28 13:30:14 UTC
Red Hat Product Errata RHSA-2023:1326 0 None None None 2023-05-17 22:47:12 UTC

Description Junqi Zhao 2022-06-10 11:10:06 UTC
Created attachment 1888623 [details]
sa from 4.11.0-0.nightly-2022-06-06-201913 cluster

Description of problem:
upgrade from 4.10.16 to 4.11.0-0.nightly-2022-06-06-201913, find the serviceaccounts are not updated, but they don't affect the function, example:
# oc -n openshift-monitoring get sa prometheus-k8s -oyaml
apiVersion: v1
imagePullSecrets:
- name: prometheus-k8s-dockercfg-tj2k7
kind: ServiceAccount
metadata:
  annotations:
    serviceaccounts.openshift.io/oauth-redirectreference.prometheus-k8s: '{"kind":"OAuthRedirectReference","apiVersion":"v1","reference":{"kind":"Route","name":"prometheus-k8s"}}'
  creationTimestamp: "2022-06-08T17:39:21Z"
  labels:
    app.kubernetes.io/component: prometheus
    app.kubernetes.io/instance: k8s
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/part-of: openshift-monitoring
    app.kubernetes.io/version: 2.32.1
  name: prometheus-k8s
  namespace: openshift-monitoring
  resourceVersion: "25206"
  uid: e59e4e8c-0a15-4fdd-96ff-a132b26e4620
secrets:
- name: prometheus-k8s-token-5p8ls
- name: prometheus-k8s-dockercfg-tj2k7
*******************************
there is not automountServiceAccountToken: false, app.kubernetes.io/version: 2.32.1, which should be 2.35.0, should not have prometheus-k8s-token secret based on bug 2093780
# oc -n openshift-monitoring logs -c prometheus prometheus-k8s-0 | head
ts=2022-06-10T00:52:24.392Z caller=main.go:542 level=info msg="Starting Prometheus" version="(version=2.35.0, branch=rhaos-4.11-rhel-8, revision=023408b0362e6fe738a42e5820ba3f4073039666)"


in a 4.11.0-0.nightly-2022-06-06-201913 cluster, it is
# oc -n openshift-monitoring get sa prometheus-k8s -oyaml
apiVersion: v1
automountServiceAccountToken: false
imagePullSecrets:
- name: prometheus-k8s-dockercfg-8znp7
kind: ServiceAccount
metadata:
  annotations:
    serviceaccounts.openshift.io/oauth-redirectreference.prometheus-k8s: '{"kind":"OAuthRedirectReference","apiVersion":"v1","reference":{"kind":"Route","name":"prometheus-k8s"}}'
  creationTimestamp: "2022-06-09T23:26:08Z"
  labels:
    app.kubernetes.io/component: prometheus
    app.kubernetes.io/instance: k8s
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/part-of: openshift-monitoring
    app.kubernetes.io/version: 2.35.0
  name: prometheus-k8s
  namespace: openshift-monitoring
  resourceVersion: "21302"
  uid: 6e652fad-76b4-43a1-9fb2-636e61e09a96
secrets:
- name: prometheus-k8s-dockercfg-8znp7


same for other monitoring serviceaccounts, for example kube-state-metrics/prometheus-operator and etc, their version label are kept the same as 4.10.16
# for i in $(oc -n openshift-monitoring get sa | grep -v NAME| awk '{print $1}'); do echo $i; oc -n openshift-monitoring get sa $i -oyaml | grep "app.kubernetes.io/version";echo -e "\n";done

alertmanager-main
    app.kubernetes.io/version: 0.23.0


builder


cluster-monitoring-operator


default


deployer


kube-state-metrics
    app.kubernetes.io/version: 2.3.0


node-exporter
    app.kubernetes.io/version: 1.3.1


openshift-state-metrics


prometheus-adapter
    app.kubernetes.io/version: 0.9.1


prometheus-k8s
    app.kubernetes.io/version: 2.32.1


prometheus-operator
    app.kubernetes.io/version: 0.53.1


prometheus-operator-admission-webhook
    app.kubernetes.io/version: 0.55.1


telemeter-client


thanos-querier
    app.kubernetes.io/version: 0.23.1

Version-Release number of selected component (if applicable):
upgrade from 4.10.16 to 4.11.0-0.nightly-2022-06-06-201913

How reproducible:
always

Steps to Reproduce:
1. upgrade from 4.10.16 to 4.11.0-0.nightly-2022-06-06-201913, and compare the sa with a 4.11.0-0.nightly-2022-06-06-201913
2.
3.

Actual results:


Expected results:


Additional info:
since must-gather does not have serviceaccounts info, only provide serviceaccounts file here

Comment 2 Junqi Zhao 2022-06-10 11:12:56 UTC
4.11.0-0.nightly-2022-06-06-201913 sa version labels
# for i in $(oc -n openshift-monitoring get sa | grep -v NAME| awk '{print $1}'); do echo $i; oc -n openshift-monitoring get sa $i -oyaml | grep "app.kubernetes.io/version";echo -e "\n";done

alertmanager-main
    app.kubernetes.io/version: 0.24.0


builder


cluster-monitoring-operator


default


deployer


kube-state-metrics
    app.kubernetes.io/version: 2.4.2


node-exporter
    app.kubernetes.io/version: 1.3.1


openshift-state-metrics


prometheus-adapter
    app.kubernetes.io/version: 0.9.1


prometheus-k8s
    app.kubernetes.io/version: 2.35.0


prometheus-operator
    app.kubernetes.io/version: 0.55.1


prometheus-operator-admission-webhook
    app.kubernetes.io/version: 0.55.1


telemeter-client


thanos-querier
    app.kubernetes.io/version: 0.25.2

Comment 3 Simon Pasquier 2022-06-10 12:46:36 UTC
Good catch! This is even documented in the CMO code base with a potential fix.

https://github.com/openshift/cluster-monitoring-operator/blob/03e8db6a55d7ecd35a8c82909a94252441b363f6/pkg/client/client.go#L1451-L1466

Comment 9 Junqi Zhao 2022-07-08 13:56:28 UTC
upgrade from 4.10.0-0.nightly-2022-06-08-150219 to 4.11.0-0.nightly-2022-07-06-145812, note down the sa, then upgrade to 4.12.0-0.nightly-2022-07-08-015358, compare the sa files, the sa files are updated for 4.12 cluster, see the attached file

Comment 18 Junqi Zhao 2022-11-29 07:05:39 UTC
upgrade from 4.12.0-0.nightly-2022-11-28-145121 to 4.13.0-0.nightly-2022-11-28-190649, serviceaccounts are updated after upgrade, see the differences for prometheus-operator/prometheus-operator-admission-webhook before and after upgrade
$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.12.0-0.nightly-2022-11-28-145121   True        False         64m     Cluster version is 4.12.0-0.nightly-2022-11-28-145121
$ for i in $(oc -n openshift-monitoring get sa | grep -v NAME| awk '{print $1}'); do echo $i; oc -n openshift-monitoring get sa $i -oyaml | grep "app.kubernetes.io/version";done
alertmanager-main
    app.kubernetes.io/version: 0.24.0
kube-state-metrics
    app.kubernetes.io/version: 2.6.0
node-exporter
    app.kubernetes.io/version: 1.4.0
prometheus-adapter
    app.kubernetes.io/version: 0.10.0
prometheus-k8s
    app.kubernetes.io/version: 2.39.1
prometheus-operator
    app.kubernetes.io/version: 0.60.1
prometheus-operator-admission-webhook
    app.kubernetes.io/version: 0.60.1
thanos-querier
    app.kubernetes.io/version: 0.28.1

$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.13.0-0.nightly-2022-11-28-190649   True        False         3h3m    Cluster version is 4.13.0-0.nightly-2022-11-28-190649

$ for i in $(oc -n openshift-monitoring get sa | grep -v NAME| awk '{print $1}'); do echo $i; oc -n openshift-monitoring get sa $i -oyaml | grep "app.kubernetes.io/version";done
alertmanager-main
    app.kubernetes.io/version: 0.24.0
kube-state-metrics
    app.kubernetes.io/version: 2.6.0
node-exporter
    app.kubernetes.io/version: 1.4.0
prometheus-adapter
    app.kubernetes.io/version: 0.10.0
prometheus-k8s
    app.kubernetes.io/version: 2.39.1
prometheus-operator
    app.kubernetes.io/version: 0.61.1
prometheus-operator-admission-webhook
    app.kubernetes.io/version: 0.61.1
thanos-querier
    app.kubernetes.io/version: 0.28.1

Comment 21 errata-xmlrpc 2023-05-17 22:46:53 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.13.0 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:1326


Note You need to log in before you can comment on or make changes to this bug.