Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2097346

Summary:	Monitoring must-gather doesn't seem to be working anymore in 4.11
Product:	OpenShift Container Platform	Reporter:	Joao Marcal <jmarcal>
Component:	Monitoring	Assignee:	Joao Marcal <jmarcal>
Status:	CLOSED ERRATA	QA Contact:	hongyan li <hongyli>
Severity:	medium	Docs Contact:
Priority:	high
Version:	4.11	CC:	amuller, anpicker, juzhao, spasquie, wking
Target Milestone:	---
Target Release:	4.11.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-08-10 11:18:19 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Joao Marcal 2022-06-15 13:34:36 UTC

Description of problem:
Monitoring must-gather not working in 4.11 due to k8s 1.24 no longer generating legacy tokens for SA.

Old command: "oc sa get-token default"
New command: "oc create token default"

Problematic line: https://github.com/openshift/must-gather/blob/87c1fd3aee6a6ec7359406a14e1ad0d8d1f8784e/collection-scripts/gather_monitoring#L29

Related PR:
https://github.com/kubernetes/kubernetes/pull/108309

Version-Release number of selected component (if applicable):
4.11.0-0.nightly-2022-06-14-172335

How reproducible:
Always 

Steps to Reproduce:
1. oc sa get-token default
2.
3.

Actual results:
error: could not find a service account token for service account "default"

Expected results:
Token we could use

Additional info:

Comment 4 hongyan li 2022-06-22 02:34:56 UTC

Test with payload 4.11.0-0.nightly-2022-06-21-151125
% ls -alR must-gather.local.3955107449144106485/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-bea7860950acd0a1490e7882884673a6839a56319f6306be17d55850712a24aa/namespaces/openshift-monitoring|head
total 8
drwxr-xr-x  17 hongyli  staff   544 Jun 22 10:14 .
drwxr-xr-x  72 hongyli  staff  2304 Jun 22 10:13 ..
drwxr-xr-x   6 hongyli  staff   192 Jun 22 10:14 apps
drwxr-xr-x   3 hongyli  staff    96 Jun 22 10:14 apps.openshift.io
drwxr-xr-x   3 hongyli  staff    96 Jun 22 10:14 autoscaling
drwxr-xr-x   4 hongyli  staff   128 Jun 22 10:14 batch
drwxr-xr-x   4 hongyli  staff   128 Jun 22 10:14 build.openshift.io
drwxr-xr-x  10 hongyli  staff   320 Jun 22 10:14 core
drwxr-xr-x   3 hongyli  staff    96 Jun 22 10:14 discovery.k8s.io

Comment 5 Junqi Zhao 2022-06-22 04:18:25 UTC

checked with 4.11.0-0.nightly-2022-06-21-151125, could see the files under monitoring
# ls -alR must-gather.local.8175929882889163820/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-bea7860950acd0a1490e7882884673a6839a56319f6306be17d55850712a24aa/monitoring/
must-gather.local.8175929882889163820/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-bea7860950acd0a1490e7882884673a6839a56319f6306be17d55850712a24aa/monitoring/:
total 0
drwxr-xr-x.  4 root root  44 Jun 21 21:13 .
drwxrwxrwx. 12 root root 277 Jun 21 21:13 ..
drwxr-xr-x.  2 root root  46 Jun 21 21:13 alertmanager
drwxr-xr-x.  5 root root 160 Jun 21 21:13 prometheus

must-gather.local.8175929882889163820/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-bea7860950acd0a1490e7882884673a6839a56319f6306be17d55850712a24aa/monitoring/alertmanager:
total 4
drwxr-xr-x. 2 root root   46 Jun 21 21:13 .
drwxr-xr-x. 4 root root   44 Jun 21 21:13 ..
-rw-r--r--. 1 root root 1825 Jun 21 21:28 status.json
-rw-r--r--. 1 root root    0 Jun 21 21:28 status.stderr

must-gather.local.8175929882889163820/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-bea7860950acd0a1490e7882884673a6839a56319f6306be17d55850712a24aa/monitoring/prometheus:
total 240
drwxr-xr-x. 5 root root    160 Jun 21 21:13 .
drwxr-xr-x. 4 root root     44 Jun 21 21:13 ..
-rw-r--r--. 1 root root   3864 Jun 21 21:28 alertmanagers.json
-rw-r--r--. 1 root root      0 Jun 21 21:28 alertmanagers.stderr
drwxr-xr-x. 3 root root     76 Jun 21 21:13 prometheus-k8s-0
drwxr-xr-x. 3 root root     76 Jun 21 21:13 prometheus-k8s-1
-rw-r--r--. 1 root root 241451 Jun 21 21:28 rules.json
-rw-r--r--. 1 root root      0 Jun 21 21:28 rules.stderr
drwxr-xr-x. 2 root root    134 Jun 21 21:13 status

must-gather.local.8175929882889163820/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-bea7860950acd0a1490e7882884673a6839a56319f6306be17d55850712a24aa/monitoring/prometheus/prometheus-k8s-0:
total 644
drwxr-xr-x. 3 root root     76 Jun 21 21:13 .
drwxr-xr-x. 5 root root    160 Jun 21 21:13 ..
-rw-r--r--. 1 root root 657118 Jun 21 21:28 active-targets.json
-rw-r--r--. 1 root root      0 Jun 21 21:28 active-targets.stderr
drwxr-xr-x. 2 root root     42 Jun 21 21:13 status

must-gather.local.8175929882889163820/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-bea7860950acd0a1490e7882884673a6839a56319f6306be17d55850712a24aa/monitoring/prometheus/prometheus-k8s-0/status:
total 4
drwxr-xr-x. 2 root root   42 Jun 21 21:13 .
drwxr-xr-x. 3 root root   76 Jun 21 21:13 ..
-rw-r--r--. 1 root root 1988 Jun 21 21:28 tsdb.json
-rw-r--r--. 1 root root    0 Jun 21 21:28 tsdb.stderr

must-gather.local.8175929882889163820/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-bea7860950acd0a1490e7882884673a6839a56319f6306be17d55850712a24aa/monitoring/prometheus/prometheus-k8s-1:
total 644
drwxr-xr-x. 3 root root     76 Jun 21 21:13 .
drwxr-xr-x. 5 root root    160 Jun 21 21:13 ..
-rw-r--r--. 1 root root 657121 Jun 21 21:28 active-targets.json
-rw-r--r--. 1 root root      0 Jun 21 21:28 active-targets.stderr
drwxr-xr-x. 2 root root     42 Jun 21 21:13 status

must-gather.local.8175929882889163820/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-bea7860950acd0a1490e7882884673a6839a56319f6306be17d55850712a24aa/monitoring/prometheus/prometheus-k8s-1/status:
total 4
drwxr-xr-x. 2 root root   42 Jun 21 21:13 .
drwxr-xr-x. 3 root root   76 Jun 21 21:13 ..
-rw-r--r--. 1 root root 1988 Jun 21 21:28 tsdb.json
-rw-r--r--. 1 root root    0 Jun 21 21:28 tsdb.stderr

must-gather.local.8175929882889163820/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-bea7860950acd0a1490e7882884673a6839a56319f6306be17d55850712a24aa/monitoring/prometheus/status:
total 248
drwxr-xr-x. 2 root root    134 Jun 21 21:13 .
drwxr-xr-x. 5 root root    160 Jun 21 21:13 ..
-rw-r--r--. 1 root root 244653 Jun 21 21:28 config.json
-rw-r--r--. 1 root root      0 Jun 21 21:28 config.stderr
-rw-r--r--. 1 root root   2100 Jun 21 21:28 flags.json
-rw-r--r--. 1 root root      0 Jun 21 21:28 flags.stderr
-rw-r--r--. 1 root root    266 Jun 21 21:28 runtimeinfo.json
-rw-r--r--. 1 root root      0 Jun 21 21:28 runtimeinfo.stderr

Comment 9 errata-xmlrpc 2022-08-10 11:18:19 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069