1928668 – Prometheus is collecting metrics for completed pods

Bug 1928668 - Prometheus is collecting metrics for completed pods

Summary: Prometheus is collecting metrics for completed pods

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Monitoring
Sub Component:
Version:	4.8
Hardware:	Unspecified
OS:	Unspecified
Priority:	low
Severity:	low
Target Milestone:	---
Target Release:	4.9.0
Assignee:	Damien Grisonnet
QA Contact:	hongyan li
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2021-02-15 10:24 UTC by Dhruv Gautam
Modified:	2022-11-15 07:27 UTC (History)
CC List:	8 users (show)
Fixed In Version:
Doc Type:	No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed:	2021-10-18 17:29:20 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Github	kubernetes-sigs metrics-server issues 734	None	open	Release v0.5.0	2021-04-30 13:19:57 UTC
Github	kubernetes-sigs metrics-server pull 651	None	closed	Watch only metadata for pods	2021-04-26 09:34:27 UTC
Github	kubernetes-sigs prometheus-adapter pull 405	None	open	Filter non-running pods	2021-06-01 17:05:59 UTC
Github	openshift cluster-monitoring-operator pull 1325	None	None	None	2021-08-18 09:15:25 UTC
Github	openshift k8s-prometheus-adapter pull 53	None	None	None	2021-08-18 09:15:25 UTC
Red Hat Product Errata	RHSA-2021:3759	None	None	None	2021-10-18 17:29:39 UTC

Description Dhruv Gautam 2021-02-15 10:24:25 UTC

Description of problem:
Prometheus is gather metrics for the pods which are in completed/succeeded state.
prometheus is producing loads of logs within a matter of minutes and elasticsearch is filling up exponentially.

The errors are not directly harmful to the environment but it should not be reported by prometheus.

2021-02-02T10:22:42.982873977Z E0202 10:22:42.982863       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-apiserver/revision-pruner-16-ip-10-0-156-87.eu-central-1.compute.internal, skipping
2021-02-02T10:22:42.982873977Z E0202 10:22:42.982870       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-apiserver/revision-pruner-16-ip-10-0-162-147.eu-central-1.compute.internal, skipping
2021-02-02T10:22:42.982881426Z E0202 10:22:42.982875       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-apiserver/revision-pruner-16-ip-10-0-202-64.eu-central-1.compute.internal, skipping
2021-02-02T10:22:42.982881426Z E0202 10:22:42.982879       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-apiserver/revision-pruner-17-ip-10-0-156-87.eu-central-1.compute.internal, skipping
2021-02-02T10:22:42.982889626Z E0202 10:22:42.982884       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-apiserver/revision-pruner-17-ip-10-0-162-147.eu-central-1.compute.internal, skipping

Version-Release number of selected component (if applicable):
4.6.8

How reproducible:
NA

Steps to Reproduce:
1.
2.
3.

Actual results:
2021-02-02T10:22:42.982873977Z E0202 10:22:42.982863       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-apiserver/revision-pruner-16-ip-10-0-156-87.eu-central-1.compute.internal, skipping


Expected results:
No error messages should be there for completed/succeeded pods.

Additional info:

Comment 2 Pawel Krupa 2021-02-15 10:48:04 UTC

Just to note - we have merged a fix for this in an upstream project in https://github.com/kubernetes-monitoring/kubernetes-mixin/pull/552. It should trickle down into OpenShift 4.8.

Comment 12 hongyan li 2021-03-02 07:08:45 UTC

encounter the issue in 4.6 cluster
[hongyli@hongyli-fed Downloads]$ oc -n openshift-monitoring logs prometheus-adapter-7c46fc8dfb-4xt9j
I0302 05:14:29.793955       1 adapter.go:94] successfully using in-cluster auth
I0302 05:14:31.593087       1 dynamic_cafile_content.go:166] Starting request-header::/etc/tls/private/requestheader-client-ca-file
I0302 05:14:31.593139       1 dynamic_cafile_content.go:166] Starting client-ca-bundle::/etc/tls/private/client-ca-file
I0302 05:14:31.593895       1 dynamic_serving_content.go:129] Starting serving-cert::/etc/tls/private/tls.crt::/etc/tls/private/tls.key
I0302 05:14:31.595559       1 secure_serving.go:178] Serving securely on [::]:6443
I0302 05:14:31.595875       1 tlsconfig.go:219] Starting DynamicServingCertificateController
E0302 05:19:44.698614       1 authentication.go:104] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, square/go-jose: error in cryptographic primitive, token lookup failed]]
E0302 05:20:14.695671       1 authentication.go:104] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, square/go-jose: error in cryptographic primitive, token lookup failed]]
E0302 05:20:44.716807       1 authentication.go:104] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, square/go-jose: error in cryptographic primitive, token lookup failed]]
E0302 05:21:14.694884       1 authentication.go:104] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, square/go-jose: error in cryptographic primitive, token lookup failed]]
E0302 05:21:44.695863       1 authentication.go:104] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, square/go-jose: error in cryptographic primitive, token lookup failed]]
E0302 05:22:14.692911       1 authentication.go:104] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, square/go-jose: error in cryptographic primitive, token lookup failed]]
E0302 06:57:35.600598       1 provider.go:187] unable to fetch CPU metrics for pod openshift-etcd/installer-2-ip-10-0-132-218.us-west-2.compute.internal, skipping
E0302 06:57:35.600626       1 provider.go:187] unable to fetch CPU metrics for pod openshift-etcd/installer-2-ip-10-0-142-172.us-west-2.compute.internal, skipping
E0302 06:57:35.600634       1 provider.go:187] unable to fetch CPU metrics for pod openshift-etcd/installer-2-ip-10-0-222-158.us-west-2.compute.internal, skipping
E0302 06:57:35.600641       1 provider.go:187] unable to fetch CPU metrics for pod openshift-etcd/installer-3-ip-10-0-132-218.us-west-2.compute.internal, skipping
E0302 06:57:35.600650       1 provider.go:187] unable to fetch CPU metrics for pod openshift-etcd/installer-3-ip-10-0-142-172.us-west-2.compute.internal, skipping
E0302 06:57:35.600657       1 provider.go:187] unable to fetch CPU metrics for pod openshift-etcd/installer-3-ip-10-0-222-158.us-west-2.compute.internal, skipping
E0302 06:57:35.600664       1 provider.go:187] unable to fetch CPU metrics for pod openshift-etcd/revision-pruner-2-ip-10-0-132-218.us-west-2.compute.internal, skipping
E0302 06:57:35.600671       1 provider.go:187] unable to fetch CPU metrics for pod openshift-etcd/revision-pruner-2-ip-10-0-142-172.us-west-2.compute.internal, skipping
E0302 06:57:35.600680       1 provider.go:187] unable to fetch CPU metrics for pod openshift-etcd/revision-pruner-2-ip-10-0-222-158.us-west-2.compute.internal, skipping
E0302 06:57:35.600692       1 provider.go:187] unable to fetch CPU metrics for pod openshift-etcd/revision-pruner-3-ip-10-0-132-218.us-west-2.compute.internal, skipping
E0302 06:57:35.600704       1 provider.go:187] unable to fetch CPU metrics for pod openshift-etcd/revision-pruner-3-ip-10-0-142-172.us-west-2.compute.internal, skipping
E0302 06:57:35.600715       1 provider.go:187] unable to fetch CPU metrics for pod openshift-etcd/revision-pruner-3-ip-10-0-222-158.us-west-2.compute.internal, skipping
E0302 06:57:35.600763       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-apiserver/installer-2-ip-10-0-132-218.us-west-2.compute.internal, skipping
E0302 06:57:35.600771       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-apiserver/installer-2-ip-10-0-142-172.us-west-2.compute.internal, skipping
E0302 06:57:35.600778       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-apiserver/installer-2-ip-10-0-222-158.us-west-2.compute.internal, skipping
E0302 06:57:35.600785       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-apiserver/installer-3-ip-10-0-222-158.us-west-2.compute.internal, skipping
E0302 06:57:35.600792       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-apiserver/installer-4-ip-10-0-222-158.us-west-2.compute.internal, skipping
E0302 06:57:35.600799       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-apiserver/installer-5-ip-10-0-132-218.us-west-2.compute.internal, skipping
E0302 06:57:35.600806       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-apiserver/installer-5-ip-10-0-142-172.us-west-2.compute.internal, skipping
E0302 06:57:35.600813       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-apiserver/installer-5-ip-10-0-222-158.us-west-2.compute.internal, skipping
E0302 06:57:35.600849       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-apiserver/revision-pruner-2-ip-10-0-132-218.us-west-2.compute.internal, skipping
E0302 06:57:35.600865       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-apiserver/revision-pruner-2-ip-10-0-142-172.us-west-2.compute.internal, skipping
E0302 06:57:35.600872       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-apiserver/revision-pruner-2-ip-10-0-222-158.us-west-2.compute.internal, skipping
E0302 06:57:35.600884       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-apiserver/revision-pruner-3-ip-10-0-222-158.us-west-2.compute.internal, skipping
E0302 06:57:35.600896       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-apiserver/revision-pruner-4-ip-10-0-222-158.us-west-2.compute.internal, skipping
E0302 06:57:35.600909       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-apiserver/revision-pruner-5-ip-10-0-132-218.us-west-2.compute.internal, skipping
E0302 06:57:35.600921       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-apiserver/revision-pruner-5-ip-10-0-142-172.us-west-2.compute.internal, skipping
E0302 06:57:35.600933       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-apiserver/revision-pruner-5-ip-10-0-222-158.us-west-2.compute.internal, skipping
E0302 06:57:35.600956       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-controller-manager/installer-4-ip-10-0-132-218.us-west-2.compute.internal, skipping
E0302 06:57:35.600968       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-controller-manager/installer-4-ip-10-0-142-172.us-west-2.compute.internal, skipping
E0302 06:57:35.600977       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-controller-manager/installer-4-ip-10-0-222-158.us-west-2.compute.internal, skipping
E0302 06:57:35.600985       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-controller-manager/installer-5-ip-10-0-132-218.us-west-2.compute.internal, skipping
E0302 06:57:35.600992       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-controller-manager/installer-5-ip-10-0-142-172.us-west-2.compute.internal, skipping
E0302 06:57:35.600999       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-controller-manager/installer-5-ip-10-0-222-158.us-west-2.compute.internal, skipping
E0302 06:57:35.601006       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-controller-manager/installer-6-ip-10-0-132-218.us-west-2.compute.internal, skipping
E0302 06:57:35.601013       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-controller-manager/installer-6-ip-10-0-142-172.us-west-2.compute.internal, skipping
E0302 06:57:35.601020       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-controller-manager/installer-6-ip-10-0-222-158.us-west-2.compute.internal, skipping
E0302 06:57:35.601044       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-controller-manager/revision-pruner-4-ip-10-0-132-218.us-west-2.compute.internal, skipping
E0302 06:57:35.601065       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-controller-manager/revision-pruner-4-ip-10-0-142-172.us-west-2.compute.internal, skipping
E0302 06:57:35.601077       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-controller-manager/revision-pruner-4-ip-10-0-222-158.us-west-2.compute.internal, skipping
E0302 06:57:35.601118       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-controller-manager/revision-pruner-5-ip-10-0-132-218.us-west-2.compute.internal, skipping
E0302 06:57:35.601153       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-controller-manager/revision-pruner-5-ip-10-0-142-172.us-west-2.compute.internal, skipping
E0302 06:57:35.601179       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-controller-manager/revision-pruner-5-ip-10-0-222-158.us-west-2.compute.internal, skipping
E0302 06:57:35.601192       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-controller-manager/revision-pruner-6-ip-10-0-132-218.us-west-2.compute.internal, skipping
E0302 06:57:35.601206       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-controller-manager/revision-pruner-6-ip-10-0-142-172.us-west-2.compute.internal, skipping
E0302 06:57:35.601220       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-controller-manager/revision-pruner-6-ip-10-0-222-158.us-west-2.compute.internal, skipping
E0302 06:57:35.601237       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-scheduler/installer-2-ip-10-0-222-158.us-west-2.compute.internal, skipping
E0302 06:57:35.601249       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-scheduler/installer-3-ip-10-0-142-172.us-west-2.compute.internal, skipping
E0302 06:57:35.601260       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-scheduler/installer-4-ip-10-0-132-218.us-west-2.compute.internal, skipping
E0302 06:57:35.601272       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-scheduler/installer-5-ip-10-0-222-158.us-west-2.compute.internal, skipping
E0302 06:57:35.601285       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-scheduler/installer-6-ip-10-0-132-218.us-west-2.compute.internal, skipping
E0302 06:57:35.601298       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-scheduler/installer-6-ip-10-0-142-172.us-west-2.compute.internal, skipping
E0302 06:57:35.601333       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-scheduler/installer-6-ip-10-0-222-158.us-west-2.compute.internal, skipping
E0302 06:57:35.601356       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-scheduler/installer-7-ip-10-0-132-218.us-west-2.compute.internal, skipping
E0302 06:57:35.601369       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-scheduler/installer-7-ip-10-0-142-172.us-west-2.compute.internal, skipping
E0302 06:57:35.601408       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-scheduler/installer-7-ip-10-0-222-158.us-west-2.compute.internal, skipping
E0302 06:57:35.601472       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-scheduler/revision-pruner-2-ip-10-0-222-158.us-west-2.compute.internal, skipping
E0302 06:57:35.601517       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-scheduler/revision-pruner-3-ip-10-0-142-172.us-west-2.compute.internal, skipping
E0302 06:57:35.601530       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-scheduler/revision-pruner-4-ip-10-0-132-218.us-west-2.compute.internal, skipping
E0302 06:57:35.601543       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-scheduler/revision-pruner-5-ip-10-0-222-158.us-west-2.compute.internal, skipping
E0302 06:57:35.601556       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-scheduler/revision-pruner-6-ip-10-0-132-218.us-west-2.compute.internal, skipping
E0302 06:57:35.601570       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-scheduler/revision-pruner-6-ip-10-0-142-172.us-west-2.compute.internal, skipping
E0302 06:57:35.601583       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-scheduler/revision-pruner-6-ip-10-0-222-158.us-west-2.compute.internal, skipping
E0302 06:57:35.601594       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-scheduler/revision-pruner-7-ip-10-0-132-218.us-west-2.compute.internal, skipping
E0302 06:57:35.601607       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-scheduler/revision-pruner-7-ip-10-0-142-172.us-west-2.compute.internal, skipping
E0302 06:57:35.601619       1 provider.go:187] unable to fetch CPU metrics for pod openshift-kube-scheduler/revision-pruner-7-ip-10-0-222-158.us-west-2.compute.internal, skipping
[hongyli@hongyli-fed Downloads]$ oc get co
NAME                                       VERSION                                           AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.6.0-0.ci.test-2021-03-02-045033-ci-ln-piym2wt   True        False         False      100m
cloud-credential                           4.6.0-0.ci.test-2021-03-02-045033-ci-ln-piym2wt   True        False         False      124m
cluster-autoscaler                         4.6.0-0.ci.test-2021-03-02-045033-ci-ln-piym2wt   True        False         False      118m
config-operator                            4.6.0-0.ci.test-2021-03-02-045033-ci-ln-piym2wt   True        False         False      119m
console                                    4.6.0-0.ci.test-2021-03-02-045033-ci-ln-piym2wt   True        False         False      104m
csi-snapshot-controller                    4.6.0-0.ci.test-2021-03-02-045033-ci-ln-piym2wt   True        False         False      119m
dns                                        4.6.0-0.ci.test-2021-03-02-045033-ci-ln-piym2wt   True        False         False      118m
etcd                                       4.6.0-0.ci.test-2021-03-02-045033-ci-ln-piym2wt   True        False         False      117m
image-registry                             4.6.0-0.ci.test-2021-03-02-045033-ci-ln-piym2wt   True        False         False      110m
ingress                                    4.6.0-0.ci.test-2021-03-02-045033-ci-ln-piym2wt   True        False         False      110m
insights                                   4.6.0-0.ci.test-2021-03-02-045033-ci-ln-piym2wt   True        False         False      119m
kube-apiserver                             4.6.0-0.ci.test-2021-03-02-045033-ci-ln-piym2wt   True        False         False      117m
kube-controller-manager                    4.6.0-0.ci.test-2021-03-02-045033-ci-ln-piym2wt   True        False         False      117m
kube-scheduler                             4.6.0-0.ci.test-2021-03-02-045033-ci-ln-piym2wt   True        False         False      116m
kube-storage-version-migrator              4.6.0-0.ci.test-2021-03-02-045033-ci-ln-piym2wt   True        False         False      109m
machine-api                                4.6.0-0.ci.test-2021-03-02-045033-ci-ln-piym2wt   True        False         False      114m
machine-approver                           4.6.0-0.ci.test-2021-03-02-045033-ci-ln-piym2wt   True        False         False      118m
machine-config                             4.6.0-0.ci.test-2021-03-02-045033-ci-ln-piym2wt   True        False         False      117m
marketplace                                4.6.0-0.ci.test-2021-03-02-045033-ci-ln-piym2wt   True        False         False      117m
monitoring                                 4.6.0-0.ci.test-2021-03-02-045033-ci-ln-piym2wt   True        False         False      104m
network                                    4.6.0-0.ci.test-2021-03-02-045033-ci-ln-piym2wt   True        False         False      120m
node-tuning                                4.6.0-0.ci.test-2021-03-02-045033-ci-ln-piym2wt   True        False         False      119m
openshift-apiserver                        4.6.0-0.ci.test-2021-03-02-045033-ci-ln-piym2wt   True        False         False      114m
openshift-controller-manager               4.6.0-0.ci.test-2021-03-02-045033-ci-ln-piym2wt   True        False         False      117m
openshift-samples                          4.6.0-0.ci.test-2021-03-02-045033-ci-ln-piym2wt   True        False         False      114m
operator-lifecycle-manager                 4.6.0-0.ci.test-2021-03-02-045033-ci-ln-piym2wt   True        False         False      118m
operator-lifecycle-manager-catalog         4.6.0-0.ci.test-2021-03-02-045033-ci-ln-piym2wt   True        False         False      118m
operator-lifecycle-manager-packageserver   4.6.0-0.ci.test-2021-03-02-045033-ci-ln-piym2wt   True        False         False      114m
service-ca                                 4.6.0-0.ci.test-2021-03-02-045033-ci-ln-piym2wt   True        False         False      119m
storage                                    4.6.0-0.ci.test-2021-03-02-045033-ci-ln-piym2wt   True        False         False      118m

Comment 13 hongyan li 2021-03-02 07:11:26 UTC

The the prometheus adapter pod has no issue
oc -n openshift-monitoring logs prometheus-adapter-7c46fc8dfb-rvzzh
I0302 05:14:29.794739       1 adapter.go:94] successfully using in-cluster auth
I0302 05:14:31.215270       1 dynamic_cafile_content.go:166] Starting request-header::/etc/tls/private/requestheader-client-ca-file
I0302 05:14:31.215337       1 dynamic_cafile_content.go:166] Starting client-ca-bundle::/etc/tls/private/client-ca-file
I0302 05:14:31.215795       1 dynamic_serving_content.go:129] Starting serving-cert::/etc/tls/private/tls.crt::/etc/tls/private/tls.key
I0302 05:14:31.217684       1 secure_serving.go:178] Serving securely on [::]:6443
I0302 05:14:31.221153       1 tlsconfig.go:219] Starting DynamicServingCertificateController

Comment 14 hongyan li 2021-03-02 08:40:37 UTC

Saw issue in comments 12 on the cluster with payload 4.6.0-0.ci.test-2021-03-02-045033-ci-ln-piym2wt and PR openshift/node_exporter#83

Comment 19 Damien Grisonnet 2021-06-03 14:22:24 UTC

The fix has been merged upstream and will be brought downstream with the next release of prometheus-adapter. It will most likely happen during the 4.9 timeframe.

Comment 23 Damien Grisonnet 2021-09-03 07:39:11 UTC

Moving to MODIFIED state since https://github.com/openshift/cluster-monitoring-operator/pull/1325 and https://github.com/openshift/k8s-prometheus-adapter/pull/53 have been merged.

Comment 24 Junqi Zhao 2021-09-09 02:40:29 UTC

is the Target Release 4.9.0 or 4.10.0, I see the fix is in 4.9 and 4.10

Comment 25 Damien Grisonnet 2021-09-09 08:43:12 UTC

Target release is 4.9.0. For some reason, the bug wasn't moved from MODIFIED to ON_QA automatically as I would have expected.

Comment 26 Junqi Zhao 2021-09-09 11:09:57 UTC

checked with 4.9.0-0.nightly-2021-09-08-233235,no error for the completed pods
# oc -n openshift-kube-scheduler get pod
NAME                                                                                READY   STATUS      RESTARTS   AGE
installer-2-master-02.pamoedo-dualtest23.qe.devcluster.openshift.com                0/1     Completed   0          148m
installer-3-master-02.pamoedo-dualtest23.qe.devcluster.openshift.com                0/1     Completed   0          148m
installer-4-master-00.pamoedo-dualtest23.qe.devcluster.openshift.com                0/1     Completed   0          146m
installer-4-master-01.pamoedo-dualtest23.qe.devcluster.openshift.com                0/1     Completed   0          145m
installer-4-master-02.pamoedo-dualtest23.qe.devcluster.openshift.com                0/1     Completed   0          147m
installer-5-master-02.pamoedo-dualtest23.qe.devcluster.openshift.com                0/1     Completed   0          143m
installer-6-master-02.pamoedo-dualtest23.qe.devcluster.openshift.com                0/1     Completed   0          142m
installer-7-master-00.pamoedo-dualtest23.qe.devcluster.openshift.com                0/1     Completed   0          140m
installer-7-master-01.pamoedo-dualtest23.qe.devcluster.openshift.com                0/1     Completed   0          139m
installer-7-master-02.pamoedo-dualtest23.qe.devcluster.openshift.com                0/1     Completed   0          141m
openshift-kube-scheduler-master-00.pamoedo-dualtest23.qe.devcluster.openshift.com   3/3     Running     0          140m
openshift-kube-scheduler-master-01.pamoedo-dualtest23.qe.devcluster.openshift.com   3/3     Running     0          139m
openshift-kube-scheduler-master-02.pamoedo-dualtest23.qe.devcluster.openshift.com   3/3     Running     0          141m
revision-pruner-4-master-00.pamoedo-dualtest23.qe.devcluster.openshift.com          0/1     Completed   0          145m
revision-pruner-4-master-01.pamoedo-dualtest23.qe.devcluster.openshift.com          0/1     Completed   0          143m
revision-pruner-4-master-02.pamoedo-dualtest23.qe.devcluster.openshift.com          0/1     Completed   0          146m
revision-pruner-7-master-00.pamoedo-dualtest23.qe.devcluster.openshift.com          0/1     Completed   0          139m
revision-pruner-7-master-01.pamoedo-dualtest23.qe.devcluster.openshift.com          0/1     Completed   0          138m
revision-pruner-7-master-02.pamoedo-dualtest23.qe.devcluster.openshift.com          0/1     Completed   0          140m

# oc adm top pod -n openshift-kube-scheduler
NAME                                                                                CPU(cores)   MEMORY(bytes)   
openshift-kube-scheduler-master-00.pamoedo-dualtest23.qe.devcluster.openshift.com   5m           127Mi           
openshift-kube-scheduler-master-01.pamoedo-dualtest23.qe.devcluster.openshift.com   3m           141Mi           
openshift-kube-scheduler-master-02.pamoedo-dualtest23.qe.devcluster.openshift.com   4m           156Mi       

# oc -n openshift-monitoring logs prometheus-adapter-bc6cc65df-64vxk
I0909 08:41:28.058691       1 adapter.go:108] successfully using in-cluster auth
I0909 08:41:28.713173       1 dynamic_cafile_content.go:155] "Starting controller" name="request-header::/etc/tls/private/requestheader-client-ca-file"
I0909 08:41:28.713195       1 dynamic_cafile_content.go:155] "Starting controller" name="client-ca-bundle::/etc/tls/private/client-ca-file"
I0909 08:41:28.714249       1 dynamic_serving_content.go:129] "Starting controller" name="serving-cert::/etc/tls/private/tls.crt::/etc/tls/private/tls.key"
I0909 08:41:28.715124       1 secure_serving.go:256] Serving securely on [::]:6443
I0909 08:41:28.715161       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"

Comment 31 errata-xmlrpc 2021-10-18 17:29:20 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759

Note You need to log in before you can comment on or make changes to this bug.