Bug 2004585
| Summary: | prometheus-k8s-0 cpu usage keeps increasing for the first 3 days | ||||||
|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | yliu1 | ||||
| Component: | kube-apiserver | Assignee: | Luis Sanchez <sanchezl> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Ke Wang <kewang> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | medium | ||||||
| Version: | 4.9 | CC: | akashem, amuller, anpicker, aos-bugs, erooth, hongyli, imiller, mfojtik, pgough, xxia | ||||
| Target Milestone: | --- | ||||||
| Target Release: | 4.10.0 | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | No Doc Update | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2022-03-10 16:10:42 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 2012346 | ||||||
| Attachments: |
|
||||||
Looking at the attached screenshot, I don't see any major issues. What alarms you about the situation and what results/threshold would you expect to see instead? We are expecting < 300 mc based on previous measurements. Verified on 4.9.1. prometheus-k8s-0 pod cpu usage has been relatively stable in the past 4 days since fresh deployment. It uses ~0.17 cpu in steady state. Verified on 4.10 nightly payload, prometheus-k8s-0 pod cpu usage is around 0.1 with some workload pods created by clusterbuster tool(./clusterbuster -P server -b 5 -p 2 -D .01 -M 1 -N 4 -r 4 -d 2 -c 6 -m 1000 -v -s 5 -x) in the past 2 days. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056 |
Created attachment 1823359 [details] prometheus pod cpu usage for the first 5 days Description of problem: On a freshly deployed SNO cluster, cpu usage of prometheus pod will keep increase from ~0.15 to ~0.45 for the first 3 days. It takes too long to settle. Version-Release number of selected component (if applicable): 4.9.0 rc.0 How reproducible: Steps to Reproduce: 1. Deploy a sno cluster 2. Deploy some workload pods 3. Observe cpu usage of each platform pod via prometheus queries for a week Actual results: 3. prometheus-k8s-0 cpu usage kept increasing steadily for the first 3 days from ~0.15 to ~0.45, and settled on the 4th day. Expected results: 3. It should settle earlier with lower cpu usage (maybe). Additional info: Prom chart for prometheus-k8s-0 pod is attached