Bug 1935586

Summary: prometheus liveness probes cause issues while replaying WAL
Product: OpenShift Container Platform Reporter: Sergiusz Urbaniak <surbania>
Component: MonitoringAssignee: Jan Fajerski <jfajersk>
Status: CLOSED ERRATA QA Contact: Junqi Zhao <juzhao>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.6CC: alegrand, anpicker, erooth, juzhao, kakkoyun, lcosic, pkrupa, surbania
Target Milestone: ---   
Target Release: 4.6.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: livenessProbe fails when prometheus is under heavy load, such as WAL replay. Consequence: Prometheus gets restarted, exasperating the load issue due to repeated WAL replay. Fix: Remove livenessProbe. Result: Heavy load no longer causes the livenessProbe to fail in turn avoiding restart loops.
Story Points: ---
Clone Of: 1935585 Environment:
Last Closed: 2021-03-25 04:45:12 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1935585    
Bug Blocks:    

Comment 1 Junqi Zhao 2021-03-09 08:38:10 UTC
tested with the PR, livenessProbe is removed from prometheus container

Comment 6 errata-xmlrpc 2021-03-25 04:45:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6.22 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:0825