Bug 1955586
| Summary: | ThanosSidecarUnhealthy will never fire if the sidecar is never healthy. | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Damien Grisonnet <dgrisonn> |
| Component: | Monitoring | Assignee: | Arunprasad Rajkumar <arajkuma> |
| Status: | CLOSED ERRATA | QA Contact: | hongyan li <hongyli> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 4.8 | CC: | anpicker, erooth, hongyli, lcosic, spasquie |
| Target Milestone: | --- | ||
| Target Release: | 4.9.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-10-18 17:30:35 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1943565 | ||
The upstream fix[1] has been pulled as part of https://github.com/openshift/cluster-monitoring-operator/pull/1244 [1] https://github.com/thanos-io/thanos/pull/4342 Test with payload 4.8.0-0.nightly-2021-07-04-112043
# oc get cm prometheus-k8s-rulefiles-0 -n openshift-monitoring -oyaml|grep -C10 'Thanos Sidecar is unhealthy'
summary: Thanos Sidecar bucket operations are failing
expr: |
rate(thanos_objstore_bucket_operation_failures_total{job=~"prometheus-(k8s|user-workload)-thanos-sidecar"}[5m]) > 0
for: 1h
labels:
severity: warning
- alert: ThanosSidecarUnhealthy
annotations:
description: Thanos Sidecar {{$labels.job}} {{$labels.pod}} is unhealthy for
more than {{ $value }} seconds.
summary: Thanos Sidecar is unhealthy.
expr: |
time() - max(timestamp(thanos_sidecar_last_heartbeat_success_time_seconds{job=~"prometheus-(k8s|user-workload)-thanos-sidecar"})) by (job,pod) >= 240
for: 1h
labels:
severity: warning
openshift-monitoring-prometheus-operator-rules.yaml: |
groups:
- name: prometheus-operator
rules:
- alert: PrometheusOperatorListErrors
Will check in 4.9 payload later than 4.9.0-0.nightly-2021-07-02-022316 Test with payload 4.9.0-0.nightly-2021-07-04-140102, the alert rule is correct
$ oc get cm prometheus-k8s-rulefiles-0 -n openshift-monitoring -oyaml|grep -C10 'Thanos Sidecar is unhealthy'
- alert: ThanosSidecarUnhealthy
annotations:
description: Thanos Sidecar {{$labels.instance}} is unhealthy for more than
{{$value}} seconds.
summary: Thanos Sidecar is unhealthy.
expr: |
time() - max by (job, instance) (thanos_sidecar_last_heartbeat_success_time_seconds{job=~"prometheus-(k8s|user-workload)-thanos-sidecar"}) >= 240
for: 1h
labels:
severity: warning
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:3759 |
Description of problem: ThanosSidecarUnhealthy will never fire if the sidecar is never healthy. The problems lies in the Prometheus query used by the alert: ``` time() - max(timestamp(thanos_sidecar_last_heartbeat_success_time_seconds{job="thanos-sidecar"})) by (job,pod) >= 240 ``` If the Thanos sidecar is unheathy from start up and remain in this state, the `thanos_sidecar_last_heartbeat_success_time_seconds` metric will not be initialized so calling `timestamp` on it will not return any value. As a result the `ThanosSidecarUnhealthy` alert will not fire even though the sidecar is unhealthy.