Description of problem: ThanosSidecarUnhealthy will never fire if the sidecar is never healthy. The problems lies in the Prometheus query used by the alert: ``` time() - max(timestamp(thanos_sidecar_last_heartbeat_success_time_seconds{job="thanos-sidecar"})) by (job,pod) >= 240 ``` If the Thanos sidecar is unheathy from start up and remain in this state, the `thanos_sidecar_last_heartbeat_success_time_seconds` metric will not be initialized so calling `timestamp` on it will not return any value. As a result the `ThanosSidecarUnhealthy` alert will not fire even though the sidecar is unhealthy.
The upstream fix[1] has been pulled as part of https://github.com/openshift/cluster-monitoring-operator/pull/1244 [1] https://github.com/thanos-io/thanos/pull/4342
Test with payload 4.8.0-0.nightly-2021-07-04-112043 # oc get cm prometheus-k8s-rulefiles-0 -n openshift-monitoring -oyaml|grep -C10 'Thanos Sidecar is unhealthy' summary: Thanos Sidecar bucket operations are failing expr: | rate(thanos_objstore_bucket_operation_failures_total{job=~"prometheus-(k8s|user-workload)-thanos-sidecar"}[5m]) > 0 for: 1h labels: severity: warning - alert: ThanosSidecarUnhealthy annotations: description: Thanos Sidecar {{$labels.job}} {{$labels.pod}} is unhealthy for more than {{ $value }} seconds. summary: Thanos Sidecar is unhealthy. expr: | time() - max(timestamp(thanos_sidecar_last_heartbeat_success_time_seconds{job=~"prometheus-(k8s|user-workload)-thanos-sidecar"})) by (job,pod) >= 240 for: 1h labels: severity: warning openshift-monitoring-prometheus-operator-rules.yaml: | groups: - name: prometheus-operator rules: - alert: PrometheusOperatorListErrors
Will check in 4.9 payload later than 4.9.0-0.nightly-2021-07-02-022316
Test with payload 4.9.0-0.nightly-2021-07-04-140102, the alert rule is correct $ oc get cm prometheus-k8s-rulefiles-0 -n openshift-monitoring -oyaml|grep -C10 'Thanos Sidecar is unhealthy' - alert: ThanosSidecarUnhealthy annotations: description: Thanos Sidecar {{$labels.instance}} is unhealthy for more than {{$value}} seconds. summary: Thanos Sidecar is unhealthy. expr: | time() - max by (job, instance) (thanos_sidecar_last_heartbeat_success_time_seconds{job=~"prometheus-(k8s|user-workload)-thanos-sidecar"}) >= 240 for: 1h labels: severity: warning
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:3759