Description of problem: Prometheus statefulset is reconciled on pod deletion but not on scale down.
Version-Release number of selected component (if applicable): multiple. I've tested this on 4 clusters with the following versions:
How reproducible: always.
Steps to Reproduce:
1. oc scale statefulset.apps/prometheus-k8s --replicas=1
2. The second sts replica is not created again. Yet the prometheus-k8s-1 pod is deleted, a new one is created instead.
Actual results: the second sts replica is not created again when the sts is scaled down.
Expected results: the statefulset to be reconciled both on pod deletion of statefulset scale down.
- Nothing suspicious in the prom operator logs
- Prometheus operator is expecting 2 replicas, but only one sts is up:
$ oc -n openshift-monitoring get prometheus k8s -o yaml | grep -i replicas
$ oc -n openshift-monitoring describe sts prometheus-k8s | grep -i replicas
Replicas: 1 desired | 1 total
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.