Bug 1734029 - Prometheus statefulset is reconciled on pod deletion but not when is scaled down
Summary: Prometheus statefulset is reconciled on pod deletion but not when is scaled down
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Monitoring
Version: 4.1.z
Hardware: Unspecified
OS: Unspecified
unspecified
low
Target Milestone: ---
: 4.3.0
Assignee: Sergiusz Urbaniak
QA Contact: Junqi Zhao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-07-29 13:27 UTC by Robert Sandu
Modified: 2020-01-23 11:05 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-01-23 11:04:29 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github coreos prometheus-operator pull 2801 'None' 'closed' 'Improve detection of StatefulSet changes' 2019-11-26 13:13:24 UTC
Red Hat Product Errata RHBA-2020:0062 None None None 2020-01-23 11:05:06 UTC

Description Robert Sandu 2019-07-29 13:27:33 UTC
Description of problem: Prometheus statefulset is reconciled on pod deletion but not on scale down.

Version-Release number of selected component (if applicable): multiple. I've tested this on 4 clusters with the following versions:

4.1.7
3.11.117
3.11.98
3.11.69

How reproducible: always.

Steps to Reproduce:
1. oc scale statefulset.apps/prometheus-k8s --replicas=1
2. The second sts replica is not created again. Yet the prometheus-k8s-1 pod is deleted, a new one is created instead.

Actual results: the second sts replica is not created again when the sts is scaled down.

Expected results: the statefulset to be reconciled both on pod deletion of statefulset scale down.

Additional info:

- Nothing suspicious in the prom operator logs
- Prometheus operator is expecting 2 replicas, but only one sts is up:

$ oc -n openshift-monitoring get prometheus k8s -o yaml | grep -i replicas
  replicas: 2
$ oc -n openshift-monitoring describe sts prometheus-k8s | grep -i replicas
Replicas:           1 desired | 1 total

Comment 14 errata-xmlrpc 2020-01-23 11:04:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0062


Note You need to log in before you can comment on or make changes to this bug.