1957703 – Prometheus Statefulsets should have 2 replicas and hard affinity set

Bug 1957703 - Prometheus Statefulsets should have 2 replicas and hard affinity set

Summary: Prometheus Statefulsets should have 2 replicas and hard affinity set

Keywords:
Status:	CLOSED WONTFIX
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Monitoring
Sub Component:
Version:	4.7
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	medium
Target Milestone:	---
Target Release:	4.7.z
Assignee:	Damien Grisonnet
QA Contact:	Junqi Zhao
Docs Contact:
URL:
Whiteboard:
Depends On:	1949262
Blocks:	1957704
TreeView+	depends on / blocked

Reported:	2021-05-06 10:27 UTC by Damien Grisonnet
Modified:	2021-10-15 12:34 UTC (History)
CC List:	12 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:	1949262
Clones:	1957704 (view as bug list)
Environment:
Last Closed:	2021-10-15 12:34:32 UTC
Target Upstream Version:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Github	openshift cluster-monitoring-operator pull 1150	0	None	open	Bug 1957703: [4.7]: Add hard anti-affinity constraints to Prometheuses	2021-05-06 11:16:28 UTC

Comment 1 Junqi Zhao 2021-05-13 06:28:09 UTC

Tested with the not merged PR, hard anti-affinity to Prometheuses is added, prometheus-k8s and prometheus-user-workload pods are scheduled to different nodes
# oc -n openshift-monitoring get sts prometheus-k8s -oyaml | grep podAntiAffinity -A10
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: prometheus
                operator: In
                values:
                - k8s
            namespaces:
            - openshift-monitoring
            topologyKey: kubernetes.io/hostname
# oc -n openshift-user-workload-monitoring get sts prometheus-user-workload -oyaml  | grep podAntiAffinity -A10
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: prometheus
                operator: In
                values:
                - user-workload
            namespaces:
            - openshift-user-workload-monitoring
            topologyKey: kubernetes.io/hostname

# oc -n openshift-monitoring get pod -o wide | grep prometheus-k8s
prometheus-k8s-0                               7/7     Running   1          18m    10.129.2.10   ci-ln-l78mzib-f76d1-x2jrt-worker-b-q6sdk   <none>           <none>
prometheus-k8s-1                               7/7     Running   1          18m    10.128.2.11   ci-ln-l78mzib-f76d1-x2jrt-worker-c-5prgg   <none>           <none>

# oc -n openshift-user-workload-monitoring get po -o wide 
NAME                                   READY   STATUS    RESTARTS   AGE     IP            NODE                                       NOMINATED NODE   READINESS GATES
prometheus-operator-559497d594-fzhps   2/2     Running   0          2m31s   10.130.0.43   ci-ln-l78mzib-f76d1-x2jrt-master-2         <none>           <none>
prometheus-user-workload-0             5/5     Running   1          2m28s   10.131.0.28   ci-ln-l78mzib-f76d1-x2jrt-worker-d-cxn8k   <none>           <none>
prometheus-user-workload-1             5/5     Running   1          2m27s   10.129.2.11   ci-ln-l78mzib-f76d1-x2jrt-worker-b-q6sdk   <none>           <none>
thanos-ruler-user-workload-0           3/3     Running   0          2m24s   10.131.0.29   ci-ln-l78mzib-f76d1-x2jrt-worker-d-cxn8k   <none>           <none>
thanos-ruler-user-workload-1           3/3     Running   0          2m24s   10.128.2.13   ci-ln-l78mzib-f76d1-x2jrt-worker-c-5prgg   <none>           <none>

Comment 2 Damien Grisonnet 2021-05-25 17:26:34 UTC

PR is waiting for patch manager approval.

Comment 4 Junqi Zhao 2021-05-31 02:07:38 UTC

fix is in 4.7.0-0.nightly-2021-05-28-013206 and later builds, based on Comment 1, move to VERIFIED

Comment 6 Siddharth Sharma 2021-06-04 18:39:19 UTC

This bug will be shipped as part of next z-stream release 4.7.15 on June 14th, as 4.7.14 was dropped due to a regression https://bugzilla.redhat.com/show_bug.cgi?id=1967614

Note You need to log in before you can comment on or make changes to this bug.