Bug 1957703 - Prometheus Statefulsets should have 2 replicas and hard affinity set
Summary: Prometheus Statefulsets should have 2 replicas and hard affinity set
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Monitoring
Version: 4.7
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.7.z
Assignee: Damien Grisonnet
QA Contact: Junqi Zhao
URL:
Whiteboard:
Depends On: 1949262
Blocks: 1957704
TreeView+ depends on / blocked
 
Reported: 2021-05-06 10:27 UTC by Damien Grisonnet
Modified: 2021-10-15 12:34 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1949262
: 1957704 (view as bug list)
Environment:
Last Closed: 2021-10-15 12:34:32 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-monitoring-operator pull 1150 0 None open Bug 1957703: [4.7]: Add hard anti-affinity constraints to Prometheuses 2021-05-06 11:16:28 UTC

Comment 1 Junqi Zhao 2021-05-13 06:28:09 UTC
Tested with the not merged PR, hard anti-affinity to Prometheuses is added, prometheus-k8s and prometheus-user-workload pods are scheduled to different nodes
# oc -n openshift-monitoring get sts prometheus-k8s -oyaml | grep podAntiAffinity -A10
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: prometheus
                operator: In
                values:
                - k8s
            namespaces:
            - openshift-monitoring
            topologyKey: kubernetes.io/hostname
# oc -n openshift-user-workload-monitoring get sts prometheus-user-workload -oyaml  | grep podAntiAffinity -A10
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: prometheus
                operator: In
                values:
                - user-workload
            namespaces:
            - openshift-user-workload-monitoring
            topologyKey: kubernetes.io/hostname

# oc -n openshift-monitoring get pod -o wide | grep prometheus-k8s
prometheus-k8s-0                               7/7     Running   1          18m    10.129.2.10   ci-ln-l78mzib-f76d1-x2jrt-worker-b-q6sdk   <none>           <none>
prometheus-k8s-1                               7/7     Running   1          18m    10.128.2.11   ci-ln-l78mzib-f76d1-x2jrt-worker-c-5prgg   <none>           <none>

# oc -n openshift-user-workload-monitoring get po -o wide 
NAME                                   READY   STATUS    RESTARTS   AGE     IP            NODE                                       NOMINATED NODE   READINESS GATES
prometheus-operator-559497d594-fzhps   2/2     Running   0          2m31s   10.130.0.43   ci-ln-l78mzib-f76d1-x2jrt-master-2         <none>           <none>
prometheus-user-workload-0             5/5     Running   1          2m28s   10.131.0.28   ci-ln-l78mzib-f76d1-x2jrt-worker-d-cxn8k   <none>           <none>
prometheus-user-workload-1             5/5     Running   1          2m27s   10.129.2.11   ci-ln-l78mzib-f76d1-x2jrt-worker-b-q6sdk   <none>           <none>
thanos-ruler-user-workload-0           3/3     Running   0          2m24s   10.131.0.29   ci-ln-l78mzib-f76d1-x2jrt-worker-d-cxn8k   <none>           <none>
thanos-ruler-user-workload-1           3/3     Running   0          2m24s   10.128.2.13   ci-ln-l78mzib-f76d1-x2jrt-worker-c-5prgg   <none>           <none>

Comment 2 Damien Grisonnet 2021-05-25 17:26:34 UTC
PR is waiting for patch manager approval.

Comment 4 Junqi Zhao 2021-05-31 02:07:38 UTC
fix is in 4.7.0-0.nightly-2021-05-28-013206 and later builds, based on Comment 1, move to VERIFIED

Comment 6 Siddharth Sharma 2021-06-04 18:39:19 UTC
This bug will be shipped as part of next z-stream release 4.7.15 on June 14th, as 4.7.14 was dropped due to a regression https://bugzilla.redhat.com/show_bug.cgi?id=1967614


Note You need to log in before you can comment on or make changes to this bug.