Bug 2008540 - HighlyAvailableWorkloadIncorrectlySpread always fires on upgrade on cluster with two workers
Summary: HighlyAvailableWorkloadIncorrectlySpread always fires on upgrade on cluster w...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Monitoring
Version: 4.8
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.10.0
Assignee: Haoyu Sun
QA Contact: Junqi Zhao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-09-28 13:56 UTC by Seth Jennings
Modified: 2022-03-10 16:14 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-10 16:13:54 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-monitoring-operator pull 1488 0 None open Bug 2008540: remove alert HighlyAvailableWorkloadIncorrectlySpread 2021-11-26 14:38:13 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-10 16:14:14 UTC

Description Seth Jennings 2021-09-28 13:56:30 UTC
Description of problem:
I have a 4.8 cluster with 2 workers.  Every time I upgrade that cluster, this alert fires and requires manual intervention to resolve.

The situation is that, during upgrade, each worker is cordoned and drain in sequence.  The result is the following:

worker-0 is cordoned and drained
prometheus-k8s-0 running on worker-0 is evicted to worker-1 where prometheus-k8s-1 is already running
worker-0 upgrades and becomes schedulable
worker-1 is cordoned and drained
both prom pods are evicted and restarted on worker-0
worker-1 upgrades and becomes schedulable again

The final result is that both prom pods are running on the same worker.  With two workers, this happens every time.

Version-Release number of selected component (if applicable):
4.8.12

How reproducible:
Always

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
https://bugzilla.redhat.com/show_bug.cgi?id=1974832

Comment 1 Damien Grisonnet 2021-09-28 14:11:23 UTC
This is intended to some extent since we want to prevent single point of failures, but I can see how this can become annoying. Ideally, we would want to switch to hard-affinity on hostname and have a PDB which would prevent this from happening, but that effort is currently blocked by bug 1995924.

That said, now that we have taken a new approach with how we want to handle high availability when persistent storage is enabled in the form of bug 1995924, we might want to consider removing the `HighlyAvailableWorkloadIncorrectlySpread` alert completely.

Comment 4 Junqi Zhao 2021-11-30 07:41:29 UTC
checked with PR, HighlyAvailableWorkloadIncorrectlySpread is removed, but we need #1489 to be merged first.

Comment 8 Junqi Zhao 2021-12-22 02:10:02 UTC
fix is in 4.10.0-0.nightly-2021-12-21-130047, HighlyAvailableWorkloadIncorrectlySpread is removed, set to VERIFIED

Comment 11 errata-xmlrpc 2022-03-10 16:13:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056


Note You need to log in before you can comment on or make changes to this bug.