Bug 2008540

Summary:	HighlyAvailableWorkloadIncorrectlySpread always fires on upgrade on cluster with two workers
Product:	OpenShift Container Platform	Reporter:	Seth Jennings <sjenning>
Component:	Monitoring	Assignee:	Haoyu Sun <hasun>
Status:	CLOSED ERRATA	QA Contact:	Junqi Zhao <juzhao>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	4.8	CC:	amuller, anpicker, aos-bugs, dgrisonn, erooth, spasquie
Target Milestone:	---
Target Release:	4.10.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-03-10 16:13:54 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Seth Jennings 2021-09-28 13:56:30 UTC

Description of problem:
I have a 4.8 cluster with 2 workers.  Every time I upgrade that cluster, this alert fires and requires manual intervention to resolve.

The situation is that, during upgrade, each worker is cordoned and drain in sequence.  The result is the following:

worker-0 is cordoned and drained
prometheus-k8s-0 running on worker-0 is evicted to worker-1 where prometheus-k8s-1 is already running
worker-0 upgrades and becomes schedulable
worker-1 is cordoned and drained
both prom pods are evicted and restarted on worker-0
worker-1 upgrades and becomes schedulable again

The final result is that both prom pods are running on the same worker.  With two workers, this happens every time.

Version-Release number of selected component (if applicable):
4.8.12

How reproducible:
Always

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
https://bugzilla.redhat.com/show_bug.cgi?id=1974832

Comment 1 Damien Grisonnet 2021-09-28 14:11:23 UTC

This is intended to some extent since we want to prevent single point of failures, but I can see how this can become annoying. Ideally, we would want to switch to hard-affinity on hostname and have a PDB which would prevent this from happening, but that effort is currently blocked by bug 1995924.

That said, now that we have taken a new approach with how we want to handle high availability when persistent storage is enabled in the form of bug 1995924, we might want to consider removing the `HighlyAvailableWorkloadIncorrectlySpread` alert completely.

Comment 4 Junqi Zhao 2021-11-30 07:41:29 UTC

checked with PR, HighlyAvailableWorkloadIncorrectlySpread is removed, but we need #1489 to be merged first.

Comment 8 Junqi Zhao 2021-12-22 02:10:02 UTC

fix is in 4.10.0-0.nightly-2021-12-21-130047, HighlyAvailableWorkloadIncorrectlySpread is removed, set to VERIFIED

Comment 11 errata-xmlrpc 2022-03-10 16:13:54 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056