Bug 1962219

Summary:	NTO relies on unreliable leader-for-life implementation.
Product:	OpenShift Container Platform	Reporter:	Jiří Mencák <jmencak>
Component:	Node Tuning Operator	Assignee:	Jiří Mencák <jmencak>
Status:	CLOSED ERRATA	QA Contact:	Simon <skordas>
Severity:	high	Docs Contact:
Priority:	high
Version:	4.8	CC:	sejug
Target Milestone:	---
Target Release:	4.8.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-07-27 23:09:14 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1958885

Description Jiří Mencák 2021-05-19 14:17:24 UTC

Description of problem:
The Node Tuning Operator (alongside with a few other SLOs) currently uses leader-for-life implementation from operator-sdk, which already caused several known upgrade failures (rhbz#1916865, rhbz#1958885) due to ConfigMap lock not being removed by the k8s garbage collection mechanism.

Version-Release number of selected component (if applicable):
All currently supported versions.

How reproducible:
Rare, but serious as it causes cluster upgrade failures.

Steps to Reproduce:
This is unfortunately hard to reproduce.

Actual results:
NTO stuck waiting for a ConfigMap lock to be removed.

Expected results:
NTO should not rely on the k8s GC mechanism for leader election.

Additional info:
Upstream fix: https://github.com/openshift/cluster-node-tuning-operator/pull/229
Related BZs:
https://bugzilla.redhat.com/show_bug.cgi?id=1916865
https://bugzilla.redhat.com/show_bug.cgi?id=1958885

Comment 2 Simon 2021-05-26 16:40:25 UTC

Cluster version: 4.8.0-0.nightly-2021-05-26-071911

Impossible to reproduce.

Regression tests 100% passed.

Comment 5 errata-xmlrpc 2021-07-27 23:09:14 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438