Bug 2011924

Summary:	[sig-node] nodes should not go unready after being upgraded and go unready only once
Product:	OpenShift Container Platform	Reporter:	Devan Goodwin <dgoodwin>
Component:	Test Framework	Assignee:	Devan Goodwin <dgoodwin>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	high	Docs Contact:
Priority:	high
Version:	4.10	CC:	aos-bugs, rphillips, scuppett, sippy, wking
Target Milestone:	---	Keywords:	DeliveryBlocker, Reopened, Upgrades
Target Release:	4.10.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-11-09 13:18:58 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Devan Goodwin 2021-10-07 18:21:21 UTC

[sig-node] nodes should not go unready after being upgraded and go unready only once

is failing frequently in CI and the problem appears related to Azure, see:
https://sippy.ci.openshift.org/sippy-ng/tests/4.10/analysis?test=%5Bsig-node%5D%20nodes%20should%20not%20go%20unready%20after%20being%20upgraded%20and%20go%20unready%20only%20once

Sample prow job failure: 

https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.10-upgrade-from-stable-4.9-e2e-azure-upgrade/1446060099507326976

Comment 12 Devan Goodwin 2021-10-14 16:56:46 UTC

I think this is a test bug, thanks to Harshal Patil analysis ruling out a node problem. I am preparing a fix and will explain more in that PR.

Comment 15 Devan Goodwin 2021-10-18 14:05:58 UTC

deads has requested we expand on this as there are still going to be tests that will fail if we do not have the expected capacity. He suggests waiting for nodes to be ready, and possibly for machines to be ready first in a third pr. Will hopefully get this completed this week.

Comment 16 Devan Goodwin 2021-11-09 13:18:58 UTC

In https://github.com/openshift/release/pull/22928 support was added to wait for number of nodes to equal number of machines before proceeding with tests.