Bug 2006360

Summary:	Origin should run k8s and OpenShift tests separately
Product:	OpenShift Container Platform	Reporter:	Stephen Benjamin <stbenjam>
Component:	Test Framework	Assignee:	Oleg Bulatov <obulatov>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	high	Docs Contact:
Priority:	high
Version:	4.9	CC:	bleanhar, wking
Target Milestone:	---	Keywords:	Triaged
Target Release:	4.10.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	2006362 (view as bug list)		Environment:
Last Closed:	2021-09-27 13:16:20 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2006362

Description Stephen Benjamin 2021-09-21 14:24:47 UTC

During a recent TRT stand-up, David pointed out the extremely high pass rates
of upstream kube tests, but those same tests in OpenShift fail quite a
bit. There's a theory that perhaps some of the OpenShift tests like
builds use up so much I/O and other resources that they introduce
instability.

In master we have a change that runs them separately, and it seems to have improved the situation overall.  4.10 today has a pass rate of 70% of all jobs running e2e tests, but 4.9 is around 51%.  This is visible in the 4th release indicator widget on the overview page:  https://sippy.ci.openshift.org/sippy-ng/release/4.10 vs https://sippy.ci.openshift.org/sippy-ng/release/4.9

It's not clear if all or any of that is attributable to that change, but we'd like to backport it to 4.9 to confirm.