Bug 2094919

Summary:	etcd vertical scaling test destabilizes vsphere
Product:	OpenShift Container Platform	Reporter:	Stephen Benjamin <stbenjam>
Component:	Etcd	Assignee:	Dean West <dwest>
Status:	CLOSED DUPLICATE	QA Contact:	ge liu <geliu>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	4.11	CC:	tjungblu, wking
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-06-16 14:19:25 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Stephen Benjamin 2022-06-08 15:47:51 UTC

vpshere serial is permafailing on `[sig-instrumentation][Late] Alerts shouldn't report any unexpected alerts in firing or pending state`, with TargetDown alerts for various components.

- This started around 5/11, which is when https://github.com/openshift/origin/pull/26917 ended up in the first payload

- TargetDown alerts always happens directly after the etcd vertical scale test. 



Example run: https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.11-e2e-vsphere-serial/1532975513713250304

If you look at the intervals chart filtered by "TargetDown|vertical" you can see the behavior.  This happens on every vsphere serial job.

Comment 1 Stephen Benjamin 2022-06-08 19:27:15 UTC

https://github.com/openshift/origin/pull/27236 disables the test on vsphere, and it passed vsphere-serial.  This does seem to be the problem.

Comment 3 Stephen Benjamin 2022-06-09 10:42:58 UTC

Setting back to new, the merged PR was just disabling it temporarily.