Bug 1996577

Summary:	Upgrading to 4.8 degrades the SDN (OpenShiftSDN or OVNKubernetes) traffic between nodes when they are on different hypervisors (vsphere)
Product:	OpenShift Container Platform	Reporter:	Khizer Naeem <knaeem>
Component:	Machine Config Operator	Assignee:	MCO Team <team-mco>
Machine Config Operator sub component:	Machine Config Operator	QA Contact:	Jian Zhang <jiazha>
Status:	CLOSED DUPLICATE	Docs Contact:
Severity:	high
Priority:	high	CC:	aconstan, aos-bugs, astoycos, bbennett, mkrejci, wking
Version:	4.8
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Other
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-09-14 11:29:51 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Comment 3 Martin Kennelly 2021-08-24 16:45:13 UTC

I am trying to repro this on a vsphere cluster. 

"Later we discovered that moving the openshift nodes to a single hypervisor resolves the issue. The upgrade to 4.8.2 was completed."

Forgive me for not understanding but how was the upgrade to 4.8.2 completed? By moving everything to a single node?

Also, I see many BZs upgrading failed on vsphere to 4.8.2. I will try to repro and apply the fix.

Comment 4 Martin Kennelly 2021-08-24 17:44:07 UTC

Cannot upgrade directly to 4.8.2 anymore. This upgrade path has been removed. 
I tried using forcing to 4.8.2 but the oc adm upgrade command fails (w/ cmd: "--allow-explicit-upgrade=true: Upgrade even if the upgrade target is not listed in the available versions list").
I upgraded from 4.7.21 -> 4.8.5 and saw no issues with pod connectivity.

Comment 5 Khizer Naeem 2021-08-24 17:46:07 UTC

(In reply to Martin Kennelly from comment #3)
> "Later we discovered that moving the openshift nodes to a single hypervisor
> resolves the issue. The upgrade to 4.8.2 was completed."
> 
> Forgive me for not understanding but how was the upgrade to 4.8.2 completed?
> By moving everything to a single node?

yes, by moving all openshift nodes to a single hypervisor

Comment 7 Martin Kennelly 2021-08-26 16:24:52 UTC

There's a fix being made to MCO to overcome this issue: https://github.com/openshift/machine-config-operator/pull/2738

Comment 8 Martin Kennelly 2021-08-26 16:29:33 UTC

You can confirm now if it will affect you. See advisory. 

Affected component:
"VMware vSphere with virtual machine hardware versions 14 and newer"