Bug 1956749

Summary:	Openshift-apiserver CO unavailable in fresh OCP 4.7.5 installations
Product:	OpenShift Container Platform	Reporter:	OpenShift BugZilla Robot <openshift-bugzilla-robot>
Component:	Machine Config Operator	Assignee:	Yu Qi Zhang <jerzhang>
Machine Config Operator sub component:	platform-vsphere	QA Contact:	Michael Nguyen <mnguyen>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	urgent
Priority:	urgent	CC:	aconstan, alexisph, alkazako, anbhat, aos-bugs, bleanhar, danw, david.karlsen, dkulkarn, huirwang, jani.eerola, jcallen, jerzhang, jima, lmohanty, mfojtik, mgugino, mkrejci, nchoudhu, oarribas, openshift-bugs-escalate, openshift-bugzilla-robot, palonsor, palshure, rbrattai, rsandu, scuppett, simore, sreber, srengan, welin, wking, yhe, zzhao
Version:	4.7	Keywords:	Upgrades
Target Milestone:	---
Target Release:	4.7.z
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-05-19 15:17:15 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1952358
Bug Blocks:

Comment 4 Ross Brattain 2021-05-12 16:58:01 UTC

Install successful on 4.7.0-0.nightly-2021-05-12-004740, VMC UPI OpenShiftSDN and OVN, vHW 14 and 15.



-- Logs begin at Wed 2021-05-12 15:24:13 UTC, end at Wed 2021-05-12 16:44:46 UTC. --
May 12 15:24:21.521265 localhost ignition[1037]: INFO     : files: createFilesystemsFiles: createFiles: op(29): [started]  writing file "/sysroot/etc/NetworkManager/dispatcher.d/99-vsphere-disable-tx-udp-tnl"
May 12 15:24:21.521265 localhost ignition[1037]: INFO     : files: createFilesystemsFiles: createFiles: op(29): [finished] writing file "/sysroot/etc/NetworkManager/dispatcher.d/99-vsphere-disable-tx-udp-tnl"
May 12 15:24:22.534366 localhost ignition[1041]: INFO     : files: createFilesystemsFiles: createFiles: op(29): [started]  writing file "/sysroot/etc/NetworkManager/dispatcher.d/99-vsphere-disable-tx-udp-tnl"
May 12 15:24:22.534366 localhost ignition[1041]: INFO     : files: createFilesystemsFiles: createFiles: op(29): [finished] writing file "/sysroot/etc/NetworkManager/dispatcher.d/99-vsphere-disable-tx-udp-tnl"
May 12 15:24:24.924947 localhost ignition[1050]: INFO     : files: createFilesystemsFiles: createFiles: op(29): [started]  writing file "/sysroot/etc/NetworkManager/dispatcher.d/99-vsphere-disable-tx-udp-tnl"
May 12 15:24:24.924947 localhost ignition[1050]: INFO     : files: createFilesystemsFiles: createFiles: op(29): [finished] writing file "/sysroot/etc/NetworkManager/dispatcher.d/99-vsphere-disable-tx-udp-tnl"
May 12 15:24:25.731030 control-plane-1 root[1879]: 99-vsphere-disable-tx-udp-tnl triggered by up on device ens192.
May 12 15:24:25.731388 control-plane-1 nm-dispatcher[1814]: <13>May 12 15:24:25 root: 99-vsphere-disable-tx-udp-tnl triggered by up on device ens192.
-- Logs begin at Wed 2021-05-12 15:24:08 UTC, end at Wed 2021-05-12 16:44:47 UTC. --
May 12 15:24:26.912035 control-plane-2 root[1887]: 99-vsphere-disable-tx-udp-tnl triggered by up on device ens192.
May 12 15:24:26.912334 control-plane-2 nm-dispatcher[1822]: <13>May 12 15:24:26 root: 99-vsphere-disable-tx-udp-tnl triggered by up on device ens192.
-- Logs begin at Wed 2021-05-12 15:24:09 UTC, end at Wed 2021-05-12 16:44:34 UTC. --
May 12 15:24:30.553673 control-plane-0 root[1892]: 99-vsphere-disable-tx-udp-tnl triggered by up on device ens192.
May 12 15:24:30.554216 control-plane-0 nm-dispatcher[1827]: <13>May 12 15:24:30 root: 99-vsphere-disable-tx-udp-tnl triggered by up on device ens192.
-- Logs begin at Wed 2021-05-12 15:24:12 UTC, end at Wed 2021-05-12 16:44:41 UTC. --
May 12 15:25:41.149794 control-plane-1 root[1452]: 99-vsphere-disable-tx-udp-tnl triggered by up on device ens192.
May 12 15:25:41.150236 control-plane-1 nm-dispatcher[1401]: <13>May 12 15:25:41 root: 99-vsphere-disable-tx-udp-tnl triggered by up on device ens192.
May 12 15:25:47.801604 control-plane-2 root[1465]: 99-vsphere-disable-tx-udp-tnl triggered by up on device ens192.
May 12 15:25:47.802604 control-plane-2 nm-dispatcher[1414]: <13>May 12 15:25:47 root: 99-vsphere-disable-tx-udp-tnl triggered by up on device ens192.
May 12 15:25:54.394815 control-plane-0 root[1460]: 99-vsphere-disable-tx-udp-tnl triggered by up on device ens192.
May 12 15:25:54.395348 control-plane-0 nm-dispatcher[1408]: <13>May 12 15:25:54 root: 99-vsphere-disable-tx-udp-tnl triggered by up on device ens192.
May 12 15:31:11.415603 localhost ignition[1048]: INFO     : files: createFilesystemsFiles: createFiles: op(24): [started]  writing file "/sysroot/etc/NetworkManager/dispatcher.d/99-vsphere-disable-tx-udp-tnl"
May 12 15:31:11.415603 localhost ignition[1048]: INFO     : files: createFilesystemsFiles: createFiles: op(24): [finished] writing file "/sysroot/etc/NetworkManager/dispatcher.d/99-vsphere-disable-tx-udp-tnl"
May 12 15:31:15.191388 compute-1 root[1901]: 99-vsphere-disable-tx-udp-tnl triggered by up on device ens192.
May 12 15:31:15.191778 compute-1 nm-dispatcher[1836]: <13>May 12 15:31:15 root: 99-vsphere-disable-tx-udp-tnl triggered by up on device ens192.
-- Logs begin at Wed 2021-05-12 15:24:13 UTC, end at Wed 2021-05-12 16:44:46 UTC. --
May 12 15:31:18.880879 localhost ignition[1053]: INFO     : files: createFilesystemsFiles: createFiles: op(24): [started]  writing file "/sysroot/etc/NetworkManager/dispatcher.d/99-vsphere-disable-tx-udp-tnl"
May 12 15:31:18.880879 localhost ignition[1053]: INFO     : files: createFilesystemsFiles: createFiles: op(24): [finished] writing file "/sysroot/etc/NetworkManager/dispatcher.d/99-vsphere-disable-tx-udp-tnl"
May 12 15:31:23.206279 compute-0 root[1901]: 99-vsphere-disable-tx-udp-tnl triggered by up on device ens192.
May 12 15:31:23.206572 compute-0 nm-dispatcher[1836]: <13>May 12 15:31:23 root: 99-vsphere-disable-tx-udp-tnl triggered by up on device ens192.
-- Logs begin at Wed 2021-05-12 15:24:12 UTC, end at Wed 2021-05-12 16:44:39 UTC. --
May 12 15:32:32.237782 compute-1 root[1456]: 99-vsphere-disable-tx-udp-tnl triggered by up on device ens192.
May 12 15:32:32.238229 compute-1 nm-dispatcher[1404]: <13>May 12 15:32:32 root: 99-vsphere-disable-tx-udp-tnl triggered by up on device ens192.
May 12 15:32:44.085886 compute-0 root[1461]: 99-vsphere-disable-tx-udp-tnl triggered by up on device ens192.
May 12 15:32:44.086366 compute-0 nm-dispatcher[1410]: <13>May 12 15:32:44 root: 99-vsphere-disable-tx-udp-tnl triggered by up on device ens192.

Comment 5 jima 2021-05-13 04:03:26 UTC

upgrade from 4.6.27 to 4.7.0-0.nightly-2021-05-11-121303 on VMC.
1. Install 4.6.27 on upi-on-vsphere and set "platform: none" in install-config.yaml. vHW is 15. 
After installation is completed, checked on all nodes with command "oc debug node/<node_name> -- ethtool -k ens192 | grep udp_tnl":
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]

2. upgrade to 4.7.0-0.nightly-2021-05-11-121303
$ ./oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.27    True        True          1s      Working towards registry.ci.openshift.org/ocp/release:4.7.0-0.nightly-2021-05-11-121303: downloading update

$ ./oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.nightly-2021-05-11-121303   True        False         71m     Cluster version is 4.7.0-0.nightly-2021-05-11-121303

3. Check again on all nodes with command "oc debug node/<node_name> -- ethtool -k ens192 | grep udp_tnl"
tx-udp_tnl-segmentation: off
tx-udp_tnl-csum-segmentation: off


Also fresh install twice with hw 15 and hw17, on upi-on-vsphere with nightly build 4.7.0-0.nightly-2021-05-11-121303, and set "platform:none" in install-config.yaml.
After installation is finished, checked on all nodes:
tx-udp_tnl-segmentation: off
tx-udp_tnl-csum-segmentation: off

Comment 7 errata-xmlrpc 2021-05-19 15:17:15 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.7.11 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1550

Comment 8 Scott Dodson 2021-08-20 15:49:00 UTC

*** Bug 1935591 has been marked as a duplicate of this bug. ***