Bug 1952358 - Openshift-apiserver CO unavailable in fresh OCP 4.7.5 installations
Summary: Openshift-apiserver CO unavailable in fresh OCP 4.7.5 installations
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Machine Config Operator
Version: 4.7
Hardware: x86_64
OS: Linux
Target Milestone: ---
: 4.8.0
Assignee: Yu Qi Zhang
QA Contact: Michael Nguyen
Whiteboard: UpdateRecommendationsBlocked
Depends On:
Blocks: 1956749
TreeView+ depends on / blocked
Reported: 2021-04-22 06:46 UTC by oarribas
Modified: 2021-11-03 03:10 UTC (History)
37 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of: 1941246
Last Closed: 2021-07-27 23:02:52 UTC
Target Upstream Version:
jerzhang: needinfo-

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift machine-config-operator pull 2559 0 None open Bug 1952358: vsphere: platform none, vmxnet3v4 fix move to base 2021-04-30 19:53:36 UTC
Red Hat Bugzilla 1935539 1 urgent CLOSED Openshift-apiserver CO unavailable during cluster upgrade from 4.6 to 4.7 2022-06-16 12:07:17 UTC
Red Hat Bugzilla 1941714 1 urgent CLOSED [ESXi][RHEL-8] VMXNET3 v4 causes invalid checksums of inner packets of VXLAN tunnel 2022-08-07 05:34:11 UTC
Red Hat Knowledge Base (Solution) 5997331 0 None None None 2021-04-27 09:30:37 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 23:03:13 UTC

Internal Links: 1941714

Comment 9 Ross Brattain 2021-05-06 21:48:30 UTC
Verified on 4.8.0-0.nightly-2021-05-06-092434

Hypervisor:	VMware ESXi, 7.0.1, 17460241
Model:	Amazon EC2 i3en.metal-2tb
Processor Type:	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz

Tested UPI install on vHW 14 OVN and OpenShiftSDN, then upgraded individual nodes to vHW 14-17

# for f in $(oc get nodes --no-headers -o custom-columns=N:.metadata.name ) ; do oc debug node/$f -- ethtool -k ens192 | grep udp_tnl | tee udp-$f & done

vHW 15
udp-compute-0:tx-udp_tnl-segmentation: off
udp-compute-0:tx-udp_tnl-csum-segmentation: off
vHW 16
udp-compute-1:tx-udp_tnl-segmentation: off
udp-compute-1:tx-udp_tnl-csum-segmentation: off
vHW 17
udp-control-plane-0:tx-udp_tnl-segmentation: off
udp-control-plane-0:tx-udp_tnl-csum-segmentation: off
vHW 14
udp-control-plane-1:tx-udp_tnl-segmentation: off
udp-control-plane-1:tx-udp_tnl-csum-segmentation: off
vHW 14
udp-control-plane-2:tx-udp_tnl-segmentation: off
udp-control-plane-2:tx-udp_tnl-csum-segmentation: off

Added secondary NIC ens224, verified offloads also disabled
# for f in $(oc get nodes --no-headers -o custom-columns=N:.metadata.name ) ; do oc debug node/$f -- ethtool -k ens224 | grep udp_tnl | tee udp-lower_up-224-$f & done

vHW 15
udp-lower_up-224-compute-0:tx-udp_tnl-segmentation: off
udp-lower_up-224-compute-0:tx-udp_tnl-csum-segmentation: off
vHW 16
udp-lower_up-224-compute-1:tx-udp_tnl-segmentation: off
udp-lower_up-224-compute-1:tx-udp_tnl-csum-segmentation: off
vHW 17
udp-lower_up-224-control-plane-0:tx-udp_tnl-segmentation: off
udp-lower_up-224-control-plane-0:tx-udp_tnl-csum-segmentation: off
vHW 14
udp-lower_up-224-control-plane-1:tx-udp_tnl-segmentation: off
udp-lower_up-224-control-plane-1:tx-udp_tnl-csum-segmentation: off
vHW 14
udp-lower_up-224-control-plane-2:tx-udp_tnl-segmentation: off
udp-lower_up-224-control-plane-2:tx-udp_tnl-csum-segmentation: off

Tested on regular OpenShiftSDN AWS IPI, verified dispatcher.d/99-vsphere-disable-tx-udp-tnl created but not activated on AWS as expected.

May 06 20:23:50.308450 ip-10-0-179-113 ignition[896]: INFO     : files: createFilesystemsFiles: createFiles: op(1a): [started]  writing file "/sysroot/etc/NetworkManager/dispatcher.d/99-vsphere-disable-tx-udp-tnl"

Comment 10 jima 2021-05-07 08:25:19 UTC
Upgrade from 4.7.0-0.nightly-2021-05-05-092347 to 4.8.0-0.nightly-2021-05-06-210840

Install vsphere upi with nightly build 4.7.0-0.nightly-2021-05-05-092347 in VMC, where:
- "platform:none" filed in install-config.yaml
- hardware version is 15

After installation is completed, checked all master/worker nodes:
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on

Then upgrade to 4.8.0-0.nightly-2021-05-06-210840
$ ./oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.nightly-2021-05-05-092347   True        True          6s      Working towards registry.ci.openshift.org/ocp/release:4.8.0-0.nightly-2021-05-06-210840: downloading update

$ ./oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-05-06-210840   True        False         99m     Cluster version is 4.8.0-0.nightly-2021-05-06-210840

Checked again on nodes after upgrade, all have below options:
tx-udp_tnl-segmentation: off
tx-udp_tnl-csum-segmentation: off

Comment 15 errata-xmlrpc 2021-07-27 23:02:52 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.