Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1845903

Summary:	gcp-routes service too slow to not route traffic into GCP SDN
Product:	OpenShift Container Platform	Reporter:	Stefan Schimanski <sttts>
Component:	Machine Config Operator	Assignee:	Antonio Murdaca <amurdaca>
Status:	CLOSED ERRATA	QA Contact:	Ke Wang <kewang>
Severity:	high	Docs Contact:
Priority:	high
Version:	4.5	CC:	kewang, kgarriso, lmohanty
Target Milestone:	---	Keywords:	Upgrades
Target Release:	4.6.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-10-27 16:06:07 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1845416, 1846647, 1847876

Description Stefan Schimanski 2020-06-10 11:58:11 UTC

When the local kube-apiserver becomes ready, the GCP LBs pick up the endpoint and route traffic to the IP. In parallel the gcp-routes service notices the local, green readyz and stops sending traffic to that LB.

The second step must happen *BEFORE* the first. Otherwise, local requests still go to the GCP LB that already picked up the endpoint. We risk to blackhole 1/3 of the requests (because GCP has no hairpinning support).

The reason is that the gcp-routes script polls every 5s (without inotify which isn't installed in the image as we know) such that we end up with 1*2s + 1.9999s + 5s until the gcp-routes script updates iptables (1.9999s because the readyz polling happens at an unfortunate time, and the 5s for the poll script to notice). Hence, 9s >> 6s of the LB. Hence, for 3s we might lose 1/3 of the requests originating from the local host.

Compare: https://github.com/openshift/installer/pull/3512/files#diff-3aaac4ae7d381237a540f05371931b76R10

Comment 3 Lalatendu Mohanty 2020-06-17 14:45:36 UTC

Do we have a fork of this bug which is targeted for 4.5.0?

Comment 4 Kirsten Garrison 2020-06-18 18:45:43 UTC

@Lala see: https://github.com/openshift/machine-config-operator/pull/1821

It's been merged into 4.5

Comment 5 Kirsten Garrison 2020-06-18 20:32:59 UTC

@Stefan is the acceptance criteria for this the same as https://bugzilla.redhat.com/show_bug.cgi?id=1845416#c13 ?

Comment 8 errata-xmlrpc 2020-10-27 16:06:07 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196