Bug 1937972

Summary:	Router HAProxy config file template is slow to render due to repetitive regex compilations
Product:	OpenShift Container Platform	Reporter:	Stephen Greene <sgreene>
Component:	Networking	Assignee:	Andrew McDermott <amcdermo>
Networking sub component:	router	QA Contact:	Hongan Li <hongli>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	high
Priority:	high	CC:	aos-bugs, bperkins
Version:	3.11.0
Target Milestone:	---
Target Release:	4.8.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Cause: The HAProxy template used by openshift-router makes repeated calls to a firstMatch() function. That function would parse and recompile a regular expression each and every time. Consequence: Parsing and recompiling the regular expression on each call to firstMatch() is expensive, particularly for configurations that have many thousands of routes. Fix: If the regular expression (in the call to firstMatch()) has already been seen then we reuse an already compiled version (i.e., this is now cached). Result: A ~60% reduction in execution time when parsing/evaluating the haproxy-config.template.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-07-27 22:53:17 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1938921

Description Stephen Greene 2021-03-11 19:52:28 UTC

The OpenShift router uses a go template [1] to render the necessary config files [2] to be consumed by HAProxy.

The HAProxy config file template uses several regular expression calls via helper functions to match on various input data [3].

When the template is being rendered, these regular expression based helper functions are called many times, depending on the number of routes available in the cluster.

Each time a regular expression call is made, a new go regular expression is compiled for the given pattern, even if a similar regex for the same pattern has already been compiled. This creates lots of unnecessary churn when rendering the HAProxy config file template. It has been estimated by Andrew McDermott via manual testing that caching compiled regular expression definitions for re-use when rendering the HAProxy config file template could expedite the rendering process by as much as 60%.

This optimization would be a huge help for clusters with route counts upwards of 5,000, where simply rendering the HAProxy config file template can take upwards of 15 seconds.

Ideally this optimization patch would be backported to all versions of OCP 4.x that are currently supported, as well as OCP 3.11.

---
[1]: https://golang.org/pkg/text/template/
[2]: https://github.com/openshift/router/blob/master/images/router/haproxy/conf/haproxy-config.template
[3]: https://github.com/openshift/router/blob/master/pkg/router/template/template_helper.go#L34-L48

Comment 2 Hongan Li 2021-03-18 08:35:41 UTC

Verified with 4.8.0-0.nightly-2021-03-17-194217 and passed.

create over 5k routes and can see logs as below

$ oc -n openshift-ingress logs router-default-754fc85487-9gj84 | grep writeConfig | tail
I0318 08:25:11.936247       1 router.go:456] template "msg"="writeConfig"  "duration"="770.211351ms"
I0318 08:25:17.019523       1 router.go:456] template "msg"="writeConfig"  "duration"="853.378275ms"
I0318 08:25:22.065676       1 router.go:456] template "msg"="writeConfig"  "duration"="899.098073ms"
I0318 08:25:27.003678       1 router.go:456] template "msg"="writeConfig"  "duration"="837.05629ms"
I0318 08:25:32.210367       1 router.go:456] template "msg"="writeConfig"  "duration"="1.043660586s"
I0318 08:25:37.192196       1 router.go:456] template "msg"="writeConfig"  "duration"="1.025401976s"


sh-4.4$ openshift-router version
openshift-router

majorFromGit: 
minorFromGit: 
commitFromGit: 0e2962eaa66cbfce9853ceb802d591778495191c
versionFromGit: 4.0.0-280-g0e2962e
gitTreeState: clean
buildDate: 2021-03-15T23:37:11Z

Comment 5 errata-xmlrpc 2021-07-27 22:53:17 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438