Bug 1937972 - Router HAProxy config file template is slow to render due to repetitive regex compilations
Summary: Router HAProxy config file template is slow to render due to repetitive regex...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 3.11.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.8.0
Assignee: Andrew McDermott
QA Contact: Hongan Li
URL:
Whiteboard:
Depends On:
Blocks: 1938921
TreeView+ depends on / blocked
 
Reported: 2021-03-11 19:52 UTC by Stephen Greene
Modified: 2022-08-04 22:32 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The HAProxy template used by openshift-router makes repeated calls to a firstMatch() function. That function would parse and recompile a regular expression each and every time. Consequence: Parsing and recompiling the regular expression on each call to firstMatch() is expensive, particularly for configurations that have many thousands of routes. Fix: If the regular expression (in the call to firstMatch()) has already been seen then we reuse an already compiled version (i.e., this is now cached). Result: A ~60% reduction in execution time when parsing/evaluating the haproxy-config.template.
Clone Of:
Environment:
Last Closed: 2021-07-27 22:53:17 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift router pull 268 0 None closed Bug 1937972: router/template: Cache compiled regular expressions 2021-03-27 18:41:27 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 22:53:36 UTC

Description Stephen Greene 2021-03-11 19:52:28 UTC
The OpenShift router uses a go template [1] to render the necessary config files [2] to be consumed by HAProxy.

The HAProxy config file template uses several regular expression calls via helper functions to match on various input data [3].

When the template is being rendered, these regular expression based helper functions are called many times, depending on the number of routes available in the cluster.

Each time a regular expression call is made, a new go regular expression is compiled for the given pattern, even if a similar regex for the same pattern has already been compiled. This creates lots of unnecessary churn when rendering the HAProxy config file template. It has been estimated by Andrew McDermott via manual testing that caching compiled regular expression definitions for re-use when rendering the HAProxy config file template could expedite the rendering process by as much as 60%.

This optimization would be a huge help for clusters with route counts upwards of 5,000, where simply rendering the HAProxy config file template can take upwards of 15 seconds.

Ideally this optimization patch would be backported to all versions of OCP 4.x that are currently supported, as well as OCP 3.11.

---
[1]: https://golang.org/pkg/text/template/
[2]: https://github.com/openshift/router/blob/master/images/router/haproxy/conf/haproxy-config.template
[3]: https://github.com/openshift/router/blob/master/pkg/router/template/template_helper.go#L34-L48

Comment 2 Hongan Li 2021-03-18 08:35:41 UTC
Verified with 4.8.0-0.nightly-2021-03-17-194217 and passed.

create over 5k routes and can see logs as below

$ oc -n openshift-ingress logs router-default-754fc85487-9gj84 | grep writeConfig | tail
I0318 08:25:11.936247       1 router.go:456] template "msg"="writeConfig"  "duration"="770.211351ms"
I0318 08:25:17.019523       1 router.go:456] template "msg"="writeConfig"  "duration"="853.378275ms"
I0318 08:25:22.065676       1 router.go:456] template "msg"="writeConfig"  "duration"="899.098073ms"
I0318 08:25:27.003678       1 router.go:456] template "msg"="writeConfig"  "duration"="837.05629ms"
I0318 08:25:32.210367       1 router.go:456] template "msg"="writeConfig"  "duration"="1.043660586s"
I0318 08:25:37.192196       1 router.go:456] template "msg"="writeConfig"  "duration"="1.025401976s"


sh-4.4$ openshift-router version
openshift-router

majorFromGit: 
minorFromGit: 
commitFromGit: 0e2962eaa66cbfce9853ceb802d591778495191c
versionFromGit: 4.0.0-280-g0e2962e
gitTreeState: clean
buildDate: 2021-03-15T23:37:11Z

Comment 5 errata-xmlrpc 2021-07-27 22:53:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438


Note You need to log in before you can comment on or make changes to this bug.