Bug 1988801 - Router HAProxy backend balance option is blank missing random argument in haproxy.config
Summary: Router HAProxy backend balance option is blank missing random argument in hap...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.8
Hardware: x86_64
OS: Linux
urgent
urgent
Target Milestone: ---
: 4.9.0
Assignee: Miciah Dashiel Butler Masters
QA Contact: Arvind iyengar
URL:
Whiteboard:
Depends On:
Blocks: 1990370
TreeView+ depends on / blocked
 
Reported: 2021-08-01 08:33 UTC by Jie Wu
Modified: 2022-10-13 10:34 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1990370 (view as bug list)
Environment:
Last Closed: 2021-10-18 17:43:44 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift router pull 323 0 None None None 2021-08-02 14:03:17 UTC
Red Hat Product Errata RHSA-2021:3759 0 None None None 2021-10-18 17:43:58 UTC

Description Jie Wu 2021-08-01 08:33:14 UTC
Description of problem:
Since the Power of Two Random Choices balancing algorithm was introduced in Red Hat OpenShift Container Platform 4.8 and has been enabled by default to achieve equal distribution of requests toward the backend pods.

https://docs.openshift.com/container-platform/4.8/release_notes/ocp-4-8-release-notes.html#ocp-4-8-ingress-controller-power-of-two-random-choices

However, the $balanceAlgoPattern var in haproxy-config.template is missing 'random' field.
https://github.com/openshift/router/blob/release-4.8/images/router/haproxy/conf/haproxy-config.template#L21-L22
~~~
{{- /* balanceAlgoPattern matches valid options for the haproxy.router.openshift.io/balance annotation. */}}
{{- $balanceAlgoPattern := "roundrobin|leastconn|source" -}}
~~~

Because currently the backend balance option is setting for 'firstMatch $balanceAlgoPattern ', if the $balanceAlgoPattern is missing 'random' filed, the balance option will be the blank value just like this:

~~~
backend be_edge_http:openshift-console:downloads
  mode http
  option redispatch
  option forwardfor
  balance               <- Here is missing 'random' field
~~~

The balance option with the blank value will be the default 'roundrobin' value, it is NOT 'random' value.
https://cbonte.github.io/haproxy-dconv/2.2/configuration.html#4.2-balance
The load balancing algorithm of a backend is set to roundrobin when no other
algorithm, mode nor option have been set. The algorithm may only be set once
for each backend.

Please refer to the sourcecode as below:

https://github.com/openshift/router/blob/release-4.8/images/router/haproxy/conf/haproxy-config.template#L474-L478
~~~
        {{- with $balanceAlgo := firstMatch $balanceAlgoPattern (index $cfg.Annotations "haproxy.router.openshift.io/balance") }}
  balance {{ $balanceAlgo }}
        {{- else }}
  balance {{ if gt $cfg.ActiveServiceUnits 1 }}roundrobin{{ else }}{{ firstMatch $balanceAlgoPattern (env "ROUTER_LOAD_BALANCE_ALGORITHM") "random" }}{{ end }}
        {{- end }}
~~~

https://github.com/openshift/router/blob/release-4.8/images/router/haproxy/conf/haproxy-config.template#L659-L665
~~~
# Secure backend, pass through
backend {{ genBackendNamePrefix $cfg.TLSTermination }}:{{ $cfgIdx }}
        {{- with $balanceAlgo := firstMatch $balanceAlgoPattern (index $cfg.Annotations "haproxy.router.openshift.io/balance") }}
  balance {{ $balanceAlgo }}
        {{- else }}
  balance {{ if gt $cfg.ActiveServiceUnits 1 }}roundrobin{{ else }}{{ firstMatch $balanceAlgoPattern (env "ROUTER_TCP_BALANCE_SCHEME") (env "ROUTER_LOAD_BALANCE_ALGORITHM") "source" }}{{ end }}
        {{- end }}
~~~


OpenShift release version:
4.8.2

Cluster Platform:
OCP4.8

How reproducible:
# oc project openshift-ingress
# oc rsh router-default-f55d8b6cb-25lqk
# view haproxy.config 

Find the backend balance option in haproxy.config.
~~~
backend be_edge_http:openshift-console:downloads
  mode http
  option redispatch
  option forwardfor
  balance
~~~

If try to change the unsupportedConfigOverrides 'loadBalancingAlgorithm' option to balance 'leastconn', it will not be the blank value.

# oc -n openshift-ingress-operator patch ingresscontroller/default --type=merge --patch='{"spec":{"unsupportedConfigOverrides":{"loadBalancingAlgorithm":"leastconn"}}}'

~~~
backend be_edge_http:openshift-console:downloads
  mode http
  option redispatch
  option forwardfor
  balance leastconn
~~~

Steps to Reproduce (in detail):
1.
2.
3.


Actual results:
Currently the backend balance option is blank in haproxy.config.
~~~
backend be_edge_http:openshift-console:downloads
  mode http
  option redispatch
  option forwardfor
  balance
~~~

Expected results:
Should set up the backend balance option is 'random' in haproxy.config.
~~~
backend be_edge_http:openshift-console:downloads
  mode http
  option redispatch
  option forwardfor
  balance random
~~~

Impact of the problem:
Since OCP4.8, the Power of Two Random Choices balancing algorithm cannot be used by default, becuase missing the balance option with 'random' that the balancing algorithm will be 'roundrobin'. Please fix the issue ASAP.


Additional info:
[release-notes]Power of Two Random Choices balancing algorithm for Ingress Controller
https://docs.openshift.com/container-platform/4.8/release_notes/ocp-4-8-release-notes.html#ocp-4-8-ingress-controller-power-of-two-random-choices

[KCS] How to change ROUTER_BACKEND_PROCESS_ENDPOINTS=shuffle in OpenShift Container Platform 4?
https://access.redhat.com/solutions/6212451

[RFE] Expose ROUTER_BACKEND_PROCESS_ENDPOINTS as this needs to be configurable
https://issues.redhat.com/browse/RFE-380


** Please do not disregard the report template; filling the template out as much as possible will allow us to help you. Please consider attaching a must-gather archive (via `oc adm must-gather`). Please review must-gather contents for sensitive information before attaching any must-gathers to a bugzilla report.  You may also mark the bug private if you wish.

Comment 2 Arvind iyengar 2021-08-05 07:36:18 UTC
Verified in "4.9.0-0.nightly-2021-08-03-135217" release version. With this payload, it is observed that the "random" option now gets set properly. 
------
oc get clusterversion                            
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.0-0.nightly-2021-08-03-135217   True        False         83m     Cluster version is 4.9.0-0.nightly-2021-08-03-135217

sh-4.4$ env  | grep ROUTER_LOAD_BALANCE_ALGORITHM
ROUTER_LOAD_BALANCE_ALGORITHM=random

Haproxy router
# Plain http backend or backend with TLS terminated at the edge or a
# secure backend with re-encryption.
backend be_http:test1:service-unsecure
  mode http
  option redispatch
  option forwardfor
  balance random <----
------

Comment 8 errata-xmlrpc 2021-10-18 17:43:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759


Note You need to log in before you can comment on or make changes to this bug.