Description of problem: Since the Power of Two Random Choices balancing algorithm was introduced in Red Hat OpenShift Container Platform 4.8 and has been enabled by default to achieve equal distribution of requests toward the backend pods. https://docs.openshift.com/container-platform/4.8/release_notes/ocp-4-8-release-notes.html#ocp-4-8-ingress-controller-power-of-two-random-choices However, the $balanceAlgoPattern var in haproxy-config.template is missing 'random' field. https://github.com/openshift/router/blob/release-4.8/images/router/haproxy/conf/haproxy-config.template#L21-L22 ~~~ {{- /* balanceAlgoPattern matches valid options for the haproxy.router.openshift.io/balance annotation. */}} {{- $balanceAlgoPattern := "roundrobin|leastconn|source" -}} ~~~ Because currently the backend balance option is setting for 'firstMatch $balanceAlgoPattern ', if the $balanceAlgoPattern is missing 'random' filed, the balance option will be the blank value just like this: ~~~ backend be_edge_http:openshift-console:downloads mode http option redispatch option forwardfor balance <- Here is missing 'random' field ~~~ The balance option with the blank value will be the default 'roundrobin' value, it is NOT 'random' value. https://cbonte.github.io/haproxy-dconv/2.2/configuration.html#4.2-balance The load balancing algorithm of a backend is set to roundrobin when no other algorithm, mode nor option have been set. The algorithm may only be set once for each backend. Please refer to the sourcecode as below: https://github.com/openshift/router/blob/release-4.8/images/router/haproxy/conf/haproxy-config.template#L474-L478 ~~~ {{- with $balanceAlgo := firstMatch $balanceAlgoPattern (index $cfg.Annotations "haproxy.router.openshift.io/balance") }} balance {{ $balanceAlgo }} {{- else }} balance {{ if gt $cfg.ActiveServiceUnits 1 }}roundrobin{{ else }}{{ firstMatch $balanceAlgoPattern (env "ROUTER_LOAD_BALANCE_ALGORITHM") "random" }}{{ end }} {{- end }} ~~~ https://github.com/openshift/router/blob/release-4.8/images/router/haproxy/conf/haproxy-config.template#L659-L665 ~~~ # Secure backend, pass through backend {{ genBackendNamePrefix $cfg.TLSTermination }}:{{ $cfgIdx }} {{- with $balanceAlgo := firstMatch $balanceAlgoPattern (index $cfg.Annotations "haproxy.router.openshift.io/balance") }} balance {{ $balanceAlgo }} {{- else }} balance {{ if gt $cfg.ActiveServiceUnits 1 }}roundrobin{{ else }}{{ firstMatch $balanceAlgoPattern (env "ROUTER_TCP_BALANCE_SCHEME") (env "ROUTER_LOAD_BALANCE_ALGORITHM") "source" }}{{ end }} {{- end }} ~~~ OpenShift release version: 4.8.2 Cluster Platform: OCP4.8 How reproducible: # oc project openshift-ingress # oc rsh router-default-f55d8b6cb-25lqk # view haproxy.config Find the backend balance option in haproxy.config. ~~~ backend be_edge_http:openshift-console:downloads mode http option redispatch option forwardfor balance ~~~ If try to change the unsupportedConfigOverrides 'loadBalancingAlgorithm' option to balance 'leastconn', it will not be the blank value. # oc -n openshift-ingress-operator patch ingresscontroller/default --type=merge --patch='{"spec":{"unsupportedConfigOverrides":{"loadBalancingAlgorithm":"leastconn"}}}' ~~~ backend be_edge_http:openshift-console:downloads mode http option redispatch option forwardfor balance leastconn ~~~ Steps to Reproduce (in detail): 1. 2. 3. Actual results: Currently the backend balance option is blank in haproxy.config. ~~~ backend be_edge_http:openshift-console:downloads mode http option redispatch option forwardfor balance ~~~ Expected results: Should set up the backend balance option is 'random' in haproxy.config. ~~~ backend be_edge_http:openshift-console:downloads mode http option redispatch option forwardfor balance random ~~~ Impact of the problem: Since OCP4.8, the Power of Two Random Choices balancing algorithm cannot be used by default, becuase missing the balance option with 'random' that the balancing algorithm will be 'roundrobin'. Please fix the issue ASAP. Additional info: [release-notes]Power of Two Random Choices balancing algorithm for Ingress Controller https://docs.openshift.com/container-platform/4.8/release_notes/ocp-4-8-release-notes.html#ocp-4-8-ingress-controller-power-of-two-random-choices [KCS] How to change ROUTER_BACKEND_PROCESS_ENDPOINTS=shuffle in OpenShift Container Platform 4? https://access.redhat.com/solutions/6212451 [RFE] Expose ROUTER_BACKEND_PROCESS_ENDPOINTS as this needs to be configurable https://issues.redhat.com/browse/RFE-380 ** Please do not disregard the report template; filling the template out as much as possible will allow us to help you. Please consider attaching a must-gather archive (via `oc adm must-gather`). Please review must-gather contents for sensitive information before attaching any must-gathers to a bugzilla report. You may also mark the bug private if you wish.
Verified in "4.9.0-0.nightly-2021-08-03-135217" release version. With this payload, it is observed that the "random" option now gets set properly. ------ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.9.0-0.nightly-2021-08-03-135217 True False 83m Cluster version is 4.9.0-0.nightly-2021-08-03-135217 sh-4.4$ env | grep ROUTER_LOAD_BALANCE_ALGORITHM ROUTER_LOAD_BALANCE_ALGORITHM=random Haproxy router # Plain http backend or backend with TLS terminated at the edge or a # secure backend with re-encryption. backend be_http:test1:service-unsecure mode http option redispatch option forwardfor balance random <---- ------
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:3759