Bug 1988801

Description of problem:
Since the Power of Two Random Choices balancing algorithm was introduced in Red Hat OpenShift Container Platform 4.8 and has been enabled by default to achieve equal distribution of requests toward the backend pods.

https://docs.openshift.com/container-platform/4.8/release_notes/ocp-4-8-release-notes.html#ocp-4-8-ingress-controller-power-of-two-random-choices

However, the $balanceAlgoPattern var in haproxy-config.template is missing 'random' field.
https://github.com/openshift/router/blob/release-4.8/images/router/haproxy/conf/haproxy-config.template#L21-L22
~~~
{{- /* balanceAlgoPattern matches valid options for the haproxy.router.openshift.io/balance annotation. */}}
{{- $balanceAlgoPattern := "roundrobin|leastconn|source" -}}
~~~

Because currently the backend balance option is setting for 'firstMatch $balanceAlgoPattern ', if the $balanceAlgoPattern is missing 'random' filed, the balance option will be the blank value just like this:

~~~
backend be_edge_http:openshift-console:downloads
  mode http
  option redispatch
  option forwardfor
  balance               <- Here is missing 'random' field
~~~

The balance option with the blank value will be the default 'roundrobin' value, it is NOT 'random' value.
https://cbonte.github.io/haproxy-dconv/2.2/configuration.html#4.2-balance
The load balancing algorithm of a backend is set to roundrobin when no other
algorithm, mode nor option have been set. The algorithm may only be set once
for each backend.

Please refer to the sourcecode as below:

https://github.com/openshift/router/blob/release-4.8/images/router/haproxy/conf/haproxy-config.template#L474-L478
~~~
        {{- with $balanceAlgo := firstMatch $balanceAlgoPattern (index $cfg.Annotations "haproxy.router.openshift.io/balance") }}
  balance {{ $balanceAlgo }}
        {{- else }}
  balance {{ if gt $cfg.ActiveServiceUnits 1 }}roundrobin{{ else }}{{ firstMatch $balanceAlgoPattern (env "ROUTER_LOAD_BALANCE_ALGORITHM") "random" }}{{ end }}
        {{- end }}
~~~

https://github.com/openshift/router/blob/release-4.8/images/router/haproxy/conf/haproxy-config.template#L659-L665
~~~
# Secure backend, pass through
backend {{ genBackendNamePrefix $cfg.TLSTermination }}:{{ $cfgIdx }}
        {{- with $balanceAlgo := firstMatch $balanceAlgoPattern (index $cfg.Annotations "haproxy.router.openshift.io/balance") }}
  balance {{ $balanceAlgo }}
        {{- else }}
  balance {{ if gt $cfg.ActiveServiceUnits 1 }}roundrobin{{ else }}{{ firstMatch $balanceAlgoPattern (env "ROUTER_TCP_BALANCE_SCHEME") (env "ROUTER_LOAD_BALANCE_ALGORITHM") "source" }}{{ end }}
        {{- end }}
~~~


OpenShift release version:
4.8.2

Cluster Platform:
OCP4.8

How reproducible:
# oc project openshift-ingress
# oc rsh router-default-f55d8b6cb-25lqk
# view haproxy.config 

Find the backend balance option in haproxy.config.
~~~
backend be_edge_http:openshift-console:downloads
  mode http
  option redispatch
  option forwardfor
  balance
~~~

If try to change the unsupportedConfigOverrides 'loadBalancingAlgorithm' option to balance 'leastconn', it will not be the blank value.

# oc -n openshift-ingress-operator patch ingresscontroller/default --type=merge --patch='{"spec":{"unsupportedConfigOverrides":{"loadBalancingAlgorithm":"leastconn"}}}'

~~~
backend be_edge_http:openshift-console:downloads
  mode http
  option redispatch
  option forwardfor
  balance leastconn
~~~

Steps to Reproduce (in detail):
1.
2.
3.


Actual results:
Currently the backend balance option is blank in haproxy.config.
~~~
backend be_edge_http:openshift-console:downloads
  mode http
  option redispatch
  option forwardfor
  balance
~~~

Expected results:
Should set up the backend balance option is 'random' in haproxy.config.
~~~
backend be_edge_http:openshift-console:downloads
  mode http
  option redispatch
  option forwardfor
  balance random
~~~

Impact of the problem:
Since OCP4.8, the Power of Two Random Choices balancing algorithm cannot be used by default, becuase missing the balance option with 'random' that the balancing algorithm will be 'roundrobin'. Please fix the issue ASAP.


Additional info:
[release-notes]Power of Two Random Choices balancing algorithm for Ingress Controller
https://docs.openshift.com/container-platform/4.8/release_notes/ocp-4-8-release-notes.html#ocp-4-8-ingress-controller-power-of-two-random-choices

[KCS] How to change ROUTER_BACKEND_PROCESS_ENDPOINTS=shuffle in OpenShift Container Platform 4?
https://access.redhat.com/solutions/6212451

[RFE] Expose ROUTER_BACKEND_PROCESS_ENDPOINTS as this needs to be configurable
https://issues.redhat.com/browse/RFE-380


** Please do not disregard the report template; filling the template out as much as possible will allow us to help you. Please consider attaching a must-gather archive (via `oc adm must-gather`). Please review must-gather contents for sensitive information before attaching any must-gathers to a bugzilla report.  You may also mark the bug private if you wish.