Description of problem: Recently, the ability to enable HTTP/2 for ingress controllers (which are based on HAProxy) was added. Enabling HAProxy to use HTTP/2 to connect to backend servers required enabling HAProxy's "HTX" option. However, enabling HTX causes HAProxy to down-case HTTP header names. HAProxy's behavior is standards-compliant because HTTP header names are case-insensitive. However, some non-compliant legacy HTTP clients and servers require that case be preserved in HTTP header names. In order to provide temporary backwards compatibility with these non-compliant clients and servers, we should turn off HTX when HTTP/2 is not enabled, so that the case of HTTP header names is preserved. Version-Release number of selected component (if applicable): HTX was first enabled in OCP 4.4. How reproducible: Reliably. Steps to Reproduce: 1. Launch an OCP 4.4, 4.5, or 4.6 cluster. 2. Create an HTTP echo server: oc adm new-project echo oc -n echo create -f https://raw.githubusercontent.com/openshift/origin/release-3.11/test/extended/testdata/router-http-echo-server.yaml 3. Curl the echo server, specifying some custom headers that include non-lowercase characters in the headers' names: INGRESS_DOMAIN="$(oc get ingress.config/cluster -o 'jsonpath={.spec.domain}')" ADDR="$(dig "xyz.$INGRESS_DOMAIN" +short)" curl http://router-headers.example.com --resolve "router-headers.example.com:80:$ADDR" -H X-foo:foo -H X-Bar:Bar -H X-BaZ:BaZ Actual results: The output of Step 3 shows that the header names are down-cased: % curl http://router-headers.example.com --resolve "router-headers.example.com:80:$ADDR" -H X-foo:foo -H X-Bar:Bar -H X-BaZ:BaZ GET / HTTP/1.1 user-agent: curl/7.29.0 accept: */* x-foo: foo x-bar: Bar x-baz: BaZ host: router-headers.example.com x-forwarded-host: router-headers.example.com x-forwarded-port: 80 x-forwarded-proto: http forwarded: for=75.170.24.197;host=router-headers.example.com;proto=http x-forwarded-for: 75.170.24.197 Expected results: The output of Step 3 should show that the case of the header names is preserved: % curl http://router-headers.example.com --resolve "router-headers.example.com:80:$ADDR" -H X-foo:foo -H X-Bar:Bar -H X-BaZ:BaZ GET / HTTP/1.1 User-Agent: curl/7.29.0 Accept: */* X-foo:foo X-Bar:Bar X-BaZ:BaZ Host: router-headers.example.com X-Forwarded-Host: router-headers.example.com X-Forwarded-Port: 80 X-Forwarded-Proto: http Forwarded: for=75.170.24.197;host=router-headers.example.com;proto=http X-Forwarded-For: 75.170.24.197 Additional info: RFC 7230 states that HTTP header names are case-insensitive: > Each header field consists of a case-insensitive field name followed > by a colon (":"), optional leading whitespace, the field value, and > optional trailing whitespace. https://tools.ietf.org/html/rfc7230#section-3.2 HTX becomes non-configurable in HAProxy 2.2, so turning off HTX is an interim solution. Ideally, legacy applications need to be updated to bring them into conformance. Alternatively, we can use HAProxy 2.2's h1-case-adjust option to preserve case for specific HTTP headers.
Tested in "4.7.0-0.nightly-2020-10-17-034503" release. With this payload, it is noted that the haproxy configuration now has the "no option http-use-htx" with http2 disabled, and the headers are no more being moderated for the lower/upper cases: ----- $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.7.0-0.nightly-2020-10-17-034503 True False 3h57m Cluster version is 4.7.0-0.nightly-2020-10-17-03450 $ oc get all NAME READY STATUS RESTARTS AGE pod/router-http-echo-1-79nll 1/1 Running 0 89m pod/router-http-echo-1-deploy 0/1 Completed 0 89m NAME DESIRED CURRENT READY AGE replicationcontroller/router-http-echo-1 1 1 1 89m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/router-http-echo ClusterIP 172.30.219.51 <none> 8676/TCP 89m NAME REVISION DESIRED CURRENT TRIGGERED BY deploymentconfig.apps.openshift.io/router-http-echo 1 1 1 config $ oc -n openshift-ingress exec router-internalapps-64688cdb7b-l8b8v -- env | grep -i http2 ROUTER_DISABLE_HTTP2=true $ oc -n openshift-ingress exec router-internalapps-64688cdb7b-l8b8v -- grep -i "htx" haproxy.config no option http-use-htx $ curl router-http-echo-test1.internalapps.aiyengar-oc47-2010.qe.devcluster.openshift.com GET / HTTP/1.1 User-Agent: curl/7.68.0 Accept: */* Host: router-http-echo-test1.internalapps.aiyengar-oc47-2010.qe.devcluster.openshift.com X-Forwarded-Host: router-http-echo-test1.internalapps.aiyengar-oc47-2010.qe.devcluster.openshift.com X-Forwarded-Port: 80 X-Forwarded-Proto: http Forwarded: for=122.169.5.75;host=router-http-echo-test1.internalapps.aiyengar-oc47-2010.qe.devcluster.openshift.com;proto=http X-Forwarded-For: 122.169.5.75 -----
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5633