Bug 1885414 - Need to disable HTX when not using HTTP/2 in order to preserve HTTP header name case
Summary: Need to disable HTX when not using HTTP/2 in order to preserve HTTP header na...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.4
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.7.0
Assignee: Miciah Dashiel Butler Masters
QA Contact: Arvind iyengar
URL:
Whiteboard:
Depends On:
Blocks: 1886213
TreeView+ depends on / blocked
 
Reported: 2020-10-05 20:57 UTC by Miciah Dashiel Butler Masters
Modified: 2022-08-04 22:30 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-02-24 15:23:10 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift router pull 194 0 None closed Bug 1885414: haproxy-config.template: Only enable HTX for HTTP/2 2021-02-16 16:33:28 UTC
Red Hat Product Errata RHSA-2020:5633 0 None None None 2021-02-24 15:23:51 UTC

Description Miciah Dashiel Butler Masters 2020-10-05 20:57:34 UTC
Description of problem: 

Recently, the ability to enable HTTP/2 for ingress controllers (which are based on HAProxy) was added.  Enabling HAProxy to use HTTP/2 to connect to backend servers required enabling HAProxy's "HTX" option.  However, enabling HTX causes HAProxy to down-case HTTP header names.  HAProxy's behavior is standards-compliant because HTTP header names are case-insensitive.  However, some non-compliant legacy HTTP clients and servers require that case be preserved in HTTP header names.  

In order to provide temporary backwards compatibility with these non-compliant clients and servers, we should turn off HTX when HTTP/2 is not enabled, so that the case of HTTP header names is preserved.  


Version-Release number of selected component (if applicable): 

HTX was first enabled in OCP 4.4.  


How reproducible: 

Reliably.  


Steps to Reproduce: 

1. Launch an OCP 4.4, 4.5, or 4.6 cluster.  

2. Create an HTTP echo server: 

    oc adm new-project echo
    oc -n echo create -f https://raw.githubusercontent.com/openshift/origin/release-3.11/test/extended/testdata/router-http-echo-server.yaml

3. Curl the echo server, specifying some custom headers that include non-lowercase characters in the headers' names: 

    INGRESS_DOMAIN="$(oc get ingress.config/cluster -o 'jsonpath={.spec.domain}')"
    ADDR="$(dig "xyz.$INGRESS_DOMAIN" +short)"
    curl http://router-headers.example.com --resolve "router-headers.example.com:80:$ADDR" -H X-foo:foo -H X-Bar:Bar -H X-BaZ:BaZ


Actual results:

The output of Step 3 shows that the header names are down-cased:

    % curl http://router-headers.example.com --resolve "router-headers.example.com:80:$ADDR" -H X-foo:foo -H X-Bar:Bar -H X-BaZ:BaZ
    GET / HTTP/1.1
    user-agent: curl/7.29.0
    accept: */*
    x-foo: foo
    x-bar: Bar
    x-baz: BaZ
    host: router-headers.example.com
    x-forwarded-host: router-headers.example.com
    x-forwarded-port: 80
    x-forwarded-proto: http
    forwarded: for=75.170.24.197;host=router-headers.example.com;proto=http
    x-forwarded-for: 75.170.24.197


Expected results:

The output of Step 3 should show that the case of the header names is preserved:

    % curl http://router-headers.example.com --resolve "router-headers.example.com:80:$ADDR" -H X-foo:foo -H X-Bar:Bar -H X-BaZ:BaZ
    GET / HTTP/1.1
    User-Agent: curl/7.29.0
    Accept: */*
    X-foo:foo
    X-Bar:Bar
    X-BaZ:BaZ
    Host: router-headers.example.com
    X-Forwarded-Host: router-headers.example.com
    X-Forwarded-Port: 80
    X-Forwarded-Proto: http
    Forwarded: for=75.170.24.197;host=router-headers.example.com;proto=http
    X-Forwarded-For: 75.170.24.197


Additional info:

RFC 7230 states that HTTP header names are case-insensitive:

>   Each header field consists of a case-insensitive field name followed
>   by a colon (":"), optional leading whitespace, the field value, and
>   optional trailing whitespace.

https://tools.ietf.org/html/rfc7230#section-3.2

HTX becomes non-configurable in HAProxy 2.2, so turning off HTX is an interim solution.  Ideally, legacy applications need to be updated to bring them into conformance.  Alternatively, we can use HAProxy 2.2's h1-case-adjust option to preserve case for specific HTTP headers.

Comment 3 Arvind iyengar 2020-10-20 10:07:18 UTC
Tested in "4.7.0-0.nightly-2020-10-17-034503" release. With this payload, it is noted that the haproxy configuration now has the "no option http-use-htx" with http2 disabled, and the headers are no more being moderated for the lower/upper cases:
-----
$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.nightly-2020-10-17-034503   True        False         3h57m   Cluster version is 4.7.0-0.nightly-2020-10-17-03450


$ oc get all
NAME                            READY   STATUS      RESTARTS   AGE
pod/router-http-echo-1-79nll    1/1     Running     0          89m
pod/router-http-echo-1-deploy   0/1     Completed   0          89m

NAME                                       DESIRED   CURRENT   READY   AGE
replicationcontroller/router-http-echo-1   1         1         1       89m

NAME                       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/router-http-echo   ClusterIP   172.30.219.51   <none>        8676/TCP   89m

NAME                                                  REVISION   DESIRED   CURRENT   TRIGGERED BY
deploymentconfig.apps.openshift.io/router-http-echo   1          1         1         config


$ oc -n openshift-ingress exec router-internalapps-64688cdb7b-l8b8v  -- env | grep -i http2
ROUTER_DISABLE_HTTP2=true
    
$ oc -n openshift-ingress exec router-internalapps-64688cdb7b-l8b8v  -- grep -i "htx" haproxy.config
  no option http-use-htx


$ curl router-http-echo-test1.internalapps.aiyengar-oc47-2010.qe.devcluster.openshift.com
GET / HTTP/1.1
User-Agent: curl/7.68.0
Accept: */*
Host: router-http-echo-test1.internalapps.aiyengar-oc47-2010.qe.devcluster.openshift.com
X-Forwarded-Host: router-http-echo-test1.internalapps.aiyengar-oc47-2010.qe.devcluster.openshift.com
X-Forwarded-Port: 80
X-Forwarded-Proto: http
Forwarded: for=122.169.5.75;host=router-http-echo-test1.internalapps.aiyengar-oc47-2010.qe.devcluster.openshift.com;proto=http
X-Forwarded-For: 122.169.5.75
-----

Comment 7 errata-xmlrpc 2021-02-24 15:23:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633


Note You need to log in before you can comment on or make changes to this bug.