Bug 1826225 - edge terminated h2 (gRPC) connections need a haproxy template change to work correctly
Summary: edge terminated h2 (gRPC) connections need a haproxy template change to work ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.5
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.10.0
Assignee: Miciah Dashiel Butler Masters
QA Contact: Arvind iyengar
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-04-21 09:22 UTC by Andrew McDermott
Modified: 2022-12-21 08:06 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-12 04:34:40 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift origin pull 26456 0 None open Bug 1826225: test/extended/router: Enable edge-terminated gRPC 2021-10-12 21:31:29 UTC
Github openshift router pull 328 0 None Merged Bug 1826225: Support edge-terminated h2 connections 2021-10-12 21:31:32 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-12 04:34:54 UTC

Description Andrew McDermott 2020-04-21 09:22:32 UTC
If your route type is "termination: edge" then your backend may still
need http2 (e.g., gRPC servers). To support such configurations the 
"server" line in the haproxy config needs to change from "alpn h2,http1.1" to "proto h2".


The "proto h2" directive allows HAProxy to communicate using HTTP/2
without TLS, such as HTTP/2-enabled backends like Varnish and H2O,
and gRPC servers.


There is a PR open that implements this:

  https://github.com/openshift/router/pull/104

Comment 4 Andrew McDermott 2020-06-17 10:05:37 UTC
I’m adding UpcomingSprint, because I was occupied by fixing bugs with
higher priority/severity, developing new features with higher
priority, or developing new features to improve stability at a macro
level. I will revisit this bug next sprint.

Comment 5 Andrew McDermott 2020-07-09 12:06:21 UTC
I’m adding UpcomingSprint, because I was occupied by fixing bugs with
higher priority/severity, developing new features with higher
priority, or developing new features to improve stability at a macro
level. I will revisit this bug next sprint.

Comment 6 Andrew McDermott 2020-07-30 10:01:30 UTC
I’m adding UpcomingSprint, because I was occupied by fixing bugs with
higher priority/severity, developing new features with higher
priority, or developing new features to improve stability at a macro
level. I will revisit this bug next sprint.

Comment 7 mfisher 2020-08-18 20:01:39 UTC
Target reset from 4.6 to 4.7 while investigation is either ongoing or not yet started.  Will be considered for earlier release versions when diagnosed and resolved.

Comment 9 Kenjiro Nakayama 2020-09-07 23:59:52 UTC
Just wanted to mention that serverless (Knative) cannot support gRPC/HTTP2 due to this issue. Here is our tracking JIRA ticket https://issues.redhat.com/browse/SRVKS-211 which has been opened since last year.
Although we haven't got any critical escalation yet, we were sometimes asked about it.

Comment 10 Andrew McDermott 2020-09-10 11:46:05 UTC
I’m adding UpcomingSprint, because I was occupied by fixing bugs with
higher priority/severity, developing new features with higher
priority, or developing new features to improve stability at a macro
level. I will revisit this bug next sprint.

Comment 12 Andrew McDermott 2020-10-02 16:39:19 UTC
Tagging with UpcomingSprint while investigation is either ongoing or
pending. Will be considered for earlier release versions when
diagnosed and resolved.

Comment 13 Andrew McDermott 2020-10-23 15:59:40 UTC
Tagging with UpcomingSprint while investigation is either ongoing or
pending. Will be considered for earlier release versions when
diagnosed and resolved.

Comment 14 Andrew McDermott 2020-11-16 08:30:18 UTC
Tagging with UpcomingSprint while investigation is either ongoing or
pending. Will be considered for earlier release versions when
diagnosed and resolved.

Comment 15 Andrew McDermott 2020-12-04 16:47:45 UTC
Tagging with UpcomingSprint while investigation is either ongoing or
pending. Will be considered for earlier release versions when
diagnosed and resolved.

Comment 22 Miciah Dashiel Butler Masters 2021-08-31 16:18:48 UTC
We'll move this to 4.10 due to lack of review bandwidth and risk to making this change this close to code freeze.  We can backport it to 4.9.z if needed.

Comment 25 Arvind iyengar 2021-11-12 09:27:13 UTC
verified in "4.10.0-0.nightly-2021-11-09-181140" release. With this version, it is observed that the "h2" flag is getting applied properly on the backend that connects over http2:
-------
oc get clusterversion                         
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.0-0.nightly-2021-11-09-181140   True        False         5h8m    Cluster version is 4.10.0-0.nightly-2021-11-09-181140


oc get all   
NAME               READY   STATUS    RESTARTS   AGE
pod/grpc-interop   1/1     Running   0          84m

NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
service/grpc-interop   ClusterIP   172.30.170.175   <none>        1110/TCP,8443/TCP   21m

NAME                                                HOST/PORT                                                                      PATH   SERVICES       PORT   TERMINATION            WILDCARD
route.route.openshift.io/grpc-interop-edge          grpc-interop-edge-test2.apps.aiyengar410b.qe.devcluster.openshift.com                 grpc-interop   1110   edge/Redirect          None


Haproxy backend configuration:

backend be_edge_http:test2:grpc-interop-edge
  mode http
  option redispatch
  option forwardfor
  balance leastconn

  timeout check 5000ms
  http-request add-header X-Forwarded-Host %[req.hdr(host)]
  http-request add-header X-Forwarded-Port %[dst_port]
  http-request add-header X-Forwarded-Proto http if !{ ssl_fc }
  http-request add-header X-Forwarded-Proto https if { ssl_fc }
  http-request add-header X-Forwarded-Proto-Version h2 if { ssl_fc_alpn -i h2 }
  http-request add-header Forwarded for=%[src];host=%[req.hdr(host)];proto=%[req.hdr(X-Forwarded-Proto)]
  cookie 9ad8d0bbc0bfab078a3056b6e254ba58 insert indirect nocache httponly secure attr SameSite=None
  server pod:grpc-interop:grpc-interop:h2c:10.128.2.34:1110 10.128.2.34:1110 cookie 223736a291101c9209aa6d0d49da8422 weight 256 proto h2 check inter 5000ms <----
-------

Comment 31 errata-xmlrpc 2022-03-12 04:34:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056


Note You need to log in before you can comment on or make changes to this bug.