Bug 1598738 - Error reloading router, too many ips included in the whitelist annotation
Summary: Error reloading router, too many ips included in the whitelist annotation
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 3.9.0
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
: 3.11.0
Assignee: Ram Ranganathan
QA Contact: zhaozhanqi
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-07-06 11:00 UTC by Victor Hernando
Modified: 2022-08-04 22:20 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: If there are too many ip addresses specified in a route via the "haproxy.router.openshift.io/ip_whitelist" annotation, then number of parameters to a haproxy config directive exceeds 63 and there is an error reloading the router. Limitations based on `#define MAX_LINE_ARGS 64` in the haproxy code. Consequence: The router does not reload (invalid config) if there too many ips in the annotation. Fix: We use an overflow map if the there are too many ips in the whitelist annotation (and pass the map to the haproxy acl). Result: Works as intended (without any changes by the user).
Clone Of:
Environment:
Last Closed: 2018-10-11 07:21:36 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github https://github.com/openshift origin pull 20357 0 None None None 2020-09-23 10:19:12 UTC
Red Hat Product Errata RHBA-2018:2652 0 None None None 2018-10-11 07:21:55 UTC

Description Victor Hernando 2018-07-06 11:00:17 UTC
Description of problem:
Adding more than 61 ips/ranges to the haproxy.router.openshift.io/ip_whitelist annotation, the router fails reloading configuration.

Version-Release number of selected component (if applicable):
3.9
Also reproducible in 3.6

How reproducible:
Create a route for your service, add the following annotation with the desired ips/ranges (more than 61) to the ip_whitelist.

metadata:
  annotations:
    haproxy.router.openshift.io/ip_whitelist:

Steps to Reproduce:
1. Create a route
$ oc expose svc test2
route "test2" exposed

2. Export the route
$ oc export route test2 -o yaml > route_test2_wlip.yml

3. Edit the route, add 62 ips/ranges and replace it.
$ cat route_test2_wlip.yml 
apiVersion: v1
kind: Route
metadata:
  annotations:
    openshift.io/host.generated: "true"
    haproxy.router.openshift.io/ip_whitelist: >-
      192.168.1.1 192.168.1.2 192.168.1.3 192.168.1.4 
      192.168.2.5 192.168.2.6 192.168.2.7 192.168.2.8 192.168.2.9 
      192.168.3.10 192.168.3.11 192.168.3.12 192.168.3.13 192.168.3.14 
      192.168.4.15 192.168.4.16 192.168.4.17 192.168.4.18 192.168.4.19 
      192.168.5.20 192.168.5.21 192.168.5.22 192.168.5.23 192.168.5.24
      192.168.6.25 192.168.6.26 192.168.6.27 192.168.6.28 192.168.6.29
      192.168.7.30 192.168.7.31 192.168.7.32 192.168.7.33 192.168.7.34
      192.168.8.35 192.168.8.36 192.168.8.37 192.168.8.38 192.168.8.39
      192.168.9.40 192.168.9.41 192.168.9.42 192.168.9.43 192.168.9.44
      192.168.10.45 192.168.10.46 192.168.10.47 192.168.10.48 192.168.10.49
      192.168.11.50 192.168.11.51 192.168.11.52 192.168.11.53 192.168.11.54
      192.168.12.55 192.168.12.56 192.168.12.57 192.168.12.58 192.168.12.59
      192.168.13.60 192.168.13.61
...

# oc logs -f router-1-cv5nd -n default
E0706 10:49:47.595860       1 limiter.go:137] error reloading router: exit status 1
[ALERT] 186/104947 (1075) : parsing [/var/lib/haproxy/conf/haproxy.config:245]: line too long, truncating at word 65, position 826: <192.168.13.62>.
[ALERT] 186/104947 (1075) : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config
[ALERT] 186/104947 (1075) : Fatal errors found in configuration.

Actual results:
The router is not able to reload the new configuration adding 62 ips to the annotation.

E0706 10:49:47.595860       1 limiter.go:137] error reloading router: exit status 1
[ALERT] 186/104947 (1075) : parsing [/var/lib/haproxy/conf/haproxy.config:245]: line too long, truncating at word 65, position 826: <192.168.13.62>.
[ALERT] 186/104947 (1075) : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config
[ALERT] 186/104947 (1075) : Fatal errors found in configuration

Adding 61 ips/ranges or less, the router is reloading successfully the configuration.

I0706 10:43:24.592134       1 router.go:441] Router reloaded:
 - Checking http://localhost:80 ...
 - Health check ok : 0 retry attempt(s).


Expected results:
Reloading the configuration successfully when more ips/ranges are added to the annotation.
If there is a known limit, should be interesting having this constraint documented. 

Additional info:

Comment 1 Ram Ranganathan 2018-07-12 00:29:43 UTC
With all those ip/ranges, this is hitting the haproxy max args per line limit (64). 
https://github.com/haproxy/haproxy/blob/673867c357260d13a8a6fcd17a3d2eb77ba10e5c/include/common/defaults.h#L75

I got the same error adding one more ip 127.0.0.1 to the list above. 
$ oc edit route <name> 
# and add the following to the metadata
metadata:
  annotations:
    haproxy.router.openshift.io/ip_whitelist: 192.168.1.1 192.168.1.2 192.168.1.3 192.168.1.4
      192.168.2.5 192.168.2.6 192.168.2.7 192.168.2.8 192.168.2.9 192.168.3.10 192.168.3.11
      192.168.3.12 192.168.3.13 192.168.3.14 192.168.4.15 192.168.4.16 192.168.4.17
      192.168.4.18 192.168.4.19 192.168.5.20 192.168.5.21 192.168.5.22 192.168.5.23
      192.168.5.24 192.168.6.25 192.168.6.26 192.168.6.27 192.168.6.28 192.168.6.29
      192.168.7.30 192.168.7.31 192.168.7.32 192.168.7.33 192.168.7.34 192.168.8.35
      192.168.8.36 192.168.8.37 192.168.8.38 192.168.8.39 192.168.9.40 192.168.9.41
      192.168.9.42 192.168.9.43 192.168.9.44 192.168.10.45 192.168.10.46 192.168.10.47
      192.168.10.48 192.168.10.49 192.168.11.50 192.168.11.51 192.168.11.52 192.168.11.53
      192.168.11.54 192.168.12.55 192.168.12.56 192.168.12.57 192.168.12.58 192.168.12.59
      192.168.13.60 192.168.13.61 127.0.0.1

Comment 2 Ram Ranganathan 2018-07-12 00:39:20 UTC
@Victor, the addresses look like examples here so wasn't sure of your actual usage.

But wanted to point out that if you are using blocks of IP addresses, an alternative here is to specify a /24 block ala: 192.168.1.0/24
instead of specifying the multiple IP addresses 192.168.1.[1-254]

Comment 3 Victor Hernando 2018-07-12 07:15:15 UTC
(In reply to Ram Ranganathan from comment #2)
> @Victor, the addresses look like examples here so wasn't sure of your actual
> usage.
> 

@Ram, Yes, those ips are examples to reproduce the issue in my lab. I was not using ip network ranges since I only wanted to reproduce the issue assigning arguments to this annotation.

> But wanted to point out that if you are using blocks of IP addresses, an
> alternative here is to specify a /24 block ala: 192.168.1.0/24
> instead of specifying the multiple IP addresses 192.168.1.[1-254]

Yes, I'm aware that it's possible to add network ranges, saving some arguments using them instead of ips. 
Unfortunately, this issue was detected in a "real" and running environment, where there is a need to include more than these 64 arguments (Ips and networks), since applying network ranges would be whitelisting many unwanted ips.

Comment 4 Ram Ranganathan 2018-07-12 18:52:54 UTC
@Victor, so if we can't use blocks of IPs, then a couple of workarounds here might be to (Warning: you will need custom templates for these):

1. Split this up into multiple annotations instead of just the single one. Aka modify this block in the haproxy config template

    {{- with $ip_whiteList := firstMatch $cidrListPattern (index $cfg.Annotations "haproxy.router.openshift.io/ip_whitelist") }}
acl whitelist src {{ $ip_whiteList }}
  tcp-request content reject if !whitelist
{{- end }}
  
with something like: 

    {{- with $ip_whiteList := firstMatch $cidrListPattern (index $cfg.Annotations "haproxy.router.openshift.io/ip_whitelist") }}
acl whitelist src {{ $ip_whiteList }}
    {{- with $ip_whiteList2 := firstMatch $cidrListPattern (index $cfg.Annotations "haproxy.router.openshift.io/ip_whitelist-2") }}
acl whitelist2 src {{ $ip_whiteList2 }}
tcp-request content reject if !whitelist !whitelist2
    {{- else }}
tcp-request content reject if !whitelist
    {{- end }}
{{- end }}
  
Basically you can add as many of those as needed (only your else conditions will need to be adjusted accordingly). 

Word of note here is that here are 2 places you will need to do this (once for secure/http and once for the passthrough backends).

Example in 3.11 this would be around: 
https://github.com/openshift/origin/blob/master/images/router/haproxy/conf/haproxy-config.template#L332
and 
https://github.com/openshift/origin/blob/master/images/router/haproxy/conf/haproxy-config.template#L430

2. Second alternative is a wee bit more involved - though the haproxy config change is small. Basically similar to the customized config template, we would mount one or more files into the container containing the list of IPs.
Example: 
$   oc create configmap my-config --from-file=whitelist1=/path/to/bar/whitelist-1.txt
--from-file=whitelist22=/path/to/whitelist-2.txt

And mount this onto /var/lib/haproxy/conf/whitelists 
and then basically use it inside a customized router template ala: 
    {{- with $ip_whiteListKey := firstMatch $cidrListPattern (index $cfg.Annotations "haproxy.router.openshift.io/ip_whitelist_file_key") }}
acl whitelist src -f /var/lib/haproxy/conf/whitelists/{{ $ip_whiteListKey }}
  tcp-request content reject if !whitelist
{{- end }}

Downside is that you need to pre-define these whitelists. If there aren't too many variants of the "haproxy.router.openshift.io/ip_whitelist" annotation value, then it might be best to pre-define them and use a map.

Side note: there's a variant of this that can be done by writing the "haproxy.router.openshift.io/ip_whitelist" annotation out to disk and then always using that file that we wrote - but it needs some template helper code changes so would be a lot more complicated as you'd need to build openshift and a new router image. So not mentioning those details here.

Instructions on for customized router templates at: https://docs.openshift.com/container-platform/3.9/install_config/router/customized_haproxy_router.html#using-configmap-replace-template

Comment 6 Ram Ranganathan 2018-07-18 21:55:31 UTC
Associated PR: https://github.com/openshift/origin/pull/20357

Comment 8 Weibin Liang 2018-08-24 15:06:59 UTC
@Ram,

Tested on v3.11.0-0.22.0, testing failed.

$ oc edit route <name> 
# and add the following to the metadata
metadata:
  annotations:
    haproxy.router.openshift.io/ip_whitelist: 192.168.1.1 192.168.1.2 192.168.1.3 192.168.1.4
      192.168.2.5 192.168.2.6 192.168.2.7 192.168.2.8 192.168.2.9 192.168.3.10 192.168.3.11
      192.168.3.12 192.168.3.13 192.168.3.14 192.168.4.15 192.168.4.16 192.168.4.17
      192.168.4.18 192.168.4.19 192.168.5.20 192.168.5.21 192.168.5.22 192.168.5.23
      192.168.5.24 192.168.6.25 192.168.6.26 192.168.6.27 192.168.6.28 192.168.6.29
      192.168.7.30 192.168.7.31 192.168.7.32 192.168.7.33 192.168.7.34 192.168.8.35
      192.168.8.36 192.168.8.37 192.168.8.38 192.168.8.39 192.168.9.40 192.168.9.41
      192.168.9.42 192.168.9.43 192.168.9.44 192.168.10.45 192.168.10.46 192.168.10.47
      192.168.10.48 192.168.10.49 192.168.11.50 192.168.11.51 192.168.11.52 192.168.11.53
      192.168.11.54 192.168.12.55 192.168.12.56 192.168.12.57 192.168.12.58 192.168.12.59
      192.168.13.60 192.168.13.61 127.0.0.1

# oc edit route <name> again
ip_whitelist is not shown up.


Tried same steps in v3.9, ip_whitelist can be shown up after oc edit route <name>

Comment 9 Ram Ranganathan 2018-09-04 21:21:23 UTC
@Weibin, what is the actual error here? I just tried it and it works fine
for me (used a dev build and image).

Is it that the annotation `haproxy.router.openshift.io/ip_whitelist` is missing when you edit the route the second time? That doesn't seem correct - we don't delete that/touch a route config in the router. 

Can you please re-test this. Thanks.

Comment 10 Weibin Liang 2018-09-05 12:55:02 UTC
Retested and verified on v3.11.0-0.25.0, no error message found in router log after deploying 66 ip address in route ip_whitelist

Comment 12 errata-xmlrpc 2018-10-11 07:21:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2652


Note You need to log in before you can comment on or make changes to this bug.