Description of problem: Adding more than 61 ips/ranges to the haproxy.router.openshift.io/ip_whitelist annotation, the router fails reloading configuration. Version-Release number of selected component (if applicable): 3.9 Also reproducible in 3.6 How reproducible: Create a route for your service, add the following annotation with the desired ips/ranges (more than 61) to the ip_whitelist. metadata: annotations: haproxy.router.openshift.io/ip_whitelist: Steps to Reproduce: 1. Create a route $ oc expose svc test2 route "test2" exposed 2. Export the route $ oc export route test2 -o yaml > route_test2_wlip.yml 3. Edit the route, add 62 ips/ranges and replace it. $ cat route_test2_wlip.yml apiVersion: v1 kind: Route metadata: annotations: openshift.io/host.generated: "true" haproxy.router.openshift.io/ip_whitelist: >- 192.168.1.1 192.168.1.2 192.168.1.3 192.168.1.4 192.168.2.5 192.168.2.6 192.168.2.7 192.168.2.8 192.168.2.9 192.168.3.10 192.168.3.11 192.168.3.12 192.168.3.13 192.168.3.14 192.168.4.15 192.168.4.16 192.168.4.17 192.168.4.18 192.168.4.19 192.168.5.20 192.168.5.21 192.168.5.22 192.168.5.23 192.168.5.24 192.168.6.25 192.168.6.26 192.168.6.27 192.168.6.28 192.168.6.29 192.168.7.30 192.168.7.31 192.168.7.32 192.168.7.33 192.168.7.34 192.168.8.35 192.168.8.36 192.168.8.37 192.168.8.38 192.168.8.39 192.168.9.40 192.168.9.41 192.168.9.42 192.168.9.43 192.168.9.44 192.168.10.45 192.168.10.46 192.168.10.47 192.168.10.48 192.168.10.49 192.168.11.50 192.168.11.51 192.168.11.52 192.168.11.53 192.168.11.54 192.168.12.55 192.168.12.56 192.168.12.57 192.168.12.58 192.168.12.59 192.168.13.60 192.168.13.61 ... # oc logs -f router-1-cv5nd -n default E0706 10:49:47.595860 1 limiter.go:137] error reloading router: exit status 1 [ALERT] 186/104947 (1075) : parsing [/var/lib/haproxy/conf/haproxy.config:245]: line too long, truncating at word 65, position 826: <192.168.13.62>. [ALERT] 186/104947 (1075) : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config [ALERT] 186/104947 (1075) : Fatal errors found in configuration. Actual results: The router is not able to reload the new configuration adding 62 ips to the annotation. E0706 10:49:47.595860 1 limiter.go:137] error reloading router: exit status 1 [ALERT] 186/104947 (1075) : parsing [/var/lib/haproxy/conf/haproxy.config:245]: line too long, truncating at word 65, position 826: <192.168.13.62>. [ALERT] 186/104947 (1075) : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config [ALERT] 186/104947 (1075) : Fatal errors found in configuration Adding 61 ips/ranges or less, the router is reloading successfully the configuration. I0706 10:43:24.592134 1 router.go:441] Router reloaded: - Checking http://localhost:80 ... - Health check ok : 0 retry attempt(s). Expected results: Reloading the configuration successfully when more ips/ranges are added to the annotation. If there is a known limit, should be interesting having this constraint documented. Additional info:
With all those ip/ranges, this is hitting the haproxy max args per line limit (64). https://github.com/haproxy/haproxy/blob/673867c357260d13a8a6fcd17a3d2eb77ba10e5c/include/common/defaults.h#L75 I got the same error adding one more ip 127.0.0.1 to the list above. $ oc edit route <name> # and add the following to the metadata metadata: annotations: haproxy.router.openshift.io/ip_whitelist: 192.168.1.1 192.168.1.2 192.168.1.3 192.168.1.4 192.168.2.5 192.168.2.6 192.168.2.7 192.168.2.8 192.168.2.9 192.168.3.10 192.168.3.11 192.168.3.12 192.168.3.13 192.168.3.14 192.168.4.15 192.168.4.16 192.168.4.17 192.168.4.18 192.168.4.19 192.168.5.20 192.168.5.21 192.168.5.22 192.168.5.23 192.168.5.24 192.168.6.25 192.168.6.26 192.168.6.27 192.168.6.28 192.168.6.29 192.168.7.30 192.168.7.31 192.168.7.32 192.168.7.33 192.168.7.34 192.168.8.35 192.168.8.36 192.168.8.37 192.168.8.38 192.168.8.39 192.168.9.40 192.168.9.41 192.168.9.42 192.168.9.43 192.168.9.44 192.168.10.45 192.168.10.46 192.168.10.47 192.168.10.48 192.168.10.49 192.168.11.50 192.168.11.51 192.168.11.52 192.168.11.53 192.168.11.54 192.168.12.55 192.168.12.56 192.168.12.57 192.168.12.58 192.168.12.59 192.168.13.60 192.168.13.61 127.0.0.1
@Victor, the addresses look like examples here so wasn't sure of your actual usage. But wanted to point out that if you are using blocks of IP addresses, an alternative here is to specify a /24 block ala: 192.168.1.0/24 instead of specifying the multiple IP addresses 192.168.1.[1-254]
(In reply to Ram Ranganathan from comment #2) > @Victor, the addresses look like examples here so wasn't sure of your actual > usage. > @Ram, Yes, those ips are examples to reproduce the issue in my lab. I was not using ip network ranges since I only wanted to reproduce the issue assigning arguments to this annotation. > But wanted to point out that if you are using blocks of IP addresses, an > alternative here is to specify a /24 block ala: 192.168.1.0/24 > instead of specifying the multiple IP addresses 192.168.1.[1-254] Yes, I'm aware that it's possible to add network ranges, saving some arguments using them instead of ips. Unfortunately, this issue was detected in a "real" and running environment, where there is a need to include more than these 64 arguments (Ips and networks), since applying network ranges would be whitelisting many unwanted ips.
@Victor, so if we can't use blocks of IPs, then a couple of workarounds here might be to (Warning: you will need custom templates for these): 1. Split this up into multiple annotations instead of just the single one. Aka modify this block in the haproxy config template {{- with $ip_whiteList := firstMatch $cidrListPattern (index $cfg.Annotations "haproxy.router.openshift.io/ip_whitelist") }} acl whitelist src {{ $ip_whiteList }} tcp-request content reject if !whitelist {{- end }} with something like: {{- with $ip_whiteList := firstMatch $cidrListPattern (index $cfg.Annotations "haproxy.router.openshift.io/ip_whitelist") }} acl whitelist src {{ $ip_whiteList }} {{- with $ip_whiteList2 := firstMatch $cidrListPattern (index $cfg.Annotations "haproxy.router.openshift.io/ip_whitelist-2") }} acl whitelist2 src {{ $ip_whiteList2 }} tcp-request content reject if !whitelist !whitelist2 {{- else }} tcp-request content reject if !whitelist {{- end }} {{- end }} Basically you can add as many of those as needed (only your else conditions will need to be adjusted accordingly). Word of note here is that here are 2 places you will need to do this (once for secure/http and once for the passthrough backends). Example in 3.11 this would be around: https://github.com/openshift/origin/blob/master/images/router/haproxy/conf/haproxy-config.template#L332 and https://github.com/openshift/origin/blob/master/images/router/haproxy/conf/haproxy-config.template#L430 2. Second alternative is a wee bit more involved - though the haproxy config change is small. Basically similar to the customized config template, we would mount one or more files into the container containing the list of IPs. Example: $ oc create configmap my-config --from-file=whitelist1=/path/to/bar/whitelist-1.txt --from-file=whitelist22=/path/to/whitelist-2.txt And mount this onto /var/lib/haproxy/conf/whitelists and then basically use it inside a customized router template ala: {{- with $ip_whiteListKey := firstMatch $cidrListPattern (index $cfg.Annotations "haproxy.router.openshift.io/ip_whitelist_file_key") }} acl whitelist src -f /var/lib/haproxy/conf/whitelists/{{ $ip_whiteListKey }} tcp-request content reject if !whitelist {{- end }} Downside is that you need to pre-define these whitelists. If there aren't too many variants of the "haproxy.router.openshift.io/ip_whitelist" annotation value, then it might be best to pre-define them and use a map. Side note: there's a variant of this that can be done by writing the "haproxy.router.openshift.io/ip_whitelist" annotation out to disk and then always using that file that we wrote - but it needs some template helper code changes so would be a lot more complicated as you'd need to build openshift and a new router image. So not mentioning those details here. Instructions on for customized router templates at: https://docs.openshift.com/container-platform/3.9/install_config/router/customized_haproxy_router.html#using-configmap-replace-template
Associated PR: https://github.com/openshift/origin/pull/20357
@Ram, Tested on v3.11.0-0.22.0, testing failed. $ oc edit route <name> # and add the following to the metadata metadata: annotations: haproxy.router.openshift.io/ip_whitelist: 192.168.1.1 192.168.1.2 192.168.1.3 192.168.1.4 192.168.2.5 192.168.2.6 192.168.2.7 192.168.2.8 192.168.2.9 192.168.3.10 192.168.3.11 192.168.3.12 192.168.3.13 192.168.3.14 192.168.4.15 192.168.4.16 192.168.4.17 192.168.4.18 192.168.4.19 192.168.5.20 192.168.5.21 192.168.5.22 192.168.5.23 192.168.5.24 192.168.6.25 192.168.6.26 192.168.6.27 192.168.6.28 192.168.6.29 192.168.7.30 192.168.7.31 192.168.7.32 192.168.7.33 192.168.7.34 192.168.8.35 192.168.8.36 192.168.8.37 192.168.8.38 192.168.8.39 192.168.9.40 192.168.9.41 192.168.9.42 192.168.9.43 192.168.9.44 192.168.10.45 192.168.10.46 192.168.10.47 192.168.10.48 192.168.10.49 192.168.11.50 192.168.11.51 192.168.11.52 192.168.11.53 192.168.11.54 192.168.12.55 192.168.12.56 192.168.12.57 192.168.12.58 192.168.12.59 192.168.13.60 192.168.13.61 127.0.0.1 # oc edit route <name> again ip_whitelist is not shown up. Tried same steps in v3.9, ip_whitelist can be shown up after oc edit route <name>
@Weibin, what is the actual error here? I just tried it and it works fine for me (used a dev build and image). Is it that the annotation `haproxy.router.openshift.io/ip_whitelist` is missing when you edit the route the second time? That doesn't seem correct - we don't delete that/touch a route config in the router. Can you please re-test this. Thanks.
Retested and verified on v3.11.0-0.25.0, no error message found in router log after deploying 66 ip address in route ip_whitelist
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:2652