Bug 1598738
| Summary: | Error reloading router, too many ips included in the whitelist annotation | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Victor Hernando <vhernand> |
| Component: | Networking | Assignee: | Ram Ranganathan <ramr> |
| Networking sub component: | router | QA Contact: | zhaozhanqi <zzhao> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | medium | ||
| Priority: | unspecified | CC: | aos-bugs, bbennett, dmace, hongli, piqin, ramr, weliang |
| Version: | 3.9.0 | ||
| Target Milestone: | --- | ||
| Target Release: | 3.11.0 | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
Cause: If there are too many ip addresses specified in a route via the "haproxy.router.openshift.io/ip_whitelist" annotation, then number of parameters to a haproxy config directive exceeds 63 and there is an error reloading the router. Limitations based on `#define MAX_LINE_ARGS 64` in the haproxy code.
Consequence: The router does not reload (invalid config) if there too many ips in the annotation.
Fix: We use an overflow map if the there are too many ips in
the whitelist annotation (and pass the map to the haproxy acl).
Result: Works as intended (without any changes by the user).
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-10-11 07:21:36 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
With all those ip/ranges, this is hitting the haproxy max args per line limit (64). https://github.com/haproxy/haproxy/blob/673867c357260d13a8a6fcd17a3d2eb77ba10e5c/include/common/defaults.h#L75 I got the same error adding one more ip 127.0.0.1 to the list above. $ oc edit route <name> # and add the following to the metadata metadata: annotations: haproxy.router.openshift.io/ip_whitelist: 192.168.1.1 192.168.1.2 192.168.1.3 192.168.1.4 192.168.2.5 192.168.2.6 192.168.2.7 192.168.2.8 192.168.2.9 192.168.3.10 192.168.3.11 192.168.3.12 192.168.3.13 192.168.3.14 192.168.4.15 192.168.4.16 192.168.4.17 192.168.4.18 192.168.4.19 192.168.5.20 192.168.5.21 192.168.5.22 192.168.5.23 192.168.5.24 192.168.6.25 192.168.6.26 192.168.6.27 192.168.6.28 192.168.6.29 192.168.7.30 192.168.7.31 192.168.7.32 192.168.7.33 192.168.7.34 192.168.8.35 192.168.8.36 192.168.8.37 192.168.8.38 192.168.8.39 192.168.9.40 192.168.9.41 192.168.9.42 192.168.9.43 192.168.9.44 192.168.10.45 192.168.10.46 192.168.10.47 192.168.10.48 192.168.10.49 192.168.11.50 192.168.11.51 192.168.11.52 192.168.11.53 192.168.11.54 192.168.12.55 192.168.12.56 192.168.12.57 192.168.12.58 192.168.12.59 192.168.13.60 192.168.13.61 127.0.0.1 @Victor, the addresses look like examples here so wasn't sure of your actual usage. But wanted to point out that if you are using blocks of IP addresses, an alternative here is to specify a /24 block ala: 192.168.1.0/24 instead of specifying the multiple IP addresses 192.168.1.[1-254] (In reply to Ram Ranganathan from comment #2) > @Victor, the addresses look like examples here so wasn't sure of your actual > usage. > @Ram, Yes, those ips are examples to reproduce the issue in my lab. I was not using ip network ranges since I only wanted to reproduce the issue assigning arguments to this annotation. > But wanted to point out that if you are using blocks of IP addresses, an > alternative here is to specify a /24 block ala: 192.168.1.0/24 > instead of specifying the multiple IP addresses 192.168.1.[1-254] Yes, I'm aware that it's possible to add network ranges, saving some arguments using them instead of ips. Unfortunately, this issue was detected in a "real" and running environment, where there is a need to include more than these 64 arguments (Ips and networks), since applying network ranges would be whitelisting many unwanted ips. @Victor, so if we can't use blocks of IPs, then a couple of workarounds here might be to (Warning: you will need custom templates for these):
1. Split this up into multiple annotations instead of just the single one. Aka modify this block in the haproxy config template
{{- with $ip_whiteList := firstMatch $cidrListPattern (index $cfg.Annotations "haproxy.router.openshift.io/ip_whitelist") }}
acl whitelist src {{ $ip_whiteList }}
tcp-request content reject if !whitelist
{{- end }}
with something like:
{{- with $ip_whiteList := firstMatch $cidrListPattern (index $cfg.Annotations "haproxy.router.openshift.io/ip_whitelist") }}
acl whitelist src {{ $ip_whiteList }}
{{- with $ip_whiteList2 := firstMatch $cidrListPattern (index $cfg.Annotations "haproxy.router.openshift.io/ip_whitelist-2") }}
acl whitelist2 src {{ $ip_whiteList2 }}
tcp-request content reject if !whitelist !whitelist2
{{- else }}
tcp-request content reject if !whitelist
{{- end }}
{{- end }}
Basically you can add as many of those as needed (only your else conditions will need to be adjusted accordingly).
Word of note here is that here are 2 places you will need to do this (once for secure/http and once for the passthrough backends).
Example in 3.11 this would be around:
https://github.com/openshift/origin/blob/master/images/router/haproxy/conf/haproxy-config.template#L332
and
https://github.com/openshift/origin/blob/master/images/router/haproxy/conf/haproxy-config.template#L430
2. Second alternative is a wee bit more involved - though the haproxy config change is small. Basically similar to the customized config template, we would mount one or more files into the container containing the list of IPs.
Example:
$ oc create configmap my-config --from-file=whitelist1=/path/to/bar/whitelist-1.txt
--from-file=whitelist22=/path/to/whitelist-2.txt
And mount this onto /var/lib/haproxy/conf/whitelists
and then basically use it inside a customized router template ala:
{{- with $ip_whiteListKey := firstMatch $cidrListPattern (index $cfg.Annotations "haproxy.router.openshift.io/ip_whitelist_file_key") }}
acl whitelist src -f /var/lib/haproxy/conf/whitelists/{{ $ip_whiteListKey }}
tcp-request content reject if !whitelist
{{- end }}
Downside is that you need to pre-define these whitelists. If there aren't too many variants of the "haproxy.router.openshift.io/ip_whitelist" annotation value, then it might be best to pre-define them and use a map.
Side note: there's a variant of this that can be done by writing the "haproxy.router.openshift.io/ip_whitelist" annotation out to disk and then always using that file that we wrote - but it needs some template helper code changes so would be a lot more complicated as you'd need to build openshift and a new router image. So not mentioning those details here.
Instructions on for customized router templates at: https://docs.openshift.com/container-platform/3.9/install_config/router/customized_haproxy_router.html#using-configmap-replace-template
Associated PR: https://github.com/openshift/origin/pull/20357 @Ram,
Tested on v3.11.0-0.22.0, testing failed.
$ oc edit route <name>
# and add the following to the metadata
metadata:
annotations:
haproxy.router.openshift.io/ip_whitelist: 192.168.1.1 192.168.1.2 192.168.1.3 192.168.1.4
192.168.2.5 192.168.2.6 192.168.2.7 192.168.2.8 192.168.2.9 192.168.3.10 192.168.3.11
192.168.3.12 192.168.3.13 192.168.3.14 192.168.4.15 192.168.4.16 192.168.4.17
192.168.4.18 192.168.4.19 192.168.5.20 192.168.5.21 192.168.5.22 192.168.5.23
192.168.5.24 192.168.6.25 192.168.6.26 192.168.6.27 192.168.6.28 192.168.6.29
192.168.7.30 192.168.7.31 192.168.7.32 192.168.7.33 192.168.7.34 192.168.8.35
192.168.8.36 192.168.8.37 192.168.8.38 192.168.8.39 192.168.9.40 192.168.9.41
192.168.9.42 192.168.9.43 192.168.9.44 192.168.10.45 192.168.10.46 192.168.10.47
192.168.10.48 192.168.10.49 192.168.11.50 192.168.11.51 192.168.11.52 192.168.11.53
192.168.11.54 192.168.12.55 192.168.12.56 192.168.12.57 192.168.12.58 192.168.12.59
192.168.13.60 192.168.13.61 127.0.0.1
# oc edit route <name> again
ip_whitelist is not shown up.
Tried same steps in v3.9, ip_whitelist can be shown up after oc edit route <name>
@Weibin, what is the actual error here? I just tried it and it works fine for me (used a dev build and image). Is it that the annotation `haproxy.router.openshift.io/ip_whitelist` is missing when you edit the route the second time? That doesn't seem correct - we don't delete that/touch a route config in the router. Can you please re-test this. Thanks. Retested and verified on v3.11.0-0.25.0, no error message found in router log after deploying 66 ip address in route ip_whitelist Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:2652 |
Description of problem: Adding more than 61 ips/ranges to the haproxy.router.openshift.io/ip_whitelist annotation, the router fails reloading configuration. Version-Release number of selected component (if applicable): 3.9 Also reproducible in 3.6 How reproducible: Create a route for your service, add the following annotation with the desired ips/ranges (more than 61) to the ip_whitelist. metadata: annotations: haproxy.router.openshift.io/ip_whitelist: Steps to Reproduce: 1. Create a route $ oc expose svc test2 route "test2" exposed 2. Export the route $ oc export route test2 -o yaml > route_test2_wlip.yml 3. Edit the route, add 62 ips/ranges and replace it. $ cat route_test2_wlip.yml apiVersion: v1 kind: Route metadata: annotations: openshift.io/host.generated: "true" haproxy.router.openshift.io/ip_whitelist: >- 192.168.1.1 192.168.1.2 192.168.1.3 192.168.1.4 192.168.2.5 192.168.2.6 192.168.2.7 192.168.2.8 192.168.2.9 192.168.3.10 192.168.3.11 192.168.3.12 192.168.3.13 192.168.3.14 192.168.4.15 192.168.4.16 192.168.4.17 192.168.4.18 192.168.4.19 192.168.5.20 192.168.5.21 192.168.5.22 192.168.5.23 192.168.5.24 192.168.6.25 192.168.6.26 192.168.6.27 192.168.6.28 192.168.6.29 192.168.7.30 192.168.7.31 192.168.7.32 192.168.7.33 192.168.7.34 192.168.8.35 192.168.8.36 192.168.8.37 192.168.8.38 192.168.8.39 192.168.9.40 192.168.9.41 192.168.9.42 192.168.9.43 192.168.9.44 192.168.10.45 192.168.10.46 192.168.10.47 192.168.10.48 192.168.10.49 192.168.11.50 192.168.11.51 192.168.11.52 192.168.11.53 192.168.11.54 192.168.12.55 192.168.12.56 192.168.12.57 192.168.12.58 192.168.12.59 192.168.13.60 192.168.13.61 ... # oc logs -f router-1-cv5nd -n default E0706 10:49:47.595860 1 limiter.go:137] error reloading router: exit status 1 [ALERT] 186/104947 (1075) : parsing [/var/lib/haproxy/conf/haproxy.config:245]: line too long, truncating at word 65, position 826: <192.168.13.62>. [ALERT] 186/104947 (1075) : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config [ALERT] 186/104947 (1075) : Fatal errors found in configuration. Actual results: The router is not able to reload the new configuration adding 62 ips to the annotation. E0706 10:49:47.595860 1 limiter.go:137] error reloading router: exit status 1 [ALERT] 186/104947 (1075) : parsing [/var/lib/haproxy/conf/haproxy.config:245]: line too long, truncating at word 65, position 826: <192.168.13.62>. [ALERT] 186/104947 (1075) : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config [ALERT] 186/104947 (1075) : Fatal errors found in configuration Adding 61 ips/ranges or less, the router is reloading successfully the configuration. I0706 10:43:24.592134 1 router.go:441] Router reloaded: - Checking http://localhost:80 ... - Health check ok : 0 retry attempt(s). Expected results: Reloading the configuration successfully when more ips/ranges are added to the annotation. If there is a known limit, should be interesting having this constraint documented. Additional info: