Bug 1983751
| Summary: | router scalability test error connection increases when client number increases in large cluster with routers run on infra nodes. | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Qiujie Li <qili> |
| Component: | Networking | Assignee: | Grant Spence <gspence> |
| Networking sub component: | router | QA Contact: | Arvind iyengar <aiyengar> |
| Status: | CLOSED NOTABUG | Docs Contact: | |
| Severity: | medium | ||
| Priority: | medium | CC: | aos-bugs, cholman, gspence, hongli, mmasters |
| Version: | 4.7 | ||
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-08-10 18:24:16 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Do we know whether this is a regression? Do we have data for 4.7.19 or earlier versions? Would it be possible to run the same tests with access logging enabled on the ingress controller? We'll follow up when the must-gather is provided so we can check for errors from HAProxy and so on. =>Update on enabling access logging on the ingress controller I enabled access logging on the ingress controller, with the 'Configure Ingress access logging to a sidecar.' approach. However, after that the test tool can't work on the environment. The reason is there are 2 containers on the router pod new, a command in the test tool failed. % oc set probe -n openshift-ingress --liveness --period-seconds=120 deploy/router-default The Deployment "router-default" is invalid: spec.template.spec.containers[1].livenessProbe: Required value: must specify a handler type I see this is a pretty old bug and hasn't gotten an update in about a year. But I have some experience with MB and haproxy errors, I'll try to help out.
Here is my guess: maxconn limits are causing an high error count. Maxconn is set at 20,000 as default, however, for reencrypt and edge, each connection is actually doubled due to an internal hop within haproxy. For MB, take LARGE_SCALE_CLIENTS * LARGE_SCALE_ROUTES and that is your approximate connection count you are driving the router at.
I see you are using AWS and are using 4 router pods, so in theory, you can 4x the 20,000 maxconn to 80,000 (given equal distribution). But remember that re-encrypt and edge use 2x the connection count.
For example, LARGE_SCALE_CLIENTS=80 and LARGE_SCALE_ROUTE=500 == 40,000 connections. Which is under your 80,000 maxconn, but keep in mind, the external load balancer may not be 100% fair and if you are driving edge/reencrypt at the same time (you are using mix which is all terminations) you'd be driving 80,000 connections due to the 2x connection count. So it's entirely possible you pushed the router over maxconn with that scenario.
The error counts you are seeing are very similar to what we saw when we pushed past maxconn and haproxy queue's the connections to drop them if not handled in time. The other important detail here, is that, your routers can also restart due to a known liveness probe failure when maxconn is reached. Haproxy will not be able respond to the kubelet's liveness probe, so kubelet will restart the router, further exacerbating the problem with more connections dropped. This is easy to identify by looking at the restart count for the router pods after the test. The must-gather appears to have expired, otherwise I'd check.
The solution I'd recommend is to patch haproxy's maxconn setting to auto (syntax depends on OCP version):
< 4.11:
oc -n openshift-ingress-operator patch ingresscontroller/default --type=merge --patch='{"spec":{"unsupportedConfigOverrides":{"maxConnections":-1}}}'
>= 4.11:
oc -n openshift-ingress-operator patch ingresscontroller/default --type=merge --patch='{"spec":{"tuningOptions":{"maxConnections":-1}}}'
Then run your MB test to see if error count drop. I wouldn't expect 0 error counts, but maybe < 100 reasonably.
Is there still interest in this bug? If so, could you try out maxconn auto and see if error count drops?
Thanks for doing the test again. To me, that looks like an incredible improvement. The rate of errors being below 100 per 1,000,000 requests is very reasonable, especially when you are pushing haproxy very hard. This is a great example of how maxconn is a very important tuning option. Do you feel satisfied with this resolution? Shall we close this bug? Great. Those results look pretty reasonable, definitely an improvement. I think the key takeaway is: if you are pushing over maxconn then you'll a lot more errors and performance degradation. Closing as NOTABUG because high error rates along are expected if maxconn is exceeded. |
Description of problem: With large cluster 25 worker nodes(m5.2xlarge) and 4 infra nodes(m5.12xlarge), router running on infra nodes, router scalability test shows that error connection increases when client number increases. OpenShift release version: 4.7.20-x86_64 Cluster Platform: AWS How reproducible: Every time Steps to Reproduce (in detail): 1.Create a cluster with 3 master nodes (m5.xlarge), 25 worker nodes(m5.2xlarge) and 4 infra nodes(m5.12xlarge). 2.Move router pods to the infra nodes. oc edit ingresscontroller default -n openshift-ingress-operator spec: nodePlacement: nodeSelector: matchLabels: node-role.kubernetes.io/infra: "" 3. Clone https://github.com/cloud-bulldozer/e2e-benchmarking on a server. 4. On the server of step1, cd /root/e2e-benchmarking/workloads/router-perf-v2 5. Edit env.sh as below export KUBECONFIG=~/.kube/config export ENGINE=podman export RUNTIME=60 export TERMINATIONS="mix" export URL_PATH="/1024.html" export TLS_REUSE=true export SAMPLES=2 export HOST_NETWORK=true export NUMBER_OF_ROUTERS=4 export NODE_SELECTOR='{node-role.kubernetes.io/workload: }' export QUIET_PERIOD=60s export ES_SERVER=https://search-perfscale-dev-chmf5l4sh66lvxbnadi4bznl3a.us-west-2.es.amazonaws.com:443 export ES_INDEX=router-test-results export PREFIX=$(oc get clusterversion version -o jsonpath="{.status.desired.version}") export ES_SERVER_BASELINE="" export SMALL_SCALE_BASELINE_UUID="" export LARGE_SCALE_BASELINE_UUID="" export SMALL_SCALE_BASELINE_PREFIX="" export LARGE_SCALE_BASELINE_PREFIX="" export GSHEET_KEY_LOCATION="" export EMAIL_ID_FOR_RESULTS_SHEET="" export THROUGHPUT_TOLERANCE=5 export LATENCY_TOLERANCE=5 export SERVICE_TYPE=NodePort export KEEPALIVE_REQUESTS="0 1 50" export LARGE_SCALE_THRESHOLD=24 export LARGE_SCALE_ROUTES=500 export LARGE_SCALE_CLIENTS="80" export LARGE_SCALE_CLIENTS_MIX="80" export SMALL_SCALE_ROUTES=200 export SMALL_SCALE_CLIENTS="10 20 80" export SMALL_SCALE_CLIENTS_MIX="10 20 80" export METADATA_COLLECTION=false 7. source ./env.sh 8. ./ingress-performance.sh 9. Check the output of ingress-performance.sh Actual results: Output of ingress-performance.sh shows Executing 'mb -i /tmp/http-scale-mix.json -d 60 -o /tmp/results.csv' failed when 'Executing sample 1/2 using termination mix with 80 clients and 0 keepalive requests' Part of Output of ingress-performance.sh ... Indexing documents in router-test-results 19-07-2021T08:59:09 Sleeping for 60s before next test 19-07-2021T09:00:09 Generating config for termination mix with 80 clients 0 keep alive requests and path /1024.html 19-07-2021T09:00:11 Copying mb config http-scale-mix.json to pod http-scale-client-6957fd584-2qvgr 19-07-2021T09:00:12 Executing sample 1/2 using termination mix with 80 clients and 0 keepalive requests Executing 'mb -i /tmp/http-scale-mix.json -d 60 -o /tmp/results.csv' Traceback (most recent call last): File "/usr/lib64/python3.6/subprocess.py", line 425, in run stdout, stderr = process.communicate(input, timeout=timeout) File "/usr/lib64/python3.6/subprocess.py", line 863, in communicate stdout, stderr = self._communicate(input, endtime, timeout) File "/usr/lib64/python3.6/subprocess.py", line 1535, in _communicate self._check_timeout(endtime, orig_timeout) File "/usr/lib64/python3.6/subprocess.py", line 891, in _check_timeout raise TimeoutExpired(self.args, orig_timeout) subprocess.TimeoutExpired: Command 'mb -i /tmp/http-scale-mix.json -d 60 -o /tmp/results.csv' timed out after 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/workload/workload.py", line 92, in <module> exit(main()) File "/workload/workload.py", line 66, in main result_codes, p95_latency, p99_latency, avg_latency = run_mb(args.mb_config, args.runtime, args.output) File "/workload/workload.py", line 35, in run_mb timeout=int(runtime) * 5) File "/usr/lib64/python3.6/subprocess.py", line 430, in run stderr=stderr) subprocess.TimeoutExpired: Command 'mb -i /tmp/http-scale-mix.json -d 60 -o /tmp/results.csv' timed out after 300 seconds ---- Executing 'mb -i /tmp/http-scale-mix.json -d 60 -o /tmp/results.csv' is actually run on a pod created by the test in the cluster. So I run the mb command directly in the pod and got the following result. There are lots of Errors connection. mb -i /tmp/http-scale-mix.json -d 60 -o /tmp/results.csv ... ... ... error: ssl_write(): error state on socket error: cannot write to [158513] (http-perf-reencrypt-323-http-scale-reencrypt.apps.qili-lr-2xlarge.qe-lrc.devcluster.openshift.com:443): Success (0) reconnecting... Time: 440.95s Sent: 340.93MiB, 791.74kiB/s Recv: 2.67GiB, 6.19MiB/s Hits: 2182722, 4950.08/s Errors connection: 177569, status: 0, parser: 0 I then tried some test with smaller clients number - 20 30 50 70 clients, the result shows that the errors connection continuously increases when clients number increases. 20 clients mb -i tmp/http-scale-mix-20client.json -d 60 -o /tmp/results.csv ...errors... Time: 60.11s Sent: 862.25MiB, 14.34MiB/s Recv: 6.62GiB, 112.82MiB/s Hits: 5326688, 88614.01/s Errors connection: 37, status: 14079, parser: 0 30 clients $ mb -i tmp/http-scale-mix-30client.json -d 60 -o /tmp/results.csv ...errors... Time: 65.42s Sent: 686.51MiB, 10.49MiB/s Recv: 5.33GiB, 83.38MiB/s Hits: 4287294, 65533.36/s Errors connection: 4334, status: 18349, parser: 0 50 clients mb -i tmp/http-scale-mix-50client.json -d 60 -o /tmp/results.csv ...errors... Time: 101.75s Sent: 368.33MiB, 3.62MiB/s Recv: 2.91GiB, 29.32MiB/s Hits: 2357992, 23174.42/s Errors connection: 122337, status: 0, parser: 0 70 clients mb -i tmp/http-scale-mix-70client.json -d 60 -o /tmp/results.csv;date ...errors... Time: 307.71s Sent: 361.01MiB, 1.17MiB/s Recv: 2.83GiB, 9.42MiB/s Hits: 2311242, 7510.99/s Errors connection: 161480, status: 0, parser: 0 ------ From the above 20 30 50 70 clients test result, I can see the tps (Hits/s) gets lower when client increases. Do I need to open another defect for this? I tried some more compare tests. 1. With no infra nodes, default 2 routers. - Reproduce. So this can be a simpler path to reproduce this bug. 2. Based on 1, with LARGE_SCALE_ROUTES=200 and 400 - NOT Reproduce. As a summary: With LARGE_SCALE_CLIENTS="80" LARGE_SCALE_ROUTES=500, Reproduce With LARGE_SCALE_CLIENTS="50" LARGE_SCALE_ROUTES=500, NOT Reproduce With LARGE_SCALE_CLIENTS="80" LARGE_SCALE_ROUTES=400, NOT Reproduce From the above tests, I think the bug is related to both client number and router number. Expected results: The ingress-performance.sh should pass. Impact of the problem: Additional info: % oc get clusterversions NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.7.20 True False 8h Cluster version is 4.7.20 % oc get machine -n openshift-machine-api NAME PHASE TYPE REGION ZONE AGE qili-lr-2xlarge-t9ng2-infra-us-east-2a-7xkjn Running m5.12xlarge us-east-2 us-east-2a 21m qili-lr-2xlarge-t9ng2-infra-us-east-2a-jxmck Running m5.12xlarge us-east-2 us-east-2a 5m34s qili-lr-2xlarge-t9ng2-infra-us-east-2a-wz6zp Running m5.12xlarge us-east-2 us-east-2a 21m qili-lr-2xlarge-t9ng2-infra-us-east-2a-zspl2 Running m5.12xlarge us-east-2 us-east-2a 21m qili-lr-2xlarge-t9ng2-master-0 Running m5.xlarge us-east-2 us-east-2a 4d1h qili-lr-2xlarge-t9ng2-master-1 Running m5.xlarge us-east-2 us-east-2b 4d1h qili-lr-2xlarge-t9ng2-master-2 Running m5.xlarge us-east-2 us-east-2c 4d1h qili-lr-2xlarge-t9ng2-worker-us-east-2a-6h2fq Running m5.2xlarge us-east-2 us-east-2a 3h54m qili-lr-2xlarge-t9ng2-worker-us-east-2a-7z2hf Running m5.2xlarge us-east-2 us-east-2a 3h54m qili-lr-2xlarge-t9ng2-worker-us-east-2a-8447r Running m5.2xlarge us-east-2 us-east-2a 3h54m qili-lr-2xlarge-t9ng2-worker-us-east-2a-c2pbn Running m5.2xlarge us-east-2 us-east-2a 3h54m qili-lr-2xlarge-t9ng2-worker-us-east-2a-ghtgs Running m5.2xlarge us-east-2 us-east-2a 3h54m qili-lr-2xlarge-t9ng2-worker-us-east-2a-jrmn7 Running m5.2xlarge us-east-2 us-east-2a 3h54m qili-lr-2xlarge-t9ng2-worker-us-east-2a-t2f8k Running m5.2xlarge us-east-2 us-east-2a 3h54m qili-lr-2xlarge-t9ng2-worker-us-east-2a-v8wp4 Running m5.2xlarge us-east-2 us-east-2a 3h54m qili-lr-2xlarge-t9ng2-worker-us-east-2a-zdkm8 Running m5.2xlarge us-east-2 us-east-2a 3d1h qili-lr-2xlarge-t9ng2-worker-us-east-2b-66rr2 Running m5.2xlarge us-east-2 us-east-2b 3h55m qili-lr-2xlarge-t9ng2-worker-us-east-2b-68z9l Running m5.2xlarge us-east-2 us-east-2b 3h55m qili-lr-2xlarge-t9ng2-worker-us-east-2b-87787 Running m5.2xlarge us-east-2 us-east-2b 3h55m qili-lr-2xlarge-t9ng2-worker-us-east-2b-mfcqb Running m5.2xlarge us-east-2 us-east-2b 3h55m qili-lr-2xlarge-t9ng2-worker-us-east-2b-p64dv Running m5.2xlarge us-east-2 us-east-2b 3h55m qili-lr-2xlarge-t9ng2-worker-us-east-2b-wl8fj Running m5.2xlarge us-east-2 us-east-2b 4d1h qili-lr-2xlarge-t9ng2-worker-us-east-2b-xgf6z Running m5.2xlarge us-east-2 us-east-2b 3h55m qili-lr-2xlarge-t9ng2-worker-us-east-2b-xhgcr Running m5.2xlarge us-east-2 us-east-2b 3h55m qili-lr-2xlarge-t9ng2-worker-us-east-2c-9x6sf Running m5.2xlarge us-east-2 us-east-2c 3h55m qili-lr-2xlarge-t9ng2-worker-us-east-2c-cc7xf Running m5.2xlarge us-east-2 us-east-2c 3h55m qili-lr-2xlarge-t9ng2-worker-us-east-2c-cjgkc Running m5.2xlarge us-east-2 us-east-2c 3h55m qili-lr-2xlarge-t9ng2-worker-us-east-2c-dnpqx Running m5.2xlarge us-east-2 us-east-2c 3h55m qili-lr-2xlarge-t9ng2-worker-us-east-2c-h8jvt Running m5.2xlarge us-east-2 us-east-2c 3h55m qili-lr-2xlarge-t9ng2-worker-us-east-2c-hkmg9 Running m5.2xlarge us-east-2 us-east-2c 3h55m qili-lr-2xlarge-t9ng2-worker-us-east-2c-mdk6w Running m5.2xlarge us-east-2 us-east-2c 3h55m qili-lr-2xlarge-t9ng2-worker-us-east-2c-xs22r Running m5.2xlarge us-east-2 us-east-2c 4d1h % oc get nodes -l node-role.kubernetes.io/infra NAME STATUS ROLES AGE VERSION ip-10-0-132-157.us-east-2.compute.internal Ready infra,worker 20m v1.20.0+01c9f3f ip-10-0-133-75.us-east-2.compute.internal Ready infra,worker 20m v1.20.0+01c9f3f ip-10-0-139-146.us-east-2.compute.internal Ready infra,worker 3m14s v1.20.0+01c9f3f ip-10-0-150-254.us-east-2.compute.internal Ready infra,worker 20m v1.20.0+01c9f3f % oc get pod -n openshift-ingress -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES router-default-5587656c5c-92tl8 1/1 Running 0 15m 10.130.42.246 ip-10-0-133-75.us-east-2.compute.internal <none> <none> router-default-5587656c5c-nbxmg 1/1 Running 0 15m 10.131.42.245 ip-10-0-150-254.us-east-2.compute.internal <none> <none> router-default-5587656c5c-pnsmd 1/1 Running 0 7m8s 10.129.44.6 ip-10-0-139-146.us-east-2.compute.internal <none> <none> router-default-5587656c5c-rdht6 1/1 Running 0 15m 10.128.44.247 ip-10-0-132-157.us-east-2.compute.internal <none> <none> Must gather will be attached later. ** Please do not disregard the report template; filling the template out as much as possible will allow us to help you. Please consider attaching a must-gather archive (via `oc adm must-gather`). Please review must-gather contents for sensitive information before attaching any must-gathers to a bugzilla report. You may also mark the bug private if you wish.