Bug 2027584
| Summary: | ovs-switchd sometimes take high cpu usage when sending ipv6 traffic from pod to pod with ovs hwoffload VF as default interface | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Ying Wang <yingwang> |
| Component: | Networking | Assignee: | William Zhao <wizhao> |
| Networking sub component: | SR-IOV | QA Contact: | Ying Wang <yingwang> |
| Status: | CLOSED DUPLICATE | Docs Contact: | |
| Severity: | medium | ||
| Priority: | medium | CC: | dosmith, mleitner |
| Version: | 4.10 | ||
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-08-18 12:13:41 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 2018001 | ||
Some questions: - Have you tried this on 4.11? Is the issue still present? - Which NIC is this? Intel/Mlx/etc... or does it matter? - Are the 2 pods on separate bare metal hosts? Or are the 2 pods residing on the same host? Hi William,
Checked on version 4.11.0-0.nightly-2022-06-30-005428, with Mellanox ConnectX-5 NIC as default network interface on worker. Tested both 2 pods on separate bm hosts and on one host, the cpu usage looks good this time, no more than 0.5.
%Cpu(s): 0.5 us, 1.5 sy, 0.0 ni, 96.4 id, 0.0 wa, 0.1 hi, 1.5 si, 0.0 st
MiB Mem : 385561.9 total, 361894.8 free, 18411.3 used, 5255.8 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 364855.9 avail Mem
The iperf traffic real-time Bandwidth is also much more stable but still has some fluctuations. Tcpdump on VF presenters of the pods, it can capture packets discontinuously, which means hardware offload is not stable.
% oc rsh iperf-rc-k2n96 iperf3 -V -c fd01:0:0:7::4 -i 2 -t 100s
iperf 3.1.3
Linux iperf-rc-k2n96 4.18.0-372.13.1.el8_6.x86_64 #1 SMP Mon Jun 6 15:05:22 EDT 2022 x86_64
Time: Wed, 13 Jul 2022 10:21:02 GMT
Connecting to host fd01:0:0:7::4, port 5201
Cookie: iperf-rc-k2n96.1657707661.937250.052
TCP MSS: 1328 (default)
[ 4] local fd01:0:0:4::1c port 42228 connected to fd01:0:0:7::4 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 100 second test
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-2.00 sec 4.53 GBytes 19.5 Gbits/sec 519 564 KBytes
[ 4] 2.00-4.00 sec 5.06 GBytes 21.7 Gbits/sec 47 752 KBytes
[ 4] 4.00-6.00 sec 5.15 GBytes 22.1 Gbits/sec 0 813 KBytes
[ 4] 6.00-8.00 sec 5.16 GBytes 22.1 Gbits/sec 0 829 KBytes
[ 4] 8.00-10.00 sec 4.44 GBytes 19.0 Gbits/sec 1374 903 KBytes
[ 4] 10.00-12.00 sec 5.14 GBytes 22.1 Gbits/sec 84 869 KBytes
[ 4] 12.00-14.00 sec 5.14 GBytes 22.1 Gbits/sec 1 846 KBytes
[ 4] 14.00-16.00 sec 4.09 GBytes 17.6 Gbits/sec 1542 2.20 MBytes
[ 4] 16.00-18.00 sec 3.66 GBytes 15.7 Gbits/sec 471 1.23 MBytes
[ 4] 18.00-20.00 sec 3.57 GBytes 15.3 Gbits/sec 0 1.30 MBytes
[ 4] 20.00-22.00 sec 3.56 GBytes 15.3 Gbits/sec 0 1.32 MBytes
[ 4] 22.00-24.00 sec 3.74 GBytes 16.1 Gbits/sec 1259 1.25 MBytes
[ 4] 24.00-26.00 sec 4.94 GBytes 21.2 Gbits/sec 560 776 KBytes
[ 4] 26.00-28.00 sec 4.96 GBytes 21.3 Gbits/sec 249 983 KBytes
[ 4] 28.00-30.00 sec 2.64 GBytes 11.4 Gbits/sec 1242 681 KBytes
[ 4] 30.00-32.00 sec 4.03 GBytes 17.3 Gbits/sec 521 944 KBytes
[ 4] 32.00-34.00 sec 3.58 GBytes 15.4 Gbits/sec 0 1.09 MBytes
[ 4] 34.00-36.00 sec 3.00 GBytes 12.9 Gbits/sec 1185 847 KBytes
[ 4] 36.00-38.00 sec 3.57 GBytes 15.3 Gbits/sec 0 1.06 MBytes
[ 4] 38.00-40.00 sec 2.68 GBytes 11.5 Gbits/sec 876 990 KBytes
[ 4] 40.00-42.00 sec 3.53 GBytes 15.2 Gbits/sec 374 1.01 MBytes
[ 4] 42.00-44.00 sec 3.57 GBytes 15.4 Gbits/sec 0 1.22 MBytes
[ 4] 44.00-46.00 sec 2.44 GBytes 10.5 Gbits/sec 1087 3.04 MBytes
[ 4] 46.00-48.00 sec 3.58 GBytes 15.4 Gbits/sec 0 3.04 MBytes
[ 4] 48.00-50.00 sec 3.58 GBytes 15.4 Gbits/sec 0 3.04 MBytes
[ 4] 50.00-52.00 sec 3.13 GBytes 13.4 Gbits/sec 1933 1.30 KBytes
[ 4] 52.00-54.00 sec 3.71 GBytes 15.9 Gbits/sec 1617 872 KBytes
[ 4] 54.00-56.00 sec 3.89 GBytes 16.7 Gbits/sec 90 1.11 MBytes
[ 4] 56.00-58.00 sec 4.93 GBytes 21.2 Gbits/sec 415 853 KBytes
[ 4] 58.00-60.00 sec 4.69 GBytes 20.2 Gbits/sec 197 961 KBytes
[ 4] 60.00-62.00 sec 3.78 GBytes 16.2 Gbits/sec 894 691 KBytes
[ 4] 62.00-64.00 sec 4.30 GBytes 18.5 Gbits/sec 519 1.63 MBytes
[ 4] 64.00-66.00 sec 5.00 GBytes 21.5 Gbits/sec 290 957 KBytes
[ 4] 66.00-68.00 sec 4.96 GBytes 21.3 Gbits/sec 20 1.07 MBytes
[ 4] 68.00-70.00 sec 3.20 GBytes 13.7 Gbits/sec 394 965 KBytes
[ 4] 70.00-72.00 sec 3.57 GBytes 15.3 Gbits/sec 76 945 KBytes
[ 4] 72.00-74.00 sec 3.58 GBytes 15.4 Gbits/sec 0 1.05 MBytes
[ 4] 74.00-76.00 sec 3.42 GBytes 14.7 Gbits/sec 154 1.10 MBytes
[ 4] 76.00-78.00 sec 4.96 GBytes 21.3 Gbits/sec 367 1.70 MBytes
[ 4] 78.00-80.00 sec 5.10 GBytes 21.9 Gbits/sec 182 1.39 MBytes
[ 4] 80.00-82.00 sec 4.44 GBytes 19.1 Gbits/sec 571 1.21 MBytes
[ 4] 82.00-84.00 sec 3.14 GBytes 13.5 Gbits/sec 35 1.05 MBytes
[ 4] 84.00-86.00 sec 5.12 GBytes 22.0 Gbits/sec 88 1.18 MBytes
[ 4] 86.00-88.00 sec 5.05 GBytes 21.7 Gbits/sec 120 1.04 MBytes
[ 4] 88.00-90.00 sec 5.04 GBytes 21.6 Gbits/sec 249 895 KBytes
[ 4] 90.00-92.00 sec 3.96 GBytes 17.0 Gbits/sec 957 1.20 MBytes
[ 4] 92.00-94.00 sec 5.06 GBytes 21.7 Gbits/sec 295 778 KBytes
[ 4] 94.00-96.00 sec 5.04 GBytes 21.6 Gbits/sec 203 977 KBytes
[ 4] 96.00-98.00 sec 5.04 GBytes 21.6 Gbits/sec 79 1.01 MBytes
[ 4] 98.00-100.00 sec 4.45 GBytes 19.1 Gbits/sec 995 1.65 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-100.00 sec 208 GBytes 17.9 Gbits/sec 22131 sender
[ 4] 0.00-100.00 sec 208 GBytes 17.9 Gbits/sec receiver
CPU Utilization: local/sender 18.3% (0.3%u/18.0%s), remote/receiver 26.2% (1.0%u/25.2%s)
Thanks,
Ying
Do you have a sample of the packets seen at the VF rep. with tcpdump? Is it zero length ack packets? Hi William,
I sent iperf traffic as below and tcpdump on iperf server and iperf client VF rep, there're not only zero length ack packets but also a lot of none-zero length packets. I attached the tcpdump result on iperf client side. The packets on server side is similar.
% oc rsh iperf-rc-9rmqg iperf3 -V -c fd01:0:0:7::1a -i 1 -t 30s
iperf 3.1.3
Linux iperf-rc-9rmqg 4.18.0-372.13.1.el8_6.x86_64 #1 SMP Mon Jun 6 15:05:22 EDT 2022 x86_64
Time: Fri, 15 Jul 2022 08:42:11 GMT
Connecting to host fd01:0:0:7::1a, port 5201
Cookie: iperf-rc-9rmqg.1657874530.676885.77e
TCP MSS: 1328 (default)
[ 4] local fd01:0:0:4::f6 port 48044 connected to fd01:0:0:7::1a port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 30 second test
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 2.41 GBytes 20.7 Gbits/sec 323 940 KBytes
[ 4] 1.00-2.00 sec 2.19 GBytes 18.8 Gbits/sec 1131 630 KBytes
[ 4] 2.00-3.00 sec 2.22 GBytes 19.1 Gbits/sec 45 1.11 MBytes
[ 4] 3.00-4.00 sec 2.22 GBytes 19.1 Gbits/sec 378 807 KBytes
[ 4] 4.00-5.00 sec 1.79 GBytes 15.3 Gbits/sec 0 884 KBytes
[ 4] 5.00-6.00 sec 551 MBytes 4.62 Gbits/sec 1 1.30 KBytes
[ 4] 6.00-7.00 sec 1.89 GBytes 16.3 Gbits/sec 125 1.14 MBytes
[ 4] 7.00-8.00 sec 1.94 GBytes 16.7 Gbits/sec 781 827 KBytes
[ 4] 8.00-9.00 sec 2.43 GBytes 20.9 Gbits/sec 511 1.16 MBytes
[ 4] 9.00-10.00 sec 2.53 GBytes 21.7 Gbits/sec 20 1.06 MBytes
[ 4] 10.00-11.00 sec 2.49 GBytes 21.4 Gbits/sec 160 1.13 MBytes
[ 4] 11.00-12.00 sec 2.44 GBytes 20.9 Gbits/sec 212 831 KBytes
[ 4] 12.00-13.00 sec 1.51 GBytes 13.0 Gbits/sec 530 1.08 MBytes
[ 4] 13.00-14.00 sec 2.50 GBytes 21.4 Gbits/sec 195 1.04 MBytes
[ 4] 14.00-15.00 sec 2.47 GBytes 21.2 Gbits/sec 97 883 KBytes
[ 4] 15.00-16.00 sec 2.02 GBytes 17.4 Gbits/sec 81 813 KBytes
[ 4] 16.00-17.00 sec 1.86 GBytes 16.0 Gbits/sec 24 874 KBytes
[ 4] 17.00-18.00 sec 1.39 GBytes 11.9 Gbits/sec 980 860 KBytes
[ 4] 18.00-19.00 sec 1.79 GBytes 15.3 Gbits/sec 0 975 KBytes
[ 4] 19.00-20.00 sec 1.79 GBytes 15.4 Gbits/sec 0 1.01 MBytes
[ 4] 20.00-21.00 sec 1.79 GBytes 15.4 Gbits/sec 90 831 KBytes
[ 4] 21.00-22.00 sec 1.76 GBytes 15.1 Gbits/sec 0 969 KBytes
[ 4] 22.00-23.00 sec 1.21 GBytes 10.4 Gbits/sec 653 814 KBytes
[ 4] 23.00-24.00 sec 1.46 GBytes 12.5 Gbits/sec 0 991 KBytes
[ 4] 24.00-25.00 sec 1.10 GBytes 9.43 Gbits/sec 322 563 KBytes
[ 4] 25.00-26.00 sec 1.41 GBytes 12.2 Gbits/sec 373 779 KBytes
[ 4] 26.00-27.00 sec 1.78 GBytes 15.3 Gbits/sec 0 908 KBytes
[ 4] 27.00-28.00 sec 1.78 GBytes 15.3 Gbits/sec 0 957 KBytes
[ 4] 28.00-29.00 sec 1.78 GBytes 15.3 Gbits/sec 0 1002 KBytes
[ 4] 29.00-30.00 sec 1.78 GBytes 15.3 Gbits/sec 0 1.02 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-30.00 sec 56.3 GBytes 16.1 Gbits/sec 7032 sender
[ 4] 0.00-30.00 sec 56.3 GBytes 16.1 Gbits/sec receiver
CPU Utilization: local/sender 16.2% (0.3%u/15.9%s), remote/receiver 19.2% (0.7%u/18.5%s)
Thanks,
Ying
Hi Ying, The zero length ack packets is a known issue, but should have been resolved in 4.11. Since you are running the nightly 4.11 release, it should have been resolved. I believe the lack of hardware offload for some packets and the zero length ack packets maybe related OVN/OVS flows instead of the sriov components. Could you provide the following information: In the case your test case is pod to pod on the same host then you only need to collect on that host. Otherwise you would collect the information on both host. The following is an example: STEP 0: Start your iperf test. STEP 1: [root@wsfd-advnetlab15 ~]# oc get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME master-0 Ready master,worker 12h v1.24.0+9546431 192.168.111.20 <none> Red Hat Enterprise Linux CoreOS 411.86.202207090519-0 (Ootpa) 4.18.0-372.13.1.el8_6.x86_64 cri-o://1.24.1-11.rhaos4.11.gitb0d2ef3.el8 master-1 Ready master,worker 12h v1.24.0+9546431 192.168.111.21 <none> Red Hat Enterprise Linux CoreOS 411.86.202207090519-0 (Ootpa) 4.18.0-372.13.1.el8_6.x86_64 cri-o://1.24.1-11.rhaos4.11.gitb0d2ef3.el8 master-2 Ready master,worker 12h v1.24.0+9546431 192.168.111.22 <none> Red Hat Enterprise Linux CoreOS 411.86.202207090519-0 (Ootpa) 4.18.0-372.13.1.el8_6.x86_64 cri-o://1.24.1-11.rhaos4.11.gitb0d2ef3.el8 worker-advnetlab48 Ready worker 11h v1.24.0+9546431 192.168.111.33 <none> Red Hat Enterprise Linux CoreOS 411.86.202207090519-0 (Ootpa) 4.18.0-372.13.1.el8_6.x86_64 cri-o://1.24.1-11.rhaos4.11.gitb0d2ef3.el8 worker-advnetlab49 Ready worker 11h v1.24.0+9546431 192.168.111.34 <none> Red Hat Enterprise Linux CoreOS 411.86.202207090519-0 (Ootpa) 4.18.0-372.13.1.el8_6.x86_64 cri-o://1.24.1-11.rhaos4.11.gitb0d2ef3.el8 STEP 2, ssh into the host: # Note you would need to ssh into the other host to gather information on that one if the client/server pods are on different hosts ssh core.111.33 STEP 3, run commands to get flows: # Note if the client/server pods are on different hosts then the commands would be repeated on the other host. sudo ovs-appctl dpctl/dump-flows --names -m # Record output # wait 20 seconds. sudo ovs-appctl dpctl/dump-flows --names -m # Record output. STEP 4, Gather Information: # Note if the client/server pods are on different hosts then the commands would be repeated on the other host. tc -s filter show dev <VF representor of the Client Iperf Pod> ingress # Record output tc -s filter show dev <VF representor of the Client Iperf Pod> egress # Record output tc -s filter show dev <VF representor of the Server Iperf Pod> ingress # Record output tc -s filter show dev <VF representor of the Server Iperf Pod> egress # Record output tc -s filter show dev br-ex egress # Record output tc -s filter show dev genev_sys_<some number> ingress # Record output tc -s filter show dev genev_sys_<some number> egress # Record output sudo ovs-appctl dpctl/show # Record output sudo ovs-vsctl show # Record output sudo ovs-ofctl dump-flows br-ex # Record output sudo ovs-ofctl dump-flows br-int # Record output Please let me know if any steps are confusing. Thanks, William Zhao Hi William, Since the pod to pod test results are the same on same host and on different hosts, so I collected the info you need on same host. I checked on version 4.11.0-rc.6 Please find the outputs of 'sudo ovs-appctl dpctl/dump-flows --names -m' with 20s interval in below link. http://pastebin.test.redhat.com/1069056 Please find tc and ovs cmd outputs in attached file. You can also check the cluster if needed. http://virt-openshift-05.lab.eng.nay.redhat.com/zzhao/kubeconfig export https_proxy=10.8.1.181:8888 export http_proxy=10.8.1.181:8888 My project is offload-testing. Sending traffic using command: oc rsh iperf-rc-ipv6-96mvl iperf3 -V -c fd01:0:0:3::5 -i 1 -t 200s lilia@liliadeMacBook-Pro offload_test % oc get nodes NAME STATUS ROLES AGE VERSION master-0 Ready master 10d v1.24.0+9546431 master-1 Ready master 10d v1.24.0+9546431 master-2 Ready master 10d v1.24.0+9546431 openshift-qe-024.lab.eng.rdu2.redhat.com Ready sriov,worker 10d v1.24.0+9546431 openshift-qe-027.lab.eng.rdu2.redhat.com Ready sriov,worker 10d v1.24.0+9546431 worker-0 Ready worker 10d v1.24.0+9546431 worker-1 Ready worker 10d v1.24.0+9546431 lilia@liliadeMacBook-Pro offload_test % oc get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES iperf-rc-ipv6-96mvl 1/1 Running 0 65m 10.128.4.6 openshift-qe-027.lab.eng.rdu2.redhat.com <none> <none> iperf-rc-ipv6-zxfnt 1/1 Running 0 65m 10.130.2.4 openshift-qe-024.lab.eng.rdu2.redhat.com <none> <none> iperf-server 1/1 Running 0 50m 10.130.2.5 openshift-qe-024.lab.eng.rdu2.redhat.com <none> <none> iperf-server-ipv6 1/1 Running 0 67m 10.128.4.5 openshift-qe-027.lab.eng.rdu2.redhat.com <none> <none> openshift-qe-027labengrdu2redhatcom-debug 1/1 Running 0 15m 192.168.111.58 openshift-qe-027.lab.eng.rdu2.redhat.com <none> <none> Please let me know if you need any other info. Thanks, Ying ============= First Run of dump-flows ============= ufid:aed183e2-4af8-4df2-a184-9146ed514bea, skb_priority(0/0),skb_mark(0/0),ct_state(0/0x23),ct_zone(0/0),ct_mark(0/0x2),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(6beacb688bd5598),packet_type(ns=0/0,id=0/0),eth(src=0a:58:0a:80:04:06,dst=0a:58:0a:80:04:05),eth_type(0x86dd),ipv6(src=fd01:0:0:3::6,dst=fd01:0:0:3::5,label=0/0,proto=6,tclass=0/0,hlimit=64,frag=no),tcp(src=0/0,dst=5201), packets:6721204, bytes:9500354728, used:0.450s, offloaded:yes, dp:tc, actions:ct(zone=16,nat),recirc(0x8ec97) ufid:3a449d7a-1d30-4205-a5f8-ef1c41894df6, skb_priority(0/0),skb_mark(0/0),ct_state(0x22/0x3e),ct_zone(0/0),ct_mark(0/0x3),ct_label(0/0),recirc_id(0x8ec97),dp_hash(0/0),in_port(6beacb688bd5598),packet_type(ns=0/0,id=0/0),eth(src=0a:58:0a:80:04:06,dst=0a:58:0a:80:04:05),eth_type(0x86dd),ipv6(src=fd00::/ffc0::,dst=fd01:0:0:3::5,label=0/0,proto=6,tclass=0/0,hlimit=64,frag=no),tcp(src=0/0,dst=0/0), packets:6721201, bytes:9500350486, used:0.450s, offloaded:yes, dp:tc, actions:ct(zone=10,nat),recirc(0x8ec98) ufid:a190e6cf-003a-439e-84a3-18b3e936ffa3, skb_priority(0/0),skb_mark(0/0),ct_state(0x22/0x3e),ct_zone(0/0),ct_mark(0/0x1),ct_label(0/0),recirc_id(0x8ec98),dp_hash(0/0),in_port(6beacb688bd5598),packet_type(ns=0/0,id=0/0),eth(src=0a:58:0a:80:04:06,dst=0a:58:0a:80:04:05),eth_type(0x86dd),ipv6(src=fd01:0:0:3::6,dst=fd01:0:0:3::5,label=0/0,proto=6,tclass=0/0,hlimit=64,frag=no),tcp(src=0/0,dst=0/0), packets:34942421, bytes:49380647522, used:0.450s, offloaded:yes, dp:tc, actions:946ab9651390e22 ============= Second Run of dump-flows (Missing flows?) ============= ufid:f247744e-1ef8-4bc4-977e-f0ca4289dc3f, skb_priority(0/0),skb_mark(0/0),ct_state(0/0x23),ct_zone(0/0),ct_mark(0/0x2),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(6beacb688bd5598),packet_type(ns=0/0,id=0/0),eth(src=0a:58:0a:80:04:06,dst=0a:58:0a:80:04:05),eth_type(0x86dd),ipv6(src=fd01:0:0:3::6,dst=fd01:0:0:3::5,label=0/0,proto=6,tclass=0/0,hlimit=64,frag=no),tcp(src=0/0,dst=5201), packets:2, bytes:2728, used:0.220s, offloaded:yes, dp:tc, actions:ct(zone=16,nat),recirc(0x8ec97) (Missing) ufid:a190e6cf-003a-439e-84a3-18b3e936ffa3, skb_priority(0/0),skb_mark(0/0),ct_state(0x22/0x3e),ct_zone(0/0),ct_mark(0/0x1),ct_label(0/0),recirc_id(0x8ec98),dp_hash(0/0),in_port(6beacb688bd5598),packet_type(ns=0/0,id=0/0),eth(src=0a:58:0a:80:04:06,dst=0a:58:0a:80:04:05),eth_type(0x86dd),ipv6(src=fd01:0:0:3::6,dst=fd01:0:0:3::5,label=0/0,proto=6,tclass=0/0,hlimit=64,frag=no),tcp(src=0/0,dst=0/0), packets:99080942, bytes:140054272276, used:0.810s, offloaded:yes, dp:tc, actions:946ab9651390e22 ============= First Run of dump-flows ============= ufid:0aeaf7fb-fa81-44ae-b796-17a2b913f41f, skb_priority(0/0),skb_mark(0/0),ct_state(0/0x23),ct_zone(0/0),ct_mark(0/0x2),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(946ab9651390e22),packet_type(ns=0/0,id=0/0),eth(src=0a:58:0a:80:04:05,dst=0a:58:0a:80:04:06),eth_type(0x86dd),ipv6(src=fd01:0:0:3::5,dst=fd01:0:0:3::6,label=0/0,proto=6,tclass=0/0,hlimit=64,frag=no),tcp(src=0/0,dst=50252), packets:93229, bytes:8020188, used:0.450s, offloaded:yes, dp:tc, actions:ct(zone=10,nat),recirc(0x8ec9b) ufid:57170943-fdb1-4a21-9181-5bfec80f0f4c, skb_priority(0/0),skb_mark(0/0),ct_state(0x2a/0x3e),ct_zone(0/0),ct_mark(0/0x3),ct_label(0/0),recirc_id(0x8ec9b),dp_hash(0/0),in_port(946ab9651390e22),packet_type(ns=0/0,id=0/0),eth(src=0a:58:0a:80:04:05,dst=0a:58:0a:80:04:06),eth_type(0x86dd),ipv6(src=fd00::/ffc0::,dst=fd01:0:0:3::6,label=0/0,proto=6,tclass=0/0,hlimit=64,frag=no),tcp(src=0/0,dst=0/0), packets:93218, bytes:7982406, used:0.450s, offloaded:yes, dp:tc, actions:ct(zone=16,nat),recirc(0x8ec9a) ufid:0d6756a8-6da9-41b0-be89-183ef846ccf7, skb_priority(0/0),skb_mark(0/0),ct_state(0x2a/0x3e),ct_zone(0/0),ct_mark(0/0x1),ct_label(0/0),recirc_id(0x8ec9a),dp_hash(0/0),in_port(946ab9651390e22),packet_type(ns=0/0,id=0/0),eth(src=00:00:00:00:00:00/00:00:00:00:00:00,dst=0a:58:0a:80:04:06),eth_type(0x86dd),ipv6(src=fd00::/ffc0::,dst=fd01:0:0:3::6,label=0/0,proto=6,tclass=0/0,hlimit=64,frag=no),tcp(src=0/0,dst=0/0), packets:708122, bytes:60735338, used:0.450s, offloaded:yes, dp:tc, actions:6beacb688bd5598 ============= Second Run of dump-flows (Missing flows?) ============= (Missing) (Missing) ufid:0d6756a8-6da9-41b0-be89-183ef846ccf7, skb_priority(0/0),skb_mark(0/0),ct_state(0x2a/0x3e),ct_zone(0/0),ct_mark(0/0x1),ct_label(0/0),recirc_id(0x8ec9a),dp_hash(0/0),in_port(946ab9651390e22),packet_type(ns=0/0,id=0/0),eth(src=00:00:00:00:00:00/00:00:00:00:00:00,dst=0a:58:0a:80:04:06),eth_type(0x86dd),ipv6(src=fd00::/ffc0::,dst=fd01:0:0:3::6,label=0/0,proto=6,tclass=0/0,hlimit=64,frag=no),tcp(src=0/0,dst=0/0), packets:2011984, bytes:172862158, used:0.810s, offloaded:yes, dp:tc, actions:6beacb688bd5598 Hi Ying, [wizhao@fedora bz2027584]$ oc get nodes NAME STATUS ROLES AGE VERSION master-0 Ready master 11d v1.24.0+9546431 master-1 Ready master 11d v1.24.0+9546431 master-2 Ready master 11d v1.24.0+9546431 openshift-qe-024.lab.eng.rdu2.redhat.com Ready sriov,worker 11d v1.24.0+9546431 openshift-qe-027.lab.eng.rdu2.redhat.com Ready sriov,worker 11d v1.24.0+9546431 worker-0 Ready worker 11d v1.24.0+9546431 worker-1 Ready worker 11d v1.24.0+9546431 [wizhao@fedora bz2027584]$ oc debug node/openshift-qe-027.lab.eng.rdu2.redhat.com Starting pod/openshift-qe-027labengrdu2redhatcom-debug ... To use host binaries, run `chroot /host` Pod IP: 192.168.111.58 If you don't see a command prompt, try pressing enter. sh-4.4# chroot /host sh-4.4# uname -a Linux openshift-qe-027 4.18.0-372.16.1.el8_6.x86_64 #1 SMP Tue Jun 28 03:02:21 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux sh-4.4# rpm -qa | grep openv openvswitch-selinux-extra-policy-1.0-29.el8fdp.noarch openvswitch2.17-2.17.0-22.el8fdp.x86_64 According to above you seem to be running openvswitch2.17-2.17.0-22.el8fdp.x86_64 on the node of interest. I talked to Marcelo, it seems that the OVS was also working on solving a similar issue in BZ 2094566 and BZ 2081773. From your dump-flows (thanks again for providing this), it seems that certain flows are evicted (missing) and I assume later on re-added (since the performance goes up and down). As stated in BZ 2094566, the fix was made available in openvswitch2.17-2.17.0-23.el8fdp. Also this issue only happens on IPv6. Could you please retest with openvswitch2.17-2.17.0-23.el8fdp? Hi William, I can not find openvswitch2.17-2.17.0-23.el8fdp in google, but I find image openvswitch-2.17.2.tar.gz, is it OK? I only need upgrade openvswitch on worker nodes, right? It would be appreciated if you can share the steps how to install openvswitch on cluster nodes. Thanks, Ying Hi Ying, Please do not use the openvswitch images from the internet. You should only install the openvswitch packages from Red Hat repositories. Official openvswitch builds for 2.17 are available here: http://download.eng.bos.redhat.com/brewroot/packages/openvswitch2.17/2.17.0/ I think you can take the latest and greatest rpm from here: http://download.eng.bos.redhat.com/brewroot/packages/openvswitch2.17/2.17.0/37.el8fdp/x86_64/ If there is something wrong, you can also try going to openvswitch2.17-2.17.0-23.el8fdp. Yes you only need to upgrade openvswitch on both worker nodes, here are the steps (very similar to upgrading the kernel on coreos): On BMH worker node (both): 1) curl -kO http://download.eng.bos.redhat.com/brewroot/packages/openvswitch2.17/2.17.0/37.el8fdp/x86_64/openvswitch2.17-2.17.0-37.el8fdp.x86_64.rpm 2) sudo rpm-ostree override replace openvswitch2.17-2.17.0-37.el8fdp.x86_64.rpm 3) sudo systemctl reboot 4) After reboot, check version with: rpm -qa | grep openv Please let me know if this works for you and if you have any other questions. Hi William, I tried openvswitch2.17-2.17.0-37.el8fdp.x86_64.rpm, the testing results look good. The performance is much more stable and no packets captured on iperf VF representors after offload take effect. You can check my cluster in comment8 for detail info. [root@openshift-qe-028 offload_test]# oc rsh iperf-rc-ipv6-jsbbf iperf3 -V -c fd01:0:0:2::4 -i 1 -t 60s iperf 3.1.3 Linux iperf-rc-ipv6-jsbbf 4.18.0-372.19.1.el8_6.x86_64 #1 SMP Mon Jul 18 11:14:02 EDT 2022 x86_64 Time: Mon, 08 Aug 2022 10:09:10 GMT Connecting to host fd01:0:0:2::4, port 5201 Cookie: iperf-rc-ipv6-jsbbf.1659953349.76346 TCP MSS: 1328 (default) [ 4] local fd01:0:0:3::5 port 49806 connected to fd01:0:0:2::4 port 5201 Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 60 second test [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 2.39 GBytes 20.5 Gbits/sec 1190 1.87 MBytes [ 4] 1.00-2.00 sec 2.44 GBytes 20.9 Gbits/sec 155 1.21 MBytes [ 4] 2.00-3.00 sec 2.38 GBytes 20.5 Gbits/sec 74 1.18 MBytes [ 4] 3.00-4.00 sec 2.42 GBytes 20.8 Gbits/sec 0 1.61 MBytes [ 4] 4.00-5.00 sec 2.27 GBytes 19.5 Gbits/sec 326 765 KBytes [ 4] 5.00-6.00 sec 1.77 GBytes 15.2 Gbits/sec 0 862 KBytes [ 4] 6.00-7.00 sec 2.39 GBytes 20.6 Gbits/sec 159 1.05 MBytes [ 4] 7.00-8.00 sec 2.40 GBytes 20.6 Gbits/sec 305 1.23 MBytes [ 4] 8.00-9.00 sec 2.50 GBytes 21.4 Gbits/sec 295 748 KBytes [ 4] 9.00-10.00 sec 2.45 GBytes 21.0 Gbits/sec 128 865 KBytes [ 4] 10.00-11.00 sec 2.27 GBytes 19.5 Gbits/sec 60 1.39 MBytes [ 4] 11.00-12.00 sec 2.32 GBytes 19.9 Gbits/sec 157 1000 KBytes [ 4] 12.00-13.00 sec 2.33 GBytes 20.0 Gbits/sec 36 1.19 MBytes [ 4] 13.00-14.00 sec 2.19 GBytes 18.8 Gbits/sec 372 770 KBytes [ 4] 14.00-15.00 sec 1.95 GBytes 16.8 Gbits/sec 0 1.42 MBytes [ 4] 15.00-16.00 sec 2.38 GBytes 20.5 Gbits/sec 473 1.07 MBytes [ 4] 16.00-17.00 sec 2.54 GBytes 21.8 Gbits/sec 0 1.45 MBytes [ 4] 17.00-18.00 sec 2.44 GBytes 21.0 Gbits/sec 44 1.12 MBytes [ 4] 18.00-19.00 sec 2.26 GBytes 19.4 Gbits/sec 0 1.93 MBytes [ 4] 19.00-20.00 sec 2.41 GBytes 20.7 Gbits/sec 73 1.92 MBytes [ 4] 20.00-21.00 sec 2.47 GBytes 21.2 Gbits/sec 134 1013 KBytes [ 4] 21.00-22.00 sec 2.23 GBytes 19.2 Gbits/sec 273 1.38 MBytes [ 4] 22.00-23.00 sec 2.25 GBytes 19.3 Gbits/sec 554 1.65 MBytes [ 4] 23.00-24.00 sec 2.42 GBytes 20.8 Gbits/sec 0 2.16 MBytes [ 4] 24.00-25.00 sec 2.53 GBytes 21.7 Gbits/sec 244 1.22 MBytes [ 4] 25.00-26.00 sec 2.43 GBytes 20.9 Gbits/sec 235 912 KBytes [ 4] 26.00-27.00 sec 2.30 GBytes 19.8 Gbits/sec 114 816 KBytes [ 4] 27.00-28.00 sec 2.00 GBytes 17.2 Gbits/sec 0 1.26 MBytes [ 4] 28.00-29.00 sec 2.37 GBytes 20.4 Gbits/sec 84 1.12 MBytes [ 4] 29.00-30.00 sec 2.37 GBytes 20.4 Gbits/sec 184 1.04 MBytes [ 4] 30.00-31.00 sec 2.51 GBytes 21.5 Gbits/sec 0 1.58 MBytes [ 4] 31.00-32.00 sec 2.51 GBytes 21.6 Gbits/sec 151 794 KBytes [ 4] 32.00-33.00 sec 2.39 GBytes 20.5 Gbits/sec 229 997 KBytes [ 4] 33.00-34.00 sec 2.50 GBytes 21.4 Gbits/sec 282 1010 KBytes [ 4] 34.00-35.00 sec 2.44 GBytes 21.0 Gbits/sec 176 829 KBytes [ 4] 35.00-36.00 sec 2.49 GBytes 21.4 Gbits/sec 138 1.08 MBytes [ 4] 36.00-37.00 sec 2.47 GBytes 21.2 Gbits/sec 211 860 KBytes [ 4] 37.00-38.00 sec 2.37 GBytes 20.4 Gbits/sec 20 1.45 MBytes [ 4] 38.00-39.00 sec 2.43 GBytes 20.9 Gbits/sec 0 2.15 MBytes [ 4] 39.00-40.00 sec 2.55 GBytes 21.9 Gbits/sec 0 2.32 MBytes [ 4] 40.00-41.00 sec 2.49 GBytes 21.4 Gbits/sec 118 1.09 MBytes [ 4] 41.00-42.00 sec 2.23 GBytes 19.2 Gbits/sec 281 1.14 MBytes [ 4] 42.00-43.00 sec 2.08 GBytes 17.8 Gbits/sec 65 1.31 MBytes [ 4] 43.00-44.00 sec 2.38 GBytes 20.5 Gbits/sec 306 1.19 MBytes [ 4] 44.00-45.00 sec 2.36 GBytes 20.3 Gbits/sec 0 1.96 MBytes [ 4] 45.00-46.00 sec 2.52 GBytes 21.6 Gbits/sec 0 2.29 MBytes [ 4] 46.00-47.00 sec 2.56 GBytes 22.0 Gbits/sec 0 2.40 MBytes [ 4] 47.00-48.00 sec 2.55 GBytes 21.9 Gbits/sec 0 2.48 MBytes [ 4] 48.00-49.00 sec 2.55 GBytes 21.9 Gbits/sec 0 2.53 MBytes [ 4] 49.00-50.00 sec 2.55 GBytes 21.9 Gbits/sec 0 2.60 MBytes [ 4] 50.00-51.00 sec 2.55 GBytes 21.9 Gbits/sec 0 2.66 MBytes [ 4] 51.00-52.00 sec 2.54 GBytes 21.9 Gbits/sec 0 2.71 MBytes [ 4] 52.00-53.00 sec 2.49 GBytes 21.4 Gbits/sec 1290 1.91 MBytes [ 4] 53.00-54.00 sec 2.58 GBytes 22.1 Gbits/sec 0 1.91 MBytes [ 4] 54.00-55.00 sec 2.58 GBytes 22.1 Gbits/sec 26 974 KBytes [ 4] 55.00-56.00 sec 2.50 GBytes 21.5 Gbits/sec 90 1.11 MBytes [ 4] 56.00-57.00 sec 2.39 GBytes 20.5 Gbits/sec 82 1.24 MBytes [ 4] 57.00-58.00 sec 2.41 GBytes 20.7 Gbits/sec 184 1.28 MBytes [ 4] 58.00-59.00 sec 2.44 GBytes 21.0 Gbits/sec 81 848 KBytes [ 4] 59.00-60.00 sec 2.46 GBytes 21.1 Gbits/sec 191 1.22 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-60.00 sec 144 GBytes 20.6 Gbits/sec 9590 sender [ 4] 0.00-60.00 sec 144 GBytes 20.6 Gbits/sec receiver CPU Utilization: local/sender 22.5% (0.4%u/22.1%s), remote/receiver 21.1% (0.8%u/20.2%s) iperf Done. Thanks, Ying Hi Ying, Yes, I agree the performance is much better. Thanks for retesting. We need to bump Openshift's openvswitch version. Hi Ying, The openvswitch "openvswitch2.17-2.17.0-31.el8fdp.x86_64" is now available in openshift 4.11 nightly build. Please retest at your convenience. If the problem does not occur anymore, I believe we can close this bug. Please let me know otherwise. Thanks again for helping to troubleshoot this. Hi William, The version I tested in Comment 14 is 4.11.0-rc.7, this issue was not seen. Thanks, Ying. Hi Ying, I logged into your cluster from Comment 8 . I noticed that the openvswitch version is openvswitch2.17-2.17.0-37.el8fdp.x86_64. Which leads me to believe you upgraded openvswitch following my steps in Comment 13 . However that version is not what comes with 4.11.0-rc.7. According to: https://openshift-release.apps.ci.l2s4.p1.openshiftapps.com/releasestream/4-stable/release/4.11.0-rc.7 and https://releases-rhcos-art.cloud.privileged.psi.redhat.com/?release=411.86.202208031059-0&stream=releases%2Frhcos-4.11#411.86.202208031059-0 Openswitch was upgraded from 2.17.0-22.el8fdp → 2.17.0-31.el8fdp. Therefore we should be testing with 2.17.0-31.el8fdp instead of 2.17.0-37.el8fdp. I guess the sure way to test if the fix is properly included in Openshift 4.11.0-rc.7 is to redeploy the worker (I think these are baremetal as well?) nodes (you do not need redeploy the cluster). This should instead a fresh coreos image on those worker nodes. Could you please verify that 2.17.0-31.el8fdp is working as well? Sorry about the confusion, I wasn't aware that openvswitch was going to be updated to 2.17.0-31.el8fdp last Friday. The upgrading of openvswitch is automated every 6 weeks. Please feel free to ping me if something is not clear here. Hi William, I tried build 4.11.0-0.nightly-2022-08-17-152830 which with openvswitch2.17-2.17.0-31.el8fdp.x86_64, ipv6 pod-to-pod traffic HW offload working fine. sh-4.4# rpm -qa | grep openv openvswitch-selinux-extra-policy-1.0-29.el8fdp.noarch openvswitch2.17-2.17.0-31.el8fdp.x86_64 Thanks, Ying Thanks Ying, Setting this bug a duplicate of BZ 2094566. *** This bug has been marked as a duplicate of bug 2094566 *** |
Description of problem: Enable ovs hardware offload on cluster. Using ovs hwoffload VF as default network. Sending ipv6 iperf traffic from one pod to another, check cpu on worker node, sometimes ovs-switchd use 100% CPU. And traffic offload is not stable, which causes the throughput fluctuate wildly. For example as below, sending ipv6 traffic from one pod to another (different node), the average bandwidth is 16.9Gbps, but check the detail bandwidth number per second, some numbers are lower than 1Gbps, if tcpdump on VF reps in real-time, data packets can be captured when the performance is low, which means those packets are not offloaded to hardware. [root@f33-h13-000-r640 ipv6]# oc rsh iperf-rc-ipv6-rfsv5 iperf3 -V -c fd01:0:0:2::4a5 -i 1 -t 60s iperf 3.1.3 Linux iperf-rc-ipv6-rfsv5 4.18.0-305.34.2.el8_4.x86_64 #1 SMP Mon Jan 17 09:42:23 EST 2022 x86_64 Time: Wed, 09 Feb 2022 07:27:04 GMT Connecting to host fd01:0:0:2::4a5, port 5201 Cookie: iperf-rc-ipv6-rfsv5.1644391624.24913 TCP MSS: 1328 (default) [ 4] local fd01:0:0:5::13 port 42280 connected to fd01:0:0:2::4a5 port 5201 Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 60 second test [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 2.35 GBytes 20.2 Gbits/sec 23 939 KBytes [ 4] 1.00-2.00 sec 1.91 GBytes 16.4 Gbits/sec 1001 633 KBytes [ 4] 2.00-3.00 sec 2.26 GBytes 19.4 Gbits/sec 125 873 KBytes [ 4] 3.00-4.00 sec 2.48 GBytes 21.3 Gbits/sec 292 891 KBytes [ 4] 4.00-5.00 sec 2.47 GBytes 21.2 Gbits/sec 462 855 KBytes [ 4] 5.00-6.00 sec 1.32 GBytes 11.3 Gbits/sec 222 865 KBytes [ 4] 6.00-7.00 sec 2.41 GBytes 20.7 Gbits/sec 178 930 KBytes [ 4] 7.00-8.00 sec 2.36 GBytes 20.3 Gbits/sec 346 821 KBytes [ 4] 8.00-9.00 sec 2.47 GBytes 21.3 Gbits/sec 210 851 KBytes [ 4] 9.00-10.00 sec 2.41 GBytes 20.7 Gbits/sec 316 919 KBytes [ 4] 10.00-11.00 sec 2.38 GBytes 20.4 Gbits/sec 360 847 KBytes [ 4] 11.00-12.00 sec 2.46 GBytes 21.1 Gbits/sec 482 1.14 MBytes [ 4] 12.00-13.00 sec 2.32 GBytes 20.0 Gbits/sec 308 1.00 MBytes [ 4] 13.00-14.00 sec 2.44 GBytes 20.9 Gbits/sec 228 1.18 MBytes [ 4] 14.00-15.00 sec 2.40 GBytes 20.6 Gbits/sec 247 966 KBytes [ 4] 15.00-16.00 sec 2.48 GBytes 21.3 Gbits/sec 507 696 KBytes [ 4] 16.00-17.00 sec 1.40 GBytes 12.0 Gbits/sec 168 689 KBytes [ 4] 17.00-18.00 sec 1.12 GBytes 9.65 Gbits/sec 249 971 KBytes [ 4] 18.00-19.00 sec 2.47 GBytes 21.3 Gbits/sec 152 997 KBytes [ 4] 19.00-20.00 sec 2.45 GBytes 21.1 Gbits/sec 152 1.10 MBytes [ 4] 20.00-21.00 sec 2.48 GBytes 21.3 Gbits/sec 213 831 KBytes [ 4] 21.00-22.00 sec 2.39 GBytes 20.5 Gbits/sec 0 1.49 MBytes [ 4] 22.00-23.00 sec 2.52 GBytes 21.7 Gbits/sec 16 1.37 MBytes [ 4] 23.00-24.00 sec 2.57 GBytes 22.1 Gbits/sec 0 1.49 MBytes [ 4] 24.00-25.00 sec 1.82 GBytes 15.7 Gbits/sec 492 792 KBytes [ 4] 25.00-26.00 sec 455 MBytes 3.81 Gbits/sec 79 493 KBytes [ 4] 26.00-27.00 sec 1.72 GBytes 14.7 Gbits/sec 439 984 KBytes [ 4] 27.00-28.00 sec 2.33 GBytes 20.0 Gbits/sec 320 1.09 MBytes [ 4] 28.00-29.00 sec 2.42 GBytes 20.8 Gbits/sec 117 883 KBytes [ 4] 29.00-30.00 sec 255 MBytes 2.14 Gbits/sec 151 456 KBytes [ 4] 30.00-31.00 sec 1.42 GBytes 12.2 Gbits/sec 269 795 KBytes [ 4] 31.00-32.00 sec 1.52 GBytes 13.1 Gbits/sec 442 875 KBytes [ 4] 32.00-33.00 sec 304 MBytes 2.55 Gbits/sec 613 599 KBytes [ 4] 33.00-34.00 sec 2.35 GBytes 20.2 Gbits/sec 22 982 KBytes [ 4] 34.00-35.00 sec 2.33 GBytes 20.0 Gbits/sec 227 827 KBytes [ 4] 35.00-36.00 sec 2.46 GBytes 21.1 Gbits/sec 40 949 KBytes [ 4] 36.00-37.00 sec 2.47 GBytes 21.2 Gbits/sec 285 741 KBytes [ 4] 37.00-38.00 sec 2.42 GBytes 20.8 Gbits/sec 276 783 KBytes [ 4] 38.00-39.00 sec 2.42 GBytes 20.8 Gbits/sec 146 689 KBytes [ 4] 39.00-40.00 sec 2.32 GBytes 19.9 Gbits/sec 227 895 KBytes [ 4] 40.00-41.00 sec 2.32 GBytes 19.9 Gbits/sec 265 756 KBytes [ 4] 41.00-42.00 sec 2.40 GBytes 20.6 Gbits/sec 399 934 KBytes [ 4] 42.00-43.00 sec 2.35 GBytes 20.2 Gbits/sec 77 960 KBytes [ 4] 43.00-44.00 sec 2.45 GBytes 21.1 Gbits/sec 209 856 KBytes [ 4] 44.00-45.00 sec 2.49 GBytes 21.4 Gbits/sec 55 975 KBytes [ 4] 45.00-46.00 sec 2.46 GBytes 21.2 Gbits/sec 381 887 KBytes [ 4] 46.00-47.00 sec 1.50 GBytes 12.9 Gbits/sec 423 685 KBytes [ 4] 47.00-48.00 sec 2.07 GBytes 17.8 Gbits/sec 909 1.03 MBytes [ 4] 48.00-49.00 sec 457 MBytes 3.83 Gbits/sec 259 751 KBytes [ 4] 49.00-50.00 sec 6.08 MBytes 51.0 Mbits/sec 4 374 KBytes [ 4] 50.00-51.00 sec 951 MBytes 7.98 Gbits/sec 157 674 KBytes [ 4] 51.00-52.00 sec 96.1 MBytes 806 Mbits/sec 0 752 KBytes [ 4] 52.00-53.00 sec 1.10 GBytes 9.48 Gbits/sec 451 824 KBytes [ 4] 53.00-54.00 sec 1.95 GBytes 16.8 Gbits/sec 678 516 KBytes [ 4] 54.00-55.00 sec 2.38 GBytes 20.4 Gbits/sec 53 1.05 MBytes [ 4] 55.00-56.00 sec 2.49 GBytes 21.4 Gbits/sec 193 1.01 MBytes [ 4] 56.00-57.00 sec 2.42 GBytes 20.8 Gbits/sec 497 912 KBytes [ 4] 57.00-58.00 sec 310 MBytes 2.60 Gbits/sec 470 393 KBytes [ 4] 58.00-59.00 sec 2.15 GBytes 18.5 Gbits/sec 416 916 KBytes [ 4] 59.00-60.00 sec 2.43 GBytes 20.9 Gbits/sec 229 931 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-60.00 sec 118 GBytes 16.9 Gbits/sec 16527 sender [ 4] 0.00-60.00 sec 118 GBytes 16.9 Gbits/sec receiver CPU Utilization: local/sender 14.3% (0.3%u/14.0%s), remote/receiver 10.7% (0.3%u/10.4%s) Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1.Enable ovs hardware offload on cluster 2.configrue ipv6 on cluster 3.create 2 pods with iperf installed using ovs hwoffload VF as default network interface. 3.Send ipv6 iperf traffic from one pod to another and check cpu on worker node. Actual results: Expected results: Additional info: