Description of problem: Under certain circunstances, some of the ovs physical queues are not used when using multiqueue. Configuration: ovs multiqueue with 3 physical queues Injection: 40 different flows in each port Most of the times I can see that traffic is distributed among all of the queues as expected, but there are times in which one of the queues is not used testpmd command /root/dpdk/build/app/dpdk-testpmd -l 1-7 -n 4 --socket-mem 1024 -- -i --nb-cores=6 --auto-start --forward-mode=rxonly --rxd=1024 --txd=1024 --rxq=3 --txq=3 testpmd output. For some reason queue 1 is not being used and queue2 is receiving 2 times more traffic than queue 0. ------- Forward Stats for RX Port= 0/Queue= 0 -> TX Port= 1/Queue= 0 ------- RX-packets: 14 TX-packets: 0 TX-dropped: 0 ------- Forward Stats for RX Port= 1/Queue= 0 -> TX Port= 0/Queue= 0 ------- RX-packets: 16 TX-packets: 0 TX-dropped: 0 ------- Forward Stats for RX Port= 0/Queue= 2 -> TX Port= 1/Queue= 2 ------- RX-packets: 38 TX-packets: 0 TX-dropped: 0 ------- Forward Stats for RX Port= 1/Queue= 2 -> TX Port= 0/Queue= 2 ------- RX-packets: 34 TX-packets: 0 TX-dropped: 0 ---------------------- Forward statistics for port 0 ---------------------- RX-packets: 52 RX-dropped: 0 RX-total: 52 TX-packets: 0 TX-dropped: 0 TX-total: 0 ---------------------------------------------------------------------------- ---------------------- Forward statistics for port 1 ---------------------- RX-packets: 50 RX-dropped: 0 RX-total: 50 TX-packets: 0 TX-dropped: 0 TX-total: 0 ---------------------------------------------------------------------------- +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++ RX-packets: 102 RX-dropped: 0 RX-total: 102 TX-packets: 0 TX-dropped: 0 TX-total: 0 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ I am sending only 40 packets per port but I am receiving around 50 because there are some ipv6 packets sent by the switch that is making some noise, but i think this is unrelated. There are times that updating the configuration it start working properly and traffic is distributed among all the queues, but I can not reproduce it all the times, not sure what is causing that sometimes it works but other times it fails. Traffic injection does not change. ovs configuration Bridge br-dpdk1 Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure datapath_type: netdev Port dpdk3 Interface dpdk3 type: dpdk options: {dpdk-devargs="0000:82:00.3", n_rxq="3"} Port br-dpdk1 Interface br-dpdk1 type: internal Port phy-br-dpdk1 Interface phy-br-dpdk1 type: patch options: {peer=int-br-dpdk1} Bridge br-dpdk0 Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure datapath_type: netdev Port dpdk2 Interface dpdk2 type: dpdk options: {dpdk-devargs="0000:82:00.2", n_rxq="3"} Port br-dpdk0 Interface br-dpdk0 type: internal Port phy-br-dpdk0 Interface phy-br-dpdk0 type: patch options: {peer=int-br-dpdk0} [root@computedpdksriov-r730-0 heat-admin]# ovs-vsctl get open_vSwitch . other_config {dpdk-extra=" -n 4", dpdk-init="true", dpdk-socket-mem="4096,1024", pmd-auto-lb="true", pmd-auto-lb-improvement-threshold="50", pmd-auto-lb-load-threshold="70", pmd-auto-lb-rebal-interval="3", pmd-cpu-mask=fc} [root@computedpdksriov-r730-0 heat-admin]# ovs-appctl dpif-netdev/pmd-rxq-show pmd thread numa_id 0 core_id 2: isolated : false port: dpdk0 queue-id: 1 (enabled) pmd usage: 0 % port: dpdk1 queue-id: 1 (enabled) pmd usage: 0 % port: vhud6341185-6a queue-id: 0 (enabled) pmd usage: 0 % port: vhud6341185-6a queue-id: 4 (disabled) pmd usage: 0 % port: vhud6341185-6a queue-id: 5 (disabled) pmd usage: 0 % port: vhue6a5fe8c-a9 queue-id: 3 (enabled) pmd usage: 0 % port: vhue6a5fe8c-a9 queue-id: 5 (enabled) pmd usage: 0 % port: vhuedf0edcd-3f queue-id: 0 (enabled) pmd usage: 0 % port: vhuedf0edcd-3f queue-id: 3 (disabled) pmd usage: 0 % port: vhuedf0edcd-3f queue-id: 4 (disabled) pmd usage: 0 % pmd thread numa_id 1 core_id 3: isolated : true port: dpdk2 queue-id: 0 (enabled) pmd usage: 0 % port: dpdk3 queue-id: 0 (enabled) pmd usage: 0 % pmd thread numa_id 0 core_id 4: isolated : false port: dpdk1 queue-id: 0 (enabled) pmd usage: 0 % port: dpdk1 queue-id: 2 (enabled) pmd usage: 0 % port: vhud6341185-6a queue-id: 3 (disabled) pmd usage: 0 % port: vhud6341185-6a queue-id: 6 (disabled) pmd usage: 0 % port: vhue6a5fe8c-a9 queue-id: 0 (enabled) pmd usage: 0 % port: vhue6a5fe8c-a9 queue-id: 1 (enabled) pmd usage: 0 % port: vhue6a5fe8c-a9 queue-id: 6 (enabled) pmd usage: 0 % port: vhue6a5fe8c-a9 queue-id: 7 (enabled) pmd usage: 0 % port: vhuedf0edcd-3f queue-id: 2 (enabled) pmd usage: 0 % port: vhuedf0edcd-3f queue-id: 5 (disabled) pmd usage: 0 % pmd thread numa_id 1 core_id 5: isolated : true port: dpdk2 queue-id: 1 (enabled) pmd usage: 0 % port: dpdk3 queue-id: 1 (enabled) pmd usage: 0 % pmd thread numa_id 0 core_id 6: isolated : false port: dpdk0 queue-id: 0 (enabled) pmd usage: 0 % port: dpdk0 queue-id: 2 (enabled) pmd usage: 0 % port: vhud6341185-6a queue-id: 1 (enabled) pmd usage: 0 % port: vhud6341185-6a queue-id: 2 (enabled) pmd usage: 0 % port: vhud6341185-6a queue-id: 7 (disabled) pmd usage: 0 % port: vhue6a5fe8c-a9 queue-id: 2 (enabled) pmd usage: 0 % port: vhue6a5fe8c-a9 queue-id: 4 (enabled) pmd usage: 0 % port: vhuedf0edcd-3f queue-id: 1 (enabled) pmd usage: 0 % port: vhuedf0edcd-3f queue-id: 6 (disabled) pmd usage: 0 % port: vhuedf0edcd-3f queue-id: 7 (disabled) pmd usage: 0 % pmd thread numa_id 1 core_id 7: isolated : true port: dpdk2 queue-id: 2 (enabled) pmd usage: 0 % port: dpdk3 queue-id: 2 (enabled) pmd usage: 0 % Version-Release number of selected component (if applicable): RHOS-16.2-RHEL-8-20211112.n.1 openvswitch2.15-2.15.0-38.el8fdp.x86_64 How reproducible: Configure ovs dpdk multiqueu Create a vm, configure dpdk and run testpmd Create other vm and send several packets changing source or destination ip in each packet Actual results: All of the queues should receive packets Expected results: One queue does not receive packets Additional info:
If I configure 4 queues, I receive traffic in 3 queues (instead or 4). I would say there is a configuration issue [root@computedpdksriov-r730-0 net]# ovs-vsctl set Interface dpdk2 options:n_rxq=4 [root@computedpdksriov-r730-0 net]# ovs-vsctl set Interface dpdk3 options:n_rxq=4 ------- Forward Stats for RX Port= 0/Queue= 0 -> TX Port= 1/Queue= 0 ------- RX-packets: 20 TX-packets: 0 TX-dropped: 0 ------- Forward Stats for RX Port= 1/Queue= 0 -> TX Port= 0/Queue= 0 ------- RX-packets: 9 TX-packets: 0 TX-dropped: 0 ------- Forward Stats for RX Port= 1/Queue= 1 -> TX Port= 0/Queue= 1 ------- RX-packets: 11 TX-packets: 0 TX-dropped: 0 ------- Forward Stats for RX Port= 0/Queue= 2 -> TX Port= 1/Queue= 2 ------- RX-packets: 20 TX-packets: 0 TX-dropped: 0 ------- Forward Stats for RX Port= 1/Queue= 2 -> TX Port= 0/Queue= 2 ------- RX-packets: 20 TX-packets: 0 TX-dropped: 0 ---------------------- Forward statistics for port 0 ---------------------- RX-packets: 40 RX-dropped: 0 RX-total: 40 TX-packets: 0 TX-dropped: 0 TX-total: 0 ---------------------------------------------------------------------------- ---------------------- Forward statistics for port 1 ---------------------- RX-packets: 40 RX-dropped: 0 RX-total: 40 TX-packets: 0 TX-dropped: 0 TX-total: 0 ---------------------------------------------------------------------------- +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++ RX-packets: 80 RX-dropped: 0 RX-total: 80 TX-packets: 0 TX-dropped: 0 TX-total: 0 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
This is the traffic sent. I am changing either source ip or destination ip, so flows are different [root@testpmd-dpdk-dut cloud-user]# tcpdump -i eth1 -nne dropped privs to tcpdump tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes 10:47:54.398672 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 65: 16.0.0.1.1025 > 48.0.0.1.32768: UDP, length 23 10:47:54.462568 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 66: 16.0.0.1.1025 > 48.0.0.2.32768: UDP, length 24 10:47:54.526590 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 67: 16.0.0.1.1025 > 48.0.0.3.32768: UDP, length 25 10:47:54.590510 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 68: 16.0.0.1.1025 > 48.0.0.4.32768: UDP, length 26 10:47:54.654374 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 69: 16.0.0.1.1025 > 48.0.0.5.32768: UDP, length 27 10:47:54.718372 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 70: 16.0.0.1.1025 > 48.0.0.6.32768: UDP, length 28 10:47:54.782555 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 71: 16.0.0.1.1025 > 48.0.0.7.32768: UDP, length 29 10:47:54.846376 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 72: 16.0.0.1.1025 > 48.0.0.8.32768: UDP, length 30 10:47:54.910399 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 73: 16.0.0.1.1025 > 48.0.0.9.32768: UDP, length 31 10:47:54.974375 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 74: 16.0.0.1.1025 > 48.0.0.10.32768: UDP, length 32 10:47:55.038369 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 75: 16.0.0.1.1025 > 48.0.0.11.32768: UDP, length 33 10:47:55.102373 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 76: 16.0.0.1.1025 > 48.0.0.12.32768: UDP, length 34 10:47:55.166512 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 77: 16.0.0.1.1025 > 48.0.0.13.32768: UDP, length 35 10:47:55.230359 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 78: 16.0.0.1.1025 > 48.0.0.14.32768: UDP, length 36 10:47:55.294374 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 79: 16.0.0.1.1025 > 48.0.0.15.32768: UDP, length 37 10:47:55.358363 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 80: 16.0.0.1.1025 > 48.0.0.16.32768: UDP, length 38 10:47:55.422376 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 81: 16.0.0.1.1025 > 48.0.0.17.32768: UDP, length 39 10:47:55.486363 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 82: 16.0.0.1.1025 > 48.0.0.18.32768: UDP, length 40 10:47:55.550361 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 83: 16.0.0.1.1025 > 48.0.0.19.32768: UDP, length 41 10:47:55.614359 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 84: 16.0.0.1.1025 > 48.0.0.20.32768: UDP, length 42 10:47:55.678358 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 85: 16.0.0.1.1025 > 48.0.0.21.32768: UDP, length 43 10:47:55.742523 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 86: 16.0.0.1.1025 > 48.0.0.22.32768: UDP, length 44 10:47:55.806368 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 87: 16.0.0.1.1025 > 48.0.0.23.32768: UDP, length 45 10:47:55.870368 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 88: 16.0.0.1.1025 > 48.0.0.24.32768: UDP, length 46 10:47:55.934374 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 89: 16.0.0.1.1025 > 48.0.0.25.32768: UDP, length 47 10:47:55.998377 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 90: 16.0.0.1.1025 > 48.0.0.26.32768: UDP, length 48 10:47:56.062361 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 91: 16.0.0.1.1025 > 48.0.0.27.32768: UDP, length 49 10:47:56.126379 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 92: 16.0.0.1.1025 > 48.0.0.28.32768: UDP, length 50 10:47:56.190362 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 93: 16.0.0.1.1025 > 48.0.0.29.32768: UDP, length 51 10:47:56.254367 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 94: 16.0.0.1.1025 > 48.0.0.30.32768: UDP, length 52 10:47:56.318517 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 95: 16.0.0.1.1025 > 48.0.0.31.32768: UDP, length 53 10:47:56.382386 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 96: 16.0.0.1.1025 > 48.0.0.32.32768: UDP, length 54 10:47:56.446382 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 97: 16.0.0.1.1025 > 48.0.0.33.32768: UDP, length 55 10:47:56.510363 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 98: 16.0.0.1.1025 > 48.0.0.34.32768: UDP, length 56 10:47:56.574391 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 99: 16.0.0.1.1025 > 48.0.0.35.32768: UDP, length 57 10:47:56.638367 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 100: 16.0.0.1.1025 > 48.0.0.36.32768: UDP, length 58 10:47:56.702369 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 101: 16.0.0.1.1025 > 48.0.0.37.32768: UDP, length 59 10:47:56.766360 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 102: 16.0.0.1.1025 > 48.0.0.38.32768: UDP, length 60 10:47:56.830368 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 103: 16.0.0.1.1025 > 48.0.0.39.32768: UDP, length 61 10:47:56.894521 f8:f2:1e:42:5d:72 > f8:f2:1e:42:5d:73, ethertype IPv4 (0x0800), length 104: 16.0.0.1.1025 > 48.0.0.40.32768: UDP, length 62 [root@testpmd-dpdk-dut cloud-user]# tcpdump -i eth2 -nne dropped privs to tcpdump tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth2, link-type EN10MB (Ethernet), capture size 262144 bytes 10:48:30.764423 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 65: 48.0.0.1.32768 > 16.0.0.1.1025: UDP, length 23 10:48:30.828428 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 66: 48.0.0.2.32768 > 16.0.0.1.1025: UDP, length 24 10:48:30.892383 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 67: 48.0.0.3.32768 > 16.0.0.1.1025: UDP, length 25 10:48:30.956819 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 68: 48.0.0.4.32768 > 16.0.0.1.1025: UDP, length 26 10:48:31.020381 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 69: 48.0.0.5.32768 > 16.0.0.1.1025: UDP, length 27 10:48:31.084382 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 70: 48.0.0.6.32768 > 16.0.0.1.1025: UDP, length 28 10:48:31.148380 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 71: 48.0.0.7.32768 > 16.0.0.1.1025: UDP, length 29 10:48:31.212552 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 72: 48.0.0.8.32768 > 16.0.0.1.1025: UDP, length 30 10:48:31.276414 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 73: 48.0.0.9.32768 > 16.0.0.1.1025: UDP, length 31 10:48:31.340378 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 74: 48.0.0.10.32768 > 16.0.0.1.1025: UDP, length 32 10:48:31.404382 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 75: 48.0.0.11.32768 > 16.0.0.1.1025: UDP, length 33 10:48:31.468530 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 76: 48.0.0.12.32768 > 16.0.0.1.1025: UDP, length 34 10:48:31.532381 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 77: 48.0.0.13.32768 > 16.0.0.1.1025: UDP, length 35 10:48:31.596375 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 78: 48.0.0.14.32768 > 16.0.0.1.1025: UDP, length 36 10:48:31.660384 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 79: 48.0.0.15.32768 > 16.0.0.1.1025: UDP, length 37 10:48:31.724526 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 80: 48.0.0.16.32768 > 16.0.0.1.1025: UDP, length 38 10:48:31.788383 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 81: 48.0.0.17.32768 > 16.0.0.1.1025: UDP, length 39 10:48:31.852385 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 82: 48.0.0.18.32768 > 16.0.0.1.1025: UDP, length 40 10:48:31.916381 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 83: 48.0.0.19.32768 > 16.0.0.1.1025: UDP, length 41 10:48:31.980532 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 84: 48.0.0.20.32768 > 16.0.0.1.1025: UDP, length 42 10:48:32.044383 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 85: 48.0.0.21.32768 > 16.0.0.1.1025: UDP, length 43 10:48:32.108379 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 86: 48.0.0.22.32768 > 16.0.0.1.1025: UDP, length 44 10:48:32.172381 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 87: 48.0.0.23.32768 > 16.0.0.1.1025: UDP, length 45 10:48:32.236537 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 88: 48.0.0.24.32768 > 16.0.0.1.1025: UDP, length 46 10:48:32.300385 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 89: 48.0.0.25.32768 > 16.0.0.1.1025: UDP, length 47 10:48:32.364385 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 90: 48.0.0.26.32768 > 16.0.0.1.1025: UDP, length 48 10:48:32.428382 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 91: 48.0.0.27.32768 > 16.0.0.1.1025: UDP, length 49 10:48:32.492536 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 92: 48.0.0.28.32768 > 16.0.0.1.1025: UDP, length 50 10:48:32.556377 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 93: 48.0.0.29.32768 > 16.0.0.1.1025: UDP, length 51 10:48:32.620384 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 94: 48.0.0.30.32768 > 16.0.0.1.1025: UDP, length 52 10:48:32.684378 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 95: 48.0.0.31.32768 > 16.0.0.1.1025: UDP, length 53 10:48:32.748538 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 96: 48.0.0.32.32768 > 16.0.0.1.1025: UDP, length 54 10:48:32.812380 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 97: 48.0.0.33.32768 > 16.0.0.1.1025: UDP, length 55 10:48:32.876385 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 98: 48.0.0.34.32768 > 16.0.0.1.1025: UDP, length 56 10:48:32.940380 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 99: 48.0.0.35.32768 > 16.0.0.1.1025: UDP, length 57 10:48:33.004546 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 100: 48.0.0.36.32768 > 16.0.0.1.1025: UDP, length 58 10:48:33.068383 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 101: 48.0.0.37.32768 > 16.0.0.1.1025: UDP, length 59 10:48:33.132381 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 102: 48.0.0.38.32768 > 16.0.0.1.1025: UDP, length 60 10:48:33.200381 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 103: 48.0.0.39.32768 > 16.0.0.1.1025: UDP, length 61 10:48:33.264548 f8:f2:1e:42:5d:73 > f8:f2:1e:42:5d:72, ethertype IPv4 (0x0800), length 104: 48.0.0.40.32768 > 16.0.0.1.1025: UDP, length 62
I have injected at higher rates and i can see checking pmd cpu usage that all physical queues are used, but some virtual queues are not used. physical queue dpdk2 is connected to virtual queue vhu8cbade5e-57 and dpdk3 to vhub27eba49-8c dpdk2 pmd cores: 3, 5, 7 dpdk3 pmd cores: 3, 5, 7 vhu8cbade5e-57 pmd_cores: 2, 7 vhub27eba49-8c pmd_cores: 5, 7 pmd thread numa_id 0 core_id 2: isolated : false port: dpdk0 queue-id: 1 (enabled) pmd usage: 0 % port: dpdk1 queue-id: 1 (enabled) pmd usage: 0 % pmd thread numa_id 1 core_id 3: isolated : false port: dpdk2 queue-id: 0 (enabled) pmd usage: 7 % port: dpdk3 queue-id: 2 (enabled) pmd usage: 10 % port: vhu8cbade5e-57 queue-id: 1 (enabled) pmd usage: 0 % port: vhu8cbade5e-57 queue-id: 2 (enabled) pmd usage: 9 % port: vhu8cbade5e-57 queue-id: 7 (disabled) pmd usage: 0 % port: vhu90e5086e-e2 queue-id: 4 (enabled) pmd usage: 0 % port: vhu90e5086e-e2 queue-id: 5 (enabled) pmd usage: 0 % port: vhu90e5086e-e2 queue-id: 6 (enabled) pmd usage: 0 % port: vhub27eba49-8c queue-id: 3 (disabled) pmd usage: 0 % port: vhub27eba49-8c queue-id: 4 (disabled) pmd usage: 0 % pmd thread numa_id 0 core_id 4: isolated : false port: dpdk0 queue-id: 0 (enabled) pmd usage: 0 % port: dpdk1 queue-id: 2 (enabled) pmd usage: 0 % pmd thread numa_id 1 core_id 5: isolated : false port: dpdk2 queue-id: 2 (enabled) pmd usage: 9 % port: dpdk3 queue-id: 0 (enabled) pmd usage: 6 % port: vhu8cbade5e-57 queue-id: 4 (disabled) pmd usage: 0 % port: vhu8cbade5e-57 queue-id: 5 (disabled) pmd usage: 0 % port: vhu90e5086e-e2 queue-id: 0 (enabled) pmd usage: 0 % port: vhu90e5086e-e2 queue-id: 1 (enabled) pmd usage: 0 % port: vhub27eba49-8c queue-id: 0 (enabled) pmd usage: 5 % port: vhub27eba49-8c queue-id: 1 (enabled) pmd usage: 0 % port: vhub27eba49-8c queue-id: 6 (disabled) pmd usage: 0 % port: vhub27eba49-8c queue-id: 7 (disabled) pmd usage: 0 % pmd thread numa_id 0 core_id 6: isolated : false port: dpdk0 queue-id: 2 (enabled) pmd usage: 0 % port: dpdk1 queue-id: 0 (enabled) pmd usage: 0 % pmd thread numa_id 1 core_id 7: isolated : false port: dpdk2 queue-id: 1 (enabled) pmd usage: 5 % port: dpdk3 queue-id: 1 (enabled) pmd usage: 6 % port: vhu8cbade5e-57 queue-id: 0 (enabled) pmd usage: 8 % port: vhu8cbade5e-57 queue-id: 3 (disabled) pmd usage: 0 % port: vhu8cbade5e-57 queue-id: 6 (disabled) pmd usage: 0 % port: vhu90e5086e-e2 queue-id: 2 (enabled) pmd usage: 0 % port: vhu90e5086e-e2 queue-id: 3 (enabled) pmd usage: 0 % port: vhu90e5086e-e2 queue-id: 7 (enabled) pmd usage: 0 % port: vhub27eba49-8c queue-id: 2 (enabled) pmd usage: 12 % port: vhub27eba49-8c queue-id: 5 (disabled) pmd usage: 0 % Waiting for lcores to finish... ------- Forward Stats for RX Port= 0/Queue= 0 -> TX Port= 1/Queue= 0 ------- RX-packets: 164304068 TX-packets: 164304068 TX-dropped: 0 ------- Forward Stats for RX Port= 1/Queue= 0 -> TX Port= 0/Queue= 0 ------- RX-packets: 88472126 TX-packets: 88472126 TX-dropped: 0 ------- Forward Stats for RX Port= 0/Queue= 2 -> TX Port= 1/Queue= 2 ------- RX-packets: 164304001 TX-packets: 164304001 TX-dropped: 0 ------- Forward Stats for RX Port= 1/Queue= 2 -> TX Port= 0/Queue= 2 ------- RX-packets: 240136758 TX-packets: 240136758 TX-dropped: 0 ---------------------- Forward statistics for port 0 ---------------------- RX-packets: 328608069 RX-dropped: 0 RX-total: 328608069 TX-packets: 328608884 TX-dropped: 0 TX-total: 328608884 ---------------------------------------------------------------------------- ---------------------- Forward statistics for port 1 ---------------------- RX-packets: 328608884 RX-dropped: 0 RX-total: 328608884 TX-packets: 328608069 TX-dropped: 0 TX-total: 328608069 ---------------------------------------------------------------------------- +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++ RX-packets: 657216953 RX-dropped: 0 RX-total: 657216953 TX-packets: 657216953 TX-dropped: 0 TX-total: 657216953 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
I have repin all queues so that physical and virtual queues are in different cores, but same behaviour pmd thread numa_id 0 core_id 2: isolated : false port: dpdk0 queue-id: 2 (enabled) pmd usage: 0 % port: dpdk1 queue-id: 0 (enabled) pmd usage: 0 % port: vhu8cbade5e-57 queue-id: 5 (disabled) pmd usage: 0 % port: vhu90e5086e-e2 queue-id: 0 (enabled) pmd usage: 0 % port: vhu90e5086e-e2 queue-id: 3 (enabled) pmd usage: 0 % port: vhu90e5086e-e2 queue-id: 7 (enabled) pmd usage: 0 % port: vhub27eba49-8c queue-id: 3 (disabled) pmd usage: 0 % port: vhub27eba49-8c queue-id: 6 (disabled) pmd usage: 0 % pmd thread numa_id 1 core_id 3: isolated : true port: dpdk2 queue-id: 0 (enabled) pmd usage: 7 % port: dpdk3 queue-id: 0 (enabled) pmd usage: 7 % port: vhu8cbade5e-57 queue-id: 0 (enabled) pmd usage: 4 % port: vhub27eba49-8c queue-id: 0 (enabled) pmd usage: 5 % pmd thread numa_id 0 core_id 4: isolated : false port: dpdk0 queue-id: 1 (enabled) pmd usage: 0 % port: dpdk1 queue-id: 1 (enabled) pmd usage: 0 % port: dpdk1 queue-id: 2 (enabled) pmd usage: 0 % port: vhu8cbade5e-57 queue-id: 3 (disabled) pmd usage: 0 % port: vhu8cbade5e-57 queue-id: 4 (disabled) pmd usage: 0 % port: vhu90e5086e-e2 queue-id: 1 (enabled) pmd usage: 0 % port: vhub27eba49-8c queue-id: 4 (disabled) pmd usage: 0 % port: vhub27eba49-8c queue-id: 5 (disabled) pmd usage: 0 % pmd thread numa_id 1 core_id 5: isolated : true port: dpdk2 queue-id: 1 (enabled) pmd usage: 5 % port: dpdk3 queue-id: 1 (enabled) pmd usage: 6 % port: vhu8cbade5e-57 queue-id: 1 (enabled) pmd usage: 0 % port: vhub27eba49-8c queue-id: 1 (enabled) pmd usage: 0 % pmd thread numa_id 0 core_id 6: isolated : false port: dpdk0 queue-id: 0 (enabled) pmd usage: 0 % port: vhu8cbade5e-57 queue-id: 6 (disabled) pmd usage: 0 % port: vhu8cbade5e-57 queue-id: 7 (disabled) pmd usage: 0 % port: vhu90e5086e-e2 queue-id: 2 (enabled) pmd usage: 0 % port: vhu90e5086e-e2 queue-id: 4 (enabled) pmd usage: 0 % port: vhu90e5086e-e2 queue-id: 5 (enabled) pmd usage: 0 % port: vhu90e5086e-e2 queue-id: 6 (enabled) pmd usage: 0 % port: vhub27eba49-8c queue-id: 7 (disabled) pmd usage: 0 % pmd thread numa_id 1 core_id 7: isolated : true port: dpdk2 queue-id: 2 (enabled) pmd usage: 8 % port: dpdk3 queue-id: 2 (enabled) pmd usage: 9 % port: vhu8cbade5e-57 queue-id: 2 (enabled) pmd usage: 12 % port: vhub27eba49-8c queue-id: 2 (enabled) pmd usage: 12 %
I think there is some confusion about the operation as it's not very intuitive. Each PMD sends to a specific Tx Queue on each port. This is done to prevent locking on the datapath. For example, if there are 3 Tx queues for a VM, and there is traffic received on 2 PMDs that must be sent to that VM, then 2 Tx queues will be used. Inside the VM testpmd would receive this traffic on 2 RxQs. If this was not done then it implies that multiple cores could send to the same Tx queue on a port. That would mean locking would be needed on the datapath between the different PMD cores which may have a large performance penalty. However, in the case where the VM has less Tx Queues than there are PMD cores, some Tx queues have to be shared with locking. If it is required that a certain amount of Tx Queues have to have traffic, then the dpdk physical port rxqs should be pinned to different PMDs to ensure this. There is a patch submitted [1] that will also allow the option of using a hash to decide the Tx Queue and hence all Tx Queues used, but also at the trade off of having locking in the datapath. [1] https://mail.openvswitch.org/pipermail/ovs-dev/2021-November/389637.html
But I would say that this case is different than the situation you are describing. I have configured 3 PMDs, 3 physical queues and 3 virtual queues. At first, what I was told was that if I had 3 physical queues in 2 PMDS (for example, 2 in one PMD and 1 in other PMD), then I would only receive traffic in 2 virtual queues But in some situations, even having physical queues in 3 PMDS (1 physical queue in each PMD) there are times where I only receive traffic in 2 virtual queues. However, other times I have seen traffic in the 3 virtual queues. I have not been able to find out what is causing the sometimes I receive traffic in 3 queues while other times I only receive traffic in 2 queues. About cpu pinning, in this test https://bugzilla.redhat.com/show_bug.cgi?id=2028142#c4 I configured cpu pinning for all physical and virtual queues and even in this case, I didnt receive traffic in all virtual queues
Moving to the NFV DFG, please reach out if I'm wrong. Thanks!
Hi Kevin, Is it possible to get per queue's (Physical and Virtual) statistics in multi queue environment? Thanks
(In reply to Haresh Khandelwal from comment #8) > Hi Kevin, > > Is it possible to get per queue's (Physical and Virtual) statistics in multi > queue environment? > > Thanks Hi Haresh, David has submitted a patch for this and I believe there may also be a patch for virtual xstats in the works. https://mail.openvswitch.org/pipermail/ovs-dev/2021-December/389876.html If it's just to confirm that traffic is on those rx queues in OVS, you can get an indication from 'ovs-appctl dpif-netdev/pmd-rxq-show'. For virtual rx/tx, you can check testpmd in the guest with 'show fwd stats all'. I believe the issue reported here is related to the mapping of pmd cores->vhost tx queues, and the combination of pmds, pmds with traffic for the guest, total tx queues for the guest, disabled tx queues for the guest. It it is not related Auto load balance and at the moment I don't believe it's a bug, but I am running some testing to confirm the operation. Will update when I have confirmation. Kevin.
I've confirmed the operation locally. There are 6 pmds and 3 enabled txqs on the vhost interface. There is not enough txqs for each of the 6 PMDs to send to the vhost interface on a different txq, so they are shared. It means that 2 PMDs can send to 1 txq. That is likely what is happening in this case. e.g. 2 of 3 PMDs with traffic for the vhost interface sending to 1 txq. Another PMD sending to a different txq, and so traffic is received on 2/3 rxqs in testpmd. This can be confirmed by getting debug on the mappings from: ovs-appctl vlog/set dpif_netdev:dbg e.g. |DBG|static_tx_qid = 2 allocated for PMD thread on core 12, numa_id 0. ovs-appctl vlog/set netdev_dpdk:dbg e.g. |netdev_dpdk|DBG|TX queue mapping for port 'dpdkvhost0': 0 --> 0 1 --> 1 2 --> 2 Use of all txqs is not guaranteed in OVS as it depends on PMDs/PMDs with traffic/txqs/enabled txqs etc. That is not new and independent of auto load balance. There is a series submitted to allow use of all txqs by distributing by flow https://mail.openvswitch.org/pipermail/ovs-dev/2021-November/389637.html
@kevin, do you know how should i calculate number of PMD I need so that queues are not shared? Should we take this into account when deciding how many PMD cores are we going to configure? What i think is that if there are shared queues, those quese will need higher cpu usage as it is receiving traffic from 2 physical queues. Somehow there will be less granularity and there is more chance that we have pmd overloaded and have packet losses.
(In reply to Miguel Angel Nieto from comment #11) > @kevin, do you know how should i calculate number of PMD I need so that > queues are not shared? Should we take this into account when deciding how > many PMD cores are we going to configure? > If you want to guarantee that txqs are never shared by PMDs then you need, enabled txqs >= PMDs. So a combination of libvirt queues e.g. <driver name='vhost' queues='8' rx_queue_size='1024' tx_queue_size='1024'/> and testpmd config in guest e.g.--rxq=6 --txq=6
Just to note the documentation for this is now updated [1] (since Maxime added Hash mode tx-steering). Will close Bz as operating as expected. Thanks all. [1] https://docs.openvswitch.org/en/latest/topics/userspace-tx-steering/?highlight=tx-steering#thread-mode