Hi Li Ting,
Regarding PvP/ovs-dpdk low Mpps performance there is one thing that I can think of. It is the pmd assignments for the the ovs-dpdk bridge script and guest's xml file.
Does it match the following CPU layout:

NUMA node0 CPU(s):   0-19,40-59
NUMA node1 CPU(s):   20-39,60-79

Thanks!
Jean

I ran P2P/ovs-dpdk/1q using Trex traffic from netqe30. The Mpps rate is only 2.8 Mpps. That's terribly low. So, essentially, I reproduced the low performance case.

*** ovs-dpdk/1q script ***
ovs-vsctl set Open_vSwitch . other_config={}
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=0x00000000020000000002
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem="4096,4096"
ovs-vsctl --no-wait set Open_vSwitch . other_config:pmd-cpu-mask=0x00000c000000000c0000
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true

# config ovs-dpdk bridge with dpdk0, dpdk1, vhost0 and vhost1
ovs-vsctl --if-exists del-br ovsbr0
ovs-vsctl add-br ovsbr0 -- set bridge ovsbr0 datapath_type=netdev
ovs-vsctl add-port ovsbr0 dpdk-10 \
    -- set interface dpdk-10 type=dpdk ofport_request=10 options:dpdk-devargs=0000:1d:00.0 \
    options:n_rxq=1 options:n_rxq_desc=256 options:n_txq_desc=256
ovs-vsctl add-port ovsbr0 dpdk-11 \
    -- set interface dpdk-11 type=dpdk ofport_request=11 options:dpdk-devargs=0000:1d:00.2 \
    options:n_rxq=1 options:n_rxq_desc=256 options:n_txq_desc=256

ovs-ofctl del-flows ovsbr0
ovs-ofctl add-flow ovsbr0 in_port=10,actions=output:11
ovs-ofctl add-flow ovsbr0 in_port=11,actions=output:10
ovs-ofctl dump-flows ovsbr0

(In reply to Jean-Tsung Hsiao from comment #2)
> I ran P2P/ovs-dpdk/1q using Trex traffic from netqe30. The Mpps rate is only
> 2.8 Mpps. That's terribly low. So, essentially, I reproduced the low
> performance case.
> 
> *** ovs-dpdk/1q script ***
> ovs-vsctl set Open_vSwitch . other_config={}
> ovs-vsctl --no-wait set Open_vSwitch .
> other_config:dpdk-lcore-mask=0x00000000020000000002
> ovs-vsctl --no-wait set Open_vSwitch .
> other_config:dpdk-socket-mem="4096,4096"
> ovs-vsctl --no-wait set Open_vSwitch .
> other_config:pmd-cpu-mask=0x00000c000000000c0000
> ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
> 
> # config ovs-dpdk bridge with dpdk0, dpdk1, vhost0 and vhost1
> ovs-vsctl --if-exists del-br ovsbr0
> ovs-vsctl add-br ovsbr0 -- set bridge ovsbr0 datapath_type=netdev
> ovs-vsctl add-port ovsbr0 dpdk-10 \
>     -- set interface dpdk-10 type=dpdk ofport_request=10
> options:dpdk-devargs=0000:1d:00.0 \
>     options:n_rxq=1 options:n_rxq_desc=256 options:n_txq_desc=256
> ovs-vsctl add-port ovsbr0 dpdk-11 \
>     -- set interface dpdk-11 type=dpdk ofport_request=11
> options:dpdk-devargs=0000:1d:00.2 \
>     options:n_rxq=1 options:n_rxq_desc=256 options:n_txq_desc=256
> 
> ovs-ofctl del-flows ovsbr0
> ovs-ofctl add-flow ovsbr0 in_port=10,actions=output:11
> ovs-ofctl add-flow ovsbr0 in_port=11,actions=output:10
> ovs-ofctl dump-flows ovsbr0

Look at the following statistics for dpdk-10 interface --- traffic at 4 Mpps rate each way. The rx_missed_errors (about 0.013% ratio) has already passed the 0.002% drop rate limit set by the trafficgen binary search. So, this explains that why extremely low performance for the 25 Gb enic.

statistics          : {ovs_rx_qos_drops=0, ovs_tx_failure_drops=205, ovs_tx_invalid_hwol_drops=0, ovs_tx_mtu_exceeded_drops=0, ovs_tx_qos_drops=0, rx_bytes=3090889080, rx_dropped=7143, rx_errors=0, rx_mbuf_allocation_errors=0, rx_missed_errors=6773, rx_packets=51514818, tx_bytes=3090867060, tx_dropped=256, tx_errors=0, tx_packets=51514451}
status              : {driver_name=net_enic, if_descr="DPDK 21.11.2 net_enic", if_type="6", link_speed="25Gbps", max_hash_mac_addrs="0", max_mac_addrs="32", max_rx_pktlen="1518", max_rx_queues="2", max_tx_queues="1", max_vfs="0", max_vmdq_pools="0", min_rx_bufsize="68", numa_id="0", pci-device_id="0x43", pci-vendor_id="0x1137", port_no="0"}
type                : dpdk

For an experiment, set the error rate limit to 0.02% (10 times of 0.002%) the binary search reached to 24.2 Mpps.

Hi - as per https://bugzilla.redhat.com/show_bug.cgi?id=2119876#c6 don't set the number of Rx descriptors.

The default is 2048 and it may run out of Rx descriptors with a setting of 256 and there is a massive amount Rx dropped packets in the results. e.g. rx_dropped=1454005215.

Please check the ovs-vswitchd.log for errors and re-test without setting 'n_rxq_desc="256"'. Thanks.

(In reply to Kevin Traynor from comment #4)
> Hi - as per https://bugzilla.redhat.com/show_bug.cgi?id=2119876#c6 don't set
> the number of Rx descriptors.
> 
> The default is 2048 and it may run out of Rx descriptors with a setting of
> 256 and there is a massive amount Rx dropped packets in the results. e.g.
> rx_dropped=1454005215.
> 
> Please check the ovs-vswitchd.log for errors and re-test without setting
> 'n_rxq_desc="256"'. Thanks.

Hi Kevin,

I add dpdk port to ovs bridge failed when without setting 'n_rxq_desc="256"'. It seems to have a minimum of 64.

[root@netqe37 perf]# ovs-vsctl add-port ovsbr0 dpdk0 -- set interface dpdk0 type=dpdk options:dpdk-devargs=0000:1d:00.0 options:n_rxq=4 
ovs-vsctl: Error detected while setting up 'dpdk0': could not add network device dpdk0 to ofproto (Invalid argument).  See ovs-vswitchd log for details.
ovs-vsctl: The default log directory is "/var/log/openvswitch".

[root@netqe37 ~]# tail -f /var/log/openvswitch/ovs-vswitchd.log
2023-03-08T03:43:44.184Z|00299|dpdk|INFO|Device with port_id=0 already stopped
2023-03-08T03:43:44.186Z|00300|dpdk|INFO|PMD: rte_enic_pmd: MTU changed from 1500 to 1500
2023-03-08T03:43:44.186Z|00301|dpdk|ERR|Invalid value for nb_tx_desc(=2048), should be: <= 256, >= 64, and a product of 32
2023-03-08T03:43:44.186Z|00302|netdev_dpdk|INFO|Interface dpdk0 unable to setup txq(0): Invalid argument
2023-03-08T03:43:44.186Z|00303|netdev_dpdk|ERR|Interface dpdk0(rxq:2 txq:1 lsc interrupt mode:false) configure error: Invalid argument
2023-03-08T03:43:44.186Z|00304|dpif_netdev|ERR|Failed to set interface dpdk0 new configuration
2023-03-08T03:43:44.186Z|00305|dpif_netdev|INFO|Performing pmd to rx queue assignment using cycles algorithm.
2023-03-08T03:43:44.186Z|00306|dpif_netdev|INFO|Core 16 on numa node 0 assigned port 'vhost1' rx queue 0 (measured processing cycles 0).
2023-03-08T03:43:44.186Z|00307|dpif_netdev|INFO|Core 17 on numa node 0 assigned port 'vhost0' rx queue 0 (measured processing cycles 0).
2023-03-08T03:43:44.186Z|00308|dpif|WARN|netdev@ovs-netdev: failed to add dpdk0 as port: Invalid argument
2023-03-08T03:43:44.186Z|00309|bridge|WARN|could not add network device dpdk0 to ofproto (Invalid argument)
2023-03-08T03:43:44.186Z|00310|dpdk|INFO|Device with port_id=0 already stopped


thanks,
Li Ting

Hi Li Ting,

The number of descriptors can be set independently set for rxq and txq.

It must be set for *txq* as the device does not support the default in OVS (2048), which is why you are seeing the error message for txq's.

It should not be set for *rxq* as the OVS default of 2048 is a good default and probably 256 is too low, leading to rx drops.

For example, from add-port command in comment#2 above for dpdk-10, it should be:

ovs-vsctl add-port ovsbr0 dpdk-10 \
    -- set interface dpdk-10 type=dpdk ofport_request=10 options:dpdk-devargs=0000:1d:00.0 \
    options:n_rxq=1 options:n_txq_desc=256

thanks,
Kevin.

The maximum of rxq_desc is 512.

[root@netqe37 jhsiao]# ovs-vsctl show
9e965d14-1012-489c-bc57-4d6625c85163
    Bridge ovsbr0
        datapath_type: netdev
        Port dpdk-11
            Interface dpdk-11
                type: dpdk
                options: {dpdk-devargs="0000:1d:00.2", n_rxq="1", n_rxq_desc="512", n_txq_desc="256"}
        Port ovsbr0
            Interface ovsbr0
                type: internal
        Port dpdk-10
            Interface dpdk-10
                type: dpdk
                options: {dpdk-devargs="0000:1d:00.0", n_rxq="1", n_rxq_desc="512", n_txq_desc="256"}
    ovs_version: "2.17.6"
[root@netqe37 jhsiao]# 

But, the Mpps throughput rate is still 2.8 Mpps(2 ways) subject to drop limit=0.002%

The following dpdk-10 interface statistics indicated that the interface had suffered both rx_drops(0.008%) and tx_drops(0.003%) in this experiment.

*** So, the bug here is that n_txq_desc and n_rxq_desc both are too low --- need to allow 2028 as default. ***

statistics          : {ovs_rx_qos_drops=0, ovs_tx_failure_drops=94670, ovs_tx_invalid_hwol_drops=0, ovs_tx_mtu_exceeded_drops=0, ovs_tx_qos_drops=0, rx_bytes=162536209320, rx_dropped=225823, rx_errors=0, rx_mbuf_allocation_errors=0, rx_missed_errors=225823, rx_packets=2708936822, tx_bytes=162530573798, tx_dropped=94670, tx_errors=0, tx_packets=2708842888}
status              : {driver_name=net_enic, if_descr="DPDK 21.11.2 net_enic", if_type="6", link_speed="25Gbps", max_hash_mac_addrs="0", max_mac_addrs="32", max_rx_pktlen="1518", max_rx_queues="2", max_tx_queues="1", max_vfs="0", max_vmdq_pools="0", min_rx_bufsize="68", numa_id="0", pci-device_id="0x43", pci-vendor_id="0x1137", port_no="0"}
type                : dpdk

(In reply to Jean-Tsung Hsiao from comment #7)
> The maximum of rxq_desc is 512.
> 

How is that conclusion drawn, can you show the log where it says that?

> [root@netqe37 jhsiao]# ovs-vsctl show
> 9e965d14-1012-489c-bc57-4d6625c85163
>     Bridge ovsbr0
>         datapath_type: netdev
>         Port dpdk-11
>             Interface dpdk-11
>                 type: dpdk
>                 options: {dpdk-devargs="0000:1d:00.2", n_rxq="1",
> n_rxq_desc="512", n_txq_desc="256"}

Similarly, did it complain when n_rxq_desc was not set?

>         Port ovsbr0
>             Interface ovsbr0
>                 type: internal
>         Port dpdk-10
>             Interface dpdk-10
>                 type: dpdk
>                 options: {dpdk-devargs="0000:1d:00.0", n_rxq="1",
> n_rxq_desc="512", n_txq_desc="256"}
>     ovs_version: "2.17.6"
> [root@netqe37 jhsiao]# 
> 
> But, the Mpps throughput rate is still 2.8 Mpps(2 ways) subject to drop
> limit=0.002%
> 
> The following dpdk-10 interface statistics indicated that the interface had
> suffered both rx_drops(0.008%) and tx_drops(0.003%) in this experiment.
> 
> *** So, the bug here is that n_txq_desc and n_rxq_desc both are too low ---
> need to allow 2028 as default. ***
> 

For tx, that is above what the hardware reports and validates. We can't just increase it.

We can ask the maintainer if this is reported correctly and why it so low.

> statistics          : {ovs_rx_qos_drops=0, ovs_tx_failure_drops=94670,
> ovs_tx_invalid_hwol_drops=0, ovs_tx_mtu_exceeded_drops=0,
> ovs_tx_qos_drops=0, rx_bytes=162536209320, rx_dropped=225823, rx_errors=0,
> rx_mbuf_allocation_errors=0, rx_missed_errors=225823, rx_packets=2708936822,
> tx_bytes=162530573798, tx_dropped=94670, tx_errors=0, tx_packets=2708842888}
> status              : {driver_name=net_enic, if_descr="DPDK 21.11.2
> net_enic", if_type="6", link_speed="25Gbps", max_hash_mac_addrs="0",
> max_mac_addrs="32", max_rx_pktlen="1518", max_rx_queues="2",
> max_tx_queues="1", max_vfs="0", max_vmdq_pools="0", min_rx_bufsize="68",
> numa_id="0", pci-device_id="0x43", pci-vendor_id="0x1137", port_no="0"}
> type                : dpdk

When I run following command without n_rxq_desc, it still failed. Please have look. thanks
[root@netqe37 ~]# ovs-vsctl add-port ovsbr0 dpdk-10     -- set interface dpdk-10 type=dpdk ofport_request=10 options:dpdk-devargs=0000:1d:00.0     options:n_rxq=1 options:n_txq_desc=256  
ovs-vsctl: Error detected while setting up 'dpdk-10': could not add network device dpdk-10 to ofproto (Invalid argument).  See ovs-vswitchd log for details.
ovs-vsctl: The default log directory is "/var/log/openvswitch".

[root@netqe37 ~]# tail -f /var/log/openvswitch/ovs-vswitchd.log
2023-03-16T03:55:42.682Z|00400|dpif_netdev|INFO|PMD thread on numa_id: 0, core id: 58 created.
2023-03-16T03:55:42.683Z|00401|dpif_netdev|INFO|PMD thread on numa_id: 0, core id: 18 created.
2023-03-16T03:55:42.684Z|00402|dpif_netdev|INFO|PMD thread on numa_id: 0, core id: 19 created.
2023-03-16T03:55:42.685Z|00403|dpif_netdev|INFO|PMD thread on numa_id: 0, core id: 59 created.
2023-03-16T03:55:42.685Z|00404|dpif_netdev|INFO|There are 4 pmd threads on numa node 0
2023-03-16T03:55:42.685Z|00405|dpdk|INFO|Device with port_id=0 already stopped
2023-03-16T03:55:42.685Z|00001|dpdk(pmd-c59/id:31)|INFO|PMD thread uses DPDK lcore 0.
2023-03-16T03:55:42.685Z|00001|dpdk(pmd-c19/id:32)|INFO|PMD thread uses DPDK lcore 2.
2023-03-16T03:55:42.685Z|00001|dpdk(pmd-c18/id:33)|INFO|PMD thread uses DPDK lcore 3.
2023-03-16T03:55:42.685Z|00001|dpdk(pmd-c58/id:34)|INFO|PMD thread uses DPDK lcore 4.
2023-03-16T03:55:42.686Z|00406|dpdk|INFO|PMD: rte_enic_pmd: MTU changed from 1500 to 1500
2023-03-16T03:55:42.686Z|00407|dpdk|INFO|PMD: rte_enic_pmd: TX Queues - effective number of descs:256
2023-03-16T03:55:42.686Z|00408|dpdk|INFO|PMD: rte_enic_pmd: vNIC resources used:  wq 1 rq 2 cq 2 intr 1
2023-03-16T03:55:42.686Z|00409|dpdk|ERR|Invalid value for nb_rx_desc(=2048), should be: <= 512, >= 64, and a product of 32
2023-03-16T03:55:42.686Z|00410|netdev_dpdk|INFO|Interface dpdk-10 unable to setup rxq(0): Invalid argument
2023-03-16T03:55:42.686Z|00411|netdev_dpdk|ERR|Interface dpdk-10(rxq:1 txq:1 lsc interrupt mode:false) configure error: Invalid argument
2023-03-16T03:55:42.686Z|00412|dpif_netdev|ERR|Failed to set interface dpdk-10 new configuration
2023-03-16T03:55:42.686Z|00413|dpif|WARN|netdev@ovs-netdev: failed to add dpdk-10 as port: Invalid argument
2023-03-16T03:55:42.686Z|00414|bridge|WARN|Dropped 1 log messages in last 105 seconds (most recently, 105 seconds ago) due to excessive rate
2023-03-16T03:55:42.686Z|00415|bridge|WARN|could not add network device dpdk-10 to ofproto (Invalid argument)
2023-03-16T03:55:42.686Z|00416|dpdk|INFO|Device with port_id=0 already stopped

Ok, thanks that confirms that the Rx descriptors are also being limited by the device.

It is likely that this is the root cause for this performance issue, but we have another Bz for tracking the number of descriptors being limited by the device, so let's track that issue there.

When that is resolved we can confirm if this Bz is a symptom or some unrelated other issue.