Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be available on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1662177 - [Azure][DPDK]There are always too many packets in tx-drop queue in testpmd tx-side
Summary: [Azure][DPDK]There are always too many packets in tx-drop queue in testpmd tx...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: dpdk
Version: 7.6
Hardware: x86_64
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Mohammed Gamal
QA Contact: Yuxin Sun
URL:
Whiteboard:
Depends On:
Blocks: 1502856 1673430
TreeView+ depends on / blocked
 
Reported: 2018-12-26 18:47 UTC by Yuhui Jiang
Modified: 2020-05-28 15:25 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1673430 (view as bug list)
Environment:
Last Closed: 2020-03-17 10:31:20 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Yuhui Jiang 2018-12-26 18:47:11 UTC
Description of problem:
While we test dpdk on Azure with testpmd, there are too many packets in tx-drop queue.
                                                 

Version-Release number of selected component (if applicable):
DPDK-18.11

How reproducible:
100%

Steps to Reproduce:
1.Prepare 2 vms on Azure with dpdk installed
2.The 1st vm, start testpmd with txonly mode. The 2nd one start testpmd with rx-only mode.
  And set 1st vm's peer mac address as the 2nd vm's mac address. Conversely, set 2nd vm's
  peer mac address as the 1st one's mac address.
#TX-SIDE
#./testpmd -c 0xf -n 1 -w 0002:00:02.0 --vdev="net_vdev_netvsc0,iface=eth1,force=1" -- --port-topology=chained --nb-cores 1 --forward-mode=txonly --eth-peer=1,00:0d:3a:a1:4b:43 --stats-period 1
#RX-SIDE
#./testpmd -c 0xf -n 1 -w 0002:00:02.0 --vdev="net_vdev_netvsc0,iface=eth1,force=1" -- --port-topology=chained --nb-cores 1 --forward-mode=rxonly --eth-peer=1,00:0d:3a:a1:41:c7 --stats-period 1

3.Keep testpmd running up to 1 minute,then stop testpmd.

4.Check 2 vms' output

Actual results:
tx-side
++++++++ Accumulated forward statistics for all ports++++++++++
   RX-packets: 0              RX-dropped: 0             RX-total: 0
  TX-packets: 340381695     TX-dropped: 241822721  TX-total: 582204416
++++++++++++++++++++++++++++++++++++++++++++++
rx-side
++++++++ Accumulated forward statistics for all ports++++++
  RX-packets: 339493162    RX-dropped: 0     RX-total: 339493162
  TX-packets: 0              TX-dropped: 0             TX-total: 0
+++++++++++++++++++++++++++++++++++++++++++++


Expected results:
There shouldn't be too many(40%-50%) packets in TX-dropped queue

Additional info:

Comment 8 Mohammed Gamal 2020-02-05 18:01:21 UTC
Opened bug against upstream dpdk
https://bugs.dpdk.org/show_bug.cgi?id=390

Comment 9 Mohammed Gamal 2020-02-11 14:10:16 UTC
Can you retest with the latest upstream DPDK with both vdev_netvsc and netvsc drivers? 
Is there a way we can check the packet drop is caused by the network rather than the driver?

Comment 10 Yuxin Sun 2020-02-12 13:51:25 UTC
(In reply to Mohammed Gamal from comment #9)
> Can you retest with the latest upstream DPDK with both vdev_netvsc and
> netvsc drivers? 
> Is there a way we can check the packet drop is caused by the network rather
> than the driver?

Hi Mohammed,

I've tested with dpdk-19.11 upstream and all of mlx4/5 PMD, VDEV_NETVSC PMD and NETVSC PMD have many tx-drop. Maybe need MSFT to have a look whether there's any configuration issue or network issue:

RHEL-7.8(3.10.0-1126.el7.x86_64)
dpdk: upstream dpdk-19.11
vm size: Standard_DS3_v2
VF card:
tx: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] (rev 80)
rx: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
MLX4/5 PMD
tx:
# timeout 30 ./testpmd -c 0xf -n 1 -w b2d3:00:02.0 -- --port-topology=chained --nb-cores 1 --forward-mode=txonly --eth-peer=1,00:0d:3a:8b:29:32 --stats-period 1
  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
  RX-packets: 0              RX-dropped: 0             RX-total: 0
  TX-packets: 161304832      TX-dropped: 989533760     TX-total: 1150838592
  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

rx:
# timeout 40 ./testpmd -c 0xf -n 1 -w 0f6d:00:02.0 -- --port-topology=chained --nb-cores 1 --forward-mode=rxonly --eth-peer=1,00:0d:3a:98:fe:df --stats-period 1
  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
  RX-packets: 159708201      RX-dropped: 0             RX-total: 159708201
  TX-packets: 0              TX-dropped: 0             TX-total: 0
  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
VDEV_NETVSC PMD
tx:
# timeout 30 ./testpmd -c 0xf -n 1 -w b2d3:00:02.0 --vdev="net_vdev_netvsc0,iface=eth1,force=1" -- --port-topology=chained --nb-cores 1 --forward-mode=txonly --eth-peer=1,00:0d:3a:8b:29:32 --stats-period 1
  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
  RX-packets: 0              RX-dropped: 0             RX-total: 0
  TX-packets: 161547040      TX-dropped: 953327680     TX-total: 1114874720
  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

rx:
# timeout 40 ./testpmd -c 0xf -n 1 -w 4dcd:00:02.0 --vdev="net_vdev_netvsc0,iface=eth1,force=1" -- --port-topology=chained --nb-cores 1 --forward-mode=rxonly --eth-peer=1,00:0d:3a:98:fe:df --stats-period 1
  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
  RX-packets: 159557769      RX-dropped: 0             RX-total: 159557769
  TX-packets: 0              TX-dropped: 0             TX-total: 0
  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
NETVSC PMD
setup:
DEV_UUID=$(basename $(readlink /sys/class/net/eth1/device))
NET_UUID="f8615163-df3e-46c5-913f-f2d2f965ed0e"
modprobe uio_hv_generic
echo $NET_UUID > /sys/bus/vmbus/drivers/uio_hv_generic/new_id
echo $DEV_UUID > /sys/bus/vmbus/drivers/hv_netvsc/unbind
echo $DEV_UUID > /sys/bus/vmbus/drivers/uio_hv_generic/bind

tx:
# timeout 30 ./testpmd -c 0xf -n 1 -w b2d3:00:02.0 -- --port-topology=chained --nb-cores 1 --forward-mode=txonly --eth-peer=1,00:0d:3a:8b:29:32 --stats-period 1
  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
  RX-packets: 0              RX-dropped: 0             RX-total: 0
  TX-packets: 161200544      TX-dropped: 961872800     TX-total: 1123073344
  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

rx:
# timeout 40 ./testpmd -c 0xf -n 1 -w 0f6d:00:02.0 -- --port-topology=chained --nb-cores 1 --forward-mode=rxonly --eth-peer=1,00:0d:3a:98:fe:df --stats-period 1
  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
  RX-packets: 159177452      RX-dropped: 0             RX-total: 159177452
  TX-packets: 0              TX-dropped: 0             TX-total: 0
  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Comment 11 Mohammed Gamal 2020-02-12 18:20:11 UTC
(In reply to Yuxin Sun from comment #10)
> (In reply to Mohammed Gamal from comment #9)
> > Can you retest with the latest upstream DPDK with both vdev_netvsc and
> > netvsc drivers? 
> > Is there a way we can check the packet drop is caused by the network rather
> > than the driver?
> 
> Hi Mohammed,
> 
> I've tested with dpdk-19.11 upstream and all of mlx4/5 PMD, VDEV_NETVSC PMD
> and NETVSC PMD have many tx-drop. Maybe need MSFT to have a look whether
> there's any configuration issue or network issue:
> 
> RHEL-7.8(3.10.0-1126.el7.x86_64)
> dpdk: upstream dpdk-19.11
> vm size: Standard_DS3_v2
> VF card:
> tx: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function]
> (rev 80)
> rx: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro
> Virtual Function]
> MLX4/5 PMD
> tx:
> # timeout 30 ./testpmd -c 0xf -n 1 -w b2d3:00:02.0 --
> --port-topology=chained --nb-cores 1 --forward-mode=txonly
> --eth-peer=1,00:0d:3a:8b:29:32 --stats-period 1
>   +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
>   RX-packets: 0              RX-dropped: 0             RX-total: 0
>   TX-packets: 161304832      TX-dropped: 989533760     TX-total: 1150838592
>  
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> 
> rx:
> # timeout 40 ./testpmd -c 0xf -n 1 -w 0f6d:00:02.0 --
> --port-topology=chained --nb-cores 1 --forward-mode=rxonly
> --eth-peer=1,00:0d:3a:98:fe:df --stats-period 1
>   +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
>   RX-packets: 159708201      RX-dropped: 0             RX-total: 159708201
>   TX-packets: 0              TX-dropped: 0             TX-total: 0
>  
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> VDEV_NETVSC PMD
> tx:
> # timeout 30 ./testpmd -c 0xf -n 1 -w b2d3:00:02.0
> --vdev="net_vdev_netvsc0,iface=eth1,force=1" -- --port-topology=chained
> --nb-cores 1 --forward-mode=txonly --eth-peer=1,00:0d:3a:8b:29:32
> --stats-period 1
>   +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
>   RX-packets: 0              RX-dropped: 0             RX-total: 0
>   TX-packets: 161547040      TX-dropped: 953327680     TX-total: 1114874720
>  
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> 
> rx:
> # timeout 40 ./testpmd -c 0xf -n 1 -w 4dcd:00:02.0
> --vdev="net_vdev_netvsc0,iface=eth1,force=1" -- --port-topology=chained
> --nb-cores 1 --forward-mode=rxonly --eth-peer=1,00:0d:3a:98:fe:df
> --stats-period 1
>   +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
>   RX-packets: 159557769      RX-dropped: 0             RX-total: 159557769
>   TX-packets: 0              TX-dropped: 0             TX-total: 0
>  
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> NETVSC PMD
> setup:
> DEV_UUID=$(basename $(readlink /sys/class/net/eth1/device))
> NET_UUID="f8615163-df3e-46c5-913f-f2d2f965ed0e"
> modprobe uio_hv_generic
> echo $NET_UUID > /sys/bus/vmbus/drivers/uio_hv_generic/new_id
> echo $DEV_UUID > /sys/bus/vmbus/drivers/hv_netvsc/unbind
> echo $DEV_UUID > /sys/bus/vmbus/drivers/uio_hv_generic/bind
> 
> tx:
> # timeout 30 ./testpmd -c 0xf -n 1 -w b2d3:00:02.0 --
> --port-topology=chained --nb-cores 1 --forward-mode=txonly
> --eth-peer=1,00:0d:3a:8b:29:32 --stats-period 1
>   +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
>   RX-packets: 0              RX-dropped: 0             RX-total: 0
>   TX-packets: 161200544      TX-dropped: 961872800     TX-total: 1123073344
>  
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> 
> rx:
> # timeout 40 ./testpmd -c 0xf -n 1 -w 0f6d:00:02.0 --
> --port-topology=chained --nb-cores 1 --forward-mode=rxonly
> --eth-peer=1,00:0d:3a:98:fe:df --stats-period 1
>   +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
>   RX-packets: 159177452      RX-dropped: 0             RX-total: 159177452
>   TX-packets: 0              TX-dropped: 0             TX-total: 0
>  
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

I've tried the netvsc driver on a local hyper-v setup and got similar results. I tried to find a way to do rate limiting, but the netvsc driver doesn't currently support dpdk rate limiting.

Comment 13 Mohammed Gamal 2020-03-16 19:05:05 UTC
Since this is an upstream (i.e. not Red Hat specific) issue and older than 11 months. I do consider closing it as WONTFIX. If there is no response from upsteam this month, I prefer if we close it after this month!

Comment 14 Mohammed Gamal 2020-03-17 10:31:20 UTC
Stephen Hemminger from Microsoft doesn't believe this is a bug, but rather caused by the fact that testpmd has no flow control and that the sending speed outpaces the driver's capacity. Upstream bug was closed as WONTFIX.
Closing here as NOTABUG


Note You need to log in before you can comment on or make changes to this bug.