| Summary: | dpdk's testpmd can not receive data with assigned VFs | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Pei Zhang <pezhang> |
| Component: | dpdk | Assignee: | Kevin Traynor <ktraynor> |
| Status: | CLOSED NOTABUG | QA Contact: | Jean-Tsung Hsiao <jhsiao> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 7.3 | CC: | ailan, alex.williamson, aloughla, atragler, chayang, fleitner, jean-mickael.guerin, juzhang, knoel, maxime.coquelin, michen, mleitner, sukulkar, thalabathy.venkatesan, wchoi, xfu, xiywang |
| Target Milestone: | rc | Keywords: | Extras |
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-06-12 10:14:37 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
(In reply to Pei Zhang from comment #0) > > (2) bind to vfio > # lspci -n -s 0000:00:02.0 > 00:02.0 0200: 8086:1515 (rev 01) ^^^^ This is an X540 VF > # echo 0000:00:02.0 > /sys/bus/pci/devices/0000\:00\:02.0/driver/unbind > # echo 0000:00:06.0 > /sys/bus/pci/devices/0000\:00\:06.0/driver/unbind > # echo "8086 1583" > /sys/bus/pci/drivers/vfio-pci/new_id > # echo "8086 1583" > /sys/bus/pci/drivers/vfio-pci/remove_id ^^^^ This is an XL710 PF ??? > # ls /sys/bus/pci/drivers/vfio-pci/ > 0000:00:02.0 0000:00:06.0 bind module new_id remove_id uevent unbind > > 5. In guest, start testpmd > # cat testpmd-ixgbe.sh > queues=1 > cores=2 > /root/dpdk-16.07/x86_64-native-linuxapp-gcc/build/app/test-pmd/testpmd -l > 0,1,2 -n 1 -d > /root/dpdk-16.07/x86_64-native-linuxapp-gcc/lib/librte_pmd_ixgbe.so \ Does librte_pmd_ixgbe.so support ixgbevf? I seem to find that this -d option is unnecessary. > -w 00:02.0 -w 00:06.0 \ > -- \ > --disable-hw-vlan -i \ > --nb-cores=${cores} \ > --disable-rss \ > --rxq=${queues} --txq=${queues} \ > --auto-start \ > --rxd=256 --txd=256 \ > > 6. In another host, start MoonGen as packets generator > # ./build/MoonGen examples/l2-load-latency.lua 0 1 150 > > 7. Check testpmd, no packets flow. > testpmd> show port stats all > > ######################## NIC statistics for port 0 > ######################## > RX-packets: 0 RX-missed: 0 RX-bytes: 0 > RX-errors: 0 > RX-nombuf: 0 > TX-packets: 0 TX-errors: 0 TX-bytes: 0 > > Throughput (since last show) > Rx-pps: 0 > Tx-pps: 0 > > ############################################################################ > > ######################## NIC statistics for port 1 > ######################## > RX-packets: 0 RX-missed: 0 RX-bytes: 0 > RX-errors: 0 > RX-nombuf: 0 > TX-packets: 0 TX-errors: 0 TX-bytes: 0 > > Throughput (since last show) > Rx-pps: 0 > Tx-pps: 0 > > ############################################################################ > > > Actual results: > testpmd can not receive data. > > Expected results: > testpmd should receive data. > > Additional info: > 1. With PFs, testpmd can receive data. > > 2. In guest, kernel driver ixgbevf and i40evf can work, #ping works. Perhaps dpdk drivers are simply broken. Who supports them? (In reply to Alex Williamson from comment #4) > (In reply to Pei Zhang from comment #0) > > > > (2) bind to vfio > > # lspci -n -s 0000:00:02.0 > > 00:02.0 0200: 8086:1515 (rev 01) > ^^^^ > This is an X540 VF > > > # echo 0000:00:02.0 > /sys/bus/pci/devices/0000\:00\:02.0/driver/unbind > > # echo 0000:00:06.0 > /sys/bus/pci/devices/0000\:00\:06.0/driver/unbind > > # echo "8086 1583" > /sys/bus/pci/drivers/vfio-pci/new_id > > # echo "8086 1583" > /sys/bus/pci/drivers/vfio-pci/remove_id > ^^^^ > This is an XL710 PF ??? Why don't you just unbind it from ixgbevf, and re-bind it in vfio-pci? The same should apply to the guest. > > # ls /sys/bus/pci/drivers/vfio-pci/ > > 0000:00:02.0 0000:00:06.0 bind module new_id remove_id uevent unbind > > > > 5. In guest, start testpmd > > # cat testpmd-ixgbe.sh > > queues=1 > > cores=2 > > /root/dpdk-16.07/x86_64-native-linuxapp-gcc/build/app/test-pmd/testpmd -l > > 0,1,2 -n 1 -d > > /root/dpdk-16.07/x86_64-native-linuxapp-gcc/lib/librte_pmd_ixgbe.so \ > > > Does librte_pmd_ixgbe.so support ixgbevf? I seem to find that this -d > option is unnecessary. Yes, the librte_pmd_ixgbe.so also support ixgbevf. I'll have a try with my HW, even if not exactly the same 10G reference. (In reply to Alex Williamson from comment #4) > (In reply to Pei Zhang from comment #0) > > > > (2) bind to vfio > > # lspci -n -s 0000:00:02.0 > > 00:02.0 0200: 8086:1515 (rev 01) > ^^^^ > This is an X540 VF > > > # echo 0000:00:02.0 > /sys/bus/pci/devices/0000\:00\:02.0/driver/unbind > > # echo 0000:00:06.0 > /sys/bus/pci/devices/0000\:00\:06.0/driver/unbind > > # echo "8086 1583" > /sys/bus/pci/drivers/vfio-pci/new_id > > # echo "8086 1583" > /sys/bus/pci/drivers/vfio-pci/remove_id > ^^^^ > This is an XL710 PF ??? > Sorry for confusing, it's typo. should be: # echo "8086 1515" > /sys/bus/pci/drivers/vfio-pci/new_id # echo "8086 1515" > /sys/bus/pci/drivers/vfio-pci/remove_id Best Regards, Pei Its working for me, and able to see the packets are landing on my guest through VF. Here VF will receive tagged packets and your test-pmd guest not enabled vlan strip. Hence packets would have dropped at VF level Could you please try below following command after binding NIC to PMD driver / dpdk set fwd mac_retry vlan set strip on 0 vlan set strip on 1 start <presuming you have only two ports connected to test-pmd guest. Regards/Thalabathy Thanks Thalabathy, it works! Summary key points: 1. When sending packets from MoonGen, the des mac of the packet should be set as MAC of VF. In other words, VF matters MAC address. Otherwise L2 network packets can not received by any VF. 2. Like Thalabathy mentioned in Comment 7 testpmd> set fwd mac retry Set mac packet forwarding mode with retry testpmd> vlan set strip on 0 testpmd> vlan set strip on 1 Results: Both VF assigned in guest can receive packets. testpmd> start mac packet forwarding with retry - ports=2 - cores=2 - streams=2 - NUMA support enabled, MP over anonymous pages disabled TX retry num: 64, delay between TX retries: 1us Logical Core 2 (socket 0) forwards packets on 1 streams: RX P=0/Q=0 (socket 0) -> TX P=1/Q=0 (socket 0) peer=02:00:00:00:00:01 Logical Core 3 (socket 0) forwards packets on 1 streams: RX P=1/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00 mac packet forwarding with retry - CRC stripping enabled - packets/burst=32 nb forwarding cores=2 - nb forwarding ports=2 RX queues=1 - RX desc=128 - RX free threshold=32 RX threshold registers: pthresh=8 hthresh=8 wthresh=0 TX queues=1 - TX desc=512 - TX free threshold=32 TX threshold registers: pthresh=32 hthresh=0 wthresh=0 TX RS bit threshold=32 - TXQ flags=0xf01 testpmd> show port stats all ######################## NIC statistics for port 0 ######################## RX-packets: 5379523 RX-missed: 0 RX-bytes: 322771380 RX-errors: 0 RX-nombuf: 0 TX-packets: 5379271 TX-errors: 0 TX-bytes: 322756260 Throughput (since last show) Rx-pps: 995257 Tx-pps: 995257 ############################################################################ ######################## NIC statistics for port 1 ######################## RX-packets: 5379518 RX-missed: 0 RX-bytes: 322771140 RX-errors: 0 RX-nombuf: 0 TX-packets: 5379321 TX-errors: 0 TX-bytes: 322759320 Throughput (since last show) Rx-pps: 995256 Tx-pps: 995257 ############################################################################ So close this bug as 'NOTABUG'. Thanks, Pei |
Description of problem: Assign VFs to guest, dpdk's testpmd can not receive data from packets generator. As VFs can not be bind to uio_pci_generic, so test with upstream dpdk which support vfio. Version-Release number of selected component (if applicable): Host: 3.10.0-507.rt56.412.el7.x86_64 qemu-kvm-rhev-2.6.0-22.el7.x86_64 Guest: 3.10.0-507.rt56.412.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1. Create 2 VFs each PF # echo 2 > /sys/bus/pci/devices/0000\:04\:00.0/sriov_numvfs # echo 2 > /sys/bus/pci/devices/0000\:04\:00.1/sriov_numvfs 2. Bind 2 VFs to vfio # ls /sys/bus/pci/drivers/vfio-pci/ 0000:04:10.2 0000:04:10.3 bind module new_id remove_id uevent unbind 3. Boot guest with 2 assigned VFs from these 2 PFs <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x04' slot='0x10' function='0x2'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x04' slot='0x10' function='0x3'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </hostdev> 4. In guest, bind them to vfio (1)load vfio # modprobe -r vfio # modprobe -r vfio_iommu_type1 # modprobe vfio enable_unsafe_noiommu_mode=Y # modprobe vfio-pci # cat /sys/module/vfio/parameters/enable_unsafe_noiommu_mode Y (2) bind to vfio # lspci -n -s 0000:00:02.0 00:02.0 0200: 8086:1515 (rev 01) # echo 0000:00:02.0 > /sys/bus/pci/devices/0000\:00\:02.0/driver/unbind # echo 0000:00:06.0 > /sys/bus/pci/devices/0000\:00\:06.0/driver/unbind # echo "8086 1583" > /sys/bus/pci/drivers/vfio-pci/new_id # echo "8086 1583" > /sys/bus/pci/drivers/vfio-pci/remove_id # ls /sys/bus/pci/drivers/vfio-pci/ 0000:00:02.0 0000:00:06.0 bind module new_id remove_id uevent unbind 5. In guest, start testpmd # cat testpmd-ixgbe.sh queues=1 cores=2 /root/dpdk-16.07/x86_64-native-linuxapp-gcc/build/app/test-pmd/testpmd -l 0,1,2 -n 1 -d /root/dpdk-16.07/x86_64-native-linuxapp-gcc/lib/librte_pmd_ixgbe.so \ -w 00:02.0 -w 00:06.0 \ -- \ --disable-hw-vlan -i \ --nb-cores=${cores} \ --disable-rss \ --rxq=${queues} --txq=${queues} \ --auto-start \ --rxd=256 --txd=256 \ 6. In another host, start MoonGen as packets generator # ./build/MoonGen examples/l2-load-latency.lua 0 1 150 7. Check testpmd, no packets flow. testpmd> show port stats all ######################## NIC statistics for port 0 ######################## RX-packets: 0 RX-missed: 0 RX-bytes: 0 RX-errors: 0 RX-nombuf: 0 TX-packets: 0 TX-errors: 0 TX-bytes: 0 Throughput (since last show) Rx-pps: 0 Tx-pps: 0 ############################################################################ ######################## NIC statistics for port 1 ######################## RX-packets: 0 RX-missed: 0 RX-bytes: 0 RX-errors: 0 RX-nombuf: 0 TX-packets: 0 TX-errors: 0 TX-bytes: 0 Throughput (since last show) Rx-pps: 0 Tx-pps: 0 ############################################################################ Actual results: testpmd can not receive data. Expected results: testpmd should receive data. Additional info: 1. With PFs, testpmd can receive data. 2. In guest, kernel driver ixgbevf and i40evf can work, #ping works. 3. X540-AT2/ixgbe and XL710/i40e hit same issue. 4. port info from testpmd (1) ixgbe testpmd> show port info 0 ********************* Infos for port 0 ********************* MAC address: 86:B3:04:E7:AB:17 Connect to socket: 0 memory allocation on the socket: 0 Link status: up Link speed: 10000 Mbps Link duplex: full-duplex Promiscuous mode: disabled Allmulticast mode: disabled Maximum number of MAC addresses: 128 Maximum number of MAC addresses of hash filtering: 4096 VLAN offload: strip off filter off qinq(extend) off No flow type is supported. Max possible RX queues: 4 Max possible number of RXDs per queue: 4096 Min possible number of RXDs per queue: 32 RXDs number alignment: 8 Max possible TX queues: 4 Max possible number of TXDs per queue: 4096 Min possible number of TXDs per queue: 32 TXDs number alignment: 8 (2) i40e testpmd> show port info 0 ********************* Infos for port 0 ********************* MAC address: 9A:D8:E0:41:30:72 Connect to socket: 0 memory allocation on the socket: 0 Link status: up Link speed: 40000 Mbps Link duplex: full-duplex Promiscuous mode: enabled Allmulticast mode: disabled Maximum number of MAC addresses: 64 Maximum number of MAC addresses of hash filtering: 0 VLAN offload: strip off filter off qinq(extend) off Hash key size in bytes: 52 Redirection table size: 64 Supported flow types: ipv4-frag ipv4-tcp ipv4-udp ipv4-sctp ipv4-other ipv6-frag ipv6-tcp ipv6-udp ipv6-sctp ipv6-other l2_payload Max possible RX queues: 4 Max possible number of RXDs per queue: 4096 Min possible number of RXDs per queue: 64 RXDs number alignment: 32 Max possible TX queues: 4 Max possible number of TXDs per queue: 4096 Min possible number of TXDs per queue: 64 TXDs number alignment: 32 5. VFs can not bind to uio_pci_generic(test with dpdk-2.2.0-3.el7.x86_64) # modprobe uio # modprobe uio_pci_generic # echo "8086 1515" > /sys/bus/pci/drivers/uio_pci_generic/new_id # echo "8086 1515" > /sys/bus/pci/drivers/uio_pci_generic/remove_id no devices show. # ls /sys/bus/pci/drivers/uio_pci_generic/ bind module new_id remove_id uevent unbind