Bug 2186755
| Summary: | VDPA: Not all ports were started when testpmd start with mq inside guest | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | liting <tli> |
| Component: | qemu-kvm | Assignee: | Virtualization Maintenance <virt-maint> |
| qemu-kvm sub component: | Networking | QA Contact: | Lei Yang <leiyang> |
| Status: | CLOSED MIGRATED | Docs Contact: | |
| Severity: | unspecified | ||
| Priority: | unspecified | CC: | chayang, coli, jinzhao, juzhang, maxime.coquelin, virt-maint, yama, zhguo |
| Version: | 9.2 | ||
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-04-18 07:40:32 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
(In reply to liting from comment #0) > vdpa dev add name vdpa0 mgmtdev pci/0000:07:00.2 mac 52:54:00:11:8f:ea > max_vqp 8 > <interface type='vdpa'> > <mac address='52:54:00:11:8f:ea'/> > <source dev='/dev/vhost-vdpa-0'/> > <model type='virtio'/> > <driver name='vhost' queues='8'/> > <address type='pci' domain='0x0000' bus='0x03' slot='0x00' > function='0x0'/> > </interface> > > [root@localhost ~]# ethtool -l eth1 > Channel parameters for eth1: > Pre-set maximums: > RX: n/a > TX: n/a > Other: n/a > Combined: 8 > Current hardware settings: > RX: n/a > TX: n/a > Other: n/a > Combined: 3 > [root@localhost ~]# ethtool -L eth1 combined 4 > netlink error: Invalid argument Hmm, in the log above, 3 queue pairs are already configured and up to 8 are available by the device. Maybe in this case it fails because you cannot set more queues than the number of vCPUs. But this is different with what you shared with me on Slack, where it looked like multiqueue was not negotiated: [root@localhost ~]# ethtool -l eth1 Channel parameters for eth1: Pre-set maximums: RX: n/a TX: n/a Other: n/a Combined: 1 Current hardware settings: RX: n/a TX: n/a Other: n/a Combined: 1 [root@localhost ~]# ethtool -L eth1 combined 4 netlink error: requested channel count exceeds maximum (offset 36) netlink error: Invalid argument (In reply to Maxime Coquelin from comment #1) > (In reply to liting from comment #0) > > > vdpa dev add name vdpa0 mgmtdev pci/0000:07:00.2 mac 52:54:00:11:8f:ea > > max_vqp 8 > > > <interface type='vdpa'> > > <mac address='52:54:00:11:8f:ea'/> > > <source dev='/dev/vhost-vdpa-0'/> > > <model type='virtio'/> > > <driver name='vhost' queues='8'/> > > <address type='pci' domain='0x0000' bus='0x03' slot='0x00' > > function='0x0'/> > > </interface> > > > > > [root@localhost ~]# ethtool -l eth1 > > Channel parameters for eth1: > > Pre-set maximums: > > RX: n/a > > TX: n/a > > Other: n/a > > Combined: 8 > > Current hardware settings: > > RX: n/a > > TX: n/a > > Other: n/a > > Combined: 3 > > [root@localhost ~]# ethtool -L eth1 combined 4 > > netlink error: Invalid argument > > Hmm, in the log above, 3 queue pairs are already configured > and up to 8 are available by the device. Maybe in this case > it fails because you cannot set more queues than the number > of vCPUs. But this is different with what you shared with me > on Slack, where it looked like multiqueue was not negotiated: > > [root@localhost ~]# ethtool -l eth1 > Channel parameters for eth1: > Pre-set maximums: > RX: n/a > TX: n/a > Other: n/a > Combined: 1 > Current hardware settings: > RX: n/a > TX: n/a > Other: n/a > Combined: 1 > [root@localhost ~]# ethtool -L eth1 combined 4 > netlink error: requested channel count exceeds maximum (offset 36) > netlink error: Invalid argument Sorry, I shared with you following result on Slack, it because the vdpa xml is <driver name='vhost'/> not <driver name='vhost' queues='8'/>, when I changed to <driver name='vhost' queues='8'/>, the current hardware settings change from "Combined: 1" to "Combined: 3". [root@localhost ~]# ethtool -l eth1 Channel parameters for eth1: Pre-set maximums: RX: n/a TX: n/a Other: n/a Combined: 1 Current hardware settings: RX: n/a TX: n/a Other: n/a Combined: 1 [root@localhost ~]# ethtool -L eth1 combined 4 netlink error: requested channel count exceeds maximum (offset 36) netlink error: Invalid argument If I boot guest with virtio-net-pci not vdpa device, then queue can be configured. So change the QA contact to leiyang since it is not dpdk related. qemu-kvm-6.2.0-11.module+el8.6.0+18167+43cf40f3.8.x86_64 -device virtio-net-pci,mac=9a:c7:a8:88:a4:a5,id=idp9fB7b,mq=on,netdev=idLyA3Zk,bus=pcie-root-port-4,addr=0x0 \ -netdev tap,id=idLyA3Zk,vhost=on,queues=8 \ [root@vm-210-192 ~]# ethtool -l eth0 Channel parameters for eth0: Pre-set maximums: RX: n/a TX: n/a Other: n/a Combined: 8 Current hardware settings: RX: n/a TX: n/a Other: n/a Combined: 8 [root@vm-210-192 ~]# ethtool -L eth0 combined 4 [root@vm-210-192 ~]# ethtool -l eth0 Channel parameters for eth0: Pre-set maximums: RX: n/a TX: n/a Other: n/a Combined: 8 Current hardware settings: RX: n/a TX: n/a Other: n/a Combined: 4 [root@vm-210-192 ~]# Hello All After testing, I think this is a duplicate issues of https://issues.redhat.com/browse/RHEL-357. It is not a qemu or dpdk issues,it belong to the kernel regression issues, which was introduced by kernel-5.14.0-278.el9.x86_64. Our develop are working on this. If you test it with libvirt, this message "qemu-kvm: Insufficient written data (0)" should be include /var/log/libvirt/qemu/*.log. Thanks Lei Hello Ting Could you please help check you test log whether include like this messages "qemu-kvm: Insufficient written data (0) in /var/log/libvirt/qemu/*.log ? If yes, I will closed the current bug. Thanks in advance. In additional, since in February, the bugs of vpda and network have been tracked by jira, so you may not be able to check whether the current bug exists through bugzilla. Just a suggestion, if you hit some issues about vdpa again, maybe we can discuss this issues via Slack(My Slack name: Lei Yang) first to avoid dup work. Best Regards Lei (In reply to Lei Yang from comment #5) > Hello Ting > > Could you please help check you test log whether include like this messages > "qemu-kvm: Insufficient written data (0) in /var/log/libvirt/qemu/*.log ? If > yes, I will closed the current bug. Thanks in advance. > In additional, since in February, the bugs of vpda and network have been > tracked by jira, so you may not be able to check whether the current bug > exists through bugzilla. Just a suggestion, if you hit some issues about > vdpa again, maybe we can discuss this issues via Slack(My Slack name: Lei > Yang) first to avoid dup work. > > Best Regards > Lei OK. thanks. I checked the /var/log/libvirt/qemu/gg.log, it indeed has followng log. 2023-04-18T06:52:38.186676Z qemu-kvm: Insufficient written data (0) (In reply to liting from comment #6) > (In reply to Lei Yang from comment #5) > > Hello Ting > > > > Could you please help check you test log whether include like this messages > > "qemu-kvm: Insufficient written data (0) in /var/log/libvirt/qemu/*.log ? If > > yes, I will closed the current bug. Thanks in advance. > > In additional, since in February, the bugs of vpda and network have been > > tracked by jira, so you may not be able to check whether the current bug > > exists through bugzilla. Just a suggestion, if you hit some issues about > > vdpa again, maybe we can discuss this issues via Slack(My Slack name: Lei > > Yang) first to avoid dup work. > > > > Best Regards > > Lei > > OK. thanks. I checked the /var/log/libvirt/qemu/gg.log, it indeed has > followng log. > 2023-04-18T06:52:38.186676Z qemu-kvm: Insufficient written data (0) Thanks for your help checked it. So based on the above result move to "MIGRATE", because this is a duplicate of https://issues.redhat.com/browse/RHEL-357. Thanks Lei |
Description of problem: VDPA: Not all ports were started when testpmd start with mq inside guest Version-Release number of selected component (if applicable): [root@dell-per730-56 ~]# uname -r 5.14.0-284.8.1.el9_2.x86_64 [root@dell-per730-56 ~]# rpm -qa|grep dpdk dpdk-22.11-1.el9.x86_64 dpdk-devel-22.11-1.el9.x86_64 dpdk-examples-22.11-1.el9.x86_64 dpdk-tools-22.11-1.el9.x86_64 [root@dell-per730-56 ~]# ethtool -i enp7s0f0np0 driver: mlx5_core version: 5.14.0-284.8.1.el9_2.x86_64 firmware-version: 22.31.1014 (MT_0000000430) expansion-rom-version: bus-info: 0000:07:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: no supports-register-dump: no supports-priv-flags: yes How reproducible: Steps to Reproduce: 1. create vhost_vdpa port ip link set enp7s0f0np0 up modprobe vdpa modprobe vhost_vdpa modprobe mlx5_vdpa echo 1 > /sys/devices/pci0000:00/0000:00:03.2/0000:07:00.0/sriov_numvfs ip li set enp7s0f0np0 vf 0 spoofchk off trust on ethtool -A enp7s0f0np0 rx off tx off echo 0000:07:00.2 >/sys/bus/pci/drivers/mlx5_core/unbind devlink dev eswitch set pci/0000:07:00.0 mode switchdev echo 0000:07:00.2 >/sys/bus/pci/drivers/mlx5_core/bind vdpa dev add name vdpa0 mgmtdev pci/0000:07:00.2 mac 52:54:00:11:8f:ea max_vqp 8 ovs-vsctl set Open_vSwitch . other_config:hw-offload=true ovs-vsctl add-br ovsbr0 ovs-vsctl add-port ovsbr0 enp7s0f0np0 ovs-vsctl add-port ovsbr0 enp7s0f0v0 ethtool -K enp7s0f0np0 hw-tc-offload on ethtool -K enp7s0f0v0 hw-tc-offload on [root@dell-per730-56 ~]# ls /dev|grep vdpa vhost-vdpa-0 [root@dell-per730-56 ~]# vdpa dev show vdpa0: type network mgmtdev pci/0000:07:00.2 vendor_id 5555 max_vqs 17 max_vq_size 256 2. start guest with following port xml <interface type='bridge'> <mac address='52:54:00:bb:63:7b'/> <source bridge='virbr0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </interface> <interface type='vdpa'> <mac address='52:54:00:11:8f:ea'/> <source dev='/dev/vhost-vdpa-0'/> <model type='virtio'/> <driver name='vhost' queues='8'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </interface> 3. bind the vdpa port inside guest to vfio-pci [root@localhost ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 52:54:00:bb:63:7b brd ff:ff:ff:ff:ff:ff inet 192.168.122.138/24 brd 192.168.122.255 scope global dynamic noprefixroute enp2s0 valid_lft 3590sec preferred_lft 3590sec inet6 fe80::785c:b7aa:ef67:920a/64 scope link noprefixroute valid_lft forever preferred_lft forever 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 52:54:00:11:8f:ea brd ff:ff:ff:ff:ff:ff altname enp3s0 inet6 fe80::c8b8:859e:d2bf:8141/64 scope link noprefixroute valid_lft forever preferred_lft forever [root@localhost ~]# ethtool -i enp2s0 driver: virtio_net version: 1.0.0 firmware-version: expansion-rom-version: bus-info: 0000:02:00.0 supports-statistics: yes supports-test: no supports-eeprom-access: no supports-register-dump: no supports-priv-flags: no [root@localhost ~]# ethtool -i eth1 driver: virtio_net version: 1.0.0 firmware-version: expansion-rom-version: bus-info: 0000:03:00.0 supports-statistics: yes supports-test: no supports-eeprom-access: no supports-register-dump: no supports-priv-flags: no [root@localhost ~]# driverctl -v set-override 0000:03:00.0 vfio-pci driverctl: setting driver override for 0000:03:00.0: vfio-pci driverctl: loading driver vfio-pci driverctl: unbinding previous driver virtio-pci driverctl: reprobing driver for 0000:03:00.0 driverctl: saving driver override for 0000:03:00.0 4. start guest with 1q: dpdk-testpmd -l 0,1 --socket-mem 1024 --legacy-mem -n 4 -- --burst=64 --forward-mode=txonly --port-topology=chained --disable-rss -i --rxq=1 --txq=1 --rxd=4096 --txd=4096 --nb-cores=1 --auto-start EAL: Detected CPU lcores: 3-start EAL: Detected NUMA nodes: 1 EAL: Detected shared linkage of DPDK EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'PA' EAL: VFIO support initialized EAL: Probe PCI driver: net_virtio (1af4:1041) device: 0000:02:00.0 (socket -1) eth_virtio_pci_init(): Failed to init PCI device EAL: Requested device 0000:02:00.0 cannot be used EAL: Probe PCI driver: net_virtio (1af4:1041) device: 0000:03:00.0 (socket -1) EAL: Using IOMMU type 8 (No-IOMMU) TELEMETRY: No legacy callbacks, legacy socket not created Set txonly packet forwarding mode Interactive-mode selected Auto-start selected Warning: NUMA should be configured manually by using --port-numa-config and --ring-numa-config parameters along with --numa. testpmd: create a new mbuf pool <mb_pool_0>: n=155456, size=2176, socket=0 testpmd: preferred mempool ops selected: ring_mp_mc Configuring Port 0 (socket 0) EAL: Error disabling MSI-X interrupts for fd 20 Port 0: 52:54:00:11:8F:EA Checking link statuses... Done Error during enabling promiscuous mode for port 0: Operation not supported - ignore Start automatic packet forwarding txonly packet forwarding - ports=1 - cores=1 - streams=1 - NUMA support enabled, MP allocation mode: native Logical Core 1 (socket 0) forwards packets on 1 streams: RX P=0/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00 txonly packet forwarding packets/burst=64 packet len=64 - nb packet segments=1 nb forwarding cores=1 - nb forwarding ports=1 port 0: RX queue number: 1 Tx queue number: 1 Rx offloads=0x0 Tx offloads=0x0 RX queue: 0 RX desc=4096 - RX free threshold=0 RX threshold registers: pthresh=0 hthresh=0 wthresh=0 RX Offloads=0x0 TX queue: 0 TX desc=4096 - TX free threshold=0 TX threshold registers: pthresh=0 hthresh=0 wthresh=0 TX offloads=0x0 - TX RS bit threshold=0 testpmd> testpmd> quit Telling cores to stop... Waiting for lcores to finish... ---------------------- Forward statistics for port 0 ---------------------- RX-packets: 0 RX-dropped: 0 RX-total: 0 TX-packets: 184288638 TX-dropped: 1806610818 TX-total: 1990899456 ---------------------------------------------------------------------------- +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++ RX-packets: 0 RX-dropped: 0 RX-total: 0 TX-packets: 184288638 TX-dropped: 1806610818 TX-total: 1990899456 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 5. start guest with 2q: dpdk-testpmd -l 0,1,2 --socket-mem 1024 --legacy-mem -n 4 -- --burst=64 --forward-mode=txonly --port-topology=chained --disable-rss -i --rxq=2 --txq=2 --rxd=4096 --txd=4096 --nb-cores=2 --auto-start Actual results: testpmd start with 1q successfully testpmd start with 2q/4q failed Step 5 run failed, following is output. EAL: Detected CPU lcores: 3 EAL: Detected NUMA nodes: 1 EAL: Detected shared linkage of DPDK EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'PA' EAL: VFIO support initialized EAL: Probe PCI driver: net_virtio (1af4:1041) device: 0000:02:00.0 (socket -1) eth_virtio_pci_init(): Failed to init PCI device EAL: Requested device 0000:02:00.0 cannot be used EAL: Probe PCI driver: net_virtio (1af4:1041) device: 0000:03:00.0 (socket -1) EAL: Using IOMMU type 8 (No-IOMMU) TELEMETRY: No legacy callbacks, legacy socket not created Set txonly packet forwarding mode Interactive-mode selected Auto-start selected Warning: NUMA should be configured manually by using --port-numa-config and --ring-numa-config parameters along with --numa. testpmd: create a new mbuf pool <mb_pool_0>: n=163456, size=2176, socket=0 testpmd: preferred mempool ops selected: ring_mp_mc Configuring Port 0 (socket 0) EAL: Error disabling MSI-X interrupts for fd 24 virtio_set_multiple_queues_auto(): Multiqueue configured but send command failed, this is too late now... Fail to start port 0: Invalid argument Please stop the ports first Done Error during enabling promiscuous mode for port 0: Operation not supported - ignore Start automatic packet forwarding Not all ports were started testpmd> stop Packet forwarding not started testpmd> start Not all ports were started testpmd> show config fwd txonly packet forwarding - ports=1 - cores=2 - streams=2 - NUMA support enabled, MP allocation mode: native Logical Core 1 (socket 0) forwards packets on 1 streams: RX P=0/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00 Logical Core 2 (socket 0) forwards packets on 1 streams: RX P=0/Q=1 (socket 0) -> TX P=0/Q=1 (socket 0) peer=02:00:00:00:00:00 Maxime sugguest to try to use ethtool to configure queue of the vdpa port. It also failed. So it may be the qemu issue not dpdk issue. [root@localhost ~]# ethtool -l eth1 Channel parameters for eth1: Pre-set maximums: RX: n/a TX: n/a Other: n/a Combined: 8 Current hardware settings: RX: n/a TX: n/a Other: n/a Combined: 3 [root@localhost ~]# ethtool -L eth1 combined 4 netlink error: Invalid argument Expected results: Testpmd start with mq successfully inside guest. Additional info: