Hide Forgot
Created attachment 1141627 [details] start-ovsdpdk.sh Description of problem: Network does not work if use vhost-user with multi-queue in vm-to-vm model (two guests in one host). Version-Release number of selected component (if applicable): guest kernel: 3.10.0-364.rt56.241.el7.x86_64 host: 3.10.0-364.rt56.241.el7.x86_64 qemu-kvm-rhev-2.5.0-3.el7.x86_64 openvswitch-dpdk-2.5.0-3.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1. mount hugetlbfs: mount -t hugetlbfs hugetlbfs /mnt/hugetlbfs/ -o size=16G 2. set up ovs-dpdk vhost-user environment with start-ovsdpdk.sh, using "vm-to-vm" model 3. enable mq: ovs-vsctl set Interface vhost-user1 options:n_rxq=$queues_nr ovs-vsctl set Interface vhost-user2 options:n_rxq=$queues_nr ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=$cpu_mask 4. boot two guests in one host, one of the guest use vhost-user1 and vhost-user2, the other one use vhost-user3 and vhost-user 4: /usr/libexec/qemu-kvm -name rhel7.2-rt-355-guest1 -machine pc-i440fx-rhel7.2.0 \ -cpu host,+tsc-deadline,pmu=off \ -m 4096 -realtime mlock=off -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/mnt/hugetlbfs,share=on,size=4096M -numa node,nodeid=0,cpus=0-3,memdev=ram-node0 \ -smp 4 \ -drive file=/home/rhel7.2-rt-355.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,snapshot=on -device virtio-blk-pci,drive=drive-virtio-disk0,id=virtio-disk0 \ -netdev tap,id=hostnet0,vhost=on -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:a1:d0:51 \ -chardev socket,id=charnet1,path=/var/run/openvswitch/vhost-user1 -netdev type=vhost-user,id=hostnet1,chardev=charnet1,queues=4 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:b2:0a:01,mq=on,vectors=10 \ -chardev socket,id=charnet2,path=/var/run/openvswitch/vhost-user2 -netdev type=vhost-user,id=hostnet2,chardev=charnet2,queues=4 -device virtio-net-pci,netdev=hostnet2,id=net2,mac=52:54:00:b2:0a:02,mq=on,vectors=10 \ -monitor stdio -device qxl-vga,id=video0 -serial unix:/tmp/console1,server,nowait -vnc :1 -spice port=5900,disable-ticketing 5. set ip for guest1 and guest2 vhost-user interface: guest1# ifconfig eth1 192.168.10.11/24 guest2# ifconfig eth1 192.168.10.12/24 6. set mq in both guests: # ethtool -L eth1 combined 4 # ethtool -l eth1 Channel parameters for eth1: Pre-set maximums: RX: 0 TX: 0 Other: 0 Combined: 4 Current hardware settings: RX: 0 TX: 0 Other: 0 Combined: 4 7. ping from guest1 to guest2, failed. Actual results: ping failed Expected results: ping and netperf should work normally Additional info: 1. tested the same case only without mq, network works fine. Are we going to support multi-queue in the future? If do, about when will this happen? And if not, I think maybe it's better that we display some kind of warnings to tell user that we're not support mq.
> 3. enable mq: > ovs-vsctl set Interface vhost-user1 options:n_rxq=$queues_nr > ovs-vsctl set Interface vhost-user2 options:n_rxq=$queues_nr This step is wrong and would explain why it didn't work. Please check: https://github.com/openvswitch/ovs/blob/branch-2.5/INSTALL.DPDK.md ---8<--- DPDK port Rx Queues ovs-vsctl set Open_vSwitch . other_config:n-dpdk-rxqs=<integer> The command above sets the number of rx queues for each DPDK interface. The rx queues are assigned to pmd threads on the same NUMA node in a round-robin fashion. ---8<---
(In reply to Flavio Leitner from comment #2) > > 3. enable mq: > > ovs-vsctl set Interface vhost-user1 options:n_rxq=$queues_nr > > ovs-vsctl set Interface vhost-user2 options:n_rxq=$queues_nr > > This step is wrong and would explain why it didn't work. > Please check: > https://github.com/openvswitch/ovs/blob/branch-2.5/INSTALL.DPDK.md > ---8<--- > DPDK port Rx Queues > > ovs-vsctl set Open_vSwitch . other_config:n-dpdk-rxqs=<integer> > > The command above sets the number of rx queues for each DPDK interface. The > rx queues are assigned to pmd threads on the same NUMA node in a round-robin > fashion. > ---8<--- Hi Pei, Since Xiyue is on KVM forum. Could you have a try when you are available? Best Regards, Junyi
Summary: After update step3, the network in VM1 and VM2 works well. Versions: Host: 3.10.0-495.rt56.397.el7.x86_64 qemu-kvm-rhev-2.6.0-22.el7.x86_64 Guest: 3.10.0-495.rt56.397.el7.x86_64 Steps: 1. Start OVS-DPDK # ovs-vsctl show f77966d3-7c88-4610-8591-a45e0ec2888d Bridge "ovsbr0" Port "vhost-user1" Interface "vhost-user1" type: dpdkvhostuser Port "vhost-user3" Interface "vhost-user3" type: dpdkvhostuser Port "ovsbr0" Interface "ovsbr0" type: internal Bridge "ovsbr1" Port "vhost-user2" Interface "vhost-user2" type: dpdkvhostuser Port "vhost-user4" Interface "vhost-user4" type: dpdkvhostuser Port "ovsbr1" Interface "ovsbr1" type: internal 2. Set ovs-dpdk 2 queues ovs-vsctl set Open_vSwitch . other_config={} ovs-vsctl set Open_vSwitch . other_config:n-dpdk-rxqs=2 ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x15554 ovs-vsctl set Open_vSwitch . other_config:dpdk-lcore-mask=0x1 3. Start 2 VMs VM1: <vcpu placement='static'>6</vcpu> <cputune> <vcpupin vcpu='0' cpuset='30'/> <vcpupin vcpu='1' cpuset='28'/> <vcpupin vcpu='2' cpuset='26'/> <vcpupin vcpu='3' cpuset='24'/> <vcpupin vcpu='4' cpuset='22'/> <vcpupin vcpu='5' cpuset='31'/> <emulatorpin cpuset='1,3,5,7,9'/> <vcpusched vcpus='0-5' scheduler='fifo' priority='1'/> </cputune> <interface type='vhostuser'> <mac address='14:18:77:48:01:02'/> <source type='unix' path='/var/run/openvswitch/vhost-user1' mode='client'/> <model type='virtio'/> <driver name='vhost' queues='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </interface> <interface type='vhostuser'> <mac address='14:18:77:48:01:03'/> <source type='unix' path='/var/run/openvswitch/vhost-user2' mode='client'/> <model type='virtio'/> <driver name='vhost' queues='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </interface> VM2: <vcpu placement='static'>6</vcpu> <cputune> <vcpupin vcpu='0' cpuset='20'/> <vcpupin vcpu='1' cpuset='18'/> <vcpupin vcpu='2' cpuset='16'/> <vcpupin vcpu='3' cpuset='14'/> <vcpupin vcpu='4' cpuset='12'/> <vcpupin vcpu='5' cpuset='29'/> <emulatorpin cpuset='1,3,5,7,9'/> <vcpusched vcpus='0-5' scheduler='fifo' priority='1'/> </cputune> <interface type='vhostuser'> <mac address='14:18:77:48:02:02'/> <source type='unix' path='/var/run/openvswitch/vhost-user3' mode='client'/> <model type='virtio'/> <driver name='vhost' queues='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </interface> <interface type='vhostuser'> <mac address='14:18:77:48:02:03'/> <source type='unix' path='/var/run/openvswitch/vhost-user4' mode='client'/> <model type='virtio'/> <driver name='vhost' queues='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </interface> 3. Set mq in VM1 and VM2 # ethtool -L eth0 combined 2 # ethtool -L eth2 combined 2 4. Set ip in VM VM1: # ifconfig eth0 192.168.1.1/24 # ifconfig eth2 192.168.2.1/24 VM2: # ifconfig eth0 192.168.1.2/24 # ifconfig eth2 192.168.2.2/24 5. Ping between VM1 and VM2. Works. So with the fixed configuration, the network works well. This bug is not a bug.
(In reply to Pei Zhang from comment #4) > Summary: After update step3, the network in VM1 and VM2 works well. [...] > So with the fixed configuration, the network works well. This bug is not a > bug. Thanks for testing it again. Closing...