| Summary: | Network does not work while using vhost-user interface with multi-queue in real-time kernel | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | xiywang | ||||
| Component: | openvswitch-dpdk | Assignee: | Flavio Leitner <fleitner> | ||||
| Status: | CLOSED NOTABUG | QA Contact: | Red Hat Kernel QE team <kernel-qe> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 7.3 | CC: | ailan, aloughla, atragler, fleitner, hhuang, juzhang, knoel, mleitner, pezhang, rkhan, xfu, xiywang | ||||
| Target Milestone: | rc | ||||||
| Target Release: | --- | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2016-08-29 18:25:32 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Attachments: |
|
||||||
> 3. enable mq: > ovs-vsctl set Interface vhost-user1 options:n_rxq=$queues_nr > ovs-vsctl set Interface vhost-user2 options:n_rxq=$queues_nr This step is wrong and would explain why it didn't work. Please check: https://github.com/openvswitch/ovs/blob/branch-2.5/INSTALL.DPDK.md ---8<--- DPDK port Rx Queues ovs-vsctl set Open_vSwitch . other_config:n-dpdk-rxqs=<integer> The command above sets the number of rx queues for each DPDK interface. The rx queues are assigned to pmd threads on the same NUMA node in a round-robin fashion. ---8<--- (In reply to Flavio Leitner from comment #2) > > 3. enable mq: > > ovs-vsctl set Interface vhost-user1 options:n_rxq=$queues_nr > > ovs-vsctl set Interface vhost-user2 options:n_rxq=$queues_nr > > This step is wrong and would explain why it didn't work. > Please check: > https://github.com/openvswitch/ovs/blob/branch-2.5/INSTALL.DPDK.md > ---8<--- > DPDK port Rx Queues > > ovs-vsctl set Open_vSwitch . other_config:n-dpdk-rxqs=<integer> > > The command above sets the number of rx queues for each DPDK interface. The > rx queues are assigned to pmd threads on the same NUMA node in a round-robin > fashion. > ---8<--- Hi Pei, Since Xiyue is on KVM forum. Could you have a try when you are available? Best Regards, Junyi Summary: After update step3, the network in VM1 and VM2 works well.
Versions:
Host:
3.10.0-495.rt56.397.el7.x86_64
qemu-kvm-rhev-2.6.0-22.el7.x86_64
Guest:
3.10.0-495.rt56.397.el7.x86_64
Steps:
1. Start OVS-DPDK
# ovs-vsctl show
f77966d3-7c88-4610-8591-a45e0ec2888d
Bridge "ovsbr0"
Port "vhost-user1"
Interface "vhost-user1"
type: dpdkvhostuser
Port "vhost-user3"
Interface "vhost-user3"
type: dpdkvhostuser
Port "ovsbr0"
Interface "ovsbr0"
type: internal
Bridge "ovsbr1"
Port "vhost-user2"
Interface "vhost-user2"
type: dpdkvhostuser
Port "vhost-user4"
Interface "vhost-user4"
type: dpdkvhostuser
Port "ovsbr1"
Interface "ovsbr1"
type: internal
2. Set ovs-dpdk 2 queues
ovs-vsctl set Open_vSwitch . other_config={}
ovs-vsctl set Open_vSwitch . other_config:n-dpdk-rxqs=2
ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x15554
ovs-vsctl set Open_vSwitch . other_config:dpdk-lcore-mask=0x1
3. Start 2 VMs
VM1:
<vcpu placement='static'>6</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='30'/>
<vcpupin vcpu='1' cpuset='28'/>
<vcpupin vcpu='2' cpuset='26'/>
<vcpupin vcpu='3' cpuset='24'/>
<vcpupin vcpu='4' cpuset='22'/>
<vcpupin vcpu='5' cpuset='31'/>
<emulatorpin cpuset='1,3,5,7,9'/>
<vcpusched vcpus='0-5' scheduler='fifo' priority='1'/>
</cputune>
<interface type='vhostuser'>
<mac address='14:18:77:48:01:02'/>
<source type='unix' path='/var/run/openvswitch/vhost-user1' mode='client'/>
<model type='virtio'/>
<driver name='vhost' queues='2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</interface>
<interface type='vhostuser'>
<mac address='14:18:77:48:01:03'/>
<source type='unix' path='/var/run/openvswitch/vhost-user2' mode='client'/>
<model type='virtio'/>
<driver name='vhost' queues='2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</interface>
VM2:
<vcpu placement='static'>6</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='20'/>
<vcpupin vcpu='1' cpuset='18'/>
<vcpupin vcpu='2' cpuset='16'/>
<vcpupin vcpu='3' cpuset='14'/>
<vcpupin vcpu='4' cpuset='12'/>
<vcpupin vcpu='5' cpuset='29'/>
<emulatorpin cpuset='1,3,5,7,9'/>
<vcpusched vcpus='0-5' scheduler='fifo' priority='1'/>
</cputune>
<interface type='vhostuser'>
<mac address='14:18:77:48:02:02'/>
<source type='unix' path='/var/run/openvswitch/vhost-user3' mode='client'/>
<model type='virtio'/>
<driver name='vhost' queues='2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</interface>
<interface type='vhostuser'>
<mac address='14:18:77:48:02:03'/>
<source type='unix' path='/var/run/openvswitch/vhost-user4' mode='client'/>
<model type='virtio'/>
<driver name='vhost' queues='2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</interface>
3. Set mq in VM1 and VM2
# ethtool -L eth0 combined 2
# ethtool -L eth2 combined 2
4. Set ip in VM
VM1:
# ifconfig eth0 192.168.1.1/24
# ifconfig eth2 192.168.2.1/24
VM2:
# ifconfig eth0 192.168.1.2/24
# ifconfig eth2 192.168.2.2/24
5. Ping between VM1 and VM2.
Works.
So with the fixed configuration, the network works well. This bug is not a bug.
(In reply to Pei Zhang from comment #4) > Summary: After update step3, the network in VM1 and VM2 works well. [...] > So with the fixed configuration, the network works well. This bug is not a > bug. Thanks for testing it again. Closing... |
Created attachment 1141627 [details] start-ovsdpdk.sh Description of problem: Network does not work if use vhost-user with multi-queue in vm-to-vm model (two guests in one host). Version-Release number of selected component (if applicable): guest kernel: 3.10.0-364.rt56.241.el7.x86_64 host: 3.10.0-364.rt56.241.el7.x86_64 qemu-kvm-rhev-2.5.0-3.el7.x86_64 openvswitch-dpdk-2.5.0-3.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1. mount hugetlbfs: mount -t hugetlbfs hugetlbfs /mnt/hugetlbfs/ -o size=16G 2. set up ovs-dpdk vhost-user environment with start-ovsdpdk.sh, using "vm-to-vm" model 3. enable mq: ovs-vsctl set Interface vhost-user1 options:n_rxq=$queues_nr ovs-vsctl set Interface vhost-user2 options:n_rxq=$queues_nr ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=$cpu_mask 4. boot two guests in one host, one of the guest use vhost-user1 and vhost-user2, the other one use vhost-user3 and vhost-user 4: /usr/libexec/qemu-kvm -name rhel7.2-rt-355-guest1 -machine pc-i440fx-rhel7.2.0 \ -cpu host,+tsc-deadline,pmu=off \ -m 4096 -realtime mlock=off -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/mnt/hugetlbfs,share=on,size=4096M -numa node,nodeid=0,cpus=0-3,memdev=ram-node0 \ -smp 4 \ -drive file=/home/rhel7.2-rt-355.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,snapshot=on -device virtio-blk-pci,drive=drive-virtio-disk0,id=virtio-disk0 \ -netdev tap,id=hostnet0,vhost=on -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:a1:d0:51 \ -chardev socket,id=charnet1,path=/var/run/openvswitch/vhost-user1 -netdev type=vhost-user,id=hostnet1,chardev=charnet1,queues=4 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:b2:0a:01,mq=on,vectors=10 \ -chardev socket,id=charnet2,path=/var/run/openvswitch/vhost-user2 -netdev type=vhost-user,id=hostnet2,chardev=charnet2,queues=4 -device virtio-net-pci,netdev=hostnet2,id=net2,mac=52:54:00:b2:0a:02,mq=on,vectors=10 \ -monitor stdio -device qxl-vga,id=video0 -serial unix:/tmp/console1,server,nowait -vnc :1 -spice port=5900,disable-ticketing 5. set ip for guest1 and guest2 vhost-user interface: guest1# ifconfig eth1 192.168.10.11/24 guest2# ifconfig eth1 192.168.10.12/24 6. set mq in both guests: # ethtool -L eth1 combined 4 # ethtool -l eth1 Channel parameters for eth1: Pre-set maximums: RX: 0 TX: 0 Other: 0 Combined: 4 Current hardware settings: RX: 0 TX: 0 Other: 0 Combined: 4 7. ping from guest1 to guest2, failed. Actual results: ping failed Expected results: ping and netperf should work normally Additional info: 1. tested the same case only without mq, network works fine. Are we going to support multi-queue in the future? If do, about when will this happen? And if not, I think maybe it's better that we display some kind of warnings to tell user that we're not support mq.