Bug 1322296

> 3. enable mq:
> ovs-vsctl set Interface vhost-user1 options:n_rxq=$queues_nr
> ovs-vsctl set Interface vhost-user2 options:n_rxq=$queues_nr

This step is wrong and would explain why it didn't work.
Please check: https://github.com/openvswitch/ovs/blob/branch-2.5/INSTALL.DPDK.md
---8<---
DPDK port Rx Queues

ovs-vsctl set Open_vSwitch . other_config:n-dpdk-rxqs=<integer>

The command above sets the number of rx queues for each DPDK interface. The rx queues are assigned to pmd threads on the same NUMA node in a round-robin fashion. 
---8<---

(In reply to Flavio Leitner from comment #2)
> > 3. enable mq:
> > ovs-vsctl set Interface vhost-user1 options:n_rxq=$queues_nr
> > ovs-vsctl set Interface vhost-user2 options:n_rxq=$queues_nr
> 
> This step is wrong and would explain why it didn't work.
> Please check:
> https://github.com/openvswitch/ovs/blob/branch-2.5/INSTALL.DPDK.md
> ---8<---
> DPDK port Rx Queues
> 
> ovs-vsctl set Open_vSwitch . other_config:n-dpdk-rxqs=<integer>
> 
> The command above sets the number of rx queues for each DPDK interface. The
> rx queues are assigned to pmd threads on the same NUMA node in a round-robin
> fashion. 
> ---8<---

Hi Pei,

Since Xiyue is on KVM forum. Could you have a try when you are available?

Best Regards,
Junyi

Summary: After update step3, the network in VM1 and VM2 works well.

Versions:
Host:
3.10.0-495.rt56.397.el7.x86_64
qemu-kvm-rhev-2.6.0-22.el7.x86_64

Guest:
3.10.0-495.rt56.397.el7.x86_64

Steps:
1. Start OVS-DPDK
# ovs-vsctl show
f77966d3-7c88-4610-8591-a45e0ec2888d
    Bridge "ovsbr0"
        Port "vhost-user1"
            Interface "vhost-user1"
                type: dpdkvhostuser
        Port "vhost-user3"
            Interface "vhost-user3"
                type: dpdkvhostuser
        Port "ovsbr0"
            Interface "ovsbr0"
                type: internal
    Bridge "ovsbr1"
        Port "vhost-user2"
            Interface "vhost-user2"
                type: dpdkvhostuser
        Port "vhost-user4"
            Interface "vhost-user4"
                type: dpdkvhostuser
        Port "ovsbr1"
            Interface "ovsbr1"
                type: internal

2. Set ovs-dpdk 2 queues
ovs-vsctl set Open_vSwitch . other_config={}  
ovs-vsctl set Open_vSwitch . other_config:n-dpdk-rxqs=2
ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x15554
ovs-vsctl set Open_vSwitch . other_config:dpdk-lcore-mask=0x1

3. Start 2 VMs
VM1:
 <vcpu placement='static'>6</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='30'/>
    <vcpupin vcpu='1' cpuset='28'/>
    <vcpupin vcpu='2' cpuset='26'/>
    <vcpupin vcpu='3' cpuset='24'/>
    <vcpupin vcpu='4' cpuset='22'/>
    <vcpupin vcpu='5' cpuset='31'/>
    <emulatorpin cpuset='1,3,5,7,9'/>
    <vcpusched vcpus='0-5' scheduler='fifo' priority='1'/>
  </cputune>

 <interface type='vhostuser'>
      <mac address='14:18:77:48:01:02'/>
      <source type='unix' path='/var/run/openvswitch/vhost-user1' mode='client'/>
      <model type='virtio'/>
      <driver name='vhost' queues='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='14:18:77:48:01:03'/>
      <source type='unix' path='/var/run/openvswitch/vhost-user2' mode='client'/>
      <model type='virtio'/>
      <driver name='vhost' queues='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </interface>

VM2:
 <vcpu placement='static'>6</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='20'/>
    <vcpupin vcpu='1' cpuset='18'/>
    <vcpupin vcpu='2' cpuset='16'/>
    <vcpupin vcpu='3' cpuset='14'/>
    <vcpupin vcpu='4' cpuset='12'/>
    <vcpupin vcpu='5' cpuset='29'/>
    <emulatorpin cpuset='1,3,5,7,9'/>
    <vcpusched vcpus='0-5' scheduler='fifo' priority='1'/>
  </cputune>

<interface type='vhostuser'>
      <mac address='14:18:77:48:02:02'/>
      <source type='unix' path='/var/run/openvswitch/vhost-user3' mode='client'/>
      <model type='virtio'/>
      <driver name='vhost' queues='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='14:18:77:48:02:03'/>
      <source type='unix' path='/var/run/openvswitch/vhost-user4' mode='client'/>
      <model type='virtio'/>
      <driver name='vhost' queues='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </interface>

3. Set mq in VM1 and VM2
# ethtool -L eth0 combined 2
# ethtool -L eth2 combined 2

4. Set ip in VM
VM1:
# ifconfig eth0 192.168.1.1/24
# ifconfig eth2 192.168.2.1/24

VM2:
# ifconfig eth0 192.168.1.2/24
# ifconfig eth2 192.168.2.2/24

5. Ping between VM1 and VM2.
Works.

So with the fixed configuration, the network works well. This bug is not a bug.

(In reply to Pei Zhang from comment #4)
> Summary: After update step3, the network in VM1 and VM2 works well.
[...]
> So with the fixed configuration, the network works well. This bug is not a
> bug.

Thanks for testing it again.
Closing...