Bug 1248777

Summary: [fdBeta 2.6] [OVS-DPDK] Support VXLAN in userspace using IPv6
Product: Red Hat Enterprise Linux 7 Reporter: Thadeu Lima de Souza Cascardo <cascardo>
Component: openvswitchAssignee: Flavio Leitner <fleitner>
Status: CLOSED CURRENTRELEASE QA Contact: Jean-Tsung Hsiao <jhsiao>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.2CC: aloughla, atragler, cascardo, fleitner, jeder, jhsiao, kzhang, mleitner, network-qe, nyechiel, rkhan, vincent.jardin
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openvswitch-2.6.1-3.git20161206.el7fdb Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1248776
: 1411939 (view as bug list) Environment:
Last Closed: 2017-06-07 17:26:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1323132, 1384672    

Description Thadeu Lima de Souza Cascardo 2015-07-30 19:41:34 UTC 
+++ This bug was initially created as a clone of Bug #1248776 +++

Using a userspace bridge with a dpdk interface, we need to support VXLAN tunnels. Notice that this requires that the tunnel destination has a route through another userspace OVS bridge. For this bug, we are interested in testing IPv6 tunnels.

There is some work that is need upstream for this to be complete.

Cascardo.
Comment 2 Thadeu Lima de Souza Cascardo 2015-10-23 13:51:59 UTC 
Patches submitted upstream:

http://openvswitch.org/pipermail/dev/2015-October/061542.html
Comment 3 Thadeu Lima de Souza Cascardo 2016-01-06 17:40:39 UTC 
Patches merged upstream, will be included when OVS 2.5 is released.
Comment 4 Thadeu Lima de Souza Cascardo 2016-01-06 17:41:02 UTC 
Patches merged upstream, will be included when OVS 2.5 is released.
Comment 5 Thadeu Lima de Souza Cascardo 2016-01-06 17:52:30 UTC 
OVS test cases include IPv6 tunnel test cases. They use the OVS testsuite.

Cascardo.
Comment 6 Thadeu Lima de Souza Cascardo 2016-07-01 15:38:12 UTC 
Userspace support is only going to OVS 2.6.

Cascardo.
Comment 7 Flavio Leitner 2016-09-29 14:37:06 UTC 
Hi Thadeu,

OVS 2.6 just got released, so could you please confirm that it includes everything we need here?
Thanks!
fbl
Comment 8 Thadeu Lima de Souza Cascardo 2016-09-29 14:48:24 UTC 
Yes, we should have everything in place, though we still need an official RPM package of OVS 2.6, right?

Cascardo.
Comment 9 Thadeu Lima de Souza Cascardo 2016-12-02 18:35:49 UTC 
We should just be waiting to go to 2.6.

Cascardo.
Comment 11 Jean-Tsung Hsiao 2017-01-14 13:04:20 UTC 
The feature has been VERIFIED.

Ping6 worked and netperf -6 has reasonable IPv6 throughput between guests.

Test 1
MIGRATED TCP STREAM TEST from ::0 (::) port 0 AF_INET6 to abcd::102 () port 0 AF_INET6
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

 87380  16384  16384    30.00    3238.42   
Test 2
MIGRATED TCP STREAM TEST from ::0 (::) port 0 AF_INET6 to abcd::102 () port 0 AF_INET6
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

 87380  16384  16384    30.00    3316.74   
Test 3
MIGRATED TCP STREAM TEST from ::0 (::) port 0 AF_INET6 to abcd::102 () port 0 AF_INET6
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

 87380  16384  16384    30.00    3041.73   
Test 4
MIGRATED TCP STREAM TEST from ::0 (::) port 0 AF_INET6 to abcd::102 () port 0 AF_INET6
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

 87380  16384  16384    30.00    3034.11   
Test 5
MIGRATED TCP STREAM TEST from ::0 (::) port 0 AF_INET6 to abcd::102 () port 0 AF_INET6
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

 87380  16384  16384    30.00    3198.90   


*** netqe9 --- Host and Guest ***

ovs-vsctl set Open_vSwitch . other_config={}
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=0xaa0000
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem="4096,1"
sleep 5
systemctl restart openvswitch
sleep 5

# Configure OVS-dpdk vxlan tunnel
ovs-vsctl --if-exists del-br ovsbr0
ovs-vsctl --if-exists del-br ovsbr1
ovs-vsctl set Open_vSwitch . other_config={}
ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0xaa0000
ovs-vsctl add-br ovsbr0 -- set bridge ovsbr0 datapath_type=netdev
ovs-vsctl add-port ovsbr0 dpdk0 -- set Interface dpdk0 type=dpdk
ovs-vsctl --timeout 10 set Interface dpdk0 options:n_rxq=4

ovs-vsctl add-br ovsbr1 -- set bridge ovsbr1 datapath_type=netdev
ovs-vsctl add-port ovsbr1 vxlan0 -- set interface vxlan0 type=vxlan \
        options:remote_ip=192.168.9.106 options:dst_port=8472 options:key=1000
ip addr add 192.168.9.105/24 dev ovsbr0
ip link set ovsbr0 up

ovs-vsctl add-port ovsbr1 vhost0 -- set interface vhost0 type=dpdkvhostuser
chown qemu /var/run/openvswitch/vhost0

ovs-vsctl show
ip r

[root@netqe9 ovs-dpdk-tunnel]# virsh dumpxml mq-vhu-tunnel
<domain type='kvm'>
  <name>mq-vhu-tunnel</name>
  <uuid>08606b6f-b987-4366-bcd5-c84e229e2854</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='1048576' unit='KiB'/>
    </hugepages>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='5'/>
    <vcpupin vcpu='1' cpuset='7'/>
    <vcpupin vcpu='2' cpuset='9'/>
    <vcpupin vcpu='3' cpuset='11'/>
    <emulatorpin cpuset='13'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.2.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Haswell-noTSX</model>
    <numa>
      <cell id='0' cpus='0' memory='4194304' unit='KiB' memAccess='shared'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/mnt/test/vhostuser/mq_vhu_tunnel.img'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <interface type='vhostuser'>
      <mac address='52:54:00:7b:99:ef'/>
      <source type='unix' path='/var/run/openvswitch/vhost0' mode='client'/>
      <model type='virtio'/>
      <driver name='vhost' queues='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='52:54:00:6b:95:f7'/>
      <source bridge='virbr0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1'>
      <listen type='address' address='127.0.0.1'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'/>
  <seclabel type='dynamic' model='dac' relabel='yes'/>
</domain>


*** netqe10 --- Host and Guest ***

ovs-vsctl set Open_vSwitch . other_config={}
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=0xaa0000
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem="4096,1"
sleep 5
systemctl restart openvswitch
sleep 5

# Configure OVS-dpdk vxlan tunnel
ovs-vsctl --if-exists del-br ovsbr0
ovs-vsctl --if-exists del-br ovsbr1
ovs-vsctl set Open_vSwitch . other_config={}
ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0xaa0000
ovs-vsctl add-br ovsbr0 -- set bridge ovsbr0 datapath_type=netdev
ovs-vsctl add-port ovsbr0 dpdk0 -- set Interface dpdk0 type=dpdk
ovs-vsctl --timeout 10 set Interface dpdk0 options:n_rxq=4

ovs-vsctl add-br ovsbr1 -- set bridge ovsbr1 datapath_type=netdev
ovs-vsctl add-port ovsbr1 vxlan0 -- set interface vxlan0 type=vxlan \
        options:remote_ip=192.168.9.105 options:dst_port=8472 options:key=1000
ip addr add 192.168.9.106/24 dev ovsbr0
ip link set ovsbr0 up

ovs-vsctl add-port ovsbr1 vhost0 -- set interface vhost0 type=dpdkvhostuser
chown qemu /var/run/openvswitch/vhost0

ovs-vsctl show
ip r

[root@netqe10 ovs-dpdk-tunnel]# virsh dumpxml mq-vhu-tunnel
<domain type='kvm' id='2'>
  <name>mq-vhu-tunnel</name>
  <uuid>a547c05d-a30b-457f-a9b0-42f8a04b6041</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='1048576' unit='KiB'/>
    </hugepages>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='5'/>
    <vcpupin vcpu='1' cpuset='7'/>
    <vcpupin vcpu='2' cpuset='9'/>
    <vcpupin vcpu='3' cpuset='11'/>
    <emulatorpin cpuset='13'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.2.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Haswell-noTSX</model>
    <numa>
      <cell id='0' cpus='0' memory='4194304' unit='KiB' memAccess='shared'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/mnt/test/vhostuser/mq_vhu_tunnel.img'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <interface type='vhostuser'>
      <mac address='52:54:00:49:da:a4'/>
      <source type='unix' path='/var/run/openvswitch/vhost0' mode='client'/>
      <model type='virtio'/>
      <driver name='vhost' queues='4'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='52:54:00:bc:21:35'/>
      <source bridge='virbr0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/2'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/2'>
      <source path='/dev/pts/2'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-2-mq-vhu-tunnel/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1'>
      <listen type='address' address='127.0.0.1'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'>
    <label>system_u:system_r:svirt_t:s0:c130,c289</label>
    <imagelabel>system_u:object_r:svirt_image_t:s0:c130,c289</imagelabel>
  </seclabel>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+107:+107</label>
    <imagelabel>+107:+107</imagelabel>
  </seclabel>
</domain>

[root@netqe10 ovs-dpdk-tunnel]#