Bug 1337993

Summary: Cannot spawn VMs with vhost_user ports (DPDK)
Product: Red Hat OpenStack Reporter: Lukasz Rzasik <lukaszx.rzasik>
Component: openstack-novaAssignee: Kashyap Chamarthy <kchamart>
Status: CLOSED ERRATA QA Contact: Prasanth Anbalagan <panbalag>
Severity: high Docs Contact:
Priority: unspecified    
Version: 8.0 (Liberty)CC: berrange, dasmith, eglynn, jwang, kchamart, sbauza, sferdjao, sgordon, srevivo, vromanso
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-nova-12.0.3-11.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-06-29 14:38:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1342229    
Attachments:
Description Flags
Nova Compute logs none

Description Lukasz Rzasik 2016-05-20 16:09:23 UTC 
Created attachment 1160007 [details]
Nova Compute logs

Description of problem:
Cannot spawn VM using vhost_user ports. 
Nova fails at _set_device_mtu (logs attached)

The bug was already reported and fixed in OpenStack project:
https://bugs.launchpad.net/nova/+bug/1533876

How reproducible:


Steps to Reproduce:
1. Deploy OSP8 using Director
2. Follow the guide to configure DPDK ( https://access.redhat.com/documentation/en/red-hat-openstack-platform/version-8/configure-dpdk-for-openstack-networking/ )
3. Try spawning VMs (e.g. cirros)

Actual results:
Spawning fails with "No host found"

Expected results:
VM spawns successfully
Comment 4 Vladik Romanovsky 2016-06-03 13:38:05 UTC 
*** Bug 1342142 has been marked as a duplicate of this bug. ***
Comment 6 Prasanth Anbalagan 2016-06-28 20:37:37 UTC 
Verified as follows - VMs spawned successfully following DPDK setup https://access.redhat.com/documentation/en/red-hat-openstack-platform/version-8/configure-dpdk-for-openstack-networking/.

***********
VERSION
***********

[root@serverA ~]# yum list installed | grep openstack-nova
openstack-nova-api.noarch            1:12.0.3-11.el7ost      @RH7-RHOS-8.0      
openstack-nova-cert.noarch           1:12.0.3-11.el7ost      @RH7-RHOS-8.0      
openstack-nova-common.noarch         1:12.0.3-11.el7ost      @RH7-RHOS-8.0      
openstack-nova-compute.noarch        1:12.0.3-11.el7ost      @RH7-RHOS-8.0      
openstack-nova-conductor.noarch      1:12.0.3-11.el7ost      @RH7-RHOS-8.0      
openstack-nova-console.noarch        1:12.0.3-11.el7ost      @RH7-RHOS-8.0      
openstack-nova-novncproxy.noarch     1:12.0.3-11.el7ost      @RH7-RHOS-8.0      
openstack-nova-scheduler.noarch      1:12.0.3-11.el7ost      @RH7-RHOS-8.0      


********
LOGS
********

[root@serverA]# yum info openvswitch-dpdk

Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Installed Packages
Name        : openvswitch-dpdk
Arch        : x86_64
Version     : 2.4.0
Release     : 0.10346.git97bab959.3.el7_2
Size        : 11 M
Repo        : installed
From repo   : RH7-RHOS-8.0
Summary     : Open vSwitch
URL         : http://www.openvswitch.org/
License     : ASL 2.0 and LGPLv2+ and SISSL
Description : Open vSwitch provides standard network bridging functions and
            : support for the OpenFlow protocol for remote per-flow control of
            : traffic.

[root@serverA]# 

[root@serverA]# ovs-vsctl  --version
ovs-vsctl (Open vSwitch) 2.4.0
Compiled Mar 22 2016 11:09:15
DB Schema 7.12.1

[root@serverA]# pgrep "ovs-vswitchd"
2407

[root@serverA]# ovs-vsctl get Open_vSwitch . iface_types
[dpdk, dpdkr, dpdkvhostuser, geneve, gre, "gre64", internal, ipsec_gre, "ipsec_gre64", lisp, patch, stt, system, tap, vxlan]

[root@serverA]# cat /proc/meminfo  | grep -i 'hugepages'
AnonHugePages:    149504 kB
HugePages_Total:      60
HugePages_Free:       59
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:    1048576 kB

[root@serverA]# dpdk_nic_bind --status

Network devices using DPDK-compatible driver
============================================
0000:03:00.1 'Ethernet Controller X710 for 10GbE backplane' drv=vfio-pci unused=

Network devices using kernel driver
===================================
0000:03:00.0 'Ethernet Controller X710 for 10GbE backplane' if=enp3s0f0 drv=i40e unused=vfio-pci *Active*

Other network devices
=====================
<none>


[root@serverA]# nova flavor-show m1.small
+----------------------------+-------------------------------+
| Property                   | Value                         |
+----------------------------+-------------------------------+
| OS-FLV-DISABLED:disabled   | False                         |
| OS-FLV-EXT-DATA:ephemeral  | 0                             |
| disk                       | 20                            |
| extra_specs                | {"hw:mem_page_size": "large"} |
| id                         | 2                             |
| name                       | m1.small                      |
| os-flavor-access:is_public | True                          |
| ram                        | 2048                          |
| rxtx_factor                | 1.0                           |
| swap                       |                               |
| vcpus                      | 1                             |
+----------------------------+-------------------------------+
[root@serverA]# 

[root@serverA]#  ovs-vsctl list bridge
_uuid               : c88cbd89-f9cd-4896-bc74-d91a7a4b361e
auto_attach         : []
controller          : []
datapath_id         : "00008abd8cc89648"
datapath_type       : netdev
datapath_version    : "<built-in>"
external_ids        : {}
fail_mode           : []
flood_vlans         : []
flow_tables         : {}
ipfix               : []
mcast_snooping_enable: false
mirrors             : []
name                : br-ex
netflow             : []
other_config        : {}
ports               : [59010605-2d88-44f6-8c5d-de3f02e0757f, e3bd6ead-a682-4652-977d-97b964c5f992]
protocols           : []
rstp_enable         : false
rstp_status         : {}
sflow               : []
status              : {}
stp_enable          : false

_uuid               : 02e130dd-4193-4be0-adac-bb00bda0f15a
auto_attach         : []
controller          : []
datapath_id         : "0000de30e102e04b"
datapath_type       : netdev
datapath_version    : "<built-in>"
external_ids        : {}
fail_mode           : secure
flood_vlans         : []
flow_tables         : {}
ipfix               : []
mcast_snooping_enable: false
mirrors             : []
name                : br-int
netflow             : []
other_config        : {}
ports               : [09b7ad0c-e4c3-477b-bd5c-9a429e098086, 892a8267-03b3-4c53-aa0b-61b8ef0eed03, ce166e34-c073-4446-adfd-e442ccf9bd31, cf20fe03-a666-4be5-8943-e9df6e0c74bb]
protocols           : ["OpenFlow10"]
rstp_enable         : false
rstp_status         : {}
sflow               : []
status              : {}
stp_enable          : false

_uuid               : 0037e749-6abe-4173-8f18-60e4512de6ff
auto_attach         : []
controller          : []
datapath_id         : "00004ae737007341"
datapath_type       : netdev
datapath_version    : "<built-in>"
external_ids        : {}
fail_mode           : secure
flood_vlans         : []
flow_tables         : {}
ipfix               : []
mcast_snooping_enable: false
mirrors             : []
name                : br-tun
netflow             : []
other_config        : {}
ports               : [09b5f3b9-2f48-47c5-ab51-8f2e39f93b44, 978b7491-82e0-4fdf-94b9-d2d5d02d35ce]
protocols           : ["OpenFlow10"]
rstp_enable         : false
rstp_status         : {}
sflow               : []
status              : {}
stp_enable          : false
[root@serverA]# 



[root@serverA]# nova boot --flavor m1.small --image cirros vm1
+--------------------------------------+-----------------------------------------------+
| Property                             | Value                                         |
+--------------------------------------+-----------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                        |
| OS-EXT-AZ:availability_zone          |                                               |
| OS-EXT-SRV-ATTR:host                 | -                                             |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | -                                             |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000001                             |
| OS-EXT-STS:power_state               | 0                                             |
| OS-EXT-STS:task_state                | scheduling                                    |
| OS-EXT-STS:vm_state                  | building                                      |
| OS-SRV-USG:launched_at               | -                                             |
| OS-SRV-USG:terminated_at             | -                                             |
| accessIPv4                           |                                               |
| accessIPv6                           |                                               |
| adminPass                            | Pxze8pgssWYR                                  |
| config_drive                         |                                               |
| created                              | 2016-06-28T20:13:16Z                          |
| flavor                               | m1.small (2)                                  |
| hostId                               |                                               |
| id                                   | eb77d2e6-d0df-48ff-99e8-a9ca841a835f          |
| image                                | cirros (4d2abbc3-9c94-4639-92f3-ad2e79386d29) |
| key_name                             | -                                             |
| metadata                             | {}                                            |
| name                                 | vm1                                           |
| os-extended-volumes:volumes_attached | []                                            |
| progress                             | 0                                             |
| security_groups                      | default                                       |
| status                               | BUILD                                         |
| tenant_id                            | 12624788715d4444a09f30ccbaf4ea24              |
| updated                              | 2016-06-28T20:13:17Z                          |
| user_id                              | 4c8b4ef553f247a8bec14a4934698e55              |
+--------------------------------------+-----------------------------------------------+
[root@serverA]# nova list
+--------------------------------------+------+--------+------------+-------------+---------------------+
| ID                                   | Name | Status | Task State | Power State | Networks            |
+--------------------------------------+------+--------+------------+-------------+---------------------+
| eb77d2e6-d0df-48ff-99e8-a9ca841a835f | vm1  | BUILD  | spawning   | NOSTATE     | public=172.24.4.227 |
+--------------------------------------+------+--------+------------+-------------+---------------------+
[root@serverA]# nova list
+--------------------------------------+------+--------+------------+-------------+---------------------+
| ID                                   | Name | Status | Task State | Power State | Networks            |
+--------------------------------------+------+--------+------------+-------------+---------------------+
| eb77d2e6-d0df-48ff-99e8-a9ca841a835f | vm1  | ACTIVE | -          | Running     | public=172.24.4.227 |
+--------------------------------------+------+--------+------------+-------------+---------------------+


[root@serverA]# virsh list --all
 Id    Name                           State
----------------------------------------------------
 2     instance-00000001              running

[root@serverA]# 
[root@serverA]# 
[root@serverA]# virsh dumpxml 2
<domain type='kvm' id='2'>
  <name>instance-00000001</name>
  <uuid>eb77d2e6-d0df-48ff-99e8-a9ca841a835f</uuid>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
      <nova:package version="12.0.3-11.el7ost"/>
      <nova:name>vm1</nova:name>
      <nova:creationTime>2016-06-28 20:13:51</nova:creationTime>
      <nova:flavor name="m1.small">
        <nova:memory>2048</nova:memory>
        <nova:disk>20</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>1</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="4c8b4ef553f247a8bec14a4934698e55">admin</nova:user>
        <nova:project uuid="12624788715d4444a09f30ccbaf4ea24">admin</nova:project>
      </nova:owner>
      <nova:root type="image" uuid="4d2abbc3-9c94-4639-92f3-ad2e79386d29"/>
    </nova:instance>
  </metadata>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='1048576' unit='KiB' nodeset='0'/>
    </hugepages>
  </memoryBacking>
  <vcpu placement='static'>1</vcpu>
  <cputune>
    <shares>1024</shares>
    <vcpupin vcpu='0' cpuset='0-11'/>
    <emulatorpin cpuset='0-11'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0'/>
    <memnode cellid='0' mode='strict' nodeset='0'/>
  </numatune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>Red Hat</entry>
      <entry name='product'>OpenStack Compute</entry>
      <entry name='version'>12.0.3-11.el7ost</entry>
      <entry name='serial'>0207bff0-616d-42c8-a2cf-0263ea85cd6c</entry>
      <entry name='uuid'>eb77d2e6-d0df-48ff-99e8-a9ca841a835f</entry>
      <entry name='family'>Virtual Machine</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.2.0'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-model'>
    <model fallback='allow'/>
    <topology sockets='1' cores='1' threads='1'/>
    <numa>
      <cell id='0' cpus='0' memory='2097152' unit='KiB' memAccess='shared'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/nova/instances/eb77d2e6-d0df-48ff-99e8-a9ca841a835f/disk'/>
      <backingStore type='file' index='1'>
        <format type='raw'/>
        <source file='/var/lib/nova/instances/_base/1738dad1a9994b0841d35655cfa65c7b00ce47ce'/>
        <backingStore/>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <controller type='usb' index='0'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>                                     *********************** INTERFACE TYPE vhostuser ******************
    <interface type='vhostuser'>
      <mac address='fa:16:3e:8e:38:81'/>
      <source type='unix' path='/var/run/openvswitch/vhub06f31d9-2a' mode='client'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>                                      ********************************************************************
    <serial type='file'>
      <source path='/var/lib/nova/instances/eb77d2e6-d0df-48ff-99e8-a9ca841a835f/console.log'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <serial type='pty'>
      <source path='/dev/pts/1'/>
      <target port='1'/>
      <alias name='serial1'/>
    </serial>
    <console type='file'>
      <source path='/var/lib/nova/instances/eb77d2e6-d0df-48ff-99e8-a9ca841a835f/console.log'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <stats period='10'/>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'>
    <label>system_u:system_r:svirt_t:s0:c57,c327</label>
    <imagelabel>system_u:object_r:svirt_image_t:s0:c57,c327</imagelabel>
  </seclabel>
</domain>

[root@serverA]#
Comment 8 errata-xmlrpc 2016-06-29 14:38:27 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1355