Bug 1342142

Summary: [OVS_DPDK]: start instance report error when config ovs_dpdk
Product: Red Hat OpenStack Reporter: jwang
Component: openstack-novaAssignee: Eoghan Glynn <eglynn>
Status: CLOSED DUPLICATE QA Contact: Prasanth Anbalagan <panbalag>
Severity: high Docs Contact:
Priority: high    
Version: 8.0 (Liberty)CC: aconole, berrange, dasmith, eglynn, jniu, jwang, kchamart, nyechiel, pliu, sbauza, sferdjao, sgordon, srevivo, vromanso, zshi
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-06-03 13:38:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1342229    

Description jwang 2016-06-02 13:58:25 UTC 
Description of problem:
[OVS_DPDK]: start instance report error when config ovs_dpdk

on compute node /var/log/nova/nova-compute.log
2016-06-01 15:12:38.108 7946 WARNING nova.compute.monitors [req-9317696f-52a8-4de6-a9ca-b8acf72f5edb - - - - -] Excluding nova.compute.monitors.cpu monitor virt_driver. Not in the list of enabled monitors (CONF.compute_monitors).
2016-06-01 15:13:36.849 7946 ERROR nova.compute.manager [req-acdc8063-c210-44eb-8155-3a2a759678aa f65119564c9d4e668404618cca10a147 0c6e0ffd9f8e41929724fb22a82b12d3 - - -] [instance: b31b3fed-8fe5-4210-a356-1c81cdfc21c4] Instance failed to spawn
2016-06-01 15:13:36.849 7946 ERROR nova.compute.manager [instance: b31b3fed-8fe5-4210-a356-1c81cdfc21c4] Traceback (most recent call last):
2016-06-01 15:13:36.849 7946 ERROR nova.compute.manager [instance: b31b3fed-8fe5-4210-a356-1c81cdfc21c4]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2156, in _build_resources
2016-06-01 15:13:36.849 7946 ERROR nova.compute.manager [instance: b31b3fed-8fe5-4210-a356-1c81cdfc21c4]     yield resources
2016-06-01 15:13:36.849 7946 ERROR nova.compute.manager [instance: b31b3fed-8fe5-4210-a356-1c81cdfc21c4]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2009, in _build_and_run_instance
2016-06-01 15:13:36.849 7946 ERROR nova.compute.manager [instance: b31b3fed-8fe5-4210-a356-1c81cdfc21c4]     block_device_info=block_device_info)
2016-06-01 15:13:36.849 7946 ERROR nova.compute.manager [instance: b31b3fed-8fe5-4210-a356-1c81cdfc21c4]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2585, in spawn
2016-06-01 15:13:36.849 7946 ERROR nova.compute.manager [instance: b31b3fed-8fe5-4210-a356-1c81cdfc21c4]     block_device_info=block_device_info)
2016-06-01 15:13:36.849 7946 ERROR nova.compute.manager [instance: b31b3fed-8fe5-4210-a356-1c81cdfc21c4]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4691, in _create_domain_and_network
2016-06-01 15:13:36.849 7946 ERROR nova.compute.manager [instance: b31b3fed-8fe5-4210-a356-1c81cdfc21c4]     self.plug_vifs(instance, network_info)
2016-06-01 15:13:36.849 7946 ERROR nova.compute.manager [instance: b31b3fed-8fe5-4210-a356-1c81cdfc21c4]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 675, in plug_vifs
2016-06-01 15:13:36.849 7946 ERROR nova.compute.manager [instance: b31b3fed-8fe5-4210-a356-1c81cdfc21c4]     self.vif_driver.plug(instance, vif)
2016-06-01 15:13:36.849 7946 ERROR nova.compute.manager [instance: b31b3fed-8fe5-4210-a356-1c81cdfc21c4]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 736, in plug
2016-06-01 15:13:36.849 7946 ERROR nova.compute.manager [instance: b31b3fed-8fe5-4210-a356-1c81cdfc21c4]     func(instance, vif)
2016-06-01 15:13:36.849 7946 ERROR nova.compute.manager [instance: b31b3fed-8fe5-4210-a356-1c81cdfc21c4]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 675, in plug_vhostuser
2016-06-01 15:13:36.849 7946 ERROR nova.compute.manager [instance: b31b3fed-8fe5-4210-a356-1c81cdfc21c4]     instance.uuid)
2016-06-01 15:13:36.849 7946 ERROR nova.compute.manager [instance: b31b3fed-8fe5-4210-a356-1c81cdfc21c4]   File "/usr/lib/python2.7/site-packages/nova/network/linux_net.py", line 1379, in create_ovs_vif_port
2016-06-01 15:13:36.849 7946 ERROR nova.compute.manager [instance: b31b3fed-8fe5-4210-a356-1c81cdfc21c4]     _set_device_mtu(dev)
2016-06-01 15:13:36.849 7946 ERROR nova.compute.manager [instance: b31b3fed-8fe5-4210-a356-1c81cdfc21c4]   File "/usr/lib/python2.7/site-packages/nova/network/linux_net.py", line 1342, in _set_device_mtu
2016-06-01 15:13:36.849 7946 ERROR nova.compute.manager [instance: b31b3fed-8fe5-4210-a356-1c81cdfc21c4]     check_exit_code=[0, 2, 254])
2016-06-01 15:13:36.849 7946 ERROR nova.compute.manager [instance: b31b3fed-8fe5-4210-a356-1c81cdfc21c4]   File "/usr/lib/python2.7/site-packages/nova/utils.py", line 389, in execute
2016-06-01 15:13:36.849 7946 ERROR nova.compute.manager [instance: b31b3fed-8fe5-4210-a356-1c81cdfc21c4]     return RootwrapProcessHelper().execute(*cmd, **kwargs)
2016-06-01 15:13:36.849 7946 ERROR nova.compute.manager [instance: b31b3fed-8fe5-4210-a356-1c81cdfc21c4]   File "/usr/lib/python2.7/site-packages/nova/utils.py", line 272, in execute
2016-06-01 15:13:36.849 7946 ERROR nova.compute.manager [instance: b31b3fed-8fe5-4210-a356-1c81cdfc21c4]     return processutils.execute(*cmd, **kwargs)
2016-06-01 15:13:36.849 7946 ERROR nova.compute.manager [instance: b31b3fed-8fe5-4210-a356-1c81cdfc21c4]   File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 275, in execute
2016-06-01 15:13:36.849 7946 ERROR nova.compute.manager [instance: b31b3fed-8fe5-4210-a356-1c81cdfc21c4]     cmd=sanitized_cmd)
2016-06-01 15:13:36.849 7946 ERROR nova.compute.manager [instance: b31b3fed-8fe5-4210-a356-1c81cdfc21c4] ProcessExecutionError: Unexpected error while running command.
2016-06-01 15:13:36.849 7946 ERROR nova.compute.manager [instance: b31b3fed-8fe5-4210-a356-1c81cdfc21c4] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf ip link set vhu47420507-77 mtu 1400
2016-06-01 15:13:36.849 7946 ERROR nova.compute.manager [instance: b31b3fed-8fe5-4210-a356-1c81cdfc21c4] Exit code: 1
2016-06-01 15:13:36.849 7946 ERROR nova.compute.manager [instance: b31b3fed-8fe5-4210-a356-1c81cdfc21c4] Stdout: u''
2016-06-01 15:13:36.849 7946 ERROR nova.compute.manager [instance: b31b3fed-8fe5-4210-a356-1c81cdfc21c4] Stderr: u'Cannot find device "vhu47420507-77"\n'
2016-06-01 15:13:36.849 7946 ERROR nova.compute.manager [instance: b31b3fed-8fe5-4210-a356-1c81cdfc21c4]


Version-Release number of selected component (if applicable):
openstack-neutron-common-7.0.1-15.el7ost.noarch
openstack-nova-api-12.0.2-5.el7ost.noarch
openstack-neutron-bigswitch-lldp-2015.3.8-1.el7ost.noarch
openstack-neutron-7.0.1-15.el7ost.noarch
openstack-nova-compute-12.0.2-5.el7ost.noarch
openstack-neutron-openvswitch-7.0.1-15.el7ost.noarch
openstack-nova-console-12.0.2-5.el7ost.noarch
openstack-nova-novncproxy-12.0.2-5.el7ost.noarch
openstack-neutron-metering-agent-7.0.1-15.el7ost.noarch
openstack-neutron-bigswitch-agent-2015.3.8-1.el7ost.noarch
openstack-nova-common-12.0.2-5.el7ost.noarch
openstack-neutron-lbaas-7.0.0-2.el7ost.noarch
openstack-nova-scheduler-12.0.2-5.el7ost.noarch
openstack-nova-conductor-12.0.2-5.el7ost.noarch
openstack-neutron-ml2-7.0.1-15.el7ost.noarch
openstack-nova-cert-12.0.2-5.el7ost.noarch
dpdk-tools-2.2.0-2.el7.x86_64
openvswitch-dpdk-2.4.0-0.10346.git97bab959.3.el7_2.x86_64
dpdk-2.2.0-2.el7.x86_64

How reproducible:
1. install rhop8
2. config ovs-dpdk as https://access.redhat.com/documentation/en/red-hat-openstack-platform/version-8/configure-dpdk-for-openstack-networking/#system_requirements
3. create instance

Steps to Reproduce:
1. install rhop8
2. config ovs-dpdk as https://access.redhat.com/documentation/en/red-hat-openstack-platform/version-8/configure-dpdk-for-openstack-networking/#system_requirements
3. create instance

Actual results:
create instance fail

Expected results:
create instance success

Additional info:
upstream has similar bug report and patch
[1] https://bugs.launchpad.net/fuel/+bug/1575091
[2] https://review.openstack.org/gitweb?p=openstack/nova.git;a=commitdiff;h=c7eb823fe73e3db5dca48df5879db18cbab5bd8d

patch shows that currently workaround is set 'network_device_mtu=0' in /etc/nova/nova.conf on compute node enabled ovs_dpdk
Comment 2 Aaron Conole 2016-06-02 14:11:44 UTC 
This is a Nova bug not an Open vSwitch bug; the dpdk driver should not be using ip link XXX to do any command changes. Reassigning.
Comment 3 Aaron Conole 2016-06-02 15:53:25 UTC 
Note: It appears this is possibly fixed upstream by commit adf7ba61dd73fe4bfffa20295be9a4b1006a1fe6 ("stop setting mtu when plugging vhost-user ports")
Comment 4 Vladik Romanovsky 2016-06-03 13:38:05 UTC 

*** This bug has been marked as a duplicate of bug 1337993 ***