Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1443092

Summary:	Not able to attach second SRIOV VF to an SRIOV instance.
Product:	Red Hat OpenStack	Reporter:	VIKRANT <vaggarwa>
Component:	openstack-neutron	Assignee:	Brent Eagles <beagles>
Status:	CLOSED DUPLICATE	QA Contact:	Ofer Blaut <oblaut>
Severity:	high	Docs Contact:
Priority:	high
Version:	10.0 (Newton)	CC:	amuller, atelang, berrange, chrisw, dasmith, eglynn, hjensas, kchamart, nyechiel, pablo.iranzo, sbauza, sferdjao, sgordon, sputhenp, srevivo, vaggarwa, vromanso
Target Milestone:	---	Keywords:	ZStream
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-04-24 12:29:56 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description VIKRANT 2017-04-18 13:01:01 UTC

Description of problem:

Not able to attach second SRIOV VF to an SRIOV instance. 

Version-Release number of selected component (if applicable):
RHEL OSP 10

How reproducible:
Everytime. 

Steps to Reproduce:
1. Spawn an instance using SRIOV Port
2. Tried to attach second SRIOV port to an instance. 
3. It's getting failed with following message: 

~~~
$ nova interface-attach --port-id 4c87802f-0f31-4087-9376-d639801be588 225b9842-aa1b-4a28-b997-537e645953b8
ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'PortBindingFailed_Remote'> (HTTP 500) (Request-ID: req-51dd3020-bbd6-4db0-bfea-3fa4e6b8ddff)
~~~

Actual results:
Not able to attach second SRIOV port to an  instance. 

Expected results:
We should be able to do that. 

Additional info:

Following call trace reported in nova-compute on compute node while trying to attach second sriov port.

~~~
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server [req-51dd3020-bbd6-4db0-bfea-3fa4e6b8ddff 8b72d740fda446458318aa9e8efe0893 2dcfcc0c0cae470593de77e1d15bd570 - - -] Exception during message handling
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4997, in attach_interface
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server     bind_host_id=bind_host_id)
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1200, in allocate_port_for_instance
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server     bind_host_id=bind_host_id)
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 881, in allocate_for_instance
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server     bind_host_id, dhcp_opts, available_macs)
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1000, in _update_ports_for_instance
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server     vif.destroy()
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 970, in _update_ports_for_instance
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server     port_client, instance, port_id, port_req_body)
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 437, in _update_port
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server     _ensure_no_port_binding_failure(port)
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 175, in _ensure_no_port_binding_failure
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server     raise exception.PortBindingFailed(port_id=port['id'])
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server PortBindingFailed: Binding failed for port 4c87802f-0f31-4087-9376-d639801be588, please check neutron logs for more information.
2017-04-18 12:26:36.713 418852 ERROR oslo_messaging.rpc.server 
~~~

Comment 1 VIKRANT 2017-04-18 13:21:49 UTC

Nothing significant reported in neutron logs.

~~~
From nova-compute.log 

2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server [req-4d70e5e2-f864-423c-8704-9ab135430ee1 8b72d740fda446458318aa9e8efe0893 2dcfcc0c0cae470593de77e1d15bd570 - - -] Exception during message handling
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4997, in attach_interface
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server     bind_host_id=bind_host_id)
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1200, in allocate_port_for_instance
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server     bind_host_id=bind_host_id)
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 881, in allocate_for_instance
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server     bind_host_id, dhcp_opts, available_macs)
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1000, in _update_ports_for_instance
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server     vif.destroy()
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server     self.force_reraise()
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 970, in _update_ports_for_instancece
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server     port_client, instance, port_id, port_req_body)
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 437, in _update_port
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server     _ensure_no_port_binding_failure(port)
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 175, in _ensure_no_port_binding_failure
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server     raise exception.PortBindingFailed(port_id=port['id'])
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server PortBindingFailed: Binding failed for port b3835f1c-e5f5-4347-89d5-3973eff5783d, please check neutron logs for more information.
2017-04-18 12:53:53.915 418852 ERROR oslo_messaging.rpc.server 
2017-04-18 12:53:53.917 418852 DEBUG oslo_messaging._drivers.amqpdriver [req-4d70e5e2-f864-423c-8704-9ab135430ee1 8b72d740fda446458318aa9e8efe0893 2dcfcc0c0cae470593de77e1d15bd570 - - -] sending reply msg_id: 33440a9eb6574919aa3f8caca787979f reply queue: reply_2f84f439b6e2493d9ed91e4877649984 time elapsed: 1.61449181591s _send_reply /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:73
~~~

From : neutron server logs. 

~~~
2017-04-18 12:53:53.829 63540 ERROR neutron.plugins.ml2.managers [req-1a6ef382-ad27-4492-a083-0983f8d7b6a2 ebd5e61315b347f88459c3fd81fb443a e7f6f6d944754ef0a1231c089cb912bd - - -] Failed to bind port b3835f1c-e5f5-4347-89d5-3973eff5783d on host overcloud-compute-1.localdomain for vnic_type direct using segments [{'segmentation_id': 172, 'physical_network': u'sriov', 'id': u'2093e6ee-b6b9-4b63-8194-200da4088649', 'network_type': u'vlan'}]
2017-04-18 12:53:53.830 63540 INFO neutron.plugins.ml2.plugin [req-1a6ef382-ad27-4492-a083-0983f8d7b6a2 ebd5e61315b347f88459c3fd81fb443a e7f6f6d944754ef0a1231c089cb912bd - - -] Attempt 10 to bind port b3835f1c-e5f5-4347-89d5-3973eff5783d
2017-04-18 12:53:53.840 63540 DEBUG neutron.plugins.ml2.managers [req-1a6ef382-ad27-4492-a083-0983f8d7b6a2 ebd5e61315b347f88459c3fd81fb443a e7f6f6d944754ef0a1231c089cb912bd - - -] Attempting to bind port b3835f1c-e5f5-4347-89d5-3973eff5783d on host overcloud-compute-1.localdomain for vnic_type direct with profile  bind_port /usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py:751
2017-04-18 12:53:53.841 63540 DEBUG neutron.plugins.ml2.managers [req-1a6ef382-ad27-4492-a083-0983f8d7b6a2 ebd5e61315b347f88459c3fd81fb443a e7f6f6d944754ef0a1231c089cb912bd - - -] Attempting to bind port b3835f1c-e5f5-4347-89d5-3973eff5783d on host overcloud-compute-1.localdomain at level 0 using segments [{'segmentation_id': 172, 'physical_network': u'sriov', 'id': u'2093e6ee-b6b9-4b63-8194-200da4088649', 'network_type': u'vlan'}] _bind_port_level /usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py:772
~~~

Comment 4 Assaf Muller 2017-04-24 12:29:56 UTC


*** This bug has been marked as a duplicate of bug 1444369 ***