Bug 1314478

Summary:	[OpenStack] [SRIOV] ERROR in VM provisioning
Product:	Red Hat OpenStack	Reporter:	Abdul Basit Alvi <Abdul_Alvi>
Component:	openstack-neutron	Assignee:	Assaf Muller <amuller>
Status:	CLOSED NOTABUG	QA Contact:	Toni Freger <tfreger>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	7.0 (Kilo)	CC:	amuller, beagles, chrisw, nyechiel, srevivo
Target Milestone:	---	Keywords:	ZStream
Target Release:	7.0 (Kilo)
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-09-27 13:04:47 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Abdul Basit Alvi 2016-03-03 17:08:00 UTC

Hi,
I am facing an issue with assigning SRIOV NICs to VMs in OpenStack Kilo environment deployed on Redhat 7.1.
I have deployed 3 control and two compute nodes in HA using Foreman. The compute nodes have Intel 82599 NICs .I have followed the following guide provided by RHEL to enable SR-IOV.
https://access.redhat.com/documentation/en/red-hat-enterprise-linux-openstack-platform/7/networking-guide/chapter-8-configure-sr-iov-networking
When I try to create a VM with SR-IOV NIC/Port the VM run into ERROR state. The error was similar when SRIOV ports were created on public and private networks in OpenStack. VMs created without SRIOV port are provisioned without any issue.
When a VM is provisioned with SRIOV NIC/Port, it runs into error state.The errors logged in neutron are given below this email.
Another peculiar behavior observed was, that changes in configuration files  (/etc/nova/nova.conf, /etc/neutron/plugin.ini, /usr/lib/systemd/system/neutron-server.service)  revert back to the default state when the services are restarted to apply changes. 
Am I doing something wrong? What steps do I need to overcome this issue? Please help.
2016-03-03 04:51:50.676 144203 ERROR oslo_messaging.rpc.dispatcher [req-8265cc48-717e-4204-a5d9-175f487acebc 56bc7b6399eb46359c2e8a7a363ec4c7 f0b29330575649f6bb95897279c429b7 - - -] Exception during message handling: PCI device request ({'requests': [InstancePCIRequest(alias_name=None,count=1,is_new=False,request_id=619304a8-9328-4c5b-914c-d624412be551,spec=[{physical_network=None}])], 'code': 500}equests)s failed
2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 142, in inner
2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher     return func(*args, **kwargs)
2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 86, in select_destinations
2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher     filter_properties)
2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 67, in select_destinations
2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher     filter_properties)
2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 163, in _schedule
2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher     chosen_host.obj.consume_from_instance(instance_properties)
2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/scheduler/host_manager.py", line 284, in consume_from_instance
2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher     self.pci_stats.apply_requests(pci_requests, instance_cells)
2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/pci/stats.py", line 234, in apply_requests
2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher     raise exception.PciDeviceRequestFailed(requests=requests)
2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher PciDeviceRequestFailed: PCI device request ({'requests': [InstancePCIRequest(alias_name=None,count=1,is_new=False,request_id=619304a8-9328-4c5b-914c-d624412be551,spec=[{physical_network=None}])], 'code': 500}equests)s failed
2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher
2016-03-03 04:51:50.677 144203 ERROR oslo_messaging._drivers.common [req-8265cc48-717e-4204-a5d9-175f487acebc 56bc7b6399eb46359c2e8a7a363ec4c7 f0b29330575649f6bb95897279c429b7 - - -] Returning exception PCI device request ({'requests': [InstancePCIRequest(alias_name=None,count=1,is_new=False,request_id=619304a8-9328-4c5b-914c-d624412be551,spec=[{physical_network=None}])], 'code': 500}equests)s failed to caller
2016-03-03 04:51:50.677 144203 ERROR oslo_messaging._drivers.common [req-8265cc48-717e-4204-a5d9-175f487acebc 56bc7b6399eb46359c2e8a7a363ec4c7 f0b29330575649f6bb95897279c429b7 - - -] ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n    executor_callback))\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n    executor_callback)\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n    result = func(ctxt, **new_args)\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 142, in inner\n    return func(*args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 86, in select_destinations\n    filter_properties)\n', '  File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 67, in select_destinations\n    filter_properties)\n', '  File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 163, in _schedule\n    chosen_host.obj.consume_from_instance(instance_properties)\n', '  File "/usr/lib/python2.7/site-packages/nova/scheduler/host_manager.py", line 284, in consume_from_instance\n    self.pci_stats.apply_requests(pci_requests, instance_cells)\n', '  File "/usr/lib/python2.7/site-packages/nova/pci/stats.py", line 234, in apply_requests\n    raise exception.PciDeviceRequestFailed(requests=requests)\n', "PciDeviceRequestFailed: PCI device request ({'requests': [InstancePCIRequest(alias_name=None,count=1,is_new=False,request_id=619304a8-9328-4c5b-914c-d624412be551,spec=[{physical_network=None}])], 'code': 500}equests)s failed\n"]

Comment 2 Assaf Muller 2016-06-05 23:52:29 UTC

@Brent, can you please help triage?

Comment 3 Brent Eagles 2016-06-06 12:33:32 UTC

The log file information provided indicates an issue in nova - possibly a misconfiguration.

That being said, if the PCI device configuration in Nova is not part of the Foreman generated configuration, it seems likely that any configuration is going to be overwritten by Foreman before it can be used.

Perhaps this should be move to the foreman installer component? @Mike, thoughts?

Comment 4 Assaf Muller 2016-06-06 20:25:59 UTC

(In reply to Brent Eagles from comment #3)
> The log file information provided indicates an issue in nova - possibly a
> misconfiguration.
> 
> That being said, if the PCI device configuration in Nova is not part of the
> Foreman generated configuration, it seems likely that any configuration is
> going to be overwritten by Foreman before it can be used.
> 
> Perhaps this should be move to the foreman installer component? @Mike,
> thoughts?

Was SRIOV claimed to be supported by Foreman? If not, I don't think it makes sense to add functionality.

Comment 5 Brent Eagles 2016-06-06 22:00:39 UTC

I'm not suggesting adding functionality. The reported error is consistent with there being a PCI white list misconfiguration on the nova side (PCI request failing). If the user has configured everything properly and foreman is overwriting the configs then it is a deployer related issue - even if it isn't supported and "not a bug".

Comment 6 Brent Eagles 2016-07-28 16:59:51 UTC

This got a little lost in the shuffle. In the end, I think this is really just "not a bug" in that SR-IOV with that installer is (AFAIK) not supported for that version of the installer. 

Thoughts?

Comment 7 Assaf Muller 2016-09-27 13:04:47 UTC

Foreman installer does not and will not support SRIOV and this looks like a configuration issue.