Hide Forgot
Hi, I am facing an issue with assigning SRIOV NICs to VMs in OpenStack Kilo environment deployed on Redhat 7.1. I have deployed 3 control and two compute nodes in HA using Foreman. The compute nodes have Intel 82599 NICs .I have followed the following guide provided by RHEL to enable SR-IOV. https://access.redhat.com/documentation/en/red-hat-enterprise-linux-openstack-platform/7/networking-guide/chapter-8-configure-sr-iov-networking When I try to create a VM with SR-IOV NIC/Port the VM run into ERROR state. The error was similar when SRIOV ports were created on public and private networks in OpenStack. VMs created without SRIOV port are provisioned without any issue. When a VM is provisioned with SRIOV NIC/Port, it runs into error state.The errors logged in neutron are given below this email. Another peculiar behavior observed was, that changes in configuration files (/etc/nova/nova.conf, /etc/neutron/plugin.ini, /usr/lib/systemd/system/neutron-server.service) revert back to the default state when the services are restarted to apply changes. Am I doing something wrong? What steps do I need to overcome this issue? Please help. 2016-03-03 04:51:50.676 144203 ERROR oslo_messaging.rpc.dispatcher [req-8265cc48-717e-4204-a5d9-175f487acebc 56bc7b6399eb46359c2e8a7a363ec4c7 f0b29330575649f6bb95897279c429b7 - - -] Exception during message handling: PCI device request ({'requests': [InstancePCIRequest(alias_name=None,count=1,is_new=False,request_id=619304a8-9328-4c5b-914c-d624412be551,spec=[{physical_network=None}])], 'code': 500}equests)s failed 2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last): 2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply 2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher executor_callback)) 2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch 2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher executor_callback) 2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch 2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args) 2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 142, in inner 2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher return func(*args, **kwargs) 2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 86, in select_destinations 2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher filter_properties) 2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 67, in select_destinations 2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher filter_properties) 2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 163, in _schedule 2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher chosen_host.obj.consume_from_instance(instance_properties) 2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/scheduler/host_manager.py", line 284, in consume_from_instance 2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher self.pci_stats.apply_requests(pci_requests, instance_cells) 2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/pci/stats.py", line 234, in apply_requests 2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher raise exception.PciDeviceRequestFailed(requests=requests) 2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher PciDeviceRequestFailed: PCI device request ({'requests': [InstancePCIRequest(alias_name=None,count=1,is_new=False,request_id=619304a8-9328-4c5b-914c-d624412be551,spec=[{physical_network=None}])], 'code': 500}equests)s failed 2016-03-03 04:51:50.676 144203 TRACE oslo_messaging.rpc.dispatcher 2016-03-03 04:51:50.677 144203 ERROR oslo_messaging._drivers.common [req-8265cc48-717e-4204-a5d9-175f487acebc 56bc7b6399eb46359c2e8a7a363ec4c7 f0b29330575649f6bb95897279c429b7 - - -] Returning exception PCI device request ({'requests': [InstancePCIRequest(alias_name=None,count=1,is_new=False,request_id=619304a8-9328-4c5b-914c-d624412be551,spec=[{physical_network=None}])], 'code': 500}equests)s failed to caller 2016-03-03 04:51:50.677 144203 ERROR oslo_messaging._drivers.common [req-8265cc48-717e-4204-a5d9-175f487acebc 56bc7b6399eb46359c2e8a7a363ec4c7 f0b29330575649f6bb95897279c429b7 - - -] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n executor_callback))\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n executor_callback)\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 142, in inner\n return func(*args, **kwargs)\n', ' File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 86, in select_destinations\n filter_properties)\n', ' File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 67, in select_destinations\n filter_properties)\n', ' File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 163, in _schedule\n chosen_host.obj.consume_from_instance(instance_properties)\n', ' File "/usr/lib/python2.7/site-packages/nova/scheduler/host_manager.py", line 284, in consume_from_instance\n self.pci_stats.apply_requests(pci_requests, instance_cells)\n', ' File "/usr/lib/python2.7/site-packages/nova/pci/stats.py", line 234, in apply_requests\n raise exception.PciDeviceRequestFailed(requests=requests)\n', "PciDeviceRequestFailed: PCI device request ({'requests': [InstancePCIRequest(alias_name=None,count=1,is_new=False,request_id=619304a8-9328-4c5b-914c-d624412be551,spec=[{physical_network=None}])], 'code': 500}equests)s failed\n"]
@Brent, can you please help triage?
The log file information provided indicates an issue in nova - possibly a misconfiguration. That being said, if the PCI device configuration in Nova is not part of the Foreman generated configuration, it seems likely that any configuration is going to be overwritten by Foreman before it can be used. Perhaps this should be move to the foreman installer component? @Mike, thoughts?
(In reply to Brent Eagles from comment #3) > The log file information provided indicates an issue in nova - possibly a > misconfiguration. > > That being said, if the PCI device configuration in Nova is not part of the > Foreman generated configuration, it seems likely that any configuration is > going to be overwritten by Foreman before it can be used. > > Perhaps this should be move to the foreman installer component? @Mike, > thoughts? Was SRIOV claimed to be supported by Foreman? If not, I don't think it makes sense to add functionality.
I'm not suggesting adding functionality. The reported error is consistent with there being a PCI white list misconfiguration on the nova side (PCI request failing). If the user has configured everything properly and foreman is overwriting the configs then it is a deployer related issue - even if it isn't supported and "not a bug".
This got a little lost in the shuffle. In the end, I think this is really just "not a bug" in that SR-IOV with that installer is (AFAIK) not supported for that version of the installer. Thoughts?
Foreman installer does not and will not support SRIOV and this looks like a configuration issue.