Hide Forgot
Created attachment 1200757 [details] dump of conf and logs Description of problem: SR-IOV functionality stooped working via neutron Version-Release number of selected component (if applicable): How reproducible: [ml2] type_drivers = vxlan,vlan tenant_network_types = vxlan,vlan mechanism_drivers =openvswitch,sriovnicswitch configure ml2_sriov in ml2_conf.ini with the following paramas [ml2_sriov] supported_pci_vendor_devs = 8086:10ed agent_required = false configure nova.conf with the following params pci_passthrough_whitelist = {"devname":"enp4s0f0","physical_network":"physnet-sriov"} Steps to Reproduce: 1. neutron net-create SRIOV --provider:network_type vlan --provider:physical_network physnet-sriov 2. neutron subnet-create SRIOV 1.1.1.0/24 --name SRIOV-SUB 3. neutron port-create --name SRIOV-PORT1 --vnic-type direct 0ce0411c-76ae-48ef-a035-d003a3dc3c03 4. nova boot --poll --flavor m1.small --image c94fc007-c017-4ff2-85ba-a03f7968c8c4 --nic port-id=e5729cd8-dc77-4ce8-8a37-792e8b07207c SRIOV Actual results: build of instance faild error in neutron server.log: 2016-09-14 10:51:54.714 12924 ERROR neutron.plugins.ml2.managers [req-71e7e5b7-7f70-4558-911b-e97bb84910a2 - - - - -] Failed to bind port ab23bde4-45da-4d7b-83ca-2087d84acd7e on host smq-187.lab.contextream.local for vnic_type normal using segments [{'segmentation_id': 110, 'physical_network': u'physnet-sriov', 'id': u'e58e0560-dadf-4fa0-b521-82fc22040a72', 'network_type': u'vlan'}] 2016-09-14 10:51:54.714 12924 INFO neutron.plugins.ml2.plugin [req-71e7e5b7-7f70-4558-911b-e97bb84910a2 - - - - -] Attempt 2 to bind port ab23bde4-45da-4d7b-83ca-2087d84acd7e 2016-09-14 10:51:54.721 12924 ERROR neutron.plugins.ml2.managers [req-71e7e5b7-7f70-4558-911b-e97bb84910a2 - - - - -] Failed to bind port ab23bde4-45da-4d7b-83ca-2087d84acd7e on host smq-187.lab.contextream.local for vnic_type normal using segments [{'segmentation_id': 110, 'physical_network': u'physnet-sriov', 'id': u'e58e0560-dadf-4fa0-b521-82fc22040a72', 'network_type': u'vlan'}] 2016-09-14 10:51:54.722 12924 INFO neutron.plugins.ml2.plugin [req-71e7e5b7-7f70-4558-911b-e97bb84910a2 - - - - -] Attempt 3 to bind port ab23bde4-45da-4d7b-83ca-2087d84acd7e 2016-09-14 10:51:54.728 12924 ERROR neutron.plugins.ml2.managers [req-71e7e5b7-7f70-4558-911b-e97bb84910a2 - - - - -] Failed to bind port ab23bde4-45da-4d7b-83ca-2087d84acd7e on host smq-187.lab.contextream.local for vnic_type normal using segments [{'segmentation_id': 110, 'physical_network': u'physnet-sriov', 'id': u'e58e0560-dadf-4fa0-b521-82fc22040a72', 'network_type': u'vlan'}] 2016-09-14 10:51:54.729 12924 INFO neutron.plugins.ml2.plugin [req-71e7e5b7-7f70-4558-911b-e97bb84910a2 - - - - -] Attempt 4 to bind port ab23bde4-45da-4d7b-83ca-2087d84acd7e 2016-09-14 10:51:54.735 12924 ERROR neutron.plugins.ml2.managers [req-71e7e5b7-7f70-4558-911b-e97bb84910a2 - - - - -] Failed to bind port ab23bde4-45da-4d7b-83ca-2087d84acd7e on host smq-187.lab.contextream.local for vnic_type normal using segments [{'segmentation_id': 110, 'physical_network': u'physnet-sriov', 'id': u'e58e0560-dadf-4fa0-b521-82fc22040a72', 'network_type': u'vlan'}] 2016-09-14 10:51:54.736 12924 INFO neutron.plugins.ml2.plugin [req-71e7e5b7-7f70-4558-911b-e97bb84910a2 - - - - -] Attempt 5 to bind port ab23bde4-45da-4d7b-83ca-2087d84acd7e 2016-09-14 10:51:54.742 12924 ERROR neutron.plugins.ml2.managers [req-71e7e5b7-7f70-4558-911b-e97bb84910a2 - - - - -] Failed to bind port ab23bde4-45da-4d7b-83ca-2087d84acd7e on host smq-187.lab.contextream.local for vnic_type normal using segments [{'segmentation_id': 110, 'physical_network': u'physnet-sriov', 'id': u'e58e0560-dadf-4fa0-b521-82fc22040a72', 'network_type': u'vlan'}] 2016-09-14 10:51:54.743 12924 INFO neutron.plugins.ml2.plugin [req-71e7e5b7-7f70-4558-911b-e97bb84910a2 - - - - -] Attempt 6 to bind port ab23bde4-45da-4d7b-83ca-2087d84acd7e 2016-09-14 10:51:54.749 12924 ERROR neutron.plugins.ml2.managers [req-71e7e5b7-7f70-4558-911b-e97bb84910a2 - - - - -] Failed to bind port ab23bde4-45da-4d7b-83ca-2087d84acd7e on host smq-187.lab.contextream.local for vnic_type normal using segments [{'segmentation_id': 110, 'physical_network': u'physnet-sriov', 'id': u'e58e0560-dadf-4fa0-b521-82fc22040a72', 'network_type': u'vlan'}] 2016-09-14 10:51:54.750 12924 INFO neutron.plugins.ml2.plugin [req-71e7e5b7-7f70-4558-911b-e97bb84910a2 - - - - -] Attempt 7 to bind port ab23bde4-45da-4d7b-83ca-2087d84acd7e 2016-09-14 10:51:54.756 12924 ERROR neutron.plugins.ml2.managers [req-71e7e5b7-7f70-4558-911b-e97bb84910a2 - - - - -] Failed to bind port ab23bde4-45da-4d7b-83ca-2087d84acd7e on host smq-187.lab.contextream.local for vnic_type normal using segments [{'segmentation_id': 110, 'physical_network': u'physnet-sriov', 'id': u'e58e0560-dadf-4fa0-b521-82fc22040a72', 'network_type': u'vlan'}] 2016-09-14 10:51:54.757 12924 INFO neutron.plugins.ml2.plugin [req-71e7e5b7-7f70-4558-911b-e97bb84910a2 - - - - -] Attempt 8 to bind port ab23bde4-45da-4d7b-83ca-2087d84acd7e 2016-09-14 10:51:54.763 12924 ERROR neutron.plugins.ml2.managers [req-71e7e5b7-7f70-4558-911b-e97bb84910a2 - - - - -] Failed to bind port ab23bde4-45da-4d7b-83ca-2087d84acd7e on host smq-187.lab.contextream.local for vnic_type normal using segments [{'segmentation_id': 110, 'physical_network': u'physnet-sriov', 'id': u'e58e0560-dadf-4fa0-b521-82fc22040a72', 'network_type': u'vlan'}] 2016-09-14 10:51:54.764 12924 INFO neutron.plugins.ml2.plugin [req-71e7e5b7-7f70-4558-911b-e97bb84910a2 - - - - -] Attempt 9 to bind port ab23bde4-45da-4d7b-83ca-2087d84acd7e 2016-09-14 10:51:54.770 12924 ERROR neutron.plugins.ml2.managers [req-71e7e5b7-7f70-4558-911b-e97bb84910a2 - - - - -] Failed to bind port ab23bde4-45da-4d7b-83ca-2087d84acd7e on host smq-187.lab.contextream.local for vnic_type normal using segments [{'segmentation_id': 110, 'physical_network': u'physnet-sriov', 'id': u'e58e0560-dadf-4fa0-b521-82fc22040a72', 'network_type': u'vlan'}] 2016-09-14 10:51:54.770 12924 INFO neutron.plugins.ml2.plugin [req-71e7e5b7-7f70-4558-911b-e97bb84910a2 - - - - -] Attempt 10 to bind port ab23bde4-45da-4d7b-83ca-2087d84acd7e 2016-09-14 10:51:54.777 12924 ERROR neutron.plugins.ml2.managers [req-71e7e5b7-7f70-4558-911b-e97bb84910a2 - - - - -] Failed to bind port ab23bde4-45da-4d7b-83ca-2087d84acd7e on host smq-187.lab.contextream.local for vnic_type normal using segments [{'segmentation_id': 110, 'physical_network': u'physnet-sriov', 'id': u'e58e0560-dadf-4fa0-b521-82fc22040a72', 'network_type': u'vlan'}] 2016-09-14 10:51:56.440 12923 ERROR neutron.plugins.ml2.managers [req-e981126b-8a5b-4271-b0d8-08860e9c8ae9 - - - - -] Failed to bind port ab23bde4-45da-4d7b-83ca-2087d84acd7e on host smq-187.lab.contextream.local for vnic_type normal using segments [{'segmentation_id': 110, 'physical_network': u'physnet-sriov', 'id': u'e58e0560-dadf-4fa0-b521-82fc22040a72', 'network_type': u'vlan'}] error in mova-compute.log 2016-09-14 10:59:35.578 19009 ERROR nova.compute.manager [req-73eab91f-f7fe-45d0-ac16-9b2e77c8e679 218aa013ecfe4e0e943da0f6b867723e f75ac3d1ac1f43f6b1391fd7a39f52a4 - - -] [instance: d857ec13-a0b2-4368-9525-1a14f7cd2128] Instance failed to spawn 2016-09-14 10:59:35.578 19009 ERROR nova.compute.manager [instance: d857ec13-a0b2-4368-9525-1a14f7cd2128] Traceback (most recent call last): 2016-09-14 10:59:35.578 19009 ERROR nova.compute.manager [instance: d857ec13-a0b2-4368-9525-1a14f7cd2128] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2218, in _build_resources 2016-09-14 10:59:35.578 19009 ERROR nova.compute.manager [instance: d857ec13-a0b2-4368-9525-1a14f7cd2128] yield resources 2016-09-14 10:59:35.578 19009 ERROR nova.compute.manager [instance: d857ec13-a0b2-4368-9525-1a14f7cd2128] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2064, in _build_and_run_instance 2016-09-14 10:59:35.578 19009 ERROR nova.compute.manager [instance: d857ec13-a0b2-4368-9525-1a14f7cd2128] block_device_info=block_device_info) 2016-09-14 10:59:35.578 19009 ERROR nova.compute.manager [instance: d857ec13-a0b2-4368-9525-1a14f7cd2128] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2777, in spawn 2016-09-14 10:59:35.578 19009 ERROR nova.compute.manager [instance: d857ec13-a0b2-4368-9525-1a14f7cd2128] write_to_disk=True) 2016-09-14 10:59:35.578 19009 ERROR nova.compute.manager [instance: d857ec13-a0b2-4368-9525-1a14f7cd2128] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4730, in _get_guest_xml 2016-09-14 10:59:35.578 19009 ERROR nova.compute.manager [instance: d857ec13-a0b2-4368-9525-1a14f7cd2128] context) 2016-09-14 10:59:35.578 19009 ERROR nova.compute.manager [instance: d857ec13-a0b2-4368-9525-1a14f7cd2128] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4596, in _get_guest_config 2016-09-14 10:59:35.578 19009 ERROR nova.compute.manager [instance: d857ec13-a0b2-4368-9525-1a14f7cd2128] flavor, virt_type, self._host) 2016-09-14 10:59:35.578 19009 ERROR nova.compute.manager [instance: d857ec13-a0b2-4368-9525-1a14f7cd2128] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 447, in get_config 2016-09-14 10:59:35.578 19009 ERROR nova.compute.manager [instance: d857ec13-a0b2-4368-9525-1a14f7cd2128] _("Unexpected vif_type=%s") % vif_type) 2016-09-14 10:59:35.578 19009 ERROR nova.compute.manager [instance: d857ec13-a0b2-4368-9525-1a14f7cd2128] NovaException: Unexpected vif_type=binding_failed 2016-09-14 10:59:35.578 19009 ERROR nova.compute.manager [instance: d857ec13-a0b2-4368-9525-1a14f7cd2128] 2016-09-14 10:59:35.581 19009 INFO nova.compute.manager [req-73eab91f-f7fe-45d0-ac16-9b2e77c8e679 218aa013ecfe4e0e943da0f6b867723e f75ac3d1ac1f43f6b1391fd7a39f52a4 - - -] [instance: d857ec13-a0b2-4368-9525-1a14f7cd2128] Terminating insta nce Expected results: build of instance success Additional info:
From the Neutron logging it looks like it's trying to bind a port of vnic_type normal and not direct. Does that look right to you?
No, I added the the neutron conf file, I tried the conf few times maybe typo but hard to believe, the same conf worked for me in kilo / liberty
[root@smq-187 ~(keystone_admin)]# neutron port-show e5729cd8-dc77-4ce8-8a37-792e8b07207c +-----------------------+--------------------------------------------------------------------------------+ | Field | Value | +-----------------------+--------------------------------------------------------------------------------+ | admin_state_up | True | | allowed_address_pairs | | | binding:host_id | | | binding:profile | {} | | binding:vif_details | {} | | binding:vif_type | unbound | | binding:vnic_type | direct | | created_at | 2016-09-14T07:52:10 | | description | | | device_id | | | device_owner | | | dns_name | | | extra_dhcp_opts | | | fixed_ips | {"subnet_id": "13df742e-9090-4c0e-8dd0-c890aa9c9eaa", "ip_address": "1.1.1.3"} | | id | e5729cd8-dc77-4ce8-8a37-792e8b07207c | | mac_address | fa:16:3e:72:1f:46 | | name | SRIOV-PORT1 | | network_id | 0ce0411c-76ae-48ef-a035-d003a3dc3c03 | | security_groups | d8d4596a-7f06-470c-bbdd-6f8725be7edf | | status | DOWN | | tenant_id | f75ac3d1ac1f43f6b1391fd7a39f52a4 | | updated_at | 2016-09-14T08:34:49 | +-----------------------+--------------------------------------------------------------------------------+ [root@smq-187 ~(keystone_admin)]#
Is the sr-iov agent running on the compute node?
No, there is no sriov agent in my topology, I'm using nova and neutron without sriov-agent since Im using intel ixgbe cards and dont need support in QOS or link status Thanks,
Running the agent isn't optional any longer. IIRC, the SR-IOV mechanism driver still uses information associated with the agent. I suspect that is why you are getting port binding failures.
Thanks, I got over this guide again http://docs.openstack.org/mitaka/networking-guide/config-sriov.html and found that the sriov agent is mandatory... but still failed to spawn even with agent up and running neutron agent-list +------------------------+--------------------+------------------------+-------------------+-------+----------------+------------------------+ | id | agent_type | host | availability_zone | alive | admin_state_up | binary | +------------------------+--------------------+------------------------+-------------------+-------+----------------+------------------------+ | 67b78385-432d-44e8 | L3 agent | smq-187.lab.contextrea | nova | :-) | True | neutron-l3-agent | | -b70a-9f213f4d1a8e | | m.local | | | | | | 9d1ee73f-5a97-4a69-8ad | Open vSwitch agent | smq-187.lab.contextrea | | :-) | True | neutron-openvswitch- | | 0-b1960cadf7a9 | | m.local | | | | agent | | adcabbc6-a04f-4f3f- | DHCP agent | smq-187.lab.contextrea | nova | :-) | True | neutron-dhcp-agent | | b1e4-06083d7ca70e | | m.local | | | | | | b3fcac1d-7e28-4b00-bd0 | Metering agent | smq-187.lab.contextrea | | :-) | True | neutron-metering-agent | | 1-a4d521c0df54 | | m.local | | | | | | d72e095c-1c63-47fa-a00 | NIC Switch agent | smq-187.lab.contextrea | | :-) | True | neutron-sriov-nic- | | 4-30bbe23dab64 | | m.local | | | | agent | | e5dad51b-bb45-4c43 | Metadata agent | smq-187.lab.contextrea | | :-) | True | neutron-metadata-agent | | -a0ac-d4ae9443f2b5 | | m.local | | | | | +------------------------+--------------------+------------------------+-------------------+-------+----------------+------------------------+
(In reply to alon from comment #7) > Thanks, > I got over this guide again > http://docs.openstack.org/mitaka/networking-guide/config-sriov.html > and found that the sriov agent is mandatory... > but still failed to spawn even with agent up and running > neutron agent-list > +------------------------+--------------------+------------------------+----- > --------------+-------+----------------+------------------------+ > | id | agent_type | host | > availability_zone | alive | admin_state_up | binary | > +------------------------+--------------------+------------------------+----- > --------------+-------+----------------+------------------------+ > | 67b78385-432d-44e8 | L3 agent | smq-187.lab.contextrea | > nova | :-) | True | neutron-l3-agent | > | -b70a-9f213f4d1a8e | | m.local | > | | | | > | 9d1ee73f-5a97-4a69-8ad | Open vSwitch agent | smq-187.lab.contextrea | > | :-) | True | neutron-openvswitch- | > | 0-b1960cadf7a9 | | m.local | > | | | agent | > | adcabbc6-a04f-4f3f- | DHCP agent | smq-187.lab.contextrea | > nova | :-) | True | neutron-dhcp-agent | > | b1e4-06083d7ca70e | | m.local | > | | | | > | b3fcac1d-7e28-4b00-bd0 | Metering agent | smq-187.lab.contextrea | > | :-) | True | neutron-metering-agent | > | 1-a4d521c0df54 | | m.local | > | | | | > | d72e095c-1c63-47fa-a00 | NIC Switch agent | smq-187.lab.contextrea | > | :-) | True | neutron-sriov-nic- | > | 4-30bbe23dab64 | | m.local | > | | | agent | > | e5dad51b-bb45-4c43 | Metadata agent | smq-187.lab.contextrea | > | :-) | True | neutron-metadata-agent | > | -a0ac-d4ae9443f2b5 | | m.local | > | | | | > +------------------------+--------------------+------------------------+----- > --------------+-------+----------------+------------------------+ [root@smq-187 ~(keystone_admin)]# grep -irn physnet-sriov /etc/neutron/ /etc/neutron/conf.d/neutron-sriov-nic-agent/sriov_agent.ini:120:physical_device_mappings = physnet-sriov:enp4s0f0 /etc/neutron/plugins/ml2/ml2_conf.ini:200:network_vlan_ranges = physnet-sriov:100:110,physnet1:100:110
(In reply to alon from comment #8) > (In reply to alon from comment #7) > > Thanks, > > I got over this guide again > > http://docs.openstack.org/mitaka/networking-guide/config-sriov.html > > and found that the sriov agent is mandatory... > > but still failed to spawn even with agent up and running > > neutron agent-list > > +------------------------+--------------------+------------------------+----- > > --------------+-------+----------------+------------------------+ > > | id | agent_type | host | > > availability_zone | alive | admin_state_up | binary | > > +------------------------+--------------------+------------------------+----- > > --------------+-------+----------------+------------------------+ > > | 67b78385-432d-44e8 | L3 agent | smq-187.lab.contextrea | > > nova | :-) | True | neutron-l3-agent | > > | -b70a-9f213f4d1a8e | | m.local | > > | | | | > > | 9d1ee73f-5a97-4a69-8ad | Open vSwitch agent | smq-187.lab.contextrea | > > | :-) | True | neutron-openvswitch- | > > | 0-b1960cadf7a9 | | m.local | > > | | | agent | > > | adcabbc6-a04f-4f3f- | DHCP agent | smq-187.lab.contextrea | > > nova | :-) | True | neutron-dhcp-agent | > > | b1e4-06083d7ca70e | | m.local | > > | | | | > > | b3fcac1d-7e28-4b00-bd0 | Metering agent | smq-187.lab.contextrea | > > | :-) | True | neutron-metering-agent | > > | 1-a4d521c0df54 | | m.local | > > | | | | > > | d72e095c-1c63-47fa-a00 | NIC Switch agent | smq-187.lab.contextrea | > > | :-) | True | neutron-sriov-nic- | > > | 4-30bbe23dab64 | | m.local | > > | | | agent | > > | e5dad51b-bb45-4c43 | Metadata agent | smq-187.lab.contextrea | > > | :-) | True | neutron-metadata-agent | > > | -a0ac-d4ae9443f2b5 | | m.local | > > | | | | > > +------------------------+--------------------+------------------------+----- > > --------------+-------+----------------+------------------------+ > > [root@smq-187 ~(keystone_admin)]# grep -irn physnet-sriov /etc/neutron/ > /etc/neutron/conf.d/neutron-sriov-nic-agent/sriov_agent.ini:120: > physical_device_mappings = physnet-sriov:enp4s0f0 > /etc/neutron/plugins/ml2/ml2_conf.ini:200:network_vlan_ranges = > physnet-sriov:100:110,physnet1:100:110 Now that you have the agent up and running, can you enable DEBUG logging on your neutron-server process, try to spawn a VM again and post the log?
Created attachment 1201150 [details] dump of logs
OK, found my typo in the /etc/neutron/plugins/ml2/sriov_agent.ini conf file there is some minor issue with this file.. this configuration line "physical_device_mappings" is under [ml2_sriov] instead [sriov_nic] so if you dont adding [sriov_nic] manually to the file and still configure the "physical_device_mappings" the conf will not take affect