Hide Forgot
Description of problem: When attempting to leverage nova's trusted VF capabilities (as per nova's spec https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/sriov-trusted-vfs.html), port creation has to be done as admin user but not all cloud operators have admin privileges. Even if the port is created as admin user on a different tenant where the cloud operator user exists, guest will be spawned without custom binding:profile. (overcloud-Admin) [stack@undercloud-0 ~]$ openstack port create --vnic-type direct --project tempest-TestNfvBasic-909024731 --binding-profile trusted=true --network sriovone port100-sriov +-----------------------+------------------------------------------------------------------------------+ | Field | Value | +-----------------------+------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | | | binding_profile | trusted='true' | | binding_vif_details | | | binding_vif_type | unbound | | binding_vnic_type | direct | | created_at | 2018-11-22T10:27:50Z | | data_plane_status | None | | description | | | device_id | | | device_owner | | | dns_assignment | None | | dns_domain | None | | dns_name | None | | extra_dhcp_opts | | | fixed_ips | ip_address='60.20.135.101', subnet_id='50c6bd91-7b1b-4c8d-a27b-f85663a5dc49' | | id | 17a9d426-1bda-47df-8d6f-2e3f6457c039 | | mac_address | fa:16:3e:6c:72:42 | | name | port100-sriov | | network_id | a03c1203-5af6-4377-942b-e5a30a1339c9 | | port_security_enabled | False | | project_id | 0540ed01fb134f28b12c7e024f5d72bd | | qos_policy_id | None | | revision_number | 2 | | security_group_ids | | | status | DOWN | | tags | | | trunk_details | None | | updated_at | 2018-11-22T10:27:51Z | +-----------------------+------------------------------------------------------------------------------+ Viewing the port as admin user will show the binding:profile: (overcloud-Admin) [stack@undercloud-0 ~]$ openstack port show 17a9d426-1bda-47df-8d6f-2e3f6457c039 +-----------------------+------------------------------------------------------------------------------+ | Field | Value | +-----------------------+------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | | | binding_profile | trusted='true' | | binding_vif_details | | | binding_vif_type | unbound | | binding_vnic_type | direct | | created_at | 2018-11-22T10:27:50Z | | data_plane_status | None | | description | | | device_id | | | device_owner | | | dns_assignment | None | | dns_domain | None | | dns_name | None | | extra_dhcp_opts | | | fixed_ips | ip_address='60.20.135.101', subnet_id='50c6bd91-7b1b-4c8d-a27b-f85663a5dc49' | | id | 17a9d426-1bda-47df-8d6f-2e3f6457c039 | | mac_address | fa:16:3e:6c:72:42 | | name | port100-sriov | | network_id | a03c1203-5af6-4377-942b-e5a30a1339c9 | | port_security_enabled | False | | project_id | 0540ed01fb134f28b12c7e024f5d72bd | | qos_policy_id | None | | revision_number | 2 | | security_group_ids | | | status | DOWN | | tags | | | trunk_details | None | | updated_at | 2018-11-22T10:27:51Z | +-----------------------+------------------------------------------------------------------------------+ Viewing the port as a non admin user will show 'None' in binding:profile: (overcloud-demo) [stack@undercloud-0 ~]$ openstack port show 17a9d426-1bda-47df-8d6f-2e3f6457c039 +-----------------------+------------------------------------------------------------------------------+ | Field | Value | +-----------------------+------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | None | | binding_profile | None | | binding_vif_details | None | | binding_vif_type | None | | binding_vnic_type | direct | | created_at | 2018-11-22T10:27:50Z | | data_plane_status | None | | description | | | device_id | | | device_owner | | | dns_assignment | None | | dns_domain | None | | dns_name | None | | extra_dhcp_opts | | | fixed_ips | ip_address='60.20.135.101', subnet_id='50c6bd91-7b1b-4c8d-a27b-f85663a5dc49' | | id | 17a9d426-1bda-47df-8d6f-2e3f6457c039 | | mac_address | fa:16:3e:6c:72:42 | | name | port100-sriov | | network_id | a03c1203-5af6-4377-942b-e5a30a1339c9 | | port_security_enabled | False | | project_id | 0540ed01fb134f28b12c7e024f5d72bd | | qos_policy_id | None | | revision_number | 2 | | security_group_ids | | | status | DOWN | | tags | | | trunk_details | None | | updated_at | 2018-11-22T10:27:51Z | +-----------------------+------------------------------------------------------------------------------+ Spawning the server as non admin user with the port: (overcloud-demo) [stack@undercloud-0 ~]$ openstack server create --image rhel-guest-image-7.5-192.x86_64.qcow2 --flavor m1.medium.huge_pages_cpu_pinning_numa_node-0 --nic port-id=17a9d426-1bda-47df-8d6f-2e3f6457c039 TrustedServer (overcloud-demo) [stack@undercloud-0 ~]$ openstack server list +--------------------------------------+---------------+--------+------------------------+---------------------------------------+----------------------------------------------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+---------------+--------+------------------------+---------------------------------------+----------------------------------------------+ | f0c6c53a-eb85-4460-883f-d799e5df8663 | TrustedServer | ACTIVE | sriovone=60.20.135.101 | rhel-guest-image-7.5-192.x86_64.qcow2 | m1.medium.huge_pages_cpu_pinning_numa_node-0 | +--------------------------------------+---------------+--------+------------------------+---------------------------------------+----------------------------------------------+ Viewing port as a non-admin: (overcloud-demo) [stack@undercloud-0 ~]$ openstack port show 17a9d426-1bda-47df-8d6f-2e3f6457c039 +-----------------------+------------------------------------------------------------------------------+ | Field | Value | +-----------------------+------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | None | | binding_profile | None | | binding_vif_details | None | | binding_vif_type | None | | binding_vnic_type | direct | | created_at | 2018-11-22T10:27:50Z | | data_plane_status | None | | description | | | device_id | f0c6c53a-eb85-4460-883f-d799e5df8663 | | device_owner | compute:nova | | dns_assignment | None | | dns_domain | None | | dns_name | None | | extra_dhcp_opts | | | fixed_ips | ip_address='60.20.135.101', subnet_id='50c6bd91-7b1b-4c8d-a27b-f85663a5dc49' | | id | 17a9d426-1bda-47df-8d6f-2e3f6457c039 | | mac_address | fa:16:3e:6c:72:42 | | name | port100-sriov | | network_id | a03c1203-5af6-4377-942b-e5a30a1339c9 | | port_security_enabled | False | | project_id | 0540ed01fb134f28b12c7e024f5d72bd | | qos_policy_id | None | | revision_number | 20 | | security_group_ids | | | status | ACTIVE | | tags | | | trunk_details | None | | updated_at | 2018-11-22T10:49:52Z | +-----------------------+------------------------------------------------------------------------------+ Viewing port as an admin: (overcloud-Admin) [stack@undercloud-0 ~]$ openstack port show 17a9d426-1bda-47df-8d6f-2e3f6457c039 +-----------------------+----------------------------------------------------------------------------------+ | Field | Value | +-----------------------+----------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | overcloud-computeovsdpdksriov-1.localdomain | | binding_profile | pci_slot='0000:05:0a.4', pci_vendor_info='8086:154c', physical_network='sriov-1' | | binding_vif_details | port_filter='False', vlan='600' | | binding_vif_type | hw_veb | | binding_vnic_type | direct | | created_at | 2018-11-22T10:27:50Z | | data_plane_status | None | | description | | | device_id | f0c6c53a-eb85-4460-883f-d799e5df8663 | | device_owner | compute:nova | | dns_assignment | None | | dns_domain | None | | dns_name | None | | extra_dhcp_opts | | | fixed_ips | ip_address='60.20.135.101', subnet_id='50c6bd91-7b1b-4c8d-a27b-f85663a5dc49' | | id | 17a9d426-1bda-47df-8d6f-2e3f6457c039 | | mac_address | fa:16:3e:6c:72:42 | | name | port100-sriov | | network_id | a03c1203-5af6-4377-942b-e5a30a1339c9 | | port_security_enabled | False | | project_id | 0540ed01fb134f28b12c7e024f5d72bd | | qos_policy_id | None | | revision_number | 20 | | security_group_ids | | | status | ACTIVE | | tags | | | trunk_details | None | | updated_at | 2018-11-22T10:49:52Z | +-----------------------+----------------------------------------------------------------------------------+ Verify that no VFs are trusted on hypervisor: [root@overcloud-computeovsdpdksriov-1 ~]# ip link | grep trust vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off vf 3 MAC 62:8a:16:26:62:65, spoof checking off, link-state enable, trust off vf 4 MAC fa:16:3e:6c:72:42, vlan 600, spoof checking off, link-state enable, trust off vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off vf 3 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off vf 4 MAC c2:29:d2:e5:34:aa, spoof checking off, link-state enable, trust off Version-Release number of selected component (if applicable): Encountered in in all OSP14 puddles so far How reproducible: Always Steps to Reproduce: 1. Create tenant with non admin user 2. Create port with custom binding:profile as admin user on the non admin tenant 3. Spawn guest instance as non admin user with the admin created port Actual results: Custom binding:profile is omitted Expected results: Custom binding:profile is kept Additional info:
Hello, thanks for the bug report! Port binding extended attributes are admin-only (both setting and getting) [1]. In addition, trusted mode on the devices is set by the libvirt driver when plugging or unplugging, so unless an instance is actually using a trusted VF, no trusted devices will show up on the host. Because of that, I think this is all working as expected. If I've misunderstood something, by all means reopen this bug. [1] https://developer.openstack.org/api-ref/network/v2/index.html#port-binding-extended-attributes
(In reply to Artom Lifshitz from comment #1) > Hello, thanks for the bug report! > > Port binding extended attributes are admin-only (both setting and getting) > [1]. In addition, trusted mode on the devices is set by the libvirt driver > when plugging or unplugging, so unless an instance is actually using a > trusted VF, no trusted devices will show up on the host. Err, apologies, I completely missed that an instance had in fact been created. The fact that no interfaces show up as trusted on the host is indeed weird. I assume passthrough_whitelist has been configured correctly? Would it be possible to reproduce the bug with debug enabled and attach nova logs from the compute host to this BZ? Thanks!
(In reply to Artom Lifshitz from comment #2) > Err, apologies, I completely missed that an instance had in fact been > created. The fact that no interfaces show up as trusted on the host is > indeed weird. I discussed this with a colleague, and without hardware capable of trusted VFs we're not entirely sure whether a VF that has been passed through to a guest would still show up on the host - after all, once passed through, it is the guest's kernel that's managing it, not the host. So it could be that what you're seeing is expected. The quickest way to make the trusted VF is working as expected is to perform an operation from within the guest that can only succeed with a trusted VF - for example, changing its MAC address. If this works, there's no bug here.
Hi Artom, thanks for your replies. We have the hardware and tested that everything works as it should when done as an admin user. I think that the binding:profile is omitted due to oslo policy(mentioned here https://docs.openstack.org/newton/config-reference/networking/samples/policy.json.html) '"get_port:binding:profile": "rule:admin_only"', I'm trying to verify if it's caused by this policy. Will update.
(In reply to Vadim Khitrin from comment #4) > Hi Artom, thanks for your replies. > > We have the hardware and tested that everything works as it should when done > as an admin user. > > I think that the binding:profile is omitted due to oslo policy(mentioned > here > https://docs.openstack.org/newton/config-reference/networking/samples/policy. > json.html) '"get_port:binding:profile": "rule:admin_only"', I'm trying to > verify if it's caused by this policy. Will update. Yeah, policy can be changed, it's just admin-only by default. If it's OK with you I'd like to close this BZ to avoid it taking up time in the compute DFG's triage process. If you do notice a bug with trusted VFs and policy, by all means re-open this. Thanks!