Bug 1652554

Summary: [RHOS14] Non admin user creation of guest instance with Trusted VF fails due to not having access to binding:profile
Product: Red Hat OpenStack Reporter: Vadim Khitrin <vkhitrin>
Component: openstack-novaAssignee: OSP DFG:Compute <osp-dfg-compute>
Status: CLOSED NOTABUG QA Contact: OSP DFG:Compute <osp-dfg-compute>
Severity: medium Docs Contact:
Priority: medium    
Version: 14.0 (Rocky)CC: berrange, cfontain, dasmith, eglynn, jhakimra, kchamart, sbauza, sgordon, vkhitrin, vromanso, yrachman
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-11-29 18:53:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Vadim Khitrin 2018-11-22 11:16:26 UTC
Description of problem:

When attempting to leverage nova's trusted VF capabilities (as per nova's spec https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/sriov-trusted-vfs.html), port creation has to be done as admin user but not all cloud operators have admin privileges. Even if the port is created as admin user on a different tenant where the cloud operator user exists, guest will be spawned without custom binding:profile.

(overcloud-Admin) [stack@undercloud-0 ~]$ openstack port create --vnic-type direct --project tempest-TestNfvBasic-909024731 --binding-profile trusted=true --network sriovone port100-sriov
+-----------------------+------------------------------------------------------------------------------+
| Field                 | Value                                                                        |
+-----------------------+------------------------------------------------------------------------------+
| admin_state_up        | UP                                                                           |
| allowed_address_pairs |                                                                              |
| binding_host_id       |                                                                              |
| binding_profile       | trusted='true'                                                               |
| binding_vif_details   |                                                                              |
| binding_vif_type      | unbound                                                                      |
| binding_vnic_type     | direct                                                                       |
| created_at            | 2018-11-22T10:27:50Z                                                         |
| data_plane_status     | None                                                                         |
| description           |                                                                              |
| device_id             |                                                                              |
| device_owner          |                                                                              |
| dns_assignment        | None                                                                         |
| dns_domain            | None                                                                         |
| dns_name              | None                                                                         |
| extra_dhcp_opts       |                                                                              |
| fixed_ips             | ip_address='60.20.135.101', subnet_id='50c6bd91-7b1b-4c8d-a27b-f85663a5dc49' |
| id                    | 17a9d426-1bda-47df-8d6f-2e3f6457c039                                         |
| mac_address           | fa:16:3e:6c:72:42                                                            |
| name                  | port100-sriov                                                                |
| network_id            | a03c1203-5af6-4377-942b-e5a30a1339c9                                         |
| port_security_enabled | False                                                                        |
| project_id            | 0540ed01fb134f28b12c7e024f5d72bd                                             |
| qos_policy_id         | None                                                                         |
| revision_number       | 2                                                                            |
| security_group_ids    |                                                                              |
| status                | DOWN                                                                         |
| tags                  |                                                                              |
| trunk_details         | None                                                                         |
| updated_at            | 2018-11-22T10:27:51Z                                                         |
+-----------------------+------------------------------------------------------------------------------+

Viewing the port as admin user will show the binding:profile:
(overcloud-Admin) [stack@undercloud-0 ~]$ openstack port show 17a9d426-1bda-47df-8d6f-2e3f6457c039
+-----------------------+------------------------------------------------------------------------------+
| Field                 | Value                                                                        |
+-----------------------+------------------------------------------------------------------------------+
| admin_state_up        | UP                                                                           |
| allowed_address_pairs |                                                                              |
| binding_host_id       |                                                                              |
| binding_profile       | trusted='true'                                                               |
| binding_vif_details   |                                                                              |
| binding_vif_type      | unbound                                                                      |
| binding_vnic_type     | direct                                                                       |
| created_at            | 2018-11-22T10:27:50Z                                                         |
| data_plane_status     | None                                                                         |
| description           |                                                                              |
| device_id             |                                                                              |
| device_owner          |                                                                              |
| dns_assignment        | None                                                                         |
| dns_domain            | None                                                                         |
| dns_name              | None                                                                         |
| extra_dhcp_opts       |                                                                              |
| fixed_ips             | ip_address='60.20.135.101', subnet_id='50c6bd91-7b1b-4c8d-a27b-f85663a5dc49' |
| id                    | 17a9d426-1bda-47df-8d6f-2e3f6457c039                                         |
| mac_address           | fa:16:3e:6c:72:42                                                            |
| name                  | port100-sriov                                                                |
| network_id            | a03c1203-5af6-4377-942b-e5a30a1339c9                                         |
| port_security_enabled | False                                                                        |
| project_id            | 0540ed01fb134f28b12c7e024f5d72bd                                             |
| qos_policy_id         | None                                                                         |
| revision_number       | 2                                                                            |
| security_group_ids    |                                                                              |
| status                | DOWN                                                                         |
| tags                  |                                                                              |
| trunk_details         | None                                                                         |
| updated_at            | 2018-11-22T10:27:51Z                                                         |
+-----------------------+------------------------------------------------------------------------------+

Viewing the port as a non admin user will show 'None' in binding:profile:
(overcloud-demo) [stack@undercloud-0 ~]$ openstack port show 17a9d426-1bda-47df-8d6f-2e3f6457c039
+-----------------------+------------------------------------------------------------------------------+
| Field                 | Value                                                                        |
+-----------------------+------------------------------------------------------------------------------+
| admin_state_up        | UP                                                                           |
| allowed_address_pairs |                                                                              |
| binding_host_id       | None                                                                         |
| binding_profile       | None                                                                         |
| binding_vif_details   | None                                                                         |
| binding_vif_type      | None                                                                         |
| binding_vnic_type     | direct                                                                       |
| created_at            | 2018-11-22T10:27:50Z                                                         |
| data_plane_status     | None                                                                         |
| description           |                                                                              |
| device_id             |                                                                              |
| device_owner          |                                                                              |
| dns_assignment        | None                                                                         |
| dns_domain            | None                                                                         |
| dns_name              | None                                                                         |
| extra_dhcp_opts       |                                                                              |
| fixed_ips             | ip_address='60.20.135.101', subnet_id='50c6bd91-7b1b-4c8d-a27b-f85663a5dc49' |
| id                    | 17a9d426-1bda-47df-8d6f-2e3f6457c039                                         |
| mac_address           | fa:16:3e:6c:72:42                                                            |
| name                  | port100-sriov                                                                |
| network_id            | a03c1203-5af6-4377-942b-e5a30a1339c9                                         |
| port_security_enabled | False                                                                        |
| project_id            | 0540ed01fb134f28b12c7e024f5d72bd                                             |
| qos_policy_id         | None                                                                         |
| revision_number       | 2                                                                            |
| security_group_ids    |                                                                              |
| status                | DOWN                                                                         |
| tags                  |                                                                              |
| trunk_details         | None                                                                         |
| updated_at            | 2018-11-22T10:27:51Z                                                         |
+-----------------------+------------------------------------------------------------------------------+

Spawning the server as non admin user with the port:
(overcloud-demo) [stack@undercloud-0 ~]$ openstack server create --image rhel-guest-image-7.5-192.x86_64.qcow2 --flavor m1.medium.huge_pages_cpu_pinning_numa_node-0 --nic port-id=17a9d426-1bda-47df-8d6f-2e3f6457c039 TrustedServer
(overcloud-demo) [stack@undercloud-0 ~]$ openstack server list
+--------------------------------------+---------------+--------+------------------------+---------------------------------------+----------------------------------------------+
| ID                                   | Name          | Status | Networks               | Image                                 | Flavor                                       |
+--------------------------------------+---------------+--------+------------------------+---------------------------------------+----------------------------------------------+
| f0c6c53a-eb85-4460-883f-d799e5df8663 | TrustedServer | ACTIVE | sriovone=60.20.135.101 | rhel-guest-image-7.5-192.x86_64.qcow2 | m1.medium.huge_pages_cpu_pinning_numa_node-0 |
+--------------------------------------+---------------+--------+------------------------+---------------------------------------+----------------------------------------------+

Viewing port as a non-admin:
(overcloud-demo) [stack@undercloud-0 ~]$ openstack port show 17a9d426-1bda-47df-8d6f-2e3f6457c039
+-----------------------+------------------------------------------------------------------------------+
| Field                 | Value                                                                        |
+-----------------------+------------------------------------------------------------------------------+
| admin_state_up        | UP                                                                           |
| allowed_address_pairs |                                                                              |
| binding_host_id       | None                                                                         |
| binding_profile       | None                                                                         |
| binding_vif_details   | None                                                                         |
| binding_vif_type      | None                                                                         |
| binding_vnic_type     | direct                                                                       |
| created_at            | 2018-11-22T10:27:50Z                                                         |
| data_plane_status     | None                                                                         |
| description           |                                                                              |
| device_id             | f0c6c53a-eb85-4460-883f-d799e5df8663                                         |
| device_owner          | compute:nova                                                                 |
| dns_assignment        | None                                                                         |
| dns_domain            | None                                                                         |
| dns_name              | None                                                                         |
| extra_dhcp_opts       |                                                                              |
| fixed_ips             | ip_address='60.20.135.101', subnet_id='50c6bd91-7b1b-4c8d-a27b-f85663a5dc49' |
| id                    | 17a9d426-1bda-47df-8d6f-2e3f6457c039                                         |
| mac_address           | fa:16:3e:6c:72:42                                                            |
| name                  | port100-sriov                                                                |
| network_id            | a03c1203-5af6-4377-942b-e5a30a1339c9                                         |
| port_security_enabled | False                                                                        |
| project_id            | 0540ed01fb134f28b12c7e024f5d72bd                                             |
| qos_policy_id         | None                                                                         |
| revision_number       | 20                                                                           |
| security_group_ids    |                                                                              |
| status                | ACTIVE                                                                       |
| tags                  |                                                                              |
| trunk_details         | None                                                                         |
| updated_at            | 2018-11-22T10:49:52Z                                                         |
+-----------------------+------------------------------------------------------------------------------+

Viewing port as an admin:
(overcloud-Admin) [stack@undercloud-0 ~]$ openstack port show 17a9d426-1bda-47df-8d6f-2e3f6457c039
+-----------------------+----------------------------------------------------------------------------------+
| Field                 | Value                                                                            |
+-----------------------+----------------------------------------------------------------------------------+
| admin_state_up        | UP                                                                               |
| allowed_address_pairs |                                                                                  |
| binding_host_id       | overcloud-computeovsdpdksriov-1.localdomain                                      |
| binding_profile       | pci_slot='0000:05:0a.4', pci_vendor_info='8086:154c', physical_network='sriov-1' |
| binding_vif_details   | port_filter='False', vlan='600'                                                  |
| binding_vif_type      | hw_veb                                                                           |
| binding_vnic_type     | direct                                                                           |
| created_at            | 2018-11-22T10:27:50Z                                                             |
| data_plane_status     | None                                                                             |
| description           |                                                                                  |
| device_id             | f0c6c53a-eb85-4460-883f-d799e5df8663                                             |
| device_owner          | compute:nova                                                                     |
| dns_assignment        | None                                                                             |
| dns_domain            | None                                                                             |
| dns_name              | None                                                                             |
| extra_dhcp_opts       |                                                                                  |
| fixed_ips             | ip_address='60.20.135.101', subnet_id='50c6bd91-7b1b-4c8d-a27b-f85663a5dc49'     |
| id                    | 17a9d426-1bda-47df-8d6f-2e3f6457c039                                             |
| mac_address           | fa:16:3e:6c:72:42                                                                |
| name                  | port100-sriov                                                                    |
| network_id            | a03c1203-5af6-4377-942b-e5a30a1339c9                                             |
| port_security_enabled | False                                                                            |
| project_id            | 0540ed01fb134f28b12c7e024f5d72bd                                                 |
| qos_policy_id         | None                                                                             |
| revision_number       | 20                                                                               |
| security_group_ids    |                                                                                  |
| status                | ACTIVE                                                                           |
| tags                  |                                                                                  |
| trunk_details         | None                                                                             |
| updated_at            | 2018-11-22T10:49:52Z                                                             |
+-----------------------+----------------------------------------------------------------------------------+

Verify that no VFs are trusted on hypervisor:
[root@overcloud-computeovsdpdksriov-1 ~]# ip link | grep trust
    vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
    vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
    vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
    vf 3 MAC 62:8a:16:26:62:65, spoof checking off, link-state enable, trust off
    vf 4 MAC fa:16:3e:6c:72:42, vlan 600, spoof checking off, link-state enable, trust off
    vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
    vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
    vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
    vf 3 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off
    vf 4 MAC c2:29:d2:e5:34:aa, spoof checking off, link-state enable, trust off


Version-Release number of selected component (if applicable):
Encountered in in all OSP14 puddles so far

How reproducible: 
Always

Steps to Reproduce:
1. Create tenant with non admin user
2. Create port with custom binding:profile as admin user on the non admin tenant
3. Spawn guest instance as non admin user with the admin created port

Actual results:
Custom binding:profile is omitted

Expected results:
Custom binding:profile is kept

Additional info:

Comment 1 Artom Lifshitz 2018-11-23 01:35:42 UTC
Hello, thanks for the bug report!

Port binding extended attributes are admin-only (both setting and getting) [1]. In addition, trusted mode on the devices is set by the libvirt driver when plugging or unplugging, so unless an instance is actually using a trusted VF, no trusted devices will show up on the host.

Because of that, I think this is all working as expected. If I've misunderstood something, by all means reopen this bug.

[1] https://developer.openstack.org/api-ref/network/v2/index.html#port-binding-extended-attributes

Comment 2 Artom Lifshitz 2018-11-23 04:48:15 UTC
(In reply to Artom Lifshitz from comment #1)
> Hello, thanks for the bug report!
> 
> Port binding extended attributes are admin-only (both setting and getting)
> [1]. In addition, trusted mode on the devices is set by the libvirt driver
> when plugging or unplugging, so unless an instance is actually using a
> trusted VF, no trusted devices will show up on the host.

Err, apologies, I completely missed that an instance had in fact been created. The fact that no interfaces show up as trusted on the host is indeed weird. I assume passthrough_whitelist has been configured correctly? Would it be possible to reproduce the bug with debug enabled and attach nova logs from the compute host to this BZ? Thanks!

Comment 3 Artom Lifshitz 2018-11-23 20:44:25 UTC
(In reply to Artom Lifshitz from comment #2)

> Err, apologies, I completely missed that an instance had in fact been
> created. The fact that no interfaces show up as trusted on the host is
> indeed weird.

I discussed this with a colleague, and without hardware capable of trusted VFs we're not entirely sure whether a VF that has been passed through to a guest would still show up on the host - after all, once passed through, it is the guest's kernel that's managing it, not the host. So it could be that what you're seeing is expected.

The quickest way to make the trusted VF is working as expected is to perform an operation from within the guest that can only succeed with a trusted VF - for example, changing its MAC address. If this works, there's no bug here.

Comment 4 Vadim Khitrin 2018-11-25 08:30:32 UTC
Hi Artom, thanks for your replies.

We have the hardware and tested that everything works as it should when done as an admin user.

I think that the binding:profile is omitted due to oslo policy(mentioned here https://docs.openstack.org/newton/config-reference/networking/samples/policy.json.html) '"get_port:binding:profile": "rule:admin_only"', I'm trying to verify if it's caused by this policy. Will update.

Comment 7 Artom Lifshitz 2018-11-29 18:53:29 UTC
(In reply to Vadim Khitrin from comment #4)
> Hi Artom, thanks for your replies.
> 
> We have the hardware and tested that everything works as it should when done
> as an admin user.
> 
> I think that the binding:profile is omitted due to oslo policy(mentioned
> here
> https://docs.openstack.org/newton/config-reference/networking/samples/policy.
> json.html) '"get_port:binding:profile": "rule:admin_only"', I'm trying to
> verify if it's caused by this policy. Will update.

Yeah, policy can be changed, it's just admin-only by default. If it's OK with you I'd like to close this BZ to avoid it taking up time in the compute DFG's triage process.  If you do notice a bug with trusted VFs and policy, by all means re-open this.

Thanks!