As an OpenStack non-admin tenant, I own a Provider Network (created by admin but with --project) and I don't want other tenants to use that network. $ openstack network show c8fd600d-7b0a-481f-b370-3d0d118fc120 +---------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +---------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | nova | | created_at | 2021-02-12T17:56:51Z | | description | Network request: RITM0843207 | | dns_domain | | | id | c8fd600d-7b0a-481f-b370-3d0d118fc120 | | ipv4_address_scope | None | | ipv6_address_scope | None | | is_default | False | | is_vlan_transparent | None | | location | cloud='upshift-sos', project.domain_id='default', project.domain_name=, project.id='c73b7097d07c46f78eb4b4dcfbac5ca8', project.name='rhos-dfg-osasinfra', region_name='regionOne', zone= | | mtu | 1500 | | name | provider_net_osasinfra | | port_security_enabled | True | | project_id | c73b7097d07c46f78eb4b4dcfbac5ca8 | | provider:network_type | None | | provider:physical_network | None | | provider:segmentation_id | None | | qos_policy_id | None | | revision_number | 17 | | router:external | External | | segments | None | | shared | False | | status | ACTIVE | | subnets | 06339bd0-b306-4bbd-b4a6-e5b35cd7433a | | tags | | | updated_at | 2021-02-24T20:01:53Z | +---------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ c73b7097d07c46f78eb4b4dcfbac5ca8 is the project ID of my tenant, where I want to plug VMs into it and don't want other tenants to have access to it. If I spawn a Nova server on this provider network, it'll fail if the network is not "shared", with this error: {'code': 500, 'created': '2021-02-24T13:31:39Z', 'message': 'Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance 0b3c1ef0-b9ba-48f6-a1ff-c22c975c58d3. Last exception: It is not allowed to create an interface on external network c8fd600d-7b0a-481f-b370-3d0d118fc120'} If I set the network to be "shared", Nova is able to create the server: $ openstack network set --share c8fd600d-7b0a-481f-b370-3d0d118fc120 This is not a desired behaviour, since I don't want other tenants to be able to use that network. Version-Release number of selected component (if applicable): OSP16, OSP17
Spelunking. This seems to be relevant to the history: https://bugs.launchpad.net/nova/+bug/1352102
https://github.com/openstack/nova/blob/3247e7b901dc791cf0396998c7f2ec1730b639f3/nova/network/neutron.py#L603-L613 This sounds like an expected behaviour...
The original LP: https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1284718 It seems this was added because nova was able to add ports but not delete them. However, reading the original bug report it doesn't seem that the provider network in question was owned by the tenant. We should test this in neutron, but it is possible to both add and delete ports, this would be a bug in Nova.
It seems that the policy was originally added because for some configuration of provider network nova could add ports but not subsequently remove them. I tested this locally using 2 provider networks in my dev system: - hostonly: not shared, owned by tenant - external: shared, owned by admin These are both provider networks! Yes, it's weird: it's a dev system. When authenticated as the tenant, I am able to both create and delete ports in both of these networks: $ openstack port create --network hostonly foo <success, full output below> $ openstack port create --network external bar <success, full output below> $ openstack port delete foo $ openstack port delete bar This seems to work just fine. I wonder if this is no longer a limitation in neutron, and this policy should be removed? ====== Full output below $ openstack network show hostonly +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | | | created_at | 2021-02-16T17:02:08Z | | description | | | dns_domain | | | id | aa50b218-35f9-4569-b71f-f8ad09557753 | | ipv4_address_scope | None | | ipv6_address_scope | None | | is_default | False | | is_vlan_transparent | None | | mtu | 1500 | | name | hostonly | | port_security_enabled | True | | project_id | d21d21c219134e75ab2be18471398908 | | provider:network_type | None | | provider:physical_network | None | | provider:segmentation_id | None | | qos_policy_id | None | | revision_number | 4 | | router:external | External | | segments | None | | shared | False | | status | ACTIVE | | subnets | 2f24818b-1469-41cc-9908-4197440fc08f | | tags | | | updated_at | 2021-02-25T15:14:35Z | +---------------------------+--------------------------------------+ $ openstack network show external +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | | | created_at | 2021-02-01T15:29:41Z | | description | | | dns_domain | | | id | 01cbfdcd-7c5e-4c9f-bf81-b090c6893f5b | | ipv4_address_scope | None | | ipv6_address_scope | None | | is_default | False | | is_vlan_transparent | None | | mtu | 1500 | | name | external | | port_security_enabled | True | | project_id | e502ec4a36184db59ef70d06281adeaf | | provider:network_type | None | | provider:physical_network | None | | provider:segmentation_id | None | | qos_policy_id | None | | revision_number | 2 | | router:external | External | | segments | None | | shared | True | | status | ACTIVE | | subnets | addbd320-de36-4cdd-a77a-765ac00075bf | | tags | | | updated_at | 2021-02-01T15:29:49Z | +---------------------------+--------------------------------------+ $ openstack port create --network hostonly foo +-------------------------+--------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+--------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | None | | binding_profile | None | | binding_vif_details | None | | binding_vif_type | None | | binding_vnic_type | normal | | created_at | 2021-02-25T15:22:33Z | | data_plane_status | None | | description | | | device_id | | | device_owner | | | dns_assignment | fqdn='host-192-168-25-32.shiftstack.', hostname='host-192-168-25-32', ip_address='192.168.25.32' | | dns_domain | None | | dns_name | | | extra_dhcp_opts | | | fixed_ips | ip_address='192.168.25.32', subnet_id='2f24818b-1469-41cc-9908-4197440fc08f' | | id | 1befc0dc-157c-498a-88c2-e5641d037c36 | | ip_allocation | immediate | | mac_address | fa:16:3e:81:41:21 | | name | foo | | network_id | aa50b218-35f9-4569-b71f-f8ad09557753 | | numa_affinity_policy | None | | port_security_enabled | True | | project_id | d21d21c219134e75ab2be18471398908 | | propagate_uplink_status | None | | qos_network_policy_id | None | | qos_policy_id | None | | resource_request | None | | revision_number | 1 | | security_group_ids | 3eac583c-88ed-472f-8a81-4e2cf25b59da | | status | DOWN | | tags | | | trunk_details | None | | updated_at | 2021-02-25T15:22:33Z | +-------------------------+--------------------------------------------------------------------------------------------------+ $ openstack port create --network external bar +-------------------------+--------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+--------------------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | None | | binding_profile | None | | binding_vif_details | None | | binding_vif_type | None | | binding_vnic_type | normal | | created_at | 2021-02-25T15:22:54Z | | data_plane_status | None | | description | | | device_id | | | device_owner | | | dns_assignment | fqdn='host-10-46-27-72.shiftstack.', hostname='host-10-46-27-72', ip_address='10.46.27.72' | | dns_domain | None | | dns_name | | | extra_dhcp_opts | | | fixed_ips | ip_address='10.46.27.72', subnet_id='addbd320-de36-4cdd-a77a-765ac00075bf' | | id | 57de4bfb-2f5e-4162-ac01-8f3b94d255bc | | ip_allocation | immediate | | mac_address | fa:16:3e:68:6a:66 | | name | bar | | network_id | 01cbfdcd-7c5e-4c9f-bf81-b090c6893f5b | | numa_affinity_policy | None | | port_security_enabled | True | | project_id | d21d21c219134e75ab2be18471398908 | | propagate_uplink_status | None | | qos_network_policy_id | None | | qos_policy_id | None | | resource_request | None | | revision_number | 1 | | security_group_ids | 3eac583c-88ed-472f-8a81-4e2cf25b59da | | status | DOWN | | tags | | | trunk_details | None | | updated_at | 2021-02-25T15:22:54Z | +-------------------------+--------------------------------------------------------------------------------------------+
To add missing context to comment 4: I am not able to boot a VM on either the hostonly or external networks, both of which are external provider networks, both of which I can create and delete ports on. I don't understand why a policy would prevent this. It seems like it may no longer be relevant?
seting dev_conditional_nack design and upstream as im not conviece this is a bug. nova is just enforcing policy as it is configured. we do not support custom policy downstream and this has been in place for many many releases. to me this seam like an RFE or policy removal which would not like be treated as a bugfix upstream. we woudl have a deprecation cycle as a minimum before removal. we may be able to add a THT parameter to allow this explicitly to workaround the need for a support exception to use custom policy. e.g. if we enable this as a featre explicitly that you can opt into we would not have to treat this as a special case but we do not allow policy to be arbitrary altered by customers directly. if we treat this as a new featrue, tested as such and configurarble via our deployment tool i have no issue with markign this as valide and using ooo to selectivly relax the policy as long as we dont do it by default so we align with upstream behavior. the current title is also miss leading nova will allow you to boot on a provider network without being admin nova will not allow you to boot on an external network. as it stand external networks are special as they are used for the gateway port neutron routers. their subnet pools are also used for floating ips. if we allow booting on external network by default you would be able to use more public ips then your floating ip quota, each port you booted directly on the external network would not be counted toward the floating ip quota but would consome an ip form the floating ip subnet and that might be seen as a billing/quota bypass by some. booting vms to an external network means that the metdata data service does not work as we configure it in neutron by defualt. neutron by default relays on the neutron router too DNAT the metadta request to the nova api. when booting on an external network you need to configure the dhcp agent to do that instead of the l3 agent otherwise the meta-data api will not work and you wont have ssh-keys injected or user data applied by cloud-init. im not sure if there are other implication to this either so before we triage this i think we need to fully understand the implications of the policy. right now i don't think this is a bug but an incomplete feature. when this policy was introduced i don't believe neutron allowd admins to create networks for other tenats. so i dont think there was a way for a non admin to own a external netowrk. when that change a parallel blueprint should have been filed in nova to track removing this policy or changing its default so that neutron could do the enforcement. the workaround right now is to mark the network as shared and then i believe neutron has an rbac extension that would allow you to limit the scope of who can see that shared network jsut to the tenant that owns it. this is a little beyond my current knowledge of neutron capabilities as i have not followed its RBAC work closely is ^ correct? i think we need to start an upstream discussion about this on the mailing list and get input form both the nova and neutron teams before we make any decisions on this. that is why i have set the dev_condtional_nack upstream for this and made this comment public.
rbac support for external netowrks was added in mitaka https://github.com/openstack/neutron/commit/49b4dd3478d782aee4260033825aa6b47eaf644a so as part of that work nova should have been updated to deprecate the policy if it was no longer relevent or to change its default. so this seam like a very latent missalingment of policy and neutron feature.
on other not form the previous bug link the policy was intoduced becuse of https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1284718 which resulted in user being ablt to both vms but not deleting them bcause accoudting to the but the prots wree crated with the admin clinet but could not be deelted by a non admin. i dont belive we still use the admin token to create the ports so that will fail if the normally user does not have permission to do this but we would need to ensure this works end to end and that any change we do does not reintroduce https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1284718
Thanks, Sean! (In reply to smooney from comment #6) > seting dev_conditional_nack design and upstream as im not conviece this is a > bug. > nova is just enforcing policy as it is configured. > we do not support custom policy downstream and this has been in place for > many many releases. > > to me this seam like an RFE or policy removal which would not like be > treated as a bugfix upstream. > we woudl have a deprecation cycle as a minimum before removal. > > we may be able to add a THT parameter to allow this explicitly to workaround > the need for a support exception to use custom policy. e.g. if we enable > this as a featre explicitly that you can opt into we would not have to treat > this as a special case but we do not allow policy to be arbitrary altered by > customers directly. > if we treat this as a new featrue, tested as such and configurarble via our > deployment tool i have no issue with markign this as valide and using ooo to > selectivly relax the policy as long as we dont do it by default so we align > with upstream behavior. It sounds like in comment 8 you might be talking yourself out of this? However, the way I see it it's either: * An RFE, which isn't going to land before feature freeze, or * A bug, which doesn't need to land before feature freeze Either way I don't think there's a panic. The RBAC workaround is a little untidy, but apart from that seems perfectly workable. The creation of a tenant-owned external network is still an admin action, so we already have a hook to get admin to do this. > the current title is also miss leading > nova will allow you to boot on a provider network without being admin > nova will not allow you to boot on an external network. Updated. > as it stand external networks are special as they are used for the gateway > port neutron routers. > their subnet pools are also used for floating ips. if we allow booting on > external network by default > you would be able to use more public ips then your floating ip quota, each > port you booted directly on the > external network would not be counted toward the floating ip quota but would > consome an ip form the floating ip subnet and that might be seen as a > billing/quota bypass by some. I don't think there's a quota issue here, because I can already create a port on the network manually, which means I can already consume the fixed ip. The only thing this policy does is prevent me from attaching a server to it. > booting vms to an external network means that the metdata data service does > not work as we configure it in neutron by defualt. neutron by default relays > on the neutron router too DNAT the metadta request to the nova api. > when booting on an external network you need to configure the dhcp agent to > do that instead of the l3 agent otherwise the meta-data api will not work > and you wont have ssh-keys injected or user data applied by cloud-init. The metadata service is another part of the puzzle. We support running without it, so I think it's worth considering separately. > im not sure if there are other implication to this either so before we > triage this i think we need to fully understand the implications of the > policy. right now i don't think this is a bug but an incomplete feature. > when this policy was introduced i don't believe neutron allowd admins to > create networks for other tenats. > so i dont think there was a way for a non admin to own a external netowrk. > when that change a parallel blueprint should have been filed in nova to > track removing this policy or changing its default so that neutron could do > the > enforcement. > > the workaround right now is to mark the network as shared and then i believe > neutron has an rbac extension that would allow you to limit the scope of who > can see that shared network jsut to the tenant that owns it. > > this is a little beyond my current knowledge of neutron capabilities as i > have not followed its RBAC work closely is ^ correct? > > i think we need to start an upstream discussion about this on the mailing > list and get input form both the nova and neutron teams before we make any > decisions on this. that is why i have set the dev_condtional_nack upstream > for this and made this comment public. Agree. As I said above, with a practical workaround and no chance of meeting the feature deadline I don't think there's a panic here. Thanks for your help!
FWIW we have a rhbz open related to the nova-compute network:attach_external_network policy check [1] which maps to an upstream ticket [2]. The want in this case is to avoid having a [legacy] policy enforcement check residing on the compute host, as updating it requires a config push to all compute hosts and policy best practice is normally only at nova-api. I did some local testing and found that it appears we could remove the policy check and instead let neutron pass/fail the port creation [3] for us. The neutron enforcement for the external network appears to make the nova-compute policy check redundant. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1910563 [2] https://bugs.launchpad.net/nova/+bug/1675486 [3] https://bugzilla.redhat.com/show_bug.cgi?id=1910563#c3
ok i think we need to discuss this again in our triage call and likely merge https://bugzilla.redhat.com/show_bug.cgi?id=1933047 and https://bugzilla.redhat.com/show_bug.cgi?id=1910563 into a single bz. from an upstream point of view we can discuss removal of the policy i think that would only be doable in Xena or Y. if we deprecate it now we can remove it in Xena but i still think this is effectivly an api change so upstream this would not be backportable and would not be a bug but likely a specless blueprint. i think we will need to wait to disucss this at the ptg and get agreement from the neutron core team that this is not needed. melanie do you have any preference which of the bzs we use for this or do you think we should keep the sperate. i do not think we would change this in 13/16 so i was considering this as a change for 17 only but we can discuss that in the bug call i guess. to me a default policy change is an api change and we do not backport api changes, we also dont backprot default changes for config options which policy files are but if other have other view on that we can discuss it.
(In reply to smooney from comment #11) > ok i think we need to discuss this again in our triage call and likely merge > https://bugzilla.redhat.com/show_bug.cgi?id=1933047 and > https://bugzilla.redhat.com/show_bug.cgi?id=1910563 > into a single bz. from an upstream point of view we can discuss removal of > the policy i think that would only be doable in Xena or Y. if we deprecate > it now we can remove it in Xena but > i still think this is effectivly an api change so upstream this would not be > backportable and would not be a bug but likely a specless blueprint. +1, agree it will require a deprecation cycle before removal and would not be backportable upstream and would be either a specless blueprint or a Wishlist bug (we have used Wishlist bugs before for minor policy rule additions, for example). > i think we will need to wait to disucss this at the ptg and get agreement > from the neutron core team that this is not needed. +1, it would be prudent to get confirmation from the neutron team that this is OK. > melanie do you have any preference which of the bzs we use for this or do > you think we should keep the sperate. > i do not think we would change this in 13/16 so i was considering this as a > change for 17 only but we can discuss that in the bug call i guess. > to me a default policy change is an api change and we do not backport api > changes, we also dont backprot default changes for config options which > policy files are but if other have other view on that we can discuss it. Hmm, each rhbz is describing a different problem AFAICT but the solution will likely be the same for both. I usually keep things separate in that case since each one contains discussion of a different problem and then use the same NVR to close both later on. But I don't feel strongly about it. If you think we should combine them, I think it is probably OK. We do have precedent for backporting "simple" policy rule _additions_ downstream but as you mention, removals/default changes I would not expect to be backportable as it could break expectation/existing behavior for older versions. Agree we can discuss.