Description of problem: The "network:attach_external_network" policy is being checked in nova-compute rather than in nova-api. Version-Release number of selected component (if applicable): Red Hat OpenStack Platform - 13 (RHOSP-13) How reproducible: Always Steps to Reproduce: policy changes are done on the controller nodes and nova_api handles most of them. But for having "network:attach_external_network" in nova_api will not be enough and we need the policy in nova_compute as well Only the API process should be doing policy checks. Someone who wants to override policy for this would have to put a policy.json file on each host, which is certainly problematic. Actual results: Policy network:attach_external_network should be set on compute nodes as well and only setting it on controller nodes (nova_api) has no effect Expected results: setting policy on controller nodes should be enough Additional info: Upstream bug: https://bugs.launchpad.net/nova/+bug/1675486 The workaround for the bug is to have the policy changed on the compute nodes as well. Also, triple-0 doesn't have the code logic to implement policy changes on the compute nodes. Meaning, " NovaApiPolicies " will only help us put policies in Controller nodes and not compute nodes.
We discussed this on the triage call today and while this is a valid issue, we need more investigation to determine if and how we can fix this. It's not as straightforward as forklifting the check to nova-api because the policy needs to be checked every time a port is allocated to an instance (for a variety of different server actions) -- that is why it's currently done by nova-compute, because nova-compute is the one that allocates/deallocates ports to/from instances. We discussed the possibility of removing the policy check altogether and letting things fail on the neutron side if a tenant isn't permitted to attach to an external network (we think the port binding step will fail in neutron if the tenant isn't allowed to attach to external networks). We need to investigate further and test the idea in a lab environment first to see what the user experience would be like with that behavior and whether users would receive adequate error related information about the failure when it fails, and so on. We will do some tests and add a comment to this rhbz with our findings.
I have tested the behavior in a devstack environment and can confirm that server create will indeed fail on the neutron side if a tenant isn't permitted to attach to an external network. This gives some support to the idea of removing the policy check altogether and letting things fail on the neutron side if a tenant is not permitted to attach to an external network. Note that the error message in the latter case is more vague and doesn't indicate that the network allocation step failed -- so a bit more work would be needed to ensure that we don't reschedule in the case where we receive the Forbidden error from neutron. Default behavior (with 'network:attach_external_network' policy check): $ openstack network list +--------------------------------------+---------+----------------------------------------------------------------------------+ | ID | Name | Subnets | +--------------------------------------+---------+----------------------------------------------------------------------------+ | 69c9167d-5fb3-458e-bce0-5934dbb2469c | public | ec1d2d9a-51f3-44ba-ac01-5723eed5e81e, fe29bcbe-3a48-4db3-bb7b-48c2ac8b55a1 | | 9b0c58a4-e843-4fd8-8256-db94893e7047 | private | a2d7ec6f-d10f-4e1a-8278-d3e6f4d5a7cf, a9ec851a-d0fe-4755-8024-d0c65619cdd1 | | 9e9ceb79-8a78-41f7-b36f-ca89b57e90fb | shared | ae67aa07-b887-452f-9574-ffb307880566 | +--------------------------------------+---------+----------------------------------------------------------------------------+ $ openstack network show 69c9167d-5fb3-458e-bce0-5934dbb2469c +---------------------------+----------------------------------------------------------------------------+ | Field | Value | +---------------------------+----------------------------------------------------------------------------+ [...] | router:external | External | [...] $ openstack server create --flavor 42 --image cirros-0.5.1-x86_64-disk --network 69c9167d-5fb3-458e-bce0-5934dbb2469c --wait pizza Error creating server: pizza Error creating server $ openstack server list +--------------------------------------+-------+--------+----------+--------------------------+---------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-------+--------+----------+--------------------------+---------+ | e60954fd-9e03-4a48-8650-606d1d4ffba8 | pizza | ERROR | | cirros-0.5.1-x86_64-disk | m1.nano | +--------------------------------------+-------+--------+----------+--------------------------+---------+ $ openstack server show pizza [...] | fault | {'code': 500, 'created': '2021-02-17T03:06:45Z', 'message': 'Build of instance e60954fd-9e03-4a48-8650-606d1d4ffba8 aborted: Failed to allocate the network(s), not rescheduling.'} | [...] In the nova-compute log: Feb 17 03:06:43 ubuntu-focal nova-compute[115582]: ERROR nova.compute.manager [None req-14a9bd99-99ce-4e74-9c43-351b09904916 demo demo] [instance: e60954fd-9e03-4a48-8650-606d1d4ffba8] Instance failed to spawn: nova.exception.ExternalNetworkAttachForbidden: It is not allowed to create an interface on external network 69c9167d-5fb3-458e-bce0-5934dbb2469c Changed behavior (with 'network:attach_external_network' policy check removed): $ openstack server create --flavor 42 --image cirros-0.5.1-x86_64-disk --network 69c9167d-5fb3-458e-bce0-5934dbb2469c --wait bigmac $ openstack server list +--------------------------------------+--------+--------+----------+--------------------------+---------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+--------+--------+----------+--------------------------+---------+ | 8552dade-9da6-4bb2-9eb5-c53a36714afb | bigmac | ERROR | | cirros-0.5.1-x86_64-disk | m1.nano | | e60954fd-9e03-4a48-8650-606d1d4ffba8 | pizza | ERROR | | cirros-0.5.1-x86_64-disk | m1.nano | +--------------------------------------+--------+--------+----------+--------------------------+---------+ $ openstack server show bigmac [...] | fault | {'code': 500, 'created': '2021-02-17T03:11:36Z', 'message': 'Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 8552dade-9da6-4bb2-9eb5-c53a36714afb.'} | [...] In the nova-compute log: Feb 17 03:11:33 ubuntu-focal nova-compute[133326]: ERROR nova.compute.manager [None req-b0b2b58f-e516-4171-8e23-555cc067b101 demo demo] [instance: 8552dade-9da6-4bb2-9eb5-c53a36714afb] Instance failed to spawn: nova.exception.Forbidden: Tenant 7c60976c662a414cb2661831ff41ee30 not allowed to create port on this network Full traceback: Traceback (most recent call last): File "/opt/stack/nova/nova/compute/manager.py", line 2392, in _build_and_ru n_instance self.driver.spawn(context, instance, image_meta, File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3643, in spawn xml = self._get_guest_xml(context, instance, network_info, File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 6395, in _get_gues t_xml network_info_str = str(network_info) File "/opt/stack/nova/nova/network/model.py", line 615, in __str__ return self._sync_wrapper(fn, *args, **kwargs) File "/opt/stack/nova/nova/network/model.py", line 598, in _sync_wrapper self.wait() File "/opt/stack/nova/nova/network/model.py", line 630, in wait self[:] = self._gt.wait() File "/usr/local/lib/python3.8/dist-packages/eventlet/greenthread.py", line 181, in wait return self._exit_event.wait() File "/usr/local/lib/python3.8/dist-packages/eventlet/event.py", line 132, in wait current.throw(*self._exc) File "/usr/local/lib/python3.8/dist-packages/eventlet/greenthread.py", line 221, in main result = function(*args, **kwargs) File "/opt/stack/nova/nova/utils.py", line 660, in context_wrapper return func(*args, **kwargs) File "/opt/stack/nova/nova/compute/manager.py", line 1765, in _allocate_net work_async raise e File "/opt/stack/nova/nova/compute/manager.py", line 1744, in _allocate_net work_async nwinfo = self.network_api.allocate_for_instance( File "/opt/stack/nova/nova/network/neutron.py", line 1075, in allocate_for_ instance requests_and_created_ports = self._create_ports_for_instance( File "/opt/stack/nova/nova/network/neutron.py", line 985, in _create_ports_ for_instance self._delete_ports( File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ self.force_reraise() File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise six.reraise(self.type_, self.value, self.tb) File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise raise value File "/opt/stack/nova/nova/network/neutron.py", line 973, in _create_ports_ for_instance created_port = self._create_port_minimal( File "/opt/stack/nova/nova/network/neutron.py", line 518, in _create_port_m inimal port_response = port_client.create_port(port_req_body) File "/opt/stack/nova/nova/network/neutron.py", line 181, in wrapper ret = obj(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/neutronclient/v2_0/client.py", line 808, in create_port return self.post(self.ports_path, body=body) File "/opt/stack/nova/nova/network/neutron.py", line 181, in wrapper ret = obj(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/neutronclient/v2_0/client.py", line 357, in post return self.do_request("POST", action, body=body, File "/opt/stack/nova/nova/network/neutron.py", line 181, in wrapper ret = obj(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/neutronclient/v2_0/client.py", line 293, in do_request self._handle_fault_response(status_code, replybody, resp) File "/opt/stack/nova/nova/network/neutron.py", line 199, in wrapper raise exception.Forbidden(str(e)) nova.exception.Forbidden: Tenant 7c60976c662a414cb2661831ff41ee30 not allowed to create port on this network Neutron server returns request_ids: ['req-251431c5-c96d-40bb-8d3c-199b492dc31 1']