Bug 1482707

Summary: Cannot set --no-share on shared network that has floating_ip, gateway AND a tenant port
Product: Red Hat OpenStack Reporter: David Hill <dhill>
Component: openstack-neutronAssignee: Slawek Kaplonski <skaplons>
Status: CLOSED ERRATA QA Contact: Roee Agiman <ragiman>
Severity: high Docs Contact:
Priority: high    
Version: 9.0 (Mitaka)CC: amuller, chrisw, dhill, nyechiel, pkovacs, skaplons, srevivo
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: 9.0 (Mitaka)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-neutron-8.4.0-18.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1570908 1570909 1570911 1790535 (view as bug list) Environment:
Last Closed: 2018-07-05 12:29:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1570908, 1570909, 1570911, 1790535    

Description David Hill 2017-08-17 23:44:49 UTC
Description of problem:
Cannot set --no-share on shared network that has floating_ip, gateway AND a tenant port.  This might have been fixed in juno and reverted in liberty [1].


[1] https://git.openstack.org/cgit/openstack/neutron/commit/?id=161d465305777226a15ced334acfd06142597f85

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Create a shared network 
2. Add some floating ip
3. Add some gateways
4. Add some tenant ports
5. Try updating the shared network to --no-share

Actual results:
Multiple tenants are using this network... 

Expected results:
Should be set to --no-share

Additional info:

Comment 1 David Hill 2017-08-17 23:54:10 UTC
This is the commit [1] that reverted the previous commit.

[1] https://github.com/openstack/neutron/commit/0186815e39f68c729e2840cfd6827f374e886a87

Comment 2 David Hill 2017-08-17 23:56:44 UTC
Sample trace of this happening in RHOSP 9:

2017-08-16 14:03:25.246 114043 DEBUG neutron.callbacks.manager [req-3c6aea20-2ac9-47ed-97ef-01fcf43806e1 223e57df62de493cb67360ad44c6d654 b60a1375569243d2be54e46adb29a21c - - -] Calling callback neutron.plugins.ml2.plugin.Ml2Plugin._validate_ext_not_in_use_by_tenant _notify_loop /usr/lib/python2.7/site-packages/neutron/callbacks/manager.py:145
2017-08-16 14:03:25.247 114043 DEBUG neutron.callbacks.manager [req-3c6aea20-2ac9-47ed-97ef-01fcf43806e1 223e57df62de493cb67360ad44c6d654 b60a1375569243d2be54e46adb29a21c - - -] Calling callback neutron.plugins.ml2.plugin.Ml2Plugin.validate_network_rbac_policy_change _notify_loop /usr/lib/python2.7/site-packages/neutron/callbacks/manager.py:145
2017-08-16 14:03:25.478 114043 ERROR neutron.callbacks.manager [req-3c6aea20-2ac9-47ed-97ef-01fcf43806e1 223e57df62de493cb67360ad44c6d654 b60a1375569243d2be54e46adb29a21c - - -] Error during notification for neutron.plugins.ml2.plugin.Ml2Plugin.validate_network_rbac_policy_change rbac-policy, before_delete
2017-08-16 14:03:25.478 114043 ERROR neutron.callbacks.manager Traceback (most recent call last):
2017-08-16 14:03:25.478 114043 ERROR neutron.callbacks.manager   File "/usr/lib/python2.7/site-packages/neutron/callbacks/manager.py", line 146, in _notify_loop
2017-08-16 14:03:25.478 114043 ERROR neutron.callbacks.manager     callback(resource, event, trigger, **kwargs)
2017-08-16 14:03:25.478 114043 ERROR neutron.callbacks.manager   File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py", line 154, in validate_network_rbac_policy_change
2017-08-16 14:03:25.478 114043 ERROR neutron.callbacks.manager     tenant_to_check)
2017-08-16 14:03:25.478 114043 ERROR neutron.callbacks.manager   File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py", line 185, in ensure_no_tenant_ports_on_network
2017-08-16 14:03:25.478 114043 ERROR neutron.callbacks.manager     raise n_exc.InvalidSharedSetting(network=network_id)
2017-08-16 14:03:25.478 114043 ERROR neutron.callbacks.manager InvalidSharedSetting: Unable to reconfigure sharing settings for network 3f357062-e8c7-4828-95e2-e911e504840a. Multiple tenants are using it.
2017-08-16 14:03:25.478 114043 ERROR neutron.callbacks.manager
2017-08-16 14:03:25.480 114043 DEBUG neutron.callbacks.manager [req-3c6aea20-2ac9-47ed-97ef-01fcf43806e1 223e57df62de493cb67360ad44c6d654 b60a1375569243d2be54e46adb29a21c - - -] Calling callback 


We confirm that the only ports present in this network are floating_ips, gateway and ONE tenant.

Comment 4 David Hill 2017-08-22 19:06:01 UTC
One other thing that we have in this issue is many RBAC entries that are granting access to this network .   The ultimate goal here is to remove the wildcard entry of all the external/shared networks and grant access on a per/tenant basis.   We are trying to reproduce this issue with RHOSP 10 and so far, our efforts were not successful.

Comment 5 Peter Kovacs 2017-08-25 17:52:49 UTC
The issue can also be reproduced on OSP10:

1, Create 2 projects (tenants): tenantA and tenantB
2, TenantA creates an external network (ext_net_A) + subnet
3, For the external network neutron automatically creates a wildcard 'access_as_external' RBAC rule
4, TenantA can create a new port on ext_net_A; TenantB is not allowed to do the same
5, Create a new 'access_as_shared' RBAC rule granting TenantB access to ext_net_A
6, TenantB is now able to create a port on ext_net_A
7, TenantA sets the shared flag to True on ext_net_A (openstack network set --share <net ID>), which creates a new wildcard 'access_as_shared' RBAC rule
8, TenantA tries to unshare ext_net_A (openstack network set --no-share <net ID>), which fails with: HttpException: Conflict

There were no ports added or any other changes made to ext_net_A between sharing and unsharing it.
Neutron should be able to unshare the network since the only tenant using it (tenantB) is already covered by a specific RBAC rule created in step 5.

Comment 6 David Hill 2017-08-25 21:21:32 UTC
I'm looking at all the traces I've found so far and this would be the problem:

    def ensure_no_tenant_ports_on_network(self, network_id, net_tenant_id,
                                          tenant_id):
        ctx_admin = ctx.get_admin_context()
        rb_model = rbac_db.NetworkRBAC
        other_rbac_entries = self._model_query(ctx_admin, rb_model).filter( and_(rb_model.object_id == network_id, rb_model.action == 'access_as_shared'))
        ports = self._model_query(ctx_admin, models_v2.Port).filter( models_v2.Port.network_id == network_id)
# we get a list of all the ports in the network
        if tenant_id == '*':
# we wan't to delete *
            other_rbac_entries = other_rbac_entries.filter( rb_model.target_tenant != tenant_id)
# we get RBAC from all other tenants
            allowed_tenants = [entry['target_tenant'] for entry in other_rbac_entries]
# we set allowed tenants to the content of target_tenant
            allowed_tenants.append(net_tenant_id)
# all ports NOT allowed are returned below
            ports = ports.filter( ~models_v2.Port.tenant_id.in_(allowed_tenants))
        else:
            query = other_rbac_entries.filter(rb_model.target_tenant == '*')
            if query.count():
                return
            ports = ports.filter(models_v2.Port.tenant_id == tenant_id)
        if ports.count():
# this fails here
            raise n_exc.InvalidSharedSetting(network=network_id)

So the problem here would be that some ports have no tenants and will never get removed.  Did I miss something?

Comment 15 Roee Agiman 2018-06-11 11:14:30 UTC
Verified.
Created an external network, added GW, FIP and tenant port; Changed the relevant setting to '--no-share', works fine.
[stack@undercloud-0 ~]$ cat /etc/yum.repos.d/latest-installed 
9   -p 2018-06-05.1
[stack@undercloud-0 ~]$ rpm -qa | grep openstack-neutron-
openstack-neutron-8.4.0-19.el7ost.noarch

Comment 17 errata-xmlrpc 2018-07-05 12:29:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2138