Bug 1680011

Summary:	neutron with BigSwitch agent. iptables rules are not being cleaned up when instances are deleted
Product:	Red Hat OpenStack	Reporter:	Andreas Karis <akaris>
Component:	python-networking-bigswitch	Assignee:	Nate Johnston <njohnston>
Status:	CLOSED ERRATA	QA Contact:	Candido Campos <ccamposr>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	10.0 (Newton)	CC:	akaris, amuller, briasmit, chrisw, ekuris, jschluet, lhh, njohnston, scohen
Target Milestone:	---	Keywords:	Reopened, Triaged, ZStream
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	python-networking-bigswitch-9.42.14-2.el7ost.src.rpm	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-10-16 09:41:17 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Andreas Karis 2019-02-22 13:49:42 UTC

Description of problem:

neutron with BigSwitch agent. iptables rules are not being cleaned up when instances are deleted. The bigswitch neutron-bsn-agent uses our downstream /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py

It seems that something with our library may actually not be working, or with the way that neutron-bsn-agent is calling it.

We can see from the BigSwitch logs:
~~~
[root@compute-5 ~]# tail -f /var/log/neutron/bsn-agent.log | grep c5ca33f7-8
2019-02-21 10:15:06.203 299564 DEBUG networking_bigswitch.plugins.bigswitch.agent.restproxy_agent [req-8b79075f-383e-4a7e-b7ce-579e7d84dd7a - - - - -] Ports on IVS: [u'ivs', u'p2p1', u'p3p2', u'storage2329', u'api2327', u'qvo69ffd313-bb', u'qvo5e6e4694-e0', u'qvoc71e7c22-83', u'qvo5958d045-50', u'qvoc5ca33f7-87', u'qvof63e4f11-e3', u'qvo522c0fc6-81', u'qvoa8674c70-80', u'qvodb56561a-c7', u'qvoe9308373-5f', u'inband'] get_port_name_list /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/agent/restproxy_agent.py:103
2019-02-21 10:15:11.204 299564 DEBUG networking_bigswitch.plugins.bigswitch.agent.restproxy_agent [req-8b79075f-383e-4a7e-b7ce-579e7d84dd7a - - - - -] Ports on IVS: [u'ivs', u'p2p1', u'p3p2', u'storage2329', u'api2327', u'qvo69ffd313-bb', u'qvo5e6e4694-e0', u'qvoc71e7c22-83', u'qvo5958d045-50', u'qvoc5ca33f7-87', u'qvof63e4f11-e3', u'qvo522c0fc6-81', u'qvoa8674c70-80', u'qvodb56561a-c7', u'qvoe9308373-5f', u'inband'] get_port_name_list /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/agent/restproxy_agent.py:103
2019-02-21 10:15:16.210 299564 DEBUG networking_bigswitch.plugins.bigswitch.agent.restproxy_agent [req-8b79075f-383e-4a7e-b7ce-579e7d84dd7a - - - - -] Ports on IVS: [u'ivs', u'p2p1', u'p3p2', u'storage2329', u'api2327', u'qvo69ffd313-bb', u'qvo5e6e4694-e0', u'qvoc71e7c22-83', u'qvo5958d045-50', u'qvoc5ca33f7-87', u'qvof63e4f11-e3', u'qvo522c0fc6-81', u'qvoa8674c70-80', u'qvodb56561a-c7', u'qvoe9308373-5f', u'inband'] get_port_name_list /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/agent/restproxy_agent.py:103
2019-02-21 10:15:21.209 299564 DEBUG networking_bigswitch.plugins.bigswitch.agent.restproxy_agent [req-8b79075f-383e-4a7e-b7ce-579e7d84dd7a - - - - -] Ports on IVS: [u'ivs', u'p2p1', u'p3p2', u'storage2329', u'api2327', u'qvo69ffd313-bb', u'qvo5e6e4694-e0', u'qvoc71e7c22-83', u'qvo5958d045-50', u'qvoc5ca33f7-87', u'qvof63e4f11-e3', u'qvo522c0fc6-81', u'qvoa8674c70-80', u'qvodb56561a-c7', u'qvoe9308373-5f', u'inband'] get_port_name_list /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/agent/restproxy_agent.py:103
2019-02-21 10:15:26.214 299564 DEBUG networking_bigswitch.plugins.bigswitch.agent.restproxy_agent [req-8b79075f-383e-4a7e-b7ce-579e7d84dd7a - - - - -] Ports on IVS: [u'ivs', u'p2p1', u'p3p2', u'storage2329', u'api2327', u'qvo69ffd313-bb', u'qvo5e6e4694-e0', u'qvoc71e7c22-83', u'qvo5958d045-50', u'qvoc5ca33f7-87', u'qvof63e4f11-e3', u'qvo522c0fc6-81', u'qvoa8674c70-80', u'qvodb56561a-c7', u'qvoe9308373-5f', u'inband'] get_port_name_list /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/agent/restproxy_agent.py:103
2019-02-21 10:15:31.218 299564 DEBUG networking_bigswitch.plugins.bigswitch.agent.restproxy_agent [req-8b79075f-383e-4a7e-b7ce-579e7d84dd7a - - - - -] Ports on IVS: [u'ivs', u'p2p1', u'p3p2', u'storage2329', u'api2327', u'qvo69ffd313-bb', u'qvo5e6e4694-e0', u'qvoc71e7c22-83', u'qvo5958d045-50', u'qvoc5ca33f7-87', u'qvof63e4f11-e3', u'qvo522c0fc6-81', u'qvoa8674c70-80', u'qvodb56561a-c7', u'qvoe9308373-5f', u'inband'] get_port_name_list /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/agent/restproxy_agent.py:103
2019-02-21 10:15:36.219 299564 DEBUG networking_bigswitch.plugins.bigswitch.agent.restproxy_agent [req-8b79075f-383e-4a7e-b7ce-579e7d84dd7a - - - - -] Ports on IVS: [u'ivs', u'p2p1', u'p3p2', u'storage2329', u'api2327', u'qvo69ffd313-bb', u'qvo5e6e4694-e0', u'qvoc71e7c22-83', u'qvo5958d045-50', u'qvoc5ca33f7-87', u'qvof63e4f11-e3', u'qvo522c0fc6-81', u'qvoa8674c70-80', u'qvodb56561a-c7', u'qvoe9308373-5f', u'inband'] get_port_name_list /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/agent/restproxy_agent.py:103
2019-02-21 10:15:37.607 299564 DEBUG neutron.api.rpc.handlers.securitygroups_rpc [req-d495916f-ddfe-4d7d-954c-b9363cd5796e 113d0c0410e7418486464ff82a60f2f0 8c2119317d3e47298c195cca47a7bfd7 - - -] Get security group information for devices via rpc [u'c5ca33f7-87'] security_group_info_for_devices /usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/securitygroups_rpc.py:52
2019-02-21 10:15:37.677 299564 DEBUG neutron.agent.linux.iptables_manager [req-d495916f-ddfe-4d7d-954c-b9363cd5796e 113d0c0410e7418486464ff82a60f2f0 8c2119317d3e47298c195cca47a7bfd7 - - -] Attempted to remove chain sc5ca33f7-8 which does not exist remove_chain /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:177
2019-02-21 10:15:41.218 299564 INFO neutron.agent.securitygroups_rpc [req-8b79075f-383e-4a7e-b7ce-579e7d84dd7a - - - - -] Remove device filter for set([u'qvoc5ca33f7-87'])
2019-02-21 10:15:41.223 299564 DEBUG neutron.agent.linux.iptables_manager [req-8b79075f-383e-4a7e-b7ce-579e7d84dd7a - - - - -] Attempted to remove chain sc5ca33f7-8 which does not exist remove_chain /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:177
^C
[root@compute-5 ~]# iptables-save | grep sc5ca33f7-8
:neutron-bsn-agen-sc5ca33f7-8 - [0:0]
-A neutron-bsn-agen-oc5ca33f7-8 -j neutron-bsn-agen-sc5ca33f7-8
-A neutron-bsn-agen-sc5ca33f7-8 -s 192.168.246.5/32 -m mac --mac-source FA:16:3E:AF:85:66 -m comment --comment "Allow traffic from defined IP/MAC pairs." -j RETURN
-A neutron-bsn-agen-sc5ca33f7-8 -m comment --comment "Drop traffic without an IP/MAC allow rule." -j DROP
[root@compute-5 ~]#
~~~

Note:
~~~
Attempted to remove chain sc5ca33f7-8 which does not exist remove_chain /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:177
~~~

The prefix `neutron-bsn-agen-` is missing, here. The issue can be resolved by a restart of the neutron-bsn-agent - init step of the agent would be taking care of that part. 
 
BSN engineering also shared following technical details which can help in clarifying the flow and simplify further debugging for RH engineers. 

1. neutron BSN agent is initialized here [1] with binary = neutron-bsn-agent. this property name is important. 
2. the binary name set above is fetched by iptables_manager.py in core neutron here [2]. as noted in the comment in [2], the binary name is limited to 16 characters == neutron-bsn-agen that you see prefixed to all rules 
3. this iptables_manager.py is used by the iptables_firewall.py 
4. the binary-name is later copied to a local variable called wrap_name [3]. and searching for wrap_name in the iptables_manager.py shows various places in adding/removing iptable chain or rules, where wrap_name is used i.e. binary-name i.e. neutron-bsn-agen 
5. so the core handling of adding or removing rules is still handled by upstream neutron code. it simply takes the binary-name of the plugin that is enabled for the agent and appends it when creating chain or rules. 


Per your request, the relevant codes can be found here, 

[1] https://github.com/openstack/networking-bigswitch/blob/stable/newton/networking_bigswitch/plugins/bigswitch/agent/restproxy_agent.py#L198-L204 

[2] https://github.com/openstack/neutron/blob/newton-eol/neutron/agent/linux/iptables_manager.py#L47-L55 

[3] https://github.com/openstack/neutron/blob/newton-eol/neutron/agent/linux/iptables_manager.py#L141 

The above is the description of how BSN plugin extends upstream firewall handler. The complete BSN agent is defined in the file [1] and its open source, so anyone should be able to see it on github. 

Additional info:

I'm still looking into this from the code side. But I think it's either:
* BigSwitch not using our libraries correctly
* our libraries containing a bug in a rarely used component

Comment 5 Andreas Karis 2019-02-25 15:26:27 UTC

Hi,

~~~
[akaris@collab-shell sosreport-20190207-103024]$ wc -l ./compute-0/sos_commands/networking/iptables_-vnxL
9208 ./compute-0/sos_commands/networking/iptables_-vnxL
~~~

iptables rules are not cleaned up and eventually there are 10 of thousands of rules in iptables leading to failures in iptables rules creation.

That's the problem. The debug log messages gives a hint to why this likely fails: `neutron-bsn-agen-` is not prepended to rules. Hence, neutron-bsn-agent cannot find the rules and as a consequence they are not deleted.

[akaris@collab-shell compute-5]$ grep neutron installed-rpms 
openstack-neutron-9.4.1-1.el7ost.noarch                     Wed Sep 20 12:21:20 2017
openstack-neutron-bigswitch-agent-9.42.7-1.el7.centos.noarch Thu Oct 19 18:51:10 2017
openstack-neutron-bigswitch-lldp-9.42.7-1.el7.centos.noarch Thu Oct 19 18:51:10 2017
openstack-neutron-common-9.4.1-1.el7ost.noarch              Wed Sep 20 11:59:08 2017
openstack-neutron-lbaas-9.2.1-4.el7ost.noarch               Wed Sep 20 12:21:20 2017
openstack-neutron-metering-agent-9.4.1-1.el7ost.noarch      Wed Sep 20 12:23:11 2017
openstack-neutron-ml2-9.4.1-1.el7ost.noarch                 Wed Sep 20 12:17:34 2017
openstack-neutron-openvswitch-9.4.1-1.el7ost.noarch         Wed Sep 20 12:21:21 2017
openstack-neutron-sriov-nic-agent-9.4.1-1.el7ost.noarch     Wed Sep 20 12:23:12 2017
puppet-neutron-9.5.0-4.el7ost.noarch                        Wed Sep 20 12:28:13 2017
python-neutron-9.4.1-1.el7ost.noarch                        Wed Sep 20 11:59:06 2017
python-neutron-lbaas-9.2.1-4.el7ost.noarch                  Wed Sep 20 12:05:54 2017
python-neutron-lib-0.4.0-1.el7ost.noarch                    Wed Sep 20 11:59:04 2017
python-neutron-tests-9.4.1-1.el7ost.noarch                  Wed Sep 20 11:59:10 2017
python-neutronclient-6.0.0-3.el7ost.noarch                  Wed Sep 20 11:58:54 2017
[akaris@collab-shell compute-5]$ 

We are aware that the neutron versions are old, we are hence updating the Red Hat neutron RPMs today to latest OSP 10.

- Andreas

Comment 7 Andreas Karis 2019-02-25 17:25:53 UTC

[heat-admin@compute-5 noarch]$ ll
total 4728
-rw-rw-r--. 1 heat-admin heat-admin   30252 Feb 25 11:17 dnsmasq-utils-2.76-2.el7_4.2.x86_64.rpm
-rw-rw-r--. 1 heat-admin heat-admin   47076 Nov 20 16:11 openstack-neutron-9.4.1-32.el7ost.noarch.rpm
-rw-rw-r--. 1 heat-admin heat-admin  288384 Nov 20 16:11 openstack-neutron-common-9.4.1-32.el7ost.noarch.rpm
-rw-rw-r--. 1 heat-admin heat-admin   30500 Nov 20 16:11 openstack-neutron-linuxbridge-9.4.1-32.el7ost.noarch.rpm
-rw-rw-r--. 1 heat-admin heat-admin   25604 Nov 20 16:11 openstack-neutron-macvtap-agent-9.4.1-32.el7ost.noarch.rpm
-rw-rw-r--. 1 heat-admin heat-admin   27504 Nov 20 16:11 openstack-neutron-metering-agent-9.4.1-32.el7ost.noarch.rpm
-rw-rw-r--. 1 heat-admin heat-admin   29460 Nov 20 16:11 openstack-neutron-ml2-9.4.1-32.el7ost.noarch.rpm
-rw-rw-r--. 1 heat-admin heat-admin   32068 Nov 20 16:11 openstack-neutron-openvswitch-9.4.1-32.el7ost.noarch.rpm
-rw-rw-r--. 1 heat-admin heat-admin   24904 Nov 20 16:11 openstack-neutron-rpc-server-9.4.1-32.el7ost.noarch.rpm
-rw-rw-r--. 1 heat-admin heat-admin   27724 Nov 20 16:11 openstack-neutron-sriov-nic-agent-9.4.1-32.el7ost.noarch.rpm
-rw-rw-r--. 1 heat-admin heat-admin 2000632 Nov 20 16:11 python-neutron-9.4.1-32.el7ost.noarch.rpm
-rw-rw-r--. 1 heat-admin heat-admin 2252232 Nov 20 16:11 python-neutron-tests-9.4.1-32.el7ost.noarch.rpm
[heat-admin@compute-5 noarch]$ sudo yum localinstall *
Loaded plugins: product-id, search-disabled-repos, subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
Examining dnsmasq-utils-2.76-2.el7_4.2.x86_64.rpm: dnsmasq-utils-2.76-2.el7_4.2.x86_64
Marking dnsmasq-utils-2.76-2.el7_4.2.x86_64.rpm as an update to dnsmasq-utils-2.66-21.el7.x86_64
Examining openstack-neutron-9.4.1-32.el7ost.noarch.rpm: 1:openstack-neutron-9.4.1-32.el7ost.noarch
Marking openstack-neutron-9.4.1-32.el7ost.noarch.rpm as an update to 1:openstack-neutron-9.4.1-1.el7ost.noarch
Examining openstack-neutron-common-9.4.1-32.el7ost.noarch.rpm: 1:openstack-neutron-common-9.4.1-32.el7ost.noarch
Marking openstack-neutron-common-9.4.1-32.el7ost.noarch.rpm as an update to 1:openstack-neutron-common-9.4.1-1.el7ost.noarch
Examining openstack-neutron-linuxbridge-9.4.1-32.el7ost.noarch.rpm: 1:openstack-neutron-linuxbridge-9.4.1-32.el7ost.noarch
Marking openstack-neutron-linuxbridge-9.4.1-32.el7ost.noarch.rpm to be installed
Examining openstack-neutron-macvtap-agent-9.4.1-32.el7ost.noarch.rpm: 1:openstack-neutron-macvtap-agent-9.4.1-32.el7ost.noarch
Marking openstack-neutron-macvtap-agent-9.4.1-32.el7ost.noarch.rpm to be installed
Examining openstack-neutron-metering-agent-9.4.1-32.el7ost.noarch.rpm: 1:openstack-neutron-metering-agent-9.4.1-32.el7ost.noarch
Marking openstack-neutron-metering-agent-9.4.1-32.el7ost.noarch.rpm as an update to 1:openstack-neutron-metering-agent-9.4.1-1.el7ost.noarch
Examining openstack-neutron-ml2-9.4.1-32.el7ost.noarch.rpm: 1:openstack-neutron-ml2-9.4.1-32.el7ost.noarch
Marking openstack-neutron-ml2-9.4.1-32.el7ost.noarch.rpm as an update to 1:openstack-neutron-ml2-9.4.1-1.el7ost.noarch
Examining openstack-neutron-openvswitch-9.4.1-32.el7ost.noarch.rpm: 1:openstack-neutron-openvswitch-9.4.1-32.el7ost.noarch
Marking openstack-neutron-openvswitch-9.4.1-32.el7ost.noarch.rpm as an update to 1:openstack-neutron-openvswitch-9.4.1-1.el7ost.noarch
Examining openstack-neutron-rpc-server-9.4.1-32.el7ost.noarch.rpm: 1:openstack-neutron-rpc-server-9.4.1-32.el7ost.noarch
Marking openstack-neutron-rpc-server-9.4.1-32.el7ost.noarch.rpm to be installed
Examining openstack-neutron-sriov-nic-agent-9.4.1-32.el7ost.noarch.rpm: 1:openstack-neutron-sriov-nic-agent-9.4.1-32.el7ost.noarch
Marking openstack-neutron-sriov-nic-agent-9.4.1-32.el7ost.noarch.rpm as an update to 1:openstack-neutron-sriov-nic-agent-9.4.1-1.el7ost.noarch
Examining python-neutron-9.4.1-32.el7ost.noarch.rpm: 1:python-neutron-9.4.1-32.el7ost.noarch
Marking python-neutron-9.4.1-32.el7ost.noarch.rpm as an update to 1:python-neutron-9.4.1-1.el7ost.noarch
Examining python-neutron-tests-9.4.1-32.el7ost.noarch.rpm: 1:python-neutron-tests-9.4.1-32.el7ost.noarch
Marking python-neutron-tests-9.4.1-32.el7ost.noarch.rpm as an update to 1:python-neutron-tests-9.4.1-1.el7ost.noarch
Resolving Dependencies
--> Running transaction check
---> Package dnsmasq-utils.x86_64 0:2.66-21.el7 will be updated
---> Package dnsmasq-utils.x86_64 0:2.76-2.el7_4.2 will be an update
---> Package openstack-neutron.noarch 1:9.4.1-1.el7ost will be updated
---> Package openstack-neutron.noarch 1:9.4.1-32.el7ost will be an update
---> Package openstack-neutron-common.noarch 1:9.4.1-1.el7ost will be updated
---> Package openstack-neutron-common.noarch 1:9.4.1-32.el7ost will be an update
---> Package openstack-neutron-linuxbridge.noarch 1:9.4.1-32.el7ost will be installed
---> Package openstack-neutron-macvtap-agent.noarch 1:9.4.1-32.el7ost will be installed
---> Package openstack-neutron-metering-agent.noarch 1:9.4.1-1.el7ost will be updated
---> Package openstack-neutron-metering-agent.noarch 1:9.4.1-32.el7ost will be an update
---> Package openstack-neutron-ml2.noarch 1:9.4.1-1.el7ost will be updated
---> Package openstack-neutron-ml2.noarch 1:9.4.1-32.el7ost will be an update
---> Package openstack-neutron-openvswitch.noarch 1:9.4.1-1.el7ost will be updated
---> Package openstack-neutron-openvswitch.noarch 1:9.4.1-32.el7ost will be an update
---> Package openstack-neutron-rpc-server.noarch 1:9.4.1-32.el7ost will be installed
---> Package openstack-neutron-sriov-nic-agent.noarch 1:9.4.1-1.el7ost will be updated
---> Package openstack-neutron-sriov-nic-agent.noarch 1:9.4.1-32.el7ost will be an update
---> Package python-neutron.noarch 1:9.4.1-1.el7ost will be updated
---> Package python-neutron.noarch 1:9.4.1-32.el7ost will be an update
---> Package python-neutron-tests.noarch 1:9.4.1-1.el7ost will be updated
---> Package python-neutron-tests.noarch 1:9.4.1-32.el7ost will be an update
--> Finished Dependency Resolution

Dependencies Resolved

=========================================================================================================================================================================================================
 Package                                              Arch                      Version                               Repository                                                                    Size
=========================================================================================================================================================================================================
Installing:
 openstack-neutron-linuxbridge                        noarch                    1:9.4.1-32.el7ost                     /openstack-neutron-linuxbridge-9.4.1-32.el7ost.noarch                         20 k
 openstack-neutron-macvtap-agent                      noarch                    1:9.4.1-32.el7ost                     /openstack-neutron-macvtap-agent-9.4.1-32.el7ost.noarch                       11 k
 openstack-neutron-rpc-server                         noarch                    1:9.4.1-32.el7ost                     /openstack-neutron-rpc-server-9.4.1-32.el7ost.noarch                          11 k
Updating:
 dnsmasq-utils                                        x86_64                    2.76-2.el7_4.2                        /dnsmasq-utils-2.76-2.el7_4.2.x86_64                                          39 k
 openstack-neutron                                    noarch                    1:9.4.1-32.el7ost                     /openstack-neutron-9.4.1-32.el7ost.noarch                                     77 k
 openstack-neutron-common                             noarch                    1:9.4.1-32.el7ost                     /openstack-neutron-common-9.4.1-32.el7ost.noarch                             1.5 M
 openstack-neutron-metering-agent                     noarch                    1:9.4.1-32.el7ost                     /openstack-neutron-metering-agent-9.4.1-32.el7ost.noarch                      15 k
 openstack-neutron-ml2                                noarch                    1:9.4.1-32.el7ost                     /openstack-neutron-ml2-9.4.1-32.el7ost.noarch                                 32 k
 openstack-neutron-openvswitch                        noarch                    1:9.4.1-32.el7ost                     /openstack-neutron-openvswitch-9.4.1-32.el7ost.noarch                         23 k
 openstack-neutron-sriov-nic-agent                    noarch                    1:9.4.1-32.el7ost                     /openstack-neutron-sriov-nic-agent-9.4.1-32.el7ost.noarch                     16 k
 python-neutron                                       noarch                    1:9.4.1-32.el7ost                     /python-neutron-9.4.1-32.el7ost.noarch                                       7.6 M
 python-neutron-tests                                 noarch                    1:9.4.1-32.el7ost                     /python-neutron-tests-9.4.1-32.el7ost.noarch                                  11 M

Transaction Summary
=========================================================================================================================================================================================================
Install  3 Packages
Upgrade  9 Packages

Total size: 21 M
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Warning: RPMDB altered outside of yum.
** Found 2 pre-existing rpmdb problem(s), 'yum check' output follows:
ivs-4.6.2-1.el7.centos.x86_64 is a duplicate with ivs-4.2.0-1.el7.centos.x86_64
ivs-debuginfo-4.6.2-1.el7.centos.x86_64 is a duplicate with ivs-debuginfo-4.2.0-1.el7.centos.x86_64
  Updating   : 1:python-neutron-9.4.1-32.el7ost.noarch                                                                                                                                                                               1/21
  Updating   : 1:openstack-neutron-common-9.4.1-32.el7ost.noarch                                                                                                                                                                     2/21
  Updating   : dnsmasq-utils-2.76-2.el7_4.2.x86_64                                                                                                                                                                                   3/21
  Updating   : 1:openstack-neutron-9.4.1-32.el7ost.noarch                                                                                                                                                                            4/21
  Updating   : 1:openstack-neutron-sriov-nic-agent-9.4.1-32.el7ost.noarch                                                                                                                                                            5/21
  Updating   : 1:openstack-neutron-ml2-9.4.1-32.el7ost.noarch                                                                                                                                                                        6/21
  Updating   : 1:openstack-neutron-openvswitch-9.4.1-32.el7ost.noarch                                                                                                                                                                7/21
  Installing : 1:openstack-neutron-rpc-server-9.4.1-32.el7ost.noarch                                                                                                                                                                 8/21
  Installing : 1:openstack-neutron-linuxbridge-9.4.1-32.el7ost.noarch                                                                                                                                                                9/21
  Updating   : 1:openstack-neutron-metering-agent-9.4.1-32.el7ost.noarch                                                                                                                                                            10/21
  Installing : 1:openstack-neutron-macvtap-agent-9.4.1-32.el7ost.noarch                                                                                                                                                             11/21
  Updating   : 1:python-neutron-tests-9.4.1-32.el7ost.noarch                                                                                                                                                                        12/21
  Cleanup    : 1:openstack-neutron-9.4.1-1.el7ost.noarch                                                                                                                                                                            13/21
  Cleanup    : 1:python-neutron-tests-9.4.1-1.el7ost.noarch                                                                                                                                                                         14/21
  Cleanup    : 1:openstack-neutron-metering-agent-9.4.1-1.el7ost.noarch                                                                                                                                                             15/21
  Cleanup    : 1:openstack-neutron-openvswitch-9.4.1-1.el7ost.noarch                                                                                                                                                                16/21
  Cleanup    : 1:openstack-neutron-ml2-9.4.1-1.el7ost.noarch                                                                                                                                                                        17/21
  Cleanup    : 1:openstack-neutron-sriov-nic-agent-9.4.1-1.el7ost.noarch                                                                                                                                                            18/21
  Cleanup    : 1:openstack-neutron-common-9.4.1-1.el7ost.noarch                                                                                                                                                                     19/21
  Cleanup    : 1:python-neutron-9.4.1-1.el7ost.noarch                                                                                                                                                                               20/21
  Cleanup    : dnsmasq-utils-2.66-21.el7.x86_64                                                                                                                                                                                     21/21
  Verifying  : 1:openstack-neutron-common-9.4.1-32.el7ost.noarch                                                                                                                                                                     1/21
  Verifying  : 1:openstack-neutron-sriov-nic-agent-9.4.1-32.el7ost.noarch                                                                                                                                                            2/21
  Verifying  : 1:openstack-neutron-ml2-9.4.1-32.el7ost.noarch                                                                                                                                                                        3/21
  Verifying  : 1:openstack-neutron-openvswitch-9.4.1-32.el7ost.noarch                                                                                                                                                                4/21
  Verifying  : 1:openstack-neutron-9.4.1-32.el7ost.noarch                                                                                                                                                                            5/21
  Verifying  : dnsmasq-utils-2.76-2.el7_4.2.x86_64                                                                                                                                                                                   6/21
  Verifying  : 1:openstack-neutron-rpc-server-9.4.1-32.el7ost.noarch                                                                                                                                                                 7/21
  Verifying  : 1:python-neutron-9.4.1-32.el7ost.noarch                                                                                                                                                                               8/21
  Verifying  : 1:openstack-neutron-linuxbridge-9.4.1-32.el7ost.noarch                                                                                                                                                                9/21
  Verifying  : 1:openstack-neutron-metering-agent-9.4.1-32.el7ost.noarch                                                                                                                                                            10/21
  Verifying  : 1:openstack-neutron-macvtap-agent-9.4.1-32.el7ost.noarch                                                                                                                                                             11/21
  Verifying  : 1:python-neutron-tests-9.4.1-32.el7ost.noarch                                                                                                                                                                        12/21
  Verifying  : 1:python-neutron-9.4.1-1.el7ost.noarch                                                                                                                                                                               13/21
  Verifying  : 1:openstack-neutron-ml2-9.4.1-1.el7ost.noarch                                                                                                                                                                        14/21
  Verifying  : 1:python-neutron-tests-9.4.1-1.el7ost.noarch                                                                                                                                                                         15/21
  Verifying  : 1:openstack-neutron-metering-agent-9.4.1-1.el7ost.noarch                                                                                                                                                             16/21
  Verifying  : dnsmasq-utils-2.66-21.el7.x86_64                                                                                                                                                                                     17/21
  Verifying  : 1:openstack-neutron-9.4.1-1.el7ost.noarch                                                                                                                                                                            18/21
  Verifying  : 1:openstack-neutron-common-9.4.1-1.el7ost.noarch                                                                                                                                                                     19/21
  Verifying  : 1:openstack-neutron-sriov-nic-agent-9.4.1-1.el7ost.noarch                                                                                                                                                            20/21
  Verifying  : 1:openstack-neutron-openvswitch-9.4.1-1.el7ost.noarch                                                                                                                                                                21/21

Installed:
  openstack-neutron-linuxbridge.noarch 1:9.4.1-32.el7ost                       openstack-neutron-macvtap-agent.noarch 1:9.4.1-32.el7ost                       openstack-neutron-rpc-server.noarch 1:9.4.1-32.el7ost

Updated:
  dnsmasq-utils.x86_64 0:2.76-2.el7_4.2             openstack-neutron.noarch 1:9.4.1-32.el7ost                openstack-neutron-common.noarch 1:9.4.1-32.el7ost             openstack-neutron-metering-agent.noarch 1:9.4.1-32.el7ost
  openstack-neutron-ml2.noarch 1:9.4.1-32.el7ost    openstack-neutron-openvswitch.noarch 1:9.4.1-32.el7ost    openstack-neutron-sriov-nic-agent.noarch 1:9.4.1-32.el7ost    python-neutron.noarch 1:9.4.1-32.el7ost
  python-neutron-tests.noarch 1:9.4.1-32.el7ost

Complete!
[heat-admin@compute-5 noarch]$ systemctl restart bsn-neutron-agent
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to manage system services or units.
Authenticating as: Cloud User (heat-admin)
Password:
[heat-admin@compute-5 noarch]$ sudo systemctl restart bsn-neutron-agent
Failed to restart bsn-neutron-agent.service: Unit not found.
[heat-admin@compute-5 noarch]$ sudo systemctl restart neutron-bsn-agent
[heat-admin@compute-5 noarch]$ sudo -i
[root@compute-5 ~]# iptables -L -nv | grep ae8db7fc
   15  1260 neutron-bsn-agen-sg-chain  all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-out tapae8db7fc-1b --physdev-is-bridged /* Direct traffic from the VM interface to the security group chain. */
   15  1260 neutron-bsn-agen-sg-chain  all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in tapae8db7fc-1b --physdev-is-bridged /* Direct traffic from the VM interface to the security group chain. */
    0     0 neutron-bsn-agen-oae8db7fc-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in tapae8db7fc-1b --physdev-is-bridged /* Direct incoming traffic from VM to the security group chain. */
Chain neutron-bsn-agen-iae8db7fc-1 (1 references)
Chain neutron-bsn-agen-oae8db7fc-1 (2 references)
   15  1260 neutron-bsn-agen-sae8db7fc-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0
Chain neutron-bsn-agen-sae8db7fc-1 (1 references)
   15  1260 neutron-bsn-agen-iae8db7fc-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-out tapae8db7fc-1b --physdev-is-bridged /* Jump to the VM specific chain. */
   15  1260 neutron-bsn-agen-oae8db7fc-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in tapae8db7fc-1b --physdev-is-bridged /* Jump to the VM specific chain. */
[root@compute-5 ~]# tail -f /var/log/neutron/bsn-agent.log | grep ae8db7fc
2019-02-25 11:23:32.009 60137 DEBUG networking_bigswitch.plugins.bigswitch.agent.restproxy_agent [req-3392ed92-b8a9-4764-90a0-c8211600c0f5 - - - - -] Ports on IVS: [u'ivs', u'p2p1', u'p3p2', u'storage2329', u'api2327', u'qvo69ffd313-bb', u'qvo5e6e4694-e0', u'qvoc71e7c22-83', u'qvo5958d045-50', u'qvoae8db7fc-1b', u'qvof63e4f11-e3', u'qvo522c0fc6-81', u'qvoa8674c70-80', u'qvodb56561a-c7', u'qvoe9308373-5f', u'inband'] get_port_name_list /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/agent/restproxy_agent.py:103
2019-02-25 11:23:37.013 60137 DEBUG networking_bigswitch.plugins.bigswitch.agent.restproxy_agent [req-3392ed92-b8a9-4764-90a0-c8211600c0f5 - - - - -] Ports on IVS: [u'ivs', u'p2p1', u'p3p2', u'storage2329', u'api2327', u'qvo69ffd313-bb', u'qvo5e6e4694-e0', u'qvoc71e7c22-83', u'qvo5958d045-50', u'qvoae8db7fc-1b', u'qvof63e4f11-e3', u'qvo522c0fc6-81', u'qvoa8674c70-80', u'qvodb56561a-c7', u'qvoe9308373-5f', u'inband'] get_port_name_list /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/agent/restproxy_agent.py:103
2019-02-25 11:23:42.013 60137 DEBUG networking_bigswitch.plugins.bigswitch.agent.restproxy_agent [req-3392ed92-b8a9-4764-90a0-c8211600c0f5 - - - - -] Ports on IVS: [u'ivs', u'p2p1', u'p3p2', u'storage2329', u'api2327', u'qvo69ffd313-bb', u'qvo5e6e4694-e0', u'qvoc71e7c22-83', u'qvo5958d045-50', u'qvoae8db7fc-1b', u'qvof63e4f11-e3', u'qvo522c0fc6-81', u'qvoa8674c70-80', u'qvodb56561a-c7', u'qvoe9308373-5f', u'inband'] get_port_name_list /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/agent/restproxy_agent.py:103
2019-02-25 11:23:47.014 60137 DEBUG networking_bigswitch.plugins.bigswitch.agent.restproxy_agent [req-3392ed92-b8a9-4764-90a0-c8211600c0f5 - - - - -] Ports on IVS: [u'ivs', u'p2p1', u'p3p2', u'storage2329', u'api2327', u'qvo69ffd313-bb', u'qvo5e6e4694-e0', u'qvoc71e7c22-83', u'qvo5958d045-50', u'qvoae8db7fc-1b', u'qvof63e4f11-e3', u'qvo522c0fc6-81', u'qvoa8674c70-80', u'qvodb56561a-c7', u'qvoe9308373-5f', u'inband'] get_port_name_list /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/agent/restproxy_agent.py:103
2019-02-25 11:23:52.019 60137 DEBUG networking_bigswitch.plugins.bigswitch.agent.restproxy_agent [req-3392ed92-b8a9-4764-90a0-c8211600c0f5 - - - - -] Ports on IVS: [u'ivs', u'p2p1', u'p3p2', u'storage2329', u'api2327', u'qvo69ffd313-bb', u'qvo5e6e4694-e0', u'qvoc71e7c22-83', u'qvo5958d045-50', u'qvoae8db7fc-1b', u'qvof63e4f11-e3', u'qvo522c0fc6-81', u'qvoa8674c70-80', u'qvodb56561a-c7', u'qvoe9308373-5f', u'inband'] get_port_name_list /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/agent/restproxy_agent.py:103
2019-02-25 11:23:57.018 60137 DEBUG networking_bigswitch.plugins.bigswitch.agent.restproxy_agent [req-3392ed92-b8a9-4764-90a0-c8211600c0f5 - - - - -] Ports on IVS: [u'ivs', u'p2p1', u'p3p2', u'storage2329', u'api2327', u'qvo69ffd313-bb', u'qvo5e6e4694-e0', u'qvoc71e7c22-83', u'qvo5958d045-50', u'qvoae8db7fc-1b', u'qvof63e4f11-e3', u'qvo522c0fc6-81', u'qvoa8674c70-80', u'qvodb56561a-c7', u'qvoe9308373-5f', u'inband'] get_port_name_list /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/agent/restproxy_agent.py:103
2019-02-25 11:24:02.021 60137 INFO neutron.agent.securitygroups_rpc [req-3392ed92-b8a9-4764-90a0-c8211600c0f5 - - - - -] Remove device filter for set([u'qvoae8db7fc-1b'])
2019-02-25 11:24:02.028 60137 DEBUG neutron.agent.linux.iptables_manager [req-3392ed92-b8a9-4764-90a0-c8211600c0f5 - - - - -] Attempted to remove chain sae8db7fc-1 which does not exist remove_chain /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:181
2019-02-25 11:24:02.787 60137 DEBUG neutron.api.rpc.handlers.securitygroups_rpc [req-7dced098-5411-49a8-b682-88b4327d4bae 113d0c0410e7418486464ff82a60f2f0 8c2119317d3e47298c195cca47a7bfd7 - - -] Get security group information for devices via rpc [u'ae8db7fc-1b'] security_group_info_for_devices /usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/securitygroups_rpc.py:52
2019-02-25 11:24:02.856 60137 DEBUG neutron.agent.linux.iptables_manager [req-7dced098-5411-49a8-b682-88b4327d4bae 113d0c0410e7418486464ff82a60f2f0 8c2119317d3e47298c195cca47a7bfd7 - - -] Attempted to remove chain sae8db7fc-1 which does not exist remove_chain /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:181
^C
[root@compute-5 ~]# iptables -L -nv | grep ae8db7fc
   15  1260 neutron-bsn-agen-sg-chain  all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-out tapae8db7fc-1b --physdev-is-bridged /* Direct traffic from the VM interface to the security group chain. */
   15  1260 neutron-bsn-agen-sg-chain  all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in tapae8db7fc-1b --physdev-is-bridged /* Direct traffic from the VM interface to the security group chain. */
    0     0 neutron-bsn-agen-oae8db7fc-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in tapae8db7fc-1b --physdev-is-bridged /* Direct incoming traffic from VM to the security group chain. */
Chain neutron-bsn-agen-iae8db7fc-1 (1 references)
Chain neutron-bsn-agen-oae8db7fc-1 (2 references)
   15  1260 neutron-bsn-agen-sae8db7fc-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0
Chain neutron-bsn-agen-sae8db7fc-1 (1 references)
   15  1260 neutron-bsn-agen-iae8db7fc-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-out tapae8db7fc-1b --physdev-is-bridged /* Jump to the VM specific chain. */
   15  1260 neutron-bsn-agen-oae8db7fc-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in tapae8db7fc-1b --physdev-is-bridged /* Jump to the VM specific chain. */
[root@compute-5 ~]#

Comment 29 Nate Johnston 2019-03-01 19:26:20 UTC

Andreas,

I just pushed a package build to brew that should generate a scratch package for you to use that has extra debugging in neutron/agent/linux/iptables_{firewall,manager}.py.  

Task info: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=20417522

This is my first time doing a scratch build so let me know if that does not work for you.

Nate

Comment 48 Candido Campos 2019-10-03 10:26:46 UTC

Verified:

[stack@undercloud-0 ~]$ cat core_puddle_version 
2019-09-19.1[stack@undercloud-0 ~]$ cat /etc/rhosp-release 
Red Hat OpenStack Platform release 10.0.13 (Newton)
[stack@undercloud-0 ~]$ 



[root@controller-0 heat-admin]# rpm -qa | grep neutron 
python-neutron-9.4.1-48.el7ost.noarch
python-neutron-lib-0.4.0-1.el7ost.noarch
openstack-neutron-lbaas-9.2.2-10.el7ost.noarch
openstack-neutron-metering-agent-9.4.1-48.el7ost.noarch
openstack-neutron-9.4.1-48.el7ost.noarch
openstack-neutron-bigswitch-agent-9.42.14-2.el7ost.noarch
python-neutronclient-6.0.1-1.el7ost.noarch
python-neutron-lbaas-9.2.2-10.el7ost.noarch
python-neutron-tests-9.4.1-48.el7ost.noarch
openstack-neutron-ml2-9.4.1-48.el7ost.noarch
puppet-neutron-9.5.0-5.el7ost.noarch
openstack-neutron-openvswitch-9.4.1-48.el7ost.noarch
openstack-neutron-sriov-nic-agent-9.4.1-48.el7ost.noarch
openstack-neutron-common-9.4.1-48.el7ost.noarch
openstack-neutron-bigswitch-lldp-9.42.14-2.el7ost.noarch


Code is included:
[root@controller-0 heat-admin]# vi /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/agent/restproxy_agent.py


..

class NFVSwitchBridge(object):
    '''
    This class does not provide parity with OVS using NFVSwitch.
    It's only the bare minimum necessary to use NFVSwitch with this agent.
    '''
    def get_vif_port_set(self):
        # Un-supported operation. Return empty set for no-op
        return set()

    def get_vif_port_by_id(self, port_id):
        # Un-supported operation. Return False for no-op
        return False


class FilterDeviceIDMixin(sg_rpc.SecurityGroupAgentRpc):
    """Override SecurityGroupAgentRpc methods that call firewall_driver.

    This is to ensure that device ID sent to firewall driver is always without
    any prefixes. Since the firewall_driver adds the prefix regardless of
    whether it already has or not AND when reading local maps/dicts in
    firewall driver, it tries to match the start of device ID _without_

...

Comment 50 errata-xmlrpc 2019-10-16 09:41:17 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3115