1032450 – When firewall and vpn co-exist on the same tenant - fwaas process services sync fails

Bug 1032450 - When firewall and vpn co-exist on the same tenant - fwaas process services sync fails

Summary: When firewall and vpn co-exist on the same tenant - fwaas process services sy...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-neutron
Sub Component:
Version:	4.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	rc
Target Release:	4.0
Assignee:	Terry Wilson
QA Contact:	Rami Vaknin
Docs Contact:
URL:
Whiteboard:	network
Depends On:	1039204 1042939
Blocks:
TreeView+	depends on / blocked

Reported:	2013-11-20 08:58 UTC by Rami Vaknin
Modified:	2016-04-26 13:33 UTC (History)
CC List:	9 users (show)
Fixed In Version:	openstack-neutron-2013.2-14.el6ost
Doc Type:	Bug Fix
Doc Text:	In previous releases, the /etc/init.d/neutron-vpn-agent script did not contain a reference to /etc/neutron/l3_agent.ini. At the same time, the Networking service packages did not provide a sample init script for FWaaS. These prevented VPNaaS and FWaaS features from being used with Networking service init scripts. This fix ensures that deploying the Networking service will also: * add a reference to /etc/neutron/l3_agent.ini in /etc/init.d/neutron-vpn-agent, * install a sample init script for FWaaS, i.e. /etc/neutron/fwaas_driver.ini, and * add a reference to /etc/neutron/fwaas_driver.ini in /etc/init.d/neutron-l3-agent. With this, VPNaaS and FWaaS features can now be configured and used properly with Networking service init scripts.
Clone Of:
Environment:
Last Closed:	2013-12-20 00:37:32 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHEA-2013:1859	0	normal	SHIPPED_LIVE	Red Hat Enterprise Linux OpenStack Platform Enhancement Advisory	2013-12-21 00:01:48 UTC

Description Rami Vaknin 2013-11-20 08:58:39 UTC

Version
=======
rhos 4.0 on rhel 6.5, puddle 2013-11-18.8
openstack-neutron-2013.2-9.el6ost
openstack-neutron-vpn-agent-2013.2-9.el6ost


Description
===========
I enable fwaas and vpnaas, I created a firewall with policies and rules, no vpnaas objects were created, the below errors appear every ~40 seconds in /var/log/neutron/vpn-agent.log (I'm not sure this is the right log for these errors).

Note that these errors disappear from the log when I remove the firewall-related objects (firewall, policy and rules).

To make sure that I got a router with this id:

# neutron router-list
+--------------------------------------+---------+-----------------------------------------------------------------------------+
| id                                   | name    | external_gateway_info                                                       |
+--------------------------------------+---------+-----------------------------------------------------------------------------+
| 002f9217-bd4f-464f-ad00-834e76db4797 | router1 | {"network_id": "597a74ee-58d4-4053-b40c-10d9b5e631bf", "enable_snat": true} |
+--------------------------------------+---------+------------------------------

The error messages:

2013-11-20 09:33:11.451 2413 ERROR neutron.services.firewall.agents.l3reference.firewall_l3_agent [-] Failed fwaas process services sync
2013-11-20 09:33:11.451 2413 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent Traceback (most recent call last):
2013-11-20 09:33:11.451 2413 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent   File "/usr/lib/python2.6/site-packages/neutron/services/firewall/agents/l3reference/firewall_l3_agent.py", line 236, in process_services_sync
2013-11-20 09:33:11.451 2413 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent     tenant_id)
2013-11-20 09:33:11.451 2413 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent   File "/usr/lib/python2.6/site-packages/neutron/services/firewall/agents/l3reference/firewall_l3_agent.py", line 97, in _get_router_info_list_for_tenant
2013-11-20 09:33:11.451 2413 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent     if self.router_info[rid].use_namespaces:
2013-11-20 09:33:11.451 2413 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent KeyError: u'002f9217-bd4f-464f-ad00-834e76db4797'
2013-11-20 09:33:11.451 2413 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent
2013-11-20 09:33:50.807 2413 ERROR neutron.services.firewall.agents.l3reference.firewall_l3_agent [-] Failed fwaas process services sync
2013-11-20 09:33:50.807 2413 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent Traceback (most recent call last):
2013-11-20 09:33:50.807 2413 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent   File "/usr/lib/python2.6/site-packages/neutron/services/firewall/agents/l3reference/firewall_l3_agent.py", line 236, in process_services_sync
2013-11-20 09:33:50.807 2413 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent     tenant_id)
2013-11-20 09:33:50.807 2413 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent   File "/usr/lib/python2.6/site-packages/neutron/services/firewall/agents/l3reference/firewall_l3_agent.py", line 97, in _get_router_info_list_for_tenant
2013-11-20 09:33:50.807 2413 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent     if self.router_info[rid].use_namespaces:
2013-11-20 09:33:50.807 2413 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent KeyError: u'002f9217-bd4f-464f-ad00-834e76db4797'
2013-11-20 09:33:50.807 2413 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent
2013-11-20 09:34:30.773 2413 ERROR neutron.services.firewall.agents.l3reference.firewall_l3_agent [-] Failed fwaas process services sync
2013-11-20 09:34:30.773 2413 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent Traceback (most recent call last):
2013-11-20 09:34:30.773 2413 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent   File "/usr/lib/python2.6/site-packages/neutron/services/firewall/agents/l3reference/firewall_l3_agent.py", line 236, in process_services_sync
2013-11-20 09:34:30.773 2413 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent     tenant_id)
2013-11-20 09:34:30.773 2413 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent   File "/usr/lib/python2.6/site-packages/neutron/services/firewall/agents/l3reference/firewall_l3_agent.py", line 97, in _get_router_info_list_for_tenant
2013-11-20 09:34:30.773 2413 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent     if self.router_info[rid].use_namespaces:
2013-11-20 09:34:30.773 2413 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent KeyError: u'002f9217-bd4f-464f-ad00-834e76db4797'
2013-11-20 09:34:30.773 2413 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent
2013-11-20 09:35:10.775 2413 ERROR neutron.services.firewall.agents.l3reference.firewall_l3_agent [-] Failed fwaas process services sync
2013-11-20 09:35:10.775 2413 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent Traceback (most recent call last):
2013-11-20 09:35:10.775 2413 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent   File "/usr/lib/python2.6/site-packages/neutron/services/firewall/agents/l3reference/firewall_l3_agent.py", line 236, in process_services_sync
2013-11-20 09:35:10.775 2413 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent     tenant_id)
2013-11-20 09:35:10.775 2413 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent   File "/usr/lib/python2.6/site-packages/neutron/services/firewall/agents/l3reference/firewall_l3_agent.py", line 97, in _get_router_info_list_for_tenant
2013-11-20 09:35:10.775 2413 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent     if self.router_info[rid].use_namespaces:
2013-11-20 09:35:10.775 2413 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent KeyError: u'002f9217-bd4f-464f-ad00-834e76db4797'
2013-11-20 09:35:10.775 2413 TRACE neutron.services.firewall.agents.l3reference.firewall_l3_agent

Comment 2 Terry Wilson 2013-12-06 23:25:45 UTC

There are some manual changes that need to be made because of some packaging issues, but basically if I:

0) packstack --allinone
1) Add the vpnaas.filters file to /usr/share/neutron/rootwrap
2) Add l3_agent.ini to the list of config files in /etc/init.d/neutron-vpn-agent
3) Add fwaas_driver.ini to /etc/neutron and set the appropriate fields
4) Add /etc/neutron/fwaas_driver.ini to the config files in /etc/init.d/neutron-l3-agent
5) Add service_plugins = neutron.services.firewall.fwaas_plugin.FirewallPlugin, neutron.services.vpn.plugin.VPNDriverPlugin to /etc/neutron.conf
6) restart all of the neutron services
7) Fix selinux issues (https://bugzilla.redhat.com/show_bug.cgi?id=1039204) with semanage fcontext -a -t neutron_exec_t /usr/bin/neutron-vpn-agent ; restorecon /usr/bin/neutron* (or disable selinux)
8) Restart all neutron services
9) Create firewall rules, policy, firewall and verify that firewall shows ACTIVE

I don't get any errors with everything on one machine. So, I'm assuming that the problem is related to one of the above steps that needs to be fixed in the packaging, which I'm working on, but involves getting a change pushed through upstream since the fwaas_driver.ini file is missing from setup.cfg.

Comment 3 Stephen Gordon 2013-12-09 15:51:04 UTC

Terry can you please clean up/verify the correctness of the "Doc Text" field as a workaround for the docs guys so they can include it in the release notes?

Thanks!

Comment 4 Terry Wilson 2013-12-09 19:33:44 UTC

Should be fixed in openstack-neutron-2013.2-14.el6ost, please make sure when testing to have:

  service_plugins = neutron.services.firewall.fwaas_plugin.FirewallPlugin, neutron.services.vpn.plugin.VPNDriverPlugin

in neutron.conf and to edit fwaas_driver.ini and vpn_agent.ini.

Comment 7 Rami Vaknin 2013-12-11 15:22:13 UTC

Verified on rhos 4.0 running on rhel 6.5 with 2013-12-09.2 puddle, openstack-neutron-2013.2-14.el6ost.

I've enabled fwaas and vpnaas, I've added a firewall to the admin tenant which holds the only router, the firewall policy contains only one rule which blocks any connection (although firewall enabled without rules should do the same), the original errors do not appear anymore - the reported issue could not reproduced on this puddle, the vpn-agent.log seems ok:

2013-12-11 17:11:25.950 15939 INFO neutron.common.config [-] Logging enabled!
2013-12-11 17:11:25.951 15939 ERROR neutron.common.legacy [-] Skipping unknown group key: firewall_driver
2013-12-11 17:11:27.572 15939 INFO neutron.openstack.common.rpc.impl_qpid [-] Connected to AMQP server on 10.35.160.29:5672
2013-12-11 17:11:27.575 15939 INFO neutron.openstack.common.rpc.impl_qpid [-] Connected to AMQP server on 10.35.160.29:5672
2013-12-11 17:11:27.602 15939 INFO neutron.openstack.common.rpc.impl_qpid [-] Connected to AMQP server on 10.35.160.29:5672
2013-12-11 17:11:27.604 15939 INFO neutron.openstack.common.rpc.impl_qpid [-] Connected to AMQP server on 10.35.160.29:5672
2013-12-11 17:11:27.623 15939 INFO neutron.agent.l3_agent [-] L3 agent started
2013-12-11 17:12:17.781 15939 WARNING neutron.openstack.common.loopingcall [-] task run outlasted interval by 45.215278 sec
2013-12-11 17:12:17.786 15939 WARNING neutron.openstack.common.loopingcall [-] task run outlasted interval by 7.162342 sec

Comment 9 errata-xmlrpc 2013-12-20 00:37:32 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2013-1859.html

Note You need to log in before you can comment on or make changes to this bug.