Bug 1399071 - Unable to create new routers if 1/2 Networker nodes goes down
Summary: Unable to create new routers if 1/2 Networker nodes goes down
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
medium
urgent
Target Milestone: rc
: 11.0 (Ocata)
Assignee: Brent Eagles
QA Contact: GenadiC
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-11-28 08:46 UTC by Marius Cornea
Modified: 2017-05-17 19:47 UTC (History)
12 users (show)

Fixed In Version: openstack-puppet-modules-10.0.0-0.20170307021643.0333c73.el7ost
Doc Type: Bug Fix
Doc Text:
Cause: Constraint on the minimum number of available L3 agents on creating new routers prevented the creation of new routers. Consequence: It is not possible to create a new Neutron router even if there is at least one L3 agent is available to handle the request. Fix: The constraint has been removed both from Neutron and installer and configuration tools. Result: User can create Neutron routers as long as there is at least on Neutron L3 agent running.
Clone Of:
Environment:
Last Closed: 2017-05-17 19:47:58 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2017:1245 normal SHIPPED_LIVE Red Hat OpenStack Platform 11.0 Bug Fix and Enhancement Advisory 2017-05-17 23:01:50 UTC
OpenStack gerrit 289925 None None None 2017-01-16 19:20:39 UTC
OpenStack gerrit 385604 None None None 2017-01-16 19:21:50 UTC
OpenStack gerrit 421521 None None None 2017-01-30 22:15:28 UTC

Description Marius Cornea 2016-11-28 08:46:43 UTC
Description of problem:
In a deployment with 2 Networker nodes(running Neutron L3 agents) if one of the nodes goes down then new routers cannnot be created. This is caused by the min_l3_agents_per_router config option which is set to 2 by default. 

Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-5.1.0-5.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy overcloud with 2 x Networker nodes which run Neutron L3 agents
2. Bring down one of the Networker nodes
3. Try to create a Neutron router

Actual results:
Router creation fails.

Expected results:
Router creation is successful.

Additional info:
The min_l3_agents_per_router config option is deprecated and it's going to be removed for the Ocata release. Nevertheless it seems that the puppet module doesn't apply it in the current version so I wasn't able to workaround the issue by overriding it with ExtraConfig.

Comment 1 Marius Cornea 2016-11-28 14:00:29 UTC
[stack@undercloud-0 ~]$ neutron router-create router01
Not enough l3 agents available to ensure HA. Minimum required 2, available 1.
Neutron server returns request_ids: ['req-c89cc076-b428-48a3-bd39-89e70a0f579f']

The error in neutron server log:
2016-11-28 13:55:34.142 326469 ERROR neutron.api.v2.resource HANotEnoughAvailableAgents: Not enough l3 agents available to ensure HA. Minimum required 2, available 1.

This is the config section in neutron.conf:

# DEPRECATED: Minimum number of L3 agents that have to be available in order to
# allow a new HA router to be scheduled. This option is deprecated in the
# Newton release and will be removed for the Ocata release where the scheduling
# of new HA routers will always be allowed. (integer value)
# This option is deprecated for removal.
# Its value may be silently ignored in the future.
#min_l3_agents_per_router = 2

In /etc/puppet/modules/neutron/manifests/server.pp:

  if $min_l3_agents_per_router {
    warning('min_l3_agents_per_router is deprecated, has no effect and will be removed for the Ocata release.')
  }

Comment 2 James Slagle 2016-11-28 14:18:08 UTC
My take on this is that the correct thing to do would be to repair the network node that went down or deploy a new one first before trying to add a new router.

However, it is odd that you can't reconfigure min_l3_agents_per_router with puppet-neutron to be set to 1 (or disable it somehow...-1?), given that logic would match the Ocata behavior which will be the new default behavior.

Regardless, I think this is a bug for DFG:Networking as it is for Neutron deployment and configuration.

Comment 3 Brent Eagles 2016-11-28 15:04:47 UTC
Since puppet doesn't support it directly, does arbitrary extra config work:


ControllerExtraConfig:
    neutron::config::server_config:
         default/min_l3_agents_per_router:
             value: '1'

If the configuration is going to be removed from neutron in Ocata, we would only reasonably add support for configuring in puppet-neutron on the newton branch and we'd introduce it as a configuration item that was deprecated from the start. Given the availability of a workaround, the nature of the issue and that it only affects <= newton, I feel this would not get much interest upstream.

Comment 4 Marius Cornea 2016-11-28 16:44:40 UTC
(In reply to Brent Eagles from comment #3)
> Since puppet doesn't support it directly, does arbitrary extra config work:
> 
> 
> ControllerExtraConfig:
>     neutron::config::server_config:
>          default/min_l3_agents_per_router:
>              value: '1'
> 

Thanks, Brent! It worked for me with a slight modification:

  ControllerExtraConfig:
    neutron::config::server_config:
      DEFAULT/min_l3_agents_per_router:
        value: '1'

I'm dropping the blocker flag. Since this option is getting removed in Ocata I think we just need to document the workaround in case users will end up in this situation.

Comment 5 Assaf Muller 2016-11-28 20:28:24 UTC
For the record this has been the behavior in OSP 7, 8 and 9 (Having a min of 2, which impacts transient issues as detailed in comment 0). Since you can set it to 1 with the snippet Marius provided in comment 4 I can't see how this would be a blocker.

Comment 9 GenadiC 2017-03-22 08:18:22 UTC
Verified with openstack-puppet-modules-10.0.0-0.20170307021643.0333c73.el7ost.noarch

Created a router when both networkers were up, then shutdown one of them and created a second router

Comment 11 errata-xmlrpc 2017-05-17 19:47:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1245


Note You need to log in before you can comment on or make changes to this bug.