1404709 – Neutron L3 Agent: AZLeastRoutersScheduler scheduler bug

Bug 1404709 - Neutron L3 Agent: AZLeastRoutersScheduler scheduler bug

Summary: Neutron L3 Agent: AZLeastRoutersScheduler scheduler bug

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-neutron
Sub Component:
Version:	10.0 (Newton)
Hardware:	x86_64
OS:	Linux
Priority:	low
Severity:	high
Target Milestone:	z9
Target Release:	10.0 (Newton)
Assignee:	Assaf Muller
QA Contact:	Toni Freger
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2016-12-14 13:17 UTC by Stephan Schultchen
Modified:	2022-07-09 10:04 UTC (History)
CC List:	7 users (show)
Fixed In Version:	openstack-neutron-9.2.0-13.el7ost
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2018-09-17 16:52:18 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Launchpad	1641879	None	None	None	2018-05-16 20:18:01 UTC
OpenStack gerrit	400192	None	None	None	2016-12-14 13:35:45 UTC
Red Hat Issue Tracker	OSP-11290	None	None	None	2021-12-10 15:08:13 UTC
Red Hat Product Errata	RHSA-2018:2715	None	None	None	2018-09-17 16:54:15 UTC

Description Stephan Schultchen 2016-12-14 13:17:52 UTC

Description of problem:
l3 agent not working with AZLeastRoutersScheduler

it is basically the same bug as this one: https://bugs.launchpad.net/neutron/+bug/1641879

Version-Release number of selected component (if applicable):


How reproducible:
always


Steps to Reproduce:
1. setup neutron l3 agent
2. start agent
3. watch logs

Actual results:


Expected results:
no exception in Log, working L3 Agent

Additional info:
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task [req-8fcbca10-2b1e-4ea4-8a4e-d355cbed8c52 - - - - -] Error during L3NATAgentWithStateReport.periodic_sync_routers_task
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task Traceback (most recent call last):
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task     task(self, context)
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 552, in periodic_sync_routers_task
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task     self.fetch_and_sync_all_routers(context, ns_manager)
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 154, in wrapper
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task     return f(*args, **kwargs)
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 563, in fetch_and_sync_all_routers
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task     router_ids = self.plugin_rpc.get_router_ids(context)
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 114, in get_router_ids
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task     return cctxt.call(context, 'get_router_ids', host=self.host)
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 138, in call
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task     return self._original_context.call(ctxt, method, **kwargs)
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task     retry=self.retry)
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task     timeout=timeout, retry=retry)
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 464, in send
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task     retry=retry)
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 455, in _send
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task     raise result
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not iterable
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task Traceback (most recent call last):
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task 
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task     res = self.dispatcher.dispatch(message)
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task 
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task     return self._do_dispatch(endpoint, method, ctxt, args)
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task 
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task     result = func(ctxt, **new_args)
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task 
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 75, in get_router_ids
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task     router_ids=None)
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task 
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 507, in auto_schedule_routers
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task     self, context, host, router_ids)
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task 
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/scheduler/l3_agent_scheduler.py", line 151, in auto_schedule_routers
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task     plugin, context, l3_agent)
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task 
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/scheduler/l3_agent_scheduler.py", line 349, in _schedule_ha_routers_to_additional_agent
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task     for router in schedulable_routers:
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task 
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not iterable

Comment 1 Assaf Muller 2016-12-14 13:35:46 UTC

Backport sent upstream.

Comment 2 Siggy Sigwald 2018-05-16 20:23:11 UTC

I have another customer with the same issue:
/var/log/neutron/l3-agent.log:

2018-05-16 14:46:30.191 18659 ERROR oslo_service.periodic_task [req-c1aaddaf-b58d-4e50-9a64-18b2b3458901 - - - - -] Error during L3NATAgentWithStateReport.periodic_sync_routers_task
2018-05-16 14:46:30.191 18659 ERROR oslo_service.periodic_task Traceback (most recent call last):
2018-05-16 14:46:30.191 18659 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks
2018-05-16 14:46:30.191 18659 ERROR oslo_service.periodic_task     task(self, context)
2018-05-16 14:46:30.191 18659 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 552, in periodic_sync_routers_task
2018-05-16 14:46:30.191 18659 ERROR oslo_service.periodic_task     self.fetch_and_sync_all_routers(context, ns_manager)
2018-05-16 14:46:30.191 18659 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 154, in wrapper
2018-05-16 14:46:30.191 18659 ERROR oslo_service.periodic_task     return f(*args, **kwargs)
2018-05-16 14:46:30.191 18659 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 586, in fetch_and_sync_all_routers
2018-05-16 14:46:30.191 18659 ERROR oslo_service.periodic_task     r['id'], r.get(l3_constants.HA_ROUTER_STATE_KEY))
2018-05-16 14:46:30.191 18659 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha.py", line 120, in check_ha_state_for_router
2018-05-16 14:46:30.191 18659 ERROR oslo_service.periodic_task     if ri and current_state != TRANSLATION_MAP[ri.ha_state]:
2018-05-16 14:46:30.191 18659 ERROR oslo_service.periodic_task KeyError: None
2018-05-16 14:46:30.191 18659 ERROR oslo_service.periodic_task
2018-05-16 14:46:30.195 18659 WARNING oslo.service.loopingcall [req-c1aaddaf-b58d-4e50-9a64-18b2b3458901 - - - - -] Function 'neutron.service.Service.periodic_tasks' run outlasted interval by 35.27 sec

Comment 4 Lon Hohberger 2018-08-10 10:38:06 UTC

According to our records, this should be resolved by openstack-neutron-9.4.1-19.el7ost.  This build is available now.

Comment 6 Alex McLeod 2018-09-03 08:00:02 UTC

Hi there,

If this bug requires doc text for errata release, please set the 'Doc Type' and provide draft text according to the template in the 'Doc Text' field.

The documentation team will review, edit, and approve the text.

If this bug does not require doc text, please set the 'requires_doc_text' flag to -.

Thanks,
Alex

Comment 8 errata-xmlrpc 2018-09-17 16:52:18 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2715

Note You need to log in before you can comment on or make changes to this bug.