Bug 1404709

Summary: Neutron L3 Agent: AZLeastRoutersScheduler scheduler bug
Product: Red Hat OpenStack Reporter: Stephan Schultchen <stephan.schultchen>
Component: openstack-neutronAssignee: Assaf Muller <amuller>
Status: CLOSED ERRATA QA Contact: Toni Freger <tfreger>
Severity: high Docs Contact:
Priority: low    
Version: 10.0 (Newton)CC: amuller, chrisw, nyechiel, ragiman, srevivo, ssigwald, stephan.schultchen
Target Milestone: z9Keywords: TestOnly, Triaged, ZStream
Target Release: 10.0 (Newton)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-neutron-9.2.0-13.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-09-17 16:52:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Stephan Schultchen 2016-12-14 13:17:52 UTC
Description of problem:
l3 agent not working with AZLeastRoutersScheduler

it is basically the same bug as this one: https://bugs.launchpad.net/neutron/+bug/1641879

Version-Release number of selected component (if applicable):


How reproducible:
always


Steps to Reproduce:
1. setup neutron l3 agent
2. start agent
3. watch logs

Actual results:


Expected results:
no exception in Log, working L3 Agent

Additional info:
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task [req-8fcbca10-2b1e-4ea4-8a4e-d355cbed8c52 - - - - -] Error during L3NATAgentWithStateReport.periodic_sync_routers_task
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task Traceback (most recent call last):
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task     task(self, context)
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 552, in periodic_sync_routers_task
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task     self.fetch_and_sync_all_routers(context, ns_manager)
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 154, in wrapper
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task     return f(*args, **kwargs)
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 563, in fetch_and_sync_all_routers
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task     router_ids = self.plugin_rpc.get_router_ids(context)
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 114, in get_router_ids
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task     return cctxt.call(context, 'get_router_ids', host=self.host)
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 138, in call
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task     return self._original_context.call(ctxt, method, **kwargs)
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task     retry=self.retry)
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task     timeout=timeout, retry=retry)
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 464, in send
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task     retry=retry)
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 455, in _send
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task     raise result
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not iterable
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task Traceback (most recent call last):
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task 
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task     res = self.dispatcher.dispatch(message)
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task 
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task     return self._do_dispatch(endpoint, method, ctxt, args)
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task 
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task     result = func(ctxt, **new_args)
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task 
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 75, in get_router_ids
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task     router_ids=None)
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task 
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 507, in auto_schedule_routers
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task     self, context, host, router_ids)
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task 
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/scheduler/l3_agent_scheduler.py", line 151, in auto_schedule_routers
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task     plugin, context, l3_agent)
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task 
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/scheduler/l3_agent_scheduler.py", line 349, in _schedule_ha_routers_to_additional_agent
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task     for router in schedulable_routers:
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task 
2016-12-14 12:49:20.294 13598 ERROR oslo_service.periodic_task TypeError: 'NoneType' object is not iterable

Comment 1 Assaf Muller 2016-12-14 13:35:46 UTC
Backport sent upstream.

Comment 2 Siggy Sigwald 2018-05-16 20:23:11 UTC
I have another customer with the same issue:
/var/log/neutron/l3-agent.log:

2018-05-16 14:46:30.191 18659 ERROR oslo_service.periodic_task [req-c1aaddaf-b58d-4e50-9a64-18b2b3458901 - - - - -] Error during L3NATAgentWithStateReport.periodic_sync_routers_task
2018-05-16 14:46:30.191 18659 ERROR oslo_service.periodic_task Traceback (most recent call last):
2018-05-16 14:46:30.191 18659 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks
2018-05-16 14:46:30.191 18659 ERROR oslo_service.periodic_task     task(self, context)
2018-05-16 14:46:30.191 18659 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 552, in periodic_sync_routers_task
2018-05-16 14:46:30.191 18659 ERROR oslo_service.periodic_task     self.fetch_and_sync_all_routers(context, ns_manager)
2018-05-16 14:46:30.191 18659 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 154, in wrapper
2018-05-16 14:46:30.191 18659 ERROR oslo_service.periodic_task     return f(*args, **kwargs)
2018-05-16 14:46:30.191 18659 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 586, in fetch_and_sync_all_routers
2018-05-16 14:46:30.191 18659 ERROR oslo_service.periodic_task     r['id'], r.get(l3_constants.HA_ROUTER_STATE_KEY))
2018-05-16 14:46:30.191 18659 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha.py", line 120, in check_ha_state_for_router
2018-05-16 14:46:30.191 18659 ERROR oslo_service.periodic_task     if ri and current_state != TRANSLATION_MAP[ri.ha_state]:
2018-05-16 14:46:30.191 18659 ERROR oslo_service.periodic_task KeyError: None
2018-05-16 14:46:30.191 18659 ERROR oslo_service.periodic_task
2018-05-16 14:46:30.195 18659 WARNING oslo.service.loopingcall [req-c1aaddaf-b58d-4e50-9a64-18b2b3458901 - - - - -] Function 'neutron.service.Service.periodic_tasks' run outlasted interval by 35.27 sec

Comment 4 Lon Hohberger 2018-08-10 10:38:06 UTC
According to our records, this should be resolved by openstack-neutron-9.4.1-19.el7ost.  This build is available now.

Comment 6 Alex McLeod 2018-09-03 08:00:02 UTC
Hi there,

If this bug requires doc text for errata release, please set the 'Doc Type' and provide draft text according to the template in the 'Doc Text' field.

The documentation team will review, edit, and approve the text.

If this bug does not require doc text, please set the 'requires_doc_text' flag to -.

Thanks,
Alex

Comment 8 errata-xmlrpc 2018-09-17 16:52:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2715