Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1823601

Summary: Failed to process compatible router:$UUID: UnboundLocalError: local variable 'filter_id' referenced before assignment
Product: Red Hat OpenStack Reporter: chrisbro <chrisbro>
Component: openstack-neutronAssignee: Slawek Kaplonski <skaplons>
Status: CLOSED ERRATA QA Contact: Alex Katz <akatz>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 13.0 (Queens)CC: amuller, bshephar, chrisw, ekuris, pveiga, rcernin, schhabdi, scohen, skaplons, tkajinam
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-neutron-12.1.1-16.el7ost.src.rpm Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-06-24 11:53:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description chrisbro@redhat.com 2020-04-14 01:55:13 UTC
Description of problem:
Failed to process compatible router:$UUID: UnboundLocalError: local variable 'filter_id' referenced before assignment

Version-Release number of selected component (if applicable):
Red Hat OpenStack Platform release 13.0.9 (Queens)
  iproute-4.11.0-25.el7_7.2.x86_64
  $IP:8787/rhosp13/openstack-neutron-l3-agent:13.0-100


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Upstream BZ https://bugs.launchpad.net/neutron/+bug/1809497
  https://review.opendev.org/gitweb?p=openstack%2Fneutron.git;a=commitdiff;h=e788d294584add2407f601a46f90b08082cccc5f

https://access.redhat.com/containers/?architecture=AMD64&tab=package-list#/registry.access.redhat.com/rhosp13/openstack-neutron-l3-agent/images/13.0-100
https://access.redhat.com/downloads/content/iproute/4.11.0-25.el7/x86_64/fd431d51/package-changelog
https://bugzilla.redhat.com/show_bug.cgi?id=1714660


Current error:

2020-04-13 12:14:46.112 25384 INFO neutron.agent.linux.interface [-] Device qg-85931fa6-a3 already exists
2020-04-13 13:03:28.422 25384 ERROR neutron.agent.l3.agent [-] Failed to process compatible router: be67feae-375e-4030-96ad-24afa7e581bb: UnboundLocalError: local variable 'filter_id' referenced before assignment
2020-04-13 13:03:28.422 25384 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2020-04-13 13:03:28.422 25384 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 628, in _process_routers_if_compatible
2020-04-13 13:03:28.422 25384 ERROR neutron.agent.l3.agent     self._process_router_if_compatible(router)
2020-04-13 13:03:28.422 25384 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 486, in _process_router_if_compatible
2020-04-13 13:03:28.422 25384 ERROR neutron.agent.l3.agent     self._process_updated_router(router)
2020-04-13 13:03:28.422 25384 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 530, in _process_updated_router
2020-04-13 13:03:28.422 25384 ERROR neutron.agent.l3.agent     self.l3_ext_manager.update_router(self.context, router)
2020-04-13 13:03:28.422 25384 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/l3_agent_extensions_manager.py", line 54, in update_router
2020-04-13 13:03:28.422 25384 ERROR neutron.agent.l3.agent     extension.obj.update_router(context, data)
2020-04-13 13:03:28.422 25384 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2020-04-13 13:03:28.422 25384 ERROR neutron.agent.l3.agent     return f(*args, **kwargs)
2020-04-13 13:03:28.422 25384 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/extensions/fip_qos.py", line 407, in update_router
2020-04-13 13:03:28.422 25384 ERROR neutron.agent.l3.agent     self.process_floating_ip_addresses(context, router_info)
2020-04-13 13:03:28.422 25384 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/extensions/fip_qos.py", line 384, in process_floating_ip_addresses
2020-04-13 13:03:28.422 25384 ERROR neutron.agent.l3.agent     self._remove_fip_rate_limit(device, fip)
2020-04-13 13:03:28.422 25384 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/extensions/fip_qos.py", line 251, in _remove_fip_rate_limit
2020-04-13 13:03:28.422 25384 ERROR neutron.agent.l3.agent     tc_wrapper.clear_ip_rate_limit(direction, fip_ip)
2020-04-13 13:03:28.422 25384 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/l3_tc_lib.py", line 188, in clear_ip_rate_limit
2020-04-13 13:03:28.422 25384 ERROR neutron.agent.l3.agent     filter_id = self._get_filterid_for_ip(qdisc_id, ip)
2020-04-13 13:03:28.422 25384 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/l3_tc_lib.py", line 82, in _get_filterid_for_ip
2020-04-13 13:03:28.422 25384 ERROR neutron.agent.l3.agent     filterids_for_ip.append(filter_id)
2020-04-13 13:03:28.422 25384 ERROR neutron.agent.l3.agent UnboundLocalError: local variable 'filter_id' referenced before assignment
2020-04-13 13:03:28.422 25384 ERROR neutron.agent.l3.agent 
2020-04-13 14:21:43.519 25384 ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 9db7a07d-011b-4c1a-afb6-339f04f8d5b5: UnboundLocalError: local variable 'filter_id' referenced before assignment
2020-04-13 14:21:43.519 25384 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2020-04-13 14:21:43.519 25384 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 628, in _process_routers_if_compatible
2020-04-13 14:21:43.519 25384 ERROR neutron.agent.l3.agent     self._process_router_if_compatible(router)
2020-04-13 14:21:43.519 25384 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 486, in _process_router_if_compatible
2020-04-13 14:21:43.519 25384 ERROR neutron.agent.l3.agent     self._process_updated_router(router)
2020-04-13 14:21:43.519 25384 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 530, in _process_updated_router
2020-04-13 14:21:43.519 25384 ERROR neutron.agent.l3.agent     self.l3_ext_manager.update_router(self.context, router)
2020-04-13 14:21:43.519 25384 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/l3_agent_extensions_manager.py", line 54, in update_router
2020-04-13 14:21:43.519 25384 ERROR neutron.agent.l3.agent     extension.obj.update_router(context, data)
2020-04-13 14:21:43.519 25384 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2020-04-13 14:21:43.519 25384 ERROR neutron.agent.l3.agent     return f(*args, **kwargs)
2020-04-13 14:21:43.519 25384 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/extensions/fip_qos.py", line 407, in update_router
2020-04-13 14:21:43.519 25384 ERROR neutron.agent.l3.agent     self.process_floating_ip_addresses(context, router_info)
2020-04-13 14:21:43.519 25384 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/extensions/fip_qos.py", line 369, in process_floating_ip_addresses
2020-04-13 14:21:43.519 25384 ERROR neutron.agent.l3.agent     self.process_ip_rates(fip_addr, device, rates)
2020-04-13 14:21:43.519 25384 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/extensions/fip_qos.py", line 296, in process_ip_rates
2020-04-13 14:21:43.519 25384 ERROR neutron.agent.l3.agent     rate['rate'], rate['burst'])
2020-04-13 14:21:43.519 25384 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/extensions/fip_qos.py", line 210, in process_ip_rate_limit
2020-04-13 14:21:43.519 25384 ERROR neutron.agent.l3.agent     tc_wrapper.clear_ip_rate_limit(direction, ip)
2020-04-13 14:21:43.519 25384 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/l3_tc_lib.py", line 188, in clear_ip_rate_limit
2020-04-13 14:21:43.519 25384 ERROR neutron.agent.l3.agent     filter_id = self._get_filterid_for_ip(qdisc_id, ip)
2020-04-13 14:21:43.519 25384 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/l3_tc_lib.py", line 82, in _get_filterid_for_ip
2020-04-13 14:21:43.519 25384 ERROR neutron.agent.l3.agent     filterids_for_ip.append(filter_id)
2020-04-13 14:21:43.519 25384 ERROR neutron.agent.l3.agent UnboundLocalError: local variable 'filter_id' referenced before assignment
2020-04-13 14:21:43.519 25384 ERROR neutron.agent.l3.agent

Comment 2 Takashi Kajinami 2020-04-14 02:37:26 UTC
As described in bug description, it seems that they hit the exact same issue
as the one reported in upsteam bz[1]
 [1] https://bugs.launchpad.net/neutron/+bug/1809497

We require fix[2] since iproute2 supports chain index in tc_filter.
 [2] https://review.opendev.org/#/c/630773/

Support for chain index in tc_filter was backported in iproute2-4.11.0-23.el7 [3],
and I think we should backport the fix[1] to downstream neutron.

 [3] https://access.redhat.com/downloads/content/iproute/4.11.0-25.el7/x86_64/fd431d51/package-changelog
     https://bugzilla.redhat.com/show_bug.cgi?id=1714660

Note openstack-neutron-l3-agent:13.0-100 is used in the deployment where we observe the issue
 https://access.redhat.com/containers/?architecture=AMD64&tab=package-list#/registry.access.redhat.com/rhosp13/openstack-neutron-l3-agent/images/13.0-100

Comment 10 Slawek Kaplonski 2020-04-15 07:15:21 UTC
@Priscila: Workaround for that issue would be to downgrade iproute2 package to 4.11.0-22.el7 which don't have patch which broke Neutron but as rpm with this new version is already available I don't think that this would be good idea. Better IMO is simply update neutron package on nodes where L3 agent is running (in case of DVR it has to be updated on all compute nodes).

Yes, openstack-neutron-12.1.1-16.el7ost.src.rpm is rpm with source code which have fix for this issue.

Comment 22 errata-xmlrpc 2020-06-24 11:53:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2724