Red Hat Bugzilla – Bug 1519783
[Netvirt] Networking-odl trying to delete a router which was never pushed to ODL
Last modified: 2018-10-18 03:20:42 EDT
Description of problem: When running Browbeat+Rally scenario with netcreate-boot-ping scenario (where we spawn a VM, associate a FIP and then try to ping the FIP with concurrency set to 5 and times set to 500), we observed the following in neutron server logs. Setup: OSP12 image ODL running in a non-cluster, non-container setup with 1 controller and 32 compute nodes. Port-status is disabled in this setup and inactivity_probe/max_backoff are appropriately configured. RPMs used: python-networking-odl-11.0.0-2.el7ost.noarch opendaylight-6.2.0-5.el7ost.noarch 2017-12-01 12:05:16.021 948413 ERROR networking_odl.common.client Traceback (most recent call last): 2017-12-01 12:05:16.021 948413 ERROR networking_odl.common.client File "/usr/lib/python2.7/site-packages/networking_odl/common/client.py", line 136, in _check_response 2017-12-01 12:05:16.021 948413 ERROR networking_odl.common.client response.raise_for_status() 2017-12-01 12:05:16.021 948413 ERROR networking_odl.common.client File "/usr/lib/python2.7/site-packages/requests/models.py", line 862, in raise_for_status 2017-12-01 12:05:16.021 948413 ERROR networking_odl.common.client raise HTTPError(http_error_msg, response=self) 2017-12-01 12:05:16.021 948413 ERROR networking_odl.common.client HTTPError: 404 Client Error: Not Found for url: http://172.16.0.13:8081/controller/nb/v2/neutron/routers/659a6590-1523-49c7-a845-b5b195e3505c 2017-12-01 12:05:16.021 948413 ERROR networking_odl.common.client 2017-12-01 12:05:16.022 948413 ERROR networking_odl.common.client [req-ec69f007-0638-46bd-b884-8aa140469c4d - - - - -] REST request ( delete ) to url ( routers/659a6590-1523-49c7-a845-b5b195e3505c ) is failed. Request body : [None] service: HTTPError: 404 Client Error: Not Found for url: http://172.16.0.13:8081/controller/nb/v2/neutron/routers/659a6590-1523-49c7-a845-b5b195e3505c 2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal [req-ec69f007-0638-46bd-b884-8aa140469c4d - - - - -] Error while processing (Entry ID: 3013) - delete router 659a6590-1523-49c7-a845-b5b195e3505c: HTTPError: 404 Client Error: Not Found for url: http://172.16.0.13:8081/controller/nb/v2/neutron/routers/659a6590-1523-49c7-a845-b5b195e3505c 2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal Traceback (most recent call last): 2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal File "/usr/lib/python2.7/site-packages/networking_odl/journal/journal.py", line 298, in _sync_entry 2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal self.client.sendjson(method, urlpath, to_send) 2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal File "/usr/lib/python2.7/site-packages/networking_odl/common/client.py", line 106, in sendjson 2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal 'body': obj}) 2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal self.force_reraise() 2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal six.reraise(self.type_, self.value, self.tb) 2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal File "/usr/lib/python2.7/site-packages/networking_odl/common/client.py", line 98, in sendjson 2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal self.request(method, urlpath, data)) 2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal File "/usr/lib/python2.7/site-packages/networking_odl/common/client.py", line 140, in _check_response 2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal {'e': error, 'text': response.text}, exc_info=1) 2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal self.force_reraise() 2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal six.reraise(self.type_, self.value, self.tb) 2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal File "/usr/lib/python2.7/site-packages/networking_odl/common/client.py", line 136, in _check_response 2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal response.raise_for_status() 2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal File "/usr/lib/python2.7/site-packages/requests/models.py", line 862, in raise_for_status 2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal raise HTTPError(http_error_msg, response=self) 2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal HTTPError: 404 Client Error: Not Found for url: http://172.16.0.13:8081/controller/nb/v2/neutron/routers/659a6590-1523-49c7-a845-b5b195e3505c 2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal I took a random neutron router UUID (i.e., 21e9ccb0-cb43-40d6-811c-f63fcbce8607) for which we were seeing the HTTPError 404. In the logs, there is no "create router *" for this UUID. The first reference to this UUID was as part of router_gateway port creation. Then we see the following log 2017-12-01 12:07:08.088 948417 DEBUG neutron.api.rpc.agentnotifiers.l3_rpc_agent_api [req-b5532c6b-c2c0-46b5-a461-1ad7a04190a8 4f3065ec9a8b40ab8291b285cdd33fdd f505a5e332c5463c92ac74c48badc286 - default default] Fanout notify agent at l3_agent the message router_deleted on router 21e9ccb0-cb43-40d6-811c-f63fcbce8607 _notification_fanout /usr/lib/python2.7/site-packages/neutron/api/rpc/agentnotifiers/l3_rpc_agent_api.py:118 So something seems to be going wrong on the neutron/networking-odl side as its trying to delete a resource (i.e., router) which was never pushed to ODL.
Created attachment 1361601 [details] Neutron Server Logs
Similar Bug from last iteration: https://bugzilla.redhat.com/show_bug.cgi?id=1487794
Is there an upstream bug for this?
Upstream bug: https://bugs.launchpad.net/networking-odl/+bug/1738246
This should be fixed as part of L3 Flavors, which should be available in OSP 14
*** Bug 1569696 has been marked as a duplicate of this bug. ***
*** Bug 1556686 has been marked as a duplicate of this bug. ***
I will work on a stopgap solution for this that sjhould cover 99% of the affected cases. The complete solution would be L3 Falvors which will be available in OSP 14.
(In reply to Mike Kolesnik from comment #9) > I will work on a stopgap solution for this that sjhould cover 99% of the > affected cases. > The complete solution would be L3 Falvors which will be available in OSP 14. Thanks Mike. Let me know if I can help. I can usually get a setup to reproduce this pretty easily and can try/verify/test things as needed.
Posted upstream for review
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:2215