Bug 1519783

Summary: [Netvirt] Networking-odl trying to delete a router which was never pushed to ODL
Product: Red Hat OpenStack Reporter: Sridhar Gaddam <sgaddam>
Component: python-networking-odlAssignee: Mike Kolesnik <mkolesni>
Status: CLOSED ERRATA QA Contact: Tomas Jamrisko <tjamrisk>
Severity: high Docs Contact:
Priority: high    
Version: 12.0 (Pike)CC: aadam, alolivei, asuryana, jluhrsen, lpeer, sclewis, sgaddam, smalleni, tjamrisk, trozet, wznoinsk
Target Milestone: z1Keywords: Triaged, ZStream
Target Release: 13.0 (Queens)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: Netvirt
Fixed In Version: python-networking-odl-12.0.1-0.20180228140715.8733cf6.el7ost Doc Type: Known Issue
Doc Text:
Neutron may issue an error claiming that the Quota has been exceed for Neutron Router creation. This is a known issue where multiple router resources are created with a single create request in Neutron DB due to a bug with networking-odl. The workaround for this issue is to delete the duplicated routers using the OpenStack Neutron CLI and create a router again, resulting with a single instance.
 Story Points: ---
Clone Of: Environment:
N/A
Last Closed: 2018-07-19 13:53:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Neutron Server Logs none

Description Sridhar Gaddam 2017-12-01 13:02:54 UTC 
Description of problem:
When running Browbeat+Rally scenario with netcreate-boot-ping scenario (where we spawn a VM, associate a FIP and then try to ping the FIP with concurrency set to 5 and times set to 500), we observed the following in neutron server logs.

Setup:
OSP12 image
ODL running in a non-cluster, non-container setup with 1 controller and 32 compute nodes.
Port-status is disabled in this setup and inactivity_probe/max_backoff are appropriately configured.

RPMs used:
python-networking-odl-11.0.0-2.el7ost.noarch
opendaylight-6.2.0-5.el7ost.noarch

2017-12-01 12:05:16.021 948413 ERROR networking_odl.common.client Traceback (most recent call last):
2017-12-01 12:05:16.021 948413 ERROR networking_odl.common.client   File "/usr/lib/python2.7/site-packages/networking_odl/common/client.py", line 136, in _check_response
2017-12-01 12:05:16.021 948413 ERROR networking_odl.common.client     response.raise_for_status()
2017-12-01 12:05:16.021 948413 ERROR networking_odl.common.client   File "/usr/lib/python2.7/site-packages/requests/models.py", line 862, in raise_for_status
2017-12-01 12:05:16.021 948413 ERROR networking_odl.common.client     raise HTTPError(http_error_msg, response=self)
2017-12-01 12:05:16.021 948413 ERROR networking_odl.common.client HTTPError: 404 Client Error: Not Found for url: http://172.16.0.13:8081/controller/nb/v2/neutron/routers/659a6590-1523-49c7-a845-b5b195e3505c
2017-12-01 12:05:16.021 948413 ERROR networking_odl.common.client
2017-12-01 12:05:16.022 948413 ERROR networking_odl.common.client [req-ec69f007-0638-46bd-b884-8aa140469c4d - - - - -] REST request ( delete ) to url ( routers/659a6590-1523-49c7-a845-b5b195e3505c ) is failed. Request body : [None] service: HTTPError: 404 Client Error: Not Found for url: http://172.16.0.13:8081/controller/nb/v2/neutron/routers/659a6590-1523-49c7-a845-b5b195e3505c
2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal [req-ec69f007-0638-46bd-b884-8aa140469c4d - - - - -] Error while processing (Entry ID: 3013) - delete router 659a6590-1523-49c7-a845-b5b195e3505c: HTTPError: 404 Client Error: Not Found for url: http://172.16.0.13:8081/controller/nb/v2/neutron/routers/659a6590-1523-49c7-a845-b5b195e3505c
2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal Traceback (most recent call last):
2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal   File "/usr/lib/python2.7/site-packages/networking_odl/journal/journal.py", line 298, in _sync_entry
2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal     self.client.sendjson(method, urlpath, to_send)
2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal   File "/usr/lib/python2.7/site-packages/networking_odl/common/client.py", line 106, in sendjson
2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal     'body': obj})
2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal     self.force_reraise()
2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal     six.reraise(self.type_, self.value, self.tb)
2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal   File "/usr/lib/python2.7/site-packages/networking_odl/common/client.py", line 98, in sendjson
2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal     self.request(method, urlpath, data))
2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal   File "/usr/lib/python2.7/site-packages/networking_odl/common/client.py", line 140, in _check_response
2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal     {'e': error, 'text': response.text}, exc_info=1)
2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal     self.force_reraise()
2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal     six.reraise(self.type_, self.value, self.tb)
2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal   File "/usr/lib/python2.7/site-packages/networking_odl/common/client.py", line 136, in _check_response
2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal     response.raise_for_status()
2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal   File "/usr/lib/python2.7/site-packages/requests/models.py", line 862, in raise_for_status
2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal     raise HTTPError(http_error_msg, response=self)
2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal HTTPError: 404 Client Error: Not Found for url: http://172.16.0.13:8081/controller/nb/v2/neutron/routers/659a6590-1523-49c7-a845-b5b195e3505c
2017-12-01 12:05:16.023 948413 ERROR networking_odl.journal.journal

I took a random neutron router UUID (i.e., 21e9ccb0-cb43-40d6-811c-f63fcbce8607) for which we were seeing the HTTPError 404.
In the logs, there is no "create router *" for this UUID. The first reference to this UUID was as part of router_gateway port creation.
Then we see the following log

2017-12-01 12:07:08.088 948417 DEBUG neutron.api.rpc.agentnotifiers.l3_rpc_agent_api [req-b5532c6b-c2c0-46b5-a461-1ad7a04190a8 4f3065ec9a8b40ab8291b285cdd33fdd f505a5e332c5463c92ac74c48badc286 - default default] Fanout notify agent at l3_agent the message router_deleted on router 21e9ccb0-cb43-40d6-811c-f63fcbce8607 _notification_fanout /usr/lib/python2.7/site-packages/neutron/api/rpc/agentnotifiers/l3_rpc_agent_api.py:118

So something seems to be going wrong on the neutron/networking-odl side as its trying to delete a resource (i.e., router) which was never pushed to ODL.
Comment 1 Sridhar Gaddam 2017-12-01 13:13:50 UTC 
Created attachment 1361601 [details]
Neutron Server Logs
Comment 2 Sai Sindhur Malleni 2017-12-01 13:22:46 UTC 
Similar Bug from last iteration: https://bugzilla.redhat.com/show_bug.cgi?id=1487794
Comment 3 Mike Kolesnik 2017-12-12 10:17:55 UTC 
Is there an upstream bug for this?
Comment 4 Sridhar Gaddam 2017-12-14 17:39:10 UTC 
Upstream bug: https://bugs.launchpad.net/networking-odl/+bug/1738246
Comment 5 Mike Kolesnik 2018-02-19 11:40:12 UTC 
This should be fixed as part of L3 Flavors, which should be available in OSP 14
Comment 7 Mike Kolesnik 2018-04-22 07:03:11 UTC 
*** Bug 1569696 has been marked as a duplicate of this bug. ***
Comment 8 Mike Kolesnik 2018-05-21 12:11:52 UTC 
*** Bug 1556686 has been marked as a duplicate of this bug. ***
Comment 9 Mike Kolesnik 2018-05-23 08:40:20 UTC 
I will work on a stopgap solution for this that sjhould cover 99% of the affected cases.
The complete solution would be L3 Falvors which will be available in OSP 14.
Comment 11 jamo luhrsen 2018-05-23 16:43:32 UTC 
(In reply to Mike Kolesnik from comment #9)
> I will work on a stopgap solution for this that sjhould cover 99% of the
> affected cases.
> The complete solution would be L3 Falvors which will be available in OSP 14.

Thanks Mike. Let me know if I can help. I can usually get a setup to reproduce
this pretty easily and can try/verify/test things as needed.
Comment 12 Mike Kolesnik 2018-05-29 11:58:27 UTC 
Posted upstream for review
Comment 23 errata-xmlrpc 2018-07-19 13:53:05 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2215