Bug 1490714

Summary: [Infra] ODL resources cleanup issue
Product: Red Hat OpenStack Reporter: Sridhar Gaddam <sgaddam>
Component: opendaylightAssignee: Victor Pickard <vpickard>
Status: CLOSED ERRATA QA Contact: Sai Sindhur Malleni <smalleni>
Severity: high Docs Contact:
Priority: high    
Version: 13.0 (Queens)CC: aadam, jluhrsen, jschluet, mkolesni, nyechiel, sclewis, smalleni, trozet, vpickard
Target Milestone: z4Keywords: Reopened, Triaged, ZStream
Target Release: 13.0 (Queens)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: Infra
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1632402 (view as bug list) Environment:
N/A
Last Closed: 2018-09-13 11:28:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1632402    

Description Sridhar Gaddam 2017-09-12 06:15:08 UTC
Description of problem:
After running rally scenarios twice (one with times set to 500, followed by times set to 1000), we noticed that a good number of resources (like ELAN interfaces, neutron ports etc) are present in ODL even though rally deleted the corresponding Neutron resources (and they are deleted from Neutron/opendaylightjournal DB).

Version-Release number of selected component (if applicable):
RH OSP12 

Opendaylight rpm used:
opendaylight-6.2.0-0.1.20170906snap55.el7.noarch
python-networking-odl-11.0.0-0.20170806093629.2e78dca.el7ost.noarch

How reproducible:
Run browbeat/rally neutron scenarios with concurrency of 8, 16, 32 and times set to 500. Allow the tests to complete.
Verify that all the neutron resources are deleted.
Verify that opendaylightjournal is clean.

Now look at ODL resources and ideally it should be empty, but you can see that elan:show, neutron-port-show (in karaf cli) returns lots of entries.

Comment 1 Victor Pickard 2017-09-12 20:49:59 UTC
Hi Sridhar,
Were these tests done with a single ODL instance, or was this an ODL cluster?

Comment 2 Sridhar Gaddam 2017-09-13 02:00:52 UTC
(In reply to Victor Pickard from comment #1)
> Hi Sridhar,
> Were these tests done with a single ODL instance, or was this an ODL cluster?

It was an ODL Cluster setup with 3 controllers and 3 compute nodes. But, I think, the resource cleanup issue would be seen even on a single controller as well.

Comment 5 Mike Kolesnik 2018-03-21 08:29:22 UTC
Vic,

Whats the status on this bug?

Comment 6 Victor Pickard 2018-03-21 11:35:35 UTC
I have added the 2 ODL Bugs and 2 gerrit patches for resources that were cleaned up.

I suspect there may be additional resources to be cleaned up, this is all that was found in local setup.

Comment 8 Victor Pickard 2018-03-21 14:07:03 UTC
Fixes merged upstream and patches are d/s. Moving to modified.

Comment 12 Sai Sindhur Malleni 2018-04-15 14:40:27 UTC
This problem still seems to exits in OSP 13 + ODL Oxygen.

==============================================================================
MariaDB [ovs_neutron]> select count(*) from opendaylightjournal;
+----------+
| count(*) |
+----------+
|        0 |
+----------+
===============================================================================
karaf@root()> show:elan
Command not found: show:elan
karaf@root()> elan:show
Elan Instance                       Mac-TimeOut          Tag                  
----------------------------------------------------------------------------------------------
bd8db3a8-2b30-4083-a8b3-b3fd46401142 300                       5500            null            
6d23f5c2-a28e-464f-87c9-246dac584ac7 300                       5501            null 
================================================================================
[root@overcloud-controller-0 neutron]# grep bd8db3a8-2b30-4083-a8b3-b3fd46401142 server.log*
server.log.6:2018-04-13 22:24:14.018 51 INFO networking_odl.journal.journal [req-774db86c-3e2a-45a2-a815-54564a8f2c25 - - - - -] Processing (Entry ID: 2) - create network bd8db3a8-2b30-4083-
a8b3-b3fd46401142 (Time stamp: 63659255054.0)
server.log.6:2018-04-13 22:24:14.131 51 INFO networking_odl.journal.journal [req-774db86c-3e2a-45a2-a815-54564a8f2c25 - - - - -] Completed (Entry ID: 2) - create network bd8db3a8-2b30-4083-a
8b3-b3fd46401142 (Time stamp: 63659255054.1)


I do not see an entry for the network deletion in the neutron-server logs, although I see an entry for creation.

Comment 13 jamo luhrsen 2018-05-01 20:35:48 UTC
you guys think this could be the same issue(s) reported here?

https://bugzilla.redhat.com/show_bug.cgi?id=1556686

Comment 14 Mike Kolesnik 2018-06-17 05:51:52 UTC
(In reply to Sai Sindhur Malleni from comment #12)
> This problem still seems to exits in OSP 13 + ODL Oxygen.
> 
> =============================================================================
> =
> MariaDB [ovs_neutron]> select count(*) from opendaylightjournal;
> +----------+
> | count(*) |
> +----------+
> |        0 |
> +----------+
> =============================================================================
> ==
> karaf@root()> show:elan
> Command not found: show:elan
> karaf@root()> elan:show
> Elan Instance                       Mac-TimeOut          Tag                
> 
> -----------------------------------------------------------------------------
> -----------------
> bd8db3a8-2b30-4083-a8b3-b3fd46401142 300                       5500         
> null            
> 6d23f5c2-a28e-464f-87c9-246dac584ac7 300                       5501         
> null 
> =============================================================================
> ===
> [root@overcloud-controller-0 neutron]# grep
> bd8db3a8-2b30-4083-a8b3-b3fd46401142 server.log*
> server.log.6:2018-04-13 22:24:14.018 51 INFO networking_odl.journal.journal
> [req-774db86c-3e2a-45a2-a815-54564a8f2c25 - - - - -] Processing (Entry ID:
> 2) - create network bd8db3a8-2b30-4083-
> a8b3-b3fd46401142 (Time stamp: 63659255054.0)
> server.log.6:2018-04-13 22:24:14.131 51 INFO networking_odl.journal.journal
> [req-774db86c-3e2a-45a2-a815-54564a8f2c25 - - - - -] Completed (Entry ID: 2)
> - create network bd8db3a8-2b30-4083-a
> 8b3-b3fd46401142 (Time stamp: 63659255054.1)
> 
> 
> I do not see an entry for the network deletion in the neutron-server logs,
> although I see an entry for creation.

Is there a chance you can reproduce this and attach the logs, so we can further investigate?

Comment 16 errata-xmlrpc 2018-06-27 13:36:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086

Comment 17 Sai Sindhur Malleni 2018-07-30 07:38:09 UTC
This problem is still present in OSP13.

Comment 18 Sai Sindhur Malleni 2018-07-30 07:40:56 UTC
Logs and ODL model dumps: http://file.rdu.redhat.com/~smalleni/rhbz-1490714.gz

Comment 21 Mike Kolesnik 2018-07-30 10:30:48 UTC
I checked for ports 3b4db743-f240-49c0-9452-ed94d92ea6ad, e876511b-dff3-4a8b-be67-172f3abd774b and they were deleted on Neutron but sopposedly still on ODL so I reckon this is a problem in ODL internals.

Comment 24 Jon Schlueter 2018-09-13 11:28:35 UTC
Please clone or open a new bug if this is still an issue.

Comment 26 Victor Pickard 2018-09-24 17:22:56 UTC
Cloned this bug, will address in OSP14.

https://bugzilla.redhat.com/show_bug.cgi?id=1632402