Bug 1490714 - [Infra] ODL resources cleanup issue
Summary: [Infra] ODL resources cleanup issue
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: opendaylight
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: z4
: 13.0 (Queens)
Assignee: Victor Pickard
QA Contact: Sai Sindhur Malleni
URL:
Whiteboard: Infra
Depends On:
Blocks: 1632402
TreeView+ depends on / blocked
 
Reported: 2017-09-12 06:15 UTC by Sridhar Gaddam
Modified: 2018-10-18 07:24 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1632402 (view as bug list)
Environment:
N/A
Last Closed: 2018-09-13 11:28:35 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
OpenDaylight Bug NETVIRT-989 None None None 2018-03-21 11:33:46 UTC
OpenDaylight Bug NETVIRT-999 None None None 2018-03-21 11:33:00 UTC
OpenDaylight gerrit 65725 None None None 2018-03-21 11:31:06 UTC
OpenDaylight gerrit 65728 None None None 2018-03-21 11:32:16 UTC
Red Hat Product Errata RHEA-2018:2086 None None None 2018-06-27 13:37:03 UTC

Description Sridhar Gaddam 2017-09-12 06:15:08 UTC
Description of problem:
After running rally scenarios twice (one with times set to 500, followed by times set to 1000), we noticed that a good number of resources (like ELAN interfaces, neutron ports etc) are present in ODL even though rally deleted the corresponding Neutron resources (and they are deleted from Neutron/opendaylightjournal DB).

Version-Release number of selected component (if applicable):
RH OSP12 

Opendaylight rpm used:
opendaylight-6.2.0-0.1.20170906snap55.el7.noarch
python-networking-odl-11.0.0-0.20170806093629.2e78dca.el7ost.noarch

How reproducible:
Run browbeat/rally neutron scenarios with concurrency of 8, 16, 32 and times set to 500. Allow the tests to complete.
Verify that all the neutron resources are deleted.
Verify that opendaylightjournal is clean.

Now look at ODL resources and ideally it should be empty, but you can see that elan:show, neutron-port-show (in karaf cli) returns lots of entries.

Comment 1 Victor Pickard 2017-09-12 20:49:59 UTC
Hi Sridhar,
Were these tests done with a single ODL instance, or was this an ODL cluster?

Comment 2 Sridhar Gaddam 2017-09-13 02:00:52 UTC
(In reply to Victor Pickard from comment #1)
> Hi Sridhar,
> Were these tests done with a single ODL instance, or was this an ODL cluster?

It was an ODL Cluster setup with 3 controllers and 3 compute nodes. But, I think, the resource cleanup issue would be seen even on a single controller as well.

Comment 5 Mike Kolesnik 2018-03-21 08:29:22 UTC
Vic,

Whats the status on this bug?

Comment 6 Victor Pickard 2018-03-21 11:35:35 UTC
I have added the 2 ODL Bugs and 2 gerrit patches for resources that were cleaned up.

I suspect there may be additional resources to be cleaned up, this is all that was found in local setup.

Comment 8 Victor Pickard 2018-03-21 14:07:03 UTC
Fixes merged upstream and patches are d/s. Moving to modified.

Comment 12 Sai Sindhur Malleni 2018-04-15 14:40:27 UTC
This problem still seems to exits in OSP 13 + ODL Oxygen.

==============================================================================
MariaDB [ovs_neutron]> select count(*) from opendaylightjournal;
+----------+
| count(*) |
+----------+
|        0 |
+----------+
===============================================================================
karaf@root()> show:elan
Command not found: show:elan
karaf@root()> elan:show
Elan Instance                       Mac-TimeOut          Tag                  
----------------------------------------------------------------------------------------------
bd8db3a8-2b30-4083-a8b3-b3fd46401142 300                       5500            null            
6d23f5c2-a28e-464f-87c9-246dac584ac7 300                       5501            null 
================================================================================
[root@overcloud-controller-0 neutron]# grep bd8db3a8-2b30-4083-a8b3-b3fd46401142 server.log*
server.log.6:2018-04-13 22:24:14.018 51 INFO networking_odl.journal.journal [req-774db86c-3e2a-45a2-a815-54564a8f2c25 - - - - -] Processing (Entry ID: 2) - create network bd8db3a8-2b30-4083-
a8b3-b3fd46401142 (Time stamp: 63659255054.0)
server.log.6:2018-04-13 22:24:14.131 51 INFO networking_odl.journal.journal [req-774db86c-3e2a-45a2-a815-54564a8f2c25 - - - - -] Completed (Entry ID: 2) - create network bd8db3a8-2b30-4083-a
8b3-b3fd46401142 (Time stamp: 63659255054.1)


I do not see an entry for the network deletion in the neutron-server logs, although I see an entry for creation.

Comment 13 jamo luhrsen 2018-05-01 20:35:48 UTC
you guys think this could be the same issue(s) reported here?

https://bugzilla.redhat.com/show_bug.cgi?id=1556686

Comment 14 Mike Kolesnik 2018-06-17 05:51:52 UTC
(In reply to Sai Sindhur Malleni from comment #12)
> This problem still seems to exits in OSP 13 + ODL Oxygen.
> 
> =============================================================================
> =
> MariaDB [ovs_neutron]> select count(*) from opendaylightjournal;
> +----------+
> | count(*) |
> +----------+
> |        0 |
> +----------+
> =============================================================================
> ==
> karaf@root()> show:elan
> Command not found: show:elan
> karaf@root()> elan:show
> Elan Instance                       Mac-TimeOut          Tag                
> 
> -----------------------------------------------------------------------------
> -----------------
> bd8db3a8-2b30-4083-a8b3-b3fd46401142 300                       5500         
> null            
> 6d23f5c2-a28e-464f-87c9-246dac584ac7 300                       5501         
> null 
> =============================================================================
> ===
> [root@overcloud-controller-0 neutron]# grep
> bd8db3a8-2b30-4083-a8b3-b3fd46401142 server.log*
> server.log.6:2018-04-13 22:24:14.018 51 INFO networking_odl.journal.journal
> [req-774db86c-3e2a-45a2-a815-54564a8f2c25 - - - - -] Processing (Entry ID:
> 2) - create network bd8db3a8-2b30-4083-
> a8b3-b3fd46401142 (Time stamp: 63659255054.0)
> server.log.6:2018-04-13 22:24:14.131 51 INFO networking_odl.journal.journal
> [req-774db86c-3e2a-45a2-a815-54564a8f2c25 - - - - -] Completed (Entry ID: 2)
> - create network bd8db3a8-2b30-4083-a
> 8b3-b3fd46401142 (Time stamp: 63659255054.1)
> 
> 
> I do not see an entry for the network deletion in the neutron-server logs,
> although I see an entry for creation.

Is there a chance you can reproduce this and attach the logs, so we can further investigate?

Comment 16 errata-xmlrpc 2018-06-27 13:36:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086

Comment 17 Sai Sindhur Malleni 2018-07-30 07:38:09 UTC
This problem is still present in OSP13.

Comment 18 Sai Sindhur Malleni 2018-07-30 07:40:56 UTC
Logs and ODL model dumps: http://file.rdu.redhat.com/~smalleni/rhbz-1490714.gz

Comment 21 Mike Kolesnik 2018-07-30 10:30:48 UTC
I checked for ports 3b4db743-f240-49c0-9452-ed94d92ea6ad, e876511b-dff3-4a8b-be67-172f3abd774b and they were deleted on Neutron but sopposedly still on ODL so I reckon this is a problem in ODL internals.

Comment 24 Jon Schlueter 2018-09-13 11:28:35 UTC
Please clone or open a new bug if this is still an issue.

Comment 26 Victor Pickard 2018-09-24 17:22:56 UTC
Cloned this bug, will address in OSP14.

https://bugzilla.redhat.com/show_bug.cgi?id=1632402


Note You need to log in before you can comment on or make changes to this bug.