Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1512073

Summary:	java.lang.OutOfMemoryError: GC overhead limit exceeded
Product:	Red Hat OpenStack	Reporter:	Sai Sindhur Malleni <smalleni>
Component:	opendaylight	Assignee:	Michael Vorburger <vorburger>
Status:	CLOSED NEXTRELEASE	QA Contact:	Itzik Brown <itbrown>
Severity:	high	Docs Contact:
Priority:	high
Version:	12.0 (Pike)	CC:	mkolesni, nyechiel, sgaddam, smalleni, vorburger
Target Milestone:	---	Keywords:	Triaged, ZStream
Target Release:	12.0 (Pike)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:	N/A
Last Closed:	2018-02-19 13:20:20 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1439320

Description Sai Sindhur Malleni 2017-11-10 19:00:23 UTC

Description of problem:
On running perf/scale tests on an OSP 12 cloud with 3 OpenStack controllers + 3 ODLs + 28 computes, by creating 100s of neutron resources and deleting them we see OOM on an ODL instance.  

Version-Release number of selected component (if applicable):
OSP12
Puddle: 2017-10-31.2
ODL RPM: opendaylight-6.2.0-3.el7ost.noarch

How reproducible:


Steps to Reproduce:
1. Run perf/scale tests using Browbeat+neutron and create large number of neutro nresources
2.
3.

Actual results:
one of the ODLs dies due to OOM

Expected results:
There should be no OOM

Additional info:

Comment 1 Sai Sindhur Malleni 2017-11-10 19:02:56 UTC

STDOUT of JVM when exiting: https://gist.github.com/smalleni/538bf2760ba9fbad5f47e76421fa6589

Comment 2 Sai Sindhur Malleni 2017-11-12 23:23:10 UTC

We hit this gain and here is a more complete output of the JVM before exiting
https://gist.github.com/smalleni/3b2febfca36c1a6ae5b41b295a9ebf84

Comment 3 Michael Vorburger 2017-11-27 14:35:51 UTC

Over the last 2 weeks, we've invested significant effort into addressing what we suspect was the root cause of this OOM (plugging MD SAL Transaction leaks found by using http://blog2.vorburger.ch/2017/09/how-to-find-transaction-related-memory.html) upstream under https://jira.opendaylight.org/browse/NETVIRT-985, and are currently awaiting confirmation from Reporter re. whether that did the trick and fixes this problem...

Comment 5 Michael Vorburger 2017-11-27 14:58:43 UTC

*** Bug 1512074 has been marked as a duplicate of this bug. ***

Comment 6 Mike Kolesnik 2018-02-19 12:38:52 UTC

Michael, has this been solved?

If so please update the bug appropriately

Comment 7 Mike Kolesnik 2018-02-19 12:39:00 UTC

*** Bug 1451401 has been marked as a duplicate of this bug. ***

Comment 8 Michael Vorburger 2018-02-19 13:14:49 UTC

Removing needinfo on my and putting back needinfo from smalleni .. will also email.

Comment 9 Sridhar Gaddam 2018-02-19 13:20:20 UTC

(In reply to Michael Vorburger from comment #3)
> Over the last 2 weeks, we've invested significant effort into addressing
> what we suspect was the root cause of this OOM (plugging MD SAL Transaction
> leaks found by using
> http://blog2.vorburger.ch/2017/09/how-to-find-transaction-related-memory.
> html) upstream under https://jira.opendaylight.org/browse/NETVIRT-985, and
> are currently awaiting confirmation from Reporter re. whether that did the
> trick and fixes this problem...

Closing this bug as there were many fixes done in Netvirt/OVSDB to handle these issues. We can re-open this or create a new one if we encounter the issue in future.