Description of problem: On running perf/scale tests on an OSP 12 cloud with 3 OpenStack controllers + 3 ODLs + 28 computes, by creating 100s of neutron resources and deleting them we see OOM on an ODL instance. Version-Release number of selected component (if applicable): OSP12 Puddle: 2017-10-31.2 ODL RPM: opendaylight-6.2.0-3.el7ost.noarch How reproducible: Steps to Reproduce: 1. Run perf/scale tests using Browbeat+neutron and create large number of neutro nresources 2. 3. Actual results: one of the ODLs dies due to OOM Expected results: There should be no OOM Additional info:
STDOUT of JVM when exiting: https://gist.github.com/smalleni/538bf2760ba9fbad5f47e76421fa6589
We hit this gain and here is a more complete output of the JVM before exiting https://gist.github.com/smalleni/3b2febfca36c1a6ae5b41b295a9ebf84
Over the last 2 weeks, we've invested significant effort into addressing what we suspect was the root cause of this OOM (plugging MD SAL Transaction leaks found by using http://blog2.vorburger.ch/2017/09/how-to-find-transaction-related-memory.html) upstream under https://jira.opendaylight.org/browse/NETVIRT-985, and are currently awaiting confirmation from Reporter re. whether that did the trick and fixes this problem...
*** Bug 1512074 has been marked as a duplicate of this bug. ***
Michael, has this been solved? If so please update the bug appropriately
*** Bug 1451401 has been marked as a duplicate of this bug. ***
Removing needinfo on my and putting back needinfo from smalleni .. will also email.
(In reply to Michael Vorburger from comment #3) > Over the last 2 weeks, we've invested significant effort into addressing > what we suspect was the root cause of this OOM (plugging MD SAL Transaction > leaks found by using > http://blog2.vorburger.ch/2017/09/how-to-find-transaction-related-memory. > html) upstream under https://jira.opendaylight.org/browse/NETVIRT-985, and > are currently awaiting confirmation from Reporter re. whether that did the > trick and fixes this problem... Closing this bug as there were many fixes done in Netvirt/OVSDB to handle these issues. We can re-open this or create a new one if we encounter the issue in future.