Bug 1479264 - Ever-increasing memory consumption causes the Controller to crash
Summary: Ever-increasing memory consumption causes the Controller to crash
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: opendaylight
Version: 12.0 (Pike)
Hardware: All
OS: Linux
urgent
urgent
Target Milestone: beta
: 13.0 (Queens)
Assignee: Stephen Kitt
QA Contact: Tomas Jamrisko
URL:
Whiteboard:
Depends On:
Blocks: 1451401
TreeView+ depends on / blocked
 
Reported: 2017-08-08 08:36 UTC by Stephen Kitt
Modified: 2018-10-18 07:20 UTC (History)
8 users (show)

Fixed In Version: opendaylight-8.0.0-1.el7ost
Doc Type: No Doc Update
Doc Text:
undefined
Clone Of:
Environment:
N/A
Last Closed: 2018-06-27 13:33:53 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenDaylight Bug 8941 0 None None None 2017-08-08 08:36:13 UTC
OpenDaylight gerrit 61284 0 None None None 2017-08-08 11:51:34 UTC
Red Hat Product Errata RHEA-2018:2086 0 None None None 2018-06-27 13:34:45 UTC

Description Stephen Kitt 2017-08-08 08:36:13 UTC
On a single-node Carbon SR1 setup, we got an OOM over the weekend. Looking at the heap dump shows 1.1GB occupied in the frontend history's closed transactions map, with 33M entries. Analysing the dump further reveals that there's a bug in the controller's transaction handling, which results in transactions being held in memory forever — which guarantees an OOM eventually, regardless of the JVM settings.

The heap dump is on https://www.sk2.org/java_pid2098.hprof.xz (269MB).

Comment 2 Stephen Kitt 2017-09-20 13:28:24 UTC
This has been fixed in Carbon upstream; it will be part of SR2.

Comment 14 errata-xmlrpc 2018-06-27 13:33:53 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086


Note You need to log in before you can comment on or make changes to this bug.