Bug 1479264 - Ever-increasing memory consumption causes the Controller to crash
Ever-increasing memory consumption causes the Controller to crash
Status: POST
Product: Red Hat OpenStack
Classification: Red Hat
Component: opendaylight (Show other bugs)
12.0 (Pike)
All Linux
urgent Severity urgent
: beta
: 12.0 (Pike)
Assigned To: Stephen Kitt
Itzik Brown
: Triaged
Depends On:
Blocks: 1451401
  Show dependency treegraph
 
Reported: 2017-08-08 04:36 EDT by Stephen Kitt
Modified: 2017-10-23 06:34 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
OpenDaylight Bug 8941 None None None 2017-08-08 04:36 EDT
OpenDaylight gerrit 61284 None None None 2017-08-08 07:51 EDT

  None (edit)
Description Stephen Kitt 2017-08-08 04:36:13 EDT
On a single-node Carbon SR1 setup, we got an OOM over the weekend. Looking at the heap dump shows 1.1GB occupied in the frontend history's closed transactions map, with 33M entries. Analysing the dump further reveals that there's a bug in the controller's transaction handling, which results in transactions being held in memory forever — which guarantees an OOM eventually, regardless of the JVM settings.

The heap dump is on https://www.sk2.org/java_pid2098.hprof.xz (269MB).
Comment 2 Stephen Kitt 2017-09-20 09:28:24 EDT
This has been fixed in Carbon upstream; it will be part of SR2.

Note You need to log in before you can comment on or make changes to this bug.