Bug 1331793

Summary: Instance of candlepin 2.0.11 unresponsive with minimal load
Product: [Community] Candlepin Reporter: Tramaine Darby <tdarby>
Component: candlepinAssignee: Chris "Ceiu" Rog <crog>
Status: CLOSED CURRENTRELEASE QA Contact: Katello QA List <katello-qa-list>
Severity: high Docs Contact:
Priority: high    
Version: 2.0CC: bcourt, redakkan, tpfromme, vrjain
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: candlepin-2.0.12-1 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-05-13 18:48:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
Heap dump from candlepin06 while tomcat was in unresponsive state
none
Thread dump none

Description Tramaine Darby 2016-04-29 14:11:09 UTC 
Description of problem:
Candlepin 2.0.11 is deployed in the IT-PnT development environment on candlepin06.web.dev.int.devlab.redhat.com.  A few requests is enough to render the tomcat unresponsive.  We suspect a memory leak

Version-Release number of selected component (if applicable):
2.0.11

How reproducible:
Becomes unresponsive approximately an hour after a restart.  Restart seems to clear it up temporarily 

Steps to Reproduce:
1. Start tomcat
2. Make status requests and pool refresh requests
3. Observe lengthening response times

Actual results:
Requests eventually time out due to an unresponsive server

Expected results:
Requests should be handled normally

Additional info:
Heap dump from unresponsive state attached
Comment 1 Tramaine Darby 2016-04-29 14:14:07 UTC 
Created attachment 1152326 [details]
Heap dump from candlepin06 while tomcat was in unresponsive state
Comment 2 Tramaine Darby 2016-05-02 19:16:57 UTC 
Looking back at the history - these pool refreshes and status calls were the only traffic before the application became unresponsive:

curl -X PUT -H "Authorization: Basic Y2FuZGxlcGluX2FkbWluOmNhbmRsZXBpbl9hZG1pbg==" -H "Cache-Control: no-cache" "http://candlepin06.web.dev.int.devlab.redhat.com:8080/candlepin/owners/7388484/subscriptions?auto_create_owner=true"


curl -X PUT -H "Authorization: Basic Y2FuZGxlcGluX2FkbWluOmNhbmRsZXBpbl9hZG1pbg==" -H "Cache-Control: no-cache" "http://candlepin06.web.dev.int.devlab.redhat.com:8080/candlepin/owners/6302666/subscriptions?auto_create_owner=true"
Comment 3 Tramaine Darby 2016-05-02 20:19:59 UTC 
Created attachment 1153034 [details]
Thread dump