Bug 1387271

Summary: Hawkular Services Running In Container Became Unresponsive
Product: [JBoss] Middleware Manager Reporter: Matt Mahoney <mmahoney>
Component: OtherAssignee: Heiko W. Rupp <hrupp>
Status: VERIFIED --- QA Contact: Matt Mahoney <mmahoney>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 7.0.0 TP2CC: jhardy, mmahoney
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 1390739    
Bug Blocks:    
Attachments:
Description Flags
hawkular-Services Log
none
evm.log.zip none

Description Matt Mahoney 2016-10-20 14:00:17 UTC
Created attachment 1212524 [details]
hawkular-Services Log

Description of problem:
After a couple of days with Hawkular-Services running in a container, HS became unresponsive. Navigating to the HS URL showed all services Green. But, attempting to access the API(s) resulted in failed connection attempts. 

After removing the HS Provider from CFME, and the attempting to add as provider again, the following Refresh error was encountered:

Error - 6 Minutes Ago
Timed out reading data from server 


Version-Release number of selected component (if applicable):
DR6

How reproducible:


Steps to Reproduce:
1. start Hawkular-Services and Cassandra nodes
2. Add EAP standalone and EAP Domain servers, under HS management.
3. Ran various automation tests (no specific tests)
4. HS API became unresponsive

Actual results:
HS API became unresponsive

Expected results:

Additional info:

Comment 2 Matt Mahoney 2016-10-20 14:09:29 UTC
Created attachment 1212541 [details]
evm.log.zip

Comment 3 Heiko W. Rupp 2016-10-20 14:12:05 UTC
Are you running with external postgres or without?
If without, it is likely memory exhaustion in the container.
If the container still exists, can you docker logs <container id> and attach?

Comment 5 Heiko W. Rupp 2016-10-20 14:39:02 UTC
So this is using hsqldb, which is not for production.
I did not mean the memory of the host, but of the JVM inside the container
and this indeed shows a out of memory exception.
There may be other issues.

Comment 6 Heiko W. Rupp 2016-11-09 11:45:29 UTC
I think the issues are fixed.