Bug 1433061

Summary: Exceptions in Hawkular-Metrics pod
Product: OpenShift Container Platform Reporter: Viet Nguyen <vnguyen>
Component: HawkularAssignee: Matt Wringe <mwringe>
Status: CLOSED NOTABUG QA Contact: Peng Li <penli>
Severity: medium Docs Contact:
Priority: unspecified    
Version: unspecifiedCC: aos-bugs
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1435436 (view as bug list) Environment:
Last Closed: 2017-03-23 16:01:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1435436    
Attachments:
Description Flags
log files none

Description Viet Nguyen 2017-03-16 16:54:45 UTC 
Description of problem:
- Single VM with 8G of RAM, 1node/1master on the same VM
- After running 30 nodejs test pods I started seeing exceptions in hawkular-metrics log. I then deleted hawkular-metrics and cassandra pods but the new pods didn't fix the problem.

Version-Release number of selected component (if applicable):
- Origin: 1.4.1 
- Metrics: 1.4.1
- Centos 7
- Deployed on RedHat public OS1, one single VM with 8GB of RAM

How reproducible:
100%

Steps to Reproduce:
1. Install Metrics as instructed
2. Install Hawkular OpenShift Agent
3. Deploy some user pods
4. After several days of running and collecting metrics CPU,RAM, Network graphs are blank. I attempted to delete Hawkular-Metrics and Cassandra pods to no avail.  

Actual results:
Exceptions in hawkular-metrics pod. No metrics data returns from a python test client

Expected results:
- UI displays metrics
- 

Additional info:
Comment 1 Viet Nguyen 2017-03-16 16:59:34 UTC 
Created attachment 1263754 [details]
log files
Comment 2 Matt Wringe 2017-03-16 17:44:50 UTC 
Do you have any resource limit applied to the Hawkular Metrics, Cassandra and Heapster pods?

If you are trying to do scalability testing, you need to be running the infrastructure components on one OpenShift node and have the pods to be monitored on another. Trying to run everything on one pod for testing purposes is not a good idea, all your pods will be competing for resources.