+++ This bug was initially created as a clone of Bug #1559448 +++ +++ This bug was initially created as a clone of Bug #1559443 +++ +++ This bug was initially created as a clone of Bug #1559440 +++ Description of problem: DeleteExpiredMetrics is a background job that runs in the hawkular-metrics server. It was introduced in OCP 3.6 to basically clean up index tables, removing rows for metrics/pods that no longer exist. This was needed because the indexes basically have unbounded growth which led to other problems. The queries that the job performs have the potential to pull back a large amount of data which makes hawkular-metrics very susceptible to OOMEs under a relatively modest sized load. It first observed the problem in a cluster of about 9k pods, but I also observed the problem in a cluster with around 2k pods. Increasing the heap size of hawkular-metrics could alleviate the issue; however, there is no need for the job to query aggressively as it does since it is not latency sensitive. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. Configure hawkular-metrics with 3 GB of memory. This should result in a JVM heap size of around 1300 MB. 2. Create about 2k pods 3. Let the DeleteExpiredMetrics job run 4. The job is only scheduled to run every 7 days. I can provide additional details to make the job run more frequently to assist with testing Actual results: Expected results: Additional info:
Verification steps followed https://bugzilla.redhat.com/show_bug.cgi?id=1559440#c3, DeleteExpiredMetrics job is already dropped from code # openshift version openshift v3.6.173.0.122 kubernetes v1.6.1+5115d708d7 etcd 3.2.1 Images: metrics-cassandra-v3.6.173.0.122-2 metrics-hawkular-metrics-v3.6.173.0.122-2 metrics-heapster-v3.6.173.0.122-2
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:1801