Bug 1479339 - Memory leak in MetricsProcessor Worker
Memory leak in MetricsProcessor Worker
Status: ON_DEV
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Performance (Show other bugs)
5.8.0
Unspecified Unspecified
high Severity high
: GA
: cfme-future
Assigned To: Nick LaMuro
Tasos Papaioannou
c&u:worker:perf
:
Depends On: 1456775
Blocks: 1479356
  Show dependency treegraph
 
Reported: 2017-08-08 07:50 EDT by Archit Sharma
Modified: 2018-01-19 18:56 EST (History)
16 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1456775
: 1479356 (view as bug list)
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: CFME Core


Attachments (Terms of Use)
Priority worker leakage on all appliances (161.22 KB, image/png)
2017-08-08 07:50 EDT, Archit Sharma
no flags Details
Generic worker leakage w.r.t stable processor queue and powered on/off vms (185.01 KB, image/png)
2017-08-08 07:51 EDT, Archit Sharma
no flags Details
all worker types memory usage comparison (132.18 KB, image/png)
2017-08-08 07:52 EDT, Archit Sharma
no flags Details
PSS & RSS utilization - 4+ day test run (475.80 KB, image/png)
2017-08-09 14:42 EDT, dmetzger
no flags Details

  None (edit)
Description Archit Sharma 2017-08-08 07:50:24 EDT
Created attachment 1310596 [details]
Priority worker leakage on all appliances

Description of problem:

Observed memory leaks in MetricsProcessor Worker and GenericWorker for a 10k VMware provider infra connected 

This ran for over 3 days but leaks occurred in the first day itself. 

Version-Release number of selected component (if applicable):
CFME 5.8.0.17

How reproducible:


Steps to Reproduce:
1. [ 1 DB, 5 worker] 6 appliance setup.
2. Turn on C&U on all worker appliances (and cluster wide C&U collection settings in config) and keep server roles to minimum on DB appliance
3. Connect to 10k vms VMware infra provider and let it run for 2-3 days while keeping an eye on C&U data collector worker memory usage.

Actual results:
MetricsProcessorWorker memory grew from about 1.5G to 2.8G. 

Expected results:
almost little or no memory growth after initial C&U/refresh period.

Additional info:
attaching screenshot for reference

Original comment (From BZ about MetricsCollector worker leak for multiple providers): https://bugzilla.redhat.com/show_bug.cgi?id=1456775#c29
Comment 2 Archit Sharma 2017-08-08 07:51 EDT
Created attachment 1310597 [details]
Generic worker leakage w.r.t stable processor queue and powered on/off vms
Comment 3 Archit Sharma 2017-08-08 07:52 EDT
Created attachment 1310598 [details]
all worker types memory usage comparison
Comment 4 Archit Sharma 2017-08-08 08:01:20 EDT
To further add to 'steps to reproduce' in description, I had increased memory thresholds / counts for specific worker processes on all appliances, just enough to accommodate those many VMs for a 6 appliance setup.

For reference:- 

----
# DB

- Generic - 2, 500 MB
- Priority - 2, 600 MB

----
# Worker appliances

- Generic - 4, 500 MB
- Priority - 2, 800 MB
- C&U Data Collectors - 6, 600 MB
- C&U Data Processors - 4, 800 MB
- Refresh - 2 GB
----

The refresh worker's (leaked?) memory grew by few MBs. Its RSS memory growth is included in the attachment https://bugzilla.redhat.com/attachment.cgi?id=1310598
Comment 5 dmetzger 2017-08-09 14:42 EDT
Created attachment 1311340 [details]
PSS & RSS utilization - 4+ day test run

Worker Config:
    Single Metrics Processor Worker
    1.5Gb Memory Threshold

Provider:
    Clusters:      10
    Hosts:         50
    Datastores:    61
    VMs:        1,000
    Type:       VMware VC 5.5.0
Comment 6 Nick LaMuro 2017-09-13 09:59:31 EDT
I think based on some talks with Dennis regarding similar tickets, I think enabling the metrics collection is the root cause to some of the "leaks" that we are seeing.

Most of my commenting will probably be done on:

https://bugzilla.redhat.com/show_bug.cgi?id=1458392



Will update here when I have more to share.
Comment 10 Nick LaMuro 2018-01-18 11:07:32 EST
A possible fix has been proposed in this related BZ:

https://bugzilla.redhat.com/show_bug.cgi?id=1535720


That is targeted for the MiqServer, and high confidence that it will fix the leak there.  Updates will probably happen there more regularly until we determine if there is a different leak in the MetricsProcessor Worker, and there is a high probability this was a leak across all workers.
Comment 11 Nick LaMuro 2018-01-19 18:53:51 EST
The fix above has been backported to 5.8:

https://bugzilla.redhat.com/show_bug.cgi?id=1536672

As well as for future releases here:

https://bugzilla.redhat.com/show_bug.cgi?id=1535720

We are going to do some testing ourselves to see if this is fixing the issue with the MetricsProcessor as well, and will update with those results.

Note You need to log in before you can comment on or make changes to this bug.