Bug 1465086

Summary: Hourly metrics_## tables grow filling up the VMDB filesystem when real-time purges fail
Product: Red Hat CloudForms Management Engine Reporter: Satoe Imaishi <simaishi>
Component: ApplianceAssignee: Jillian Tullo <jtullo>
Status: CLOSED ERRATA QA Contact: Tasos Papaioannou <tpapaioa>
Severity: high Docs Contact:
Priority: medium    
Version: 5.7.0CC: abellott, cpelland, gblomqui, jdeubel, jhardy, jocarter, jtullo, mfeifer, myoder, obarenbo, saali, simaishi, thenness, tpapaioa, yrudman
Target Milestone: GAKeywords: ZStream
Target Release: 5.7.4   
Hardware: x86_64   
OS: Linux   
Whiteboard: c&u
Fixed In Version: 5.7.4.0 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1445498 Environment:
Last Closed: 2017-12-18 20:26:17 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: CFME Core Target Upstream Version:
Embargoed:
Bug Depends On: 1445498    
Bug Blocks:    

Comment 2 Jillian Tullo 2017-08-04 17:01:59 UTC
PR https://github.com/ManageIQ/manageiq/pull/15702

Comment 3 Tasos Papaioannou 2017-09-19 14:16:21 UTC
Verified on 5.8.1.0. Metric purging now uses 20 minute timeout, e.g.,

****
[----] I, [2017-09-19T03:50:12.715632 #3456:e3d12c]  INFO -- : MIQ(MiqQueue.put) Message id: [39327],  id: [], Zone: [], Role: [database_operations], Server: [], Ident: [generic], Target id: [], Instance id: [], Task id: [], Command: [VmdbMetric.purge_hourly], Timeout: [1200], Priority: [100], State: [ready], Deliver On: [], Data: [], Args: [2017-03-23 07:50:12 UTC]

[----] I, [2017-09-19T04:04:19.428016 #3447:e3d12c]  INFO -- : MIQ(MiqQueue.put) Message id: [39551],  id: [], Zone: [default], Role: [ems_metrics_processor], Server: [], Ident: [ems_metrics_processor], Target id: [], Instance id: [], Task id: [], Command: [Metric::Purging.purge_realtime], Timeout: [1200], Priority: [100], State: [ready], Deliver On: [], Data: [], Args: [2017-09-19 04:04:19 UTC]
****

After changing the database.metrics_history.queue_timeout and performance.history.queue_timeout settings to 10.minutes and restarting EVM server processes, the timeouts change to 10 minutes.

****
[----] I, [2017-09-19T09:23:21.079515 #19931:373138]  INFO -- : MIQ(MiqQueue.put) Message id: [47427],  id: [], Zone: [default], Role: [ems_metrics_processor], Server: [], Ident: [ems_metrics_processor], Target id: [], Instance id: [], Task id: [], Command: [Metric::Purging.purge_realtime], Timeout: [600], Priority: [100], State: [ready], Deliver On: [], Data: [], Args: [2017-09-19 09:23:21 UTC]

[----] I, [2017-09-19T09:50:09.367103 #19931:373138]  INFO -- : MIQ(MiqQueue.put) Message id: [47912],  id: [], Zone: [], Role: [database_operations], Server: [], Ident: [generic], Target id: [], Instance id: [], Task id: [], Command: [VmdbMetric.purge_hourly], Timeout: [600], Priority: [100], State: [ready], Deliver On: [], Data: [], Args: [2017-03-23 13:50:09 UTC]
****

Comment 6 errata-xmlrpc 2017-12-18 20:26:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3484