Bug 1462358

Summary:	Hourly metrics_## tables grow filling up the VMDB filesystem when real-time purges fail
Product:	Red Hat CloudForms Management Engine	Reporter:	Satoe Imaishi <simaishi>
Component:	Appliance	Assignee:	Jillian Tullo <jtullo>
Status:	CLOSED ERRATA	QA Contact:	Tasos Papaioannou <tpapaioa>
Severity:	high	Docs Contact:
Priority:	medium
Version:	5.7.0	CC:	abellott, cpelland, gblomqui, jdeubel, jhardy, jocarter, jtullo, mfeifer, myoder, obarenbo, saali, simaishi, thenness, tpapaioa, yrudman
Target Milestone:	GA	Keywords:	ZStream
Target Release:	5.8.1
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	5.8.1.0	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1445498	Environment:
Last Closed:	2017-08-02 17:32:21 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	CFME Core	Target Upstream Version:
Embargoed:
Bug Depends On:	1445498
Bug Blocks:

Comment 2 CFME Bot 2017-06-16 20:16:06 UTC

New commit detected on ManageIQ/manageiq/fine:
https://github.com/ManageIQ/manageiq/commit/6332db37fc931b7c4acc135d0a56dfe96f55d883

commit 6332db37fc931b7c4acc135d0a56dfe96f55d883
Author:     Gregg Tanzillo <gtanzill>
AuthorDate: Wed Jun 14 11:12:57 2017 -0400
Commit:     Satoe Imaishi <simaishi>
CommitDate: Fri Jun 16 16:09:36 2017 -0400

    Merge pull request #15312 from jntullo/bz/purge_timeout
    
    Increase timeout for metric purging
    (cherry picked from commit 62719214d86bf68b169571af9a761fea6331f5f4)
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1462358

 app/models/metric/purging.rb       |  5 +++++
 app/models/vmdb_metric/purging.rb  |  7 ++++++-
 config/settings.yml                |  4 +++-
 spec/models/metric/purging_spec.rb | 33 +++++++++++++++++++--------------
 4 files changed, 33 insertions(+), 16 deletions(-)

Comment 3 Tasos Papaioannou 2017-07-03 20:09:13 UTC

Verified on 5.8.1.0. Metric purging now uses 20 minute timeout, e.g.,

****
[----] I, [2017-07-03T12:50:04.889211 #3062:459138]  INFO -- : MIQ(MiqQueue.put) Message id: [175205],  id: [], Zone: [default], Role: [], Server: [], Ident: [generic], Target id: [], Instance id: [], Task id: [], Command: [VmdbDatabaseMetric.purge_hourly], Timeout: [1200], Priority: [100], State: [ready], Deliver On: [], Data: [], Args: [2017-01-04 16:50:04 UTC]

[----] I, [2017-07-03T13:43:31.468162 #3062:459138]  INFO -- : MIQ(MiqQueue.put) Message id: [177438],  id: [], Zone: [default], Role: [], Server: [], Ident: [generic], Target id: [], Instance id: [], Task id: [], Command: [Metric::Purging.purge_realtime], Timeout: [1200], Priority: [100], State: [ready], Deliver On: [], Data: [], Args: [2017-07-02 17:43:31 UTC]
****

After changing the database.metrics_history.queue_timeout and performance.history.queue_timeout settings to 10.minutes and restarting EVM server processes, the timeouts change to 10 minutes.

Comment 5 errata-xmlrpc 2017-08-02 17:32:21 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:1758