Bug 1510484 - vmdb size constantly increasing 1+gb a day
Summary: vmdb size constantly increasing 1+gb a day
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Appliance
Version: 5.8.0
Hardware: All
OS: All
urgent
urgent
Target Milestone: GA
: 5.10.0
Assignee: Nick Carboni
QA Contact: Tasos Papaioannou
URL:
Whiteboard:
Depends On:
Blocks: 1512705 1512706
TreeView+ depends on / blocked
 
Reported: 2017-11-07 14:30 UTC by Felix Dewaleyne
Modified: 2021-09-09 12:48 UTC (History)
9 users (show)

Fixed In Version: 5.10.0.0
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1512705 1512706 (view as bug list)
Environment:
Last Closed: 2019-02-11 13:54:39 UTC
Category: ---
Cloudforms Team: ---
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
vim_performance_tag_values purge tool (3.63 KB, application/x-ruby)
2017-11-09 16:08 UTC, Nick Carboni
no flags Details

Comment 4 Nick Carboni 2017-11-07 17:24:46 UTC
The setting for the metrics retention time needs to be set on every appliance.

This is because the request to execute a purge is based on the presence of the ems_metrics_coordinator role in the environment, but the setting is not taken from the appliance with that role.

The safest option would be to set the value everywhere, but the setting is being taken from the appliance with the scheduler role active. Because the the scheduler role could failover to a different server, this should just always be set for all appliances in a region (or ideally set as a region wide setting).

This should take care of the purging issue; the growth of the table is a separate issue.

To stop the growth of that particular table a user should be able to stop collecting metrics based on tags be turning off the "Capture C & U Data by Tag" value described by the "4.1.1.4.2. Creating a Tag Category" section in https://access.redhat.com/documentation/en-us/red_hat_cloudforms/4.5/html-single/general_configuration/#regions for all tag categories.

Comment 8 CFME Bot 2017-11-09 02:51:52 UTC
New commit detected on ManageIQ/manageiq/master:
https://github.com/ManageIQ/manageiq/commit/e04a60eb67c81f39944b417c450290ee74deea68

commit e04a60eb67c81f39944b417c450290ee74deea68
Author:     Nick Carboni <ncarboni>
AuthorDate: Wed Nov 8 17:28:58 2017 -0500
Commit:     Nick Carboni <ncarboni>
CommitDate: Wed Nov 8 17:28:58 2017 -0500

    Add purging for vim_performance_tag_values with disabled tags
    
    When a user stops collecting C&U data for a particular tag category,
    the records hang around in vim_performance_tag_values until they are
    purged by date.
    
    This commit adds an option to the purge_orphaned_tag_values.rb
    tool to remove such values immediately.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1510484

 tools/purge_orphaned_tag_values.rb | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

Comment 9 Nick Carboni 2017-11-09 16:08:33 UTC
Created attachment 1349998 [details]
vim_performance_tag_values purge tool

This is a replacement for the current file at /var/www/miq/vmdb/tools/purge_orphaned_tag_values.rb

Running this file with no options will purge rows from the vim_performance_tag_values table which no longer have a corresponding metric.

This new version adds an option to remove rows which map to a tag category which is no longer set to collect data from metrics. These rows could still be used in reports (as their metrics likely still exist), but if the C&U collection was enabled for many tags by mistake, this is how you can remove some of the data which would accumulate in this table.

To delete both types of rows you can run the following from the vmdb directory:
`tools/purge_orphaned_tag_values.rb --purge-disabled-tag-values`

Comment 10 Gregg Tanzillo 2017-11-09 16:34:08 UTC
Please provide script and instructions in comment #9 to customer

Comment 16 CFME Bot 2018-01-03 16:26:39 UTC
New commit detected on ManageIQ/manageiq/master:
https://github.com/ManageIQ/manageiq/commit/d0bc3369c04d7196e97a54176505299109fb2107

commit d0bc3369c04d7196e97a54176505299109fb2107
Author:     Gregg Tanzillo <gtanzill>
AuthorDate: Tue Dec 19 15:36:36 2017 -0500
Commit:     Gregg Tanzillo <gtanzill>
CommitDate: Tue Jan 2 14:17:14 2018 -0500

    Stop generating `vim_performance_tag_values` rows
    
    Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1510484
    Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1514505

 app/models/metric/ci_mixin/rollup.rb                        |  1 -
 app/models/storage.rb                                       |  2 --
 .../connection_adapters/miq_postgres_adapter.rb             |  4 ----
 .../connection_adapters/miq_postgres_legacy_adapter.rb      |  4 ----
 spec/models/vim_performance_tag_spec.rb                     |  2 --
 tools/metrics_populate_retro_tags.rb                        | 13 -------------
 6 files changed, 26 deletions(-)

Comment 17 CFME Bot 2018-01-03 16:33:51 UTC
New commit detected on ManageIQ/manageiq/master:
https://github.com/ManageIQ/manageiq/commit/d0bc3369c04d7196e97a54176505299109fb2107

commit d0bc3369c04d7196e97a54176505299109fb2107
Author:     Gregg Tanzillo <gtanzill>
AuthorDate: Tue Dec 19 15:36:36 2017 -0500
Commit:     Gregg Tanzillo <gtanzill>
CommitDate: Tue Jan 2 14:17:14 2018 -0500

    Stop generating `vim_performance_tag_values` rows
    
    Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1510484
    Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1514505

 app/models/metric/ci_mixin/rollup.rb                        |  1 -
 app/models/storage.rb                                       |  2 --
 .../connection_adapters/miq_postgres_adapter.rb             |  4 ----
 .../connection_adapters/miq_postgres_legacy_adapter.rb      |  4 ----
 spec/models/vim_performance_tag_spec.rb                     |  2 --
 tools/metrics_populate_retro_tags.rb                        | 13 -------------
 6 files changed, 26 deletions(-)

Comment 19 Tasos Papaioannou 2018-07-11 17:06:08 UTC
Verified on 5.10.0.3.


Note You need to log in before you can comment on or make changes to this bug.