Bug 1348625 - We might not be purging all tables that we should be
Summary: We might not be purging all tables that we should be
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Performance
Version: 5.5.0
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: GA
: 5.8.0
Assignee: Keenan Brock
QA Contact: Tasos Papaioannou
URL:
Whiteboard: perf:database
Depends On:
Blocks: 1410846 1412448
TreeView+ depends on / blocked
 
Reported: 2016-06-21 15:41 UTC by Joe Rafaniello
Modified: 2018-04-11 18:16 UTC (History)
6 users (show)

Fixed In Version: 5.8.0.0
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1410846 1412448 (view as bug list)
Environment:
Last Closed: 2017-06-12 16:09:22 UTC
Category: ---
Cloudforms Team: ---
Target Upstream Version:


Attachments (Terms of Use)

Comment 3 Joe Rafaniello 2016-10-07 20:54:44 UTC
In addition, others have found that vim_performance_states is not automatically purged but the upgrade docs often tell users to truncate this table.  We should probably purge this table also.

Comment 4 Keenan Brock 2016-10-10 21:13:23 UTC
I got the following information.

|                         relation                         | size mb | rows       |
|:---------------------------------------------------------|--------:|------:     |
| **event_streams**                                        |  8,934  |  6,933,269 |
| event_streams_pkey                                       |    149  |            |
| index_event_streams_on_chain_id_and_ems_id               |    209  |            |
| index_event_streams_on_dest_host_id                      |    149  |            |
| index_event_streams_on_dest_vm_or_template_id            |    149  |            |
| index_event_streams_on_ems_cluster_id                    |    149  |            |
| index_event_streams_on_ems_id                            |    149  |            |
| index_event_streams_on_event_type                        |    359  |            |
| index_event_streams_on_host_id                           |    149  |            |
| index_event_streams_on_timestamp                         |    149  |            |
| index_event_streams_on_vm_or_template_id                 |    149  |            |
| **miq_report_result_details**                            |  5,582  | 18,383,604 |
| miq_report_result_details_idx                            |    712  |            |
| miq_report_result_details_pkey                           |    394  |            |
| **miq_report_results**                                   |     47  |     27,841 |
| **audit_events**                                         |  1,078  |  6,725,715 |
| audit_events_pkey                                        |    144  |            |
| index_audit_events_on_target_id_and_target_class         |    145  |            |
| **vmdb_metrics**                                         |    937  |  6,013,446 |
| vmdb_metrics_pkey                                        |    129  |            |
| **advanced_settings**                                    |    312  |  1,738,917 |
| index_advanced_settings_on_resource_id_and_resource_type |     54  |            |
| index_vmdb_metrics_on_resource_and_timestamp             |    285  |            |
| **guest_applications**                                   |    493  |  1,378,972 |
| index_guest_applications_on_typename                     |     58  |            |

Comment 5 Keenan Brock 2016-10-10 21:14:04 UTC
heh. oops. table was markdown centric

EventStreams: https://github.com/ManageIQ/manageiq/pull/11813

Comment 6 CFME Bot 2016-10-17 16:21:10 UTC
New commit detected on ManageIQ/manageiq/euwe:
https://github.com/ManageIQ/manageiq/commit/f915117f045f9e2308428cb1b03d921bae5a6b7b

commit f915117f045f9e2308428cb1b03d921bae5a6b7b
Author:     Jason Frey <fryguy9>
AuthorDate: Tue Oct 11 18:19:43 2016 -0400
Commit:     Oleg Barenboim <chessbyte>
CommitDate: Thu Oct 13 16:45:04 2016 -0400

    Merge pull request #11813 from kbrock/prune_system_events
    
    Prune all EventStream events
    (cherry picked from commit a9497c3d792341e1c81113a7a071e634a92bdc03)
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1348625

 app/models/ems_event.rb                        | 44 --------------------------
 app/models/event_stream.rb                     | 44 ++++++++++++++++++++++++++
 app/models/miq_schedule_worker/jobs.rb         |  2 +-
 spec/models/miq_schedule_worker/runner_spec.rb |  2 +-
 4 files changed, 46 insertions(+), 46 deletions(-)

Comment 11 CFME Bot 2016-12-13 18:56:09 UTC
New commit detected on ManageIQ/manageiq/master:
https://github.com/ManageIQ/manageiq/commit/f84a42594a446a92ce04fe7c7829ac598ed5ae9d

commit f84a42594a446a92ce04fe7c7829ac598ed5ae9d
Author:     Keenan Brock <kbrock>
AuthorDate: Thu Dec 8 16:48:43 2016 -0500
Commit:     Keenan Brock <kbrock>
CommitDate: Fri Dec 9 16:02:10 2016 -0500

    Introduce purge timer for drift states
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1348625

 app/models/miq_schedule_worker/jobs.rb   | 4 ++++
 app/models/miq_schedule_worker/runner.rb | 5 +++++
 config/settings.yml                      | 1 +
 3 files changed, 10 insertions(+)

Comment 12 CFME Bot 2016-12-14 21:56:18 UTC
New commit detected on ManageIQ/manageiq/master:
https://github.com/ManageIQ/manageiq/commit/2f7824930787304c1b77511dfd5cf46f542d0ae8

commit 2f7824930787304c1b77511dfd5cf46f542d0ae8
Author:     Keenan Brock <kbrock>
AuthorDate: Wed Dec 7 11:11:50 2016 -0500
Commit:     Keenan Brock <kbrock>
CommitDate: Wed Dec 14 14:11:25 2016 -0500

    Introduce report result purging timer
    
    The resources required for report result purging has been reduced,
    so make sure this is scheduled.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1348625

 app/models/miq_schedule_worker/jobs.rb   | 4 ++++
 app/models/miq_schedule_worker/runner.rb | 6 ++++++
 config/settings.yml                      | 1 +
 3 files changed, 11 insertions(+)

Comment 13 Keenan Brock 2017-01-03 15:27:33 UTC
Purging tables in the following PRs

event streams: https://github.com/ManageIQ/manageiq/pull/11813
miq_report_results https://github.com/ManageIQ/manageiq/pull/13044
drift_states https://github.com/ManageIQ/manageiq/pull/13086

all on master

Comment 16 CFME Bot 2017-01-12 15:50:50 UTC
New commit detected on ManageIQ/manageiq/euwe:
https://github.com/ManageIQ/manageiq/commit/5a82e28f3ef52cb9082f2eaaa529e0bf4757dce0

commit 5a82e28f3ef52cb9082f2eaaa529e0bf4757dce0
Author:     Keenan Brock <kbrock>
AuthorDate: Wed Dec 7 11:11:50 2016 -0500
Commit:     Keenan Brock <kbrock>
CommitDate: Wed Jan 11 20:26:48 2017 -0500

    Introduce report result purging timer
    
    The resources required for report result purging has been reduced,
    so make sure this is scheduled.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1348625

 app/models/miq_schedule_worker/jobs.rb   | 4 ++++
 app/models/miq_schedule_worker/runner.rb | 6 ++++++
 config/settings.yml                      | 1 +
 3 files changed, 11 insertions(+)

Comment 17 Tasos Papaioannou 2017-05-19 19:43:10 UTC
[----] I, [2017-05-19T07:48:23.994536 #63471:985134]  INFO -- : MIQ(DriftState.purge_by_date) Purging Drift states older than [2016-11-20 11:48:20 UTC]...

[----] I, [2017-05-19T14:35:49.443990 #2923:5c1138]  INFO -- : MIQ(EventStream.purge_by_date) Purging Event streams older than [2017-05-19 16:35:44 UTC]...

[----] I, [2017-05-19T14:35:49.456240 #2923:5c1138]  INFO -- : MIQ(PolicyEvent.purge_by_date) Purging Policy events older than [2017-05-19 16:35:44 UTC]...

No ReportResult purge activity in evm.log, even with report_result_purge_interval set to 10.minutes (and server restart).

Comment 18 Tasos Papaioannou 2017-05-19 20:05:49 UTC
ReportResult purge activity showing up as well:

[----] I, [2017-05-19T15:57:21.768484 #2932:112513c]  INFO -- : MIQ(MiqReportResult.purge_by_date) Purging Miq report results older than [2016-11-20 19:57:17 UTC]...


Verified on 5.8.0.15.


Note You need to log in before you can comment on or make changes to this bug.