Bug 1618808

Summary: Migrations linking jobs and miq_tasks could take long time when upgrading to 5.9
Product: Red Hat CloudForms Management Engine Reporter: Satoe Imaishi <simaishi>
Component: ApplianceAssignee: Yuri Rudman <yrudman>
Status: CLOSED ERRATA QA Contact: Jaroslav Henner <jhenner>
Severity: medium Docs Contact:
Priority: medium    
Version: 5.9.0CC: abellott, dmetzger, obarenbo, smallamp, yrudman
Target Milestone: GAKeywords: ZStream
Target Release: 5.9.5   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 5.9.5.0 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1614023 Environment:
Last Closed: 2018-11-05 13:58:28 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1614023    
Bug Blocks:    

Comment 2 CFME Bot 2018-09-11 21:26:33 UTC
New commit detected on ManageIQ/manageiq/gaprindashvili:

https://github.com/ManageIQ/manageiq/commit/b28bb84022819b3252fb96099abead785011456a
commit b28bb84022819b3252fb96099abead785011456a
Author:     Keenan Brock <keenan>
AuthorDate: Thu Aug  2 11:02:37 2018 -0400
Commit:     Keenan Brock <keenan>
CommitDate: Thu Aug  2 11:02:37 2018 -0400

    Merge pull request #17727 from yrudman/purge-old-jobs-and-tasks

    Purge old MiqTasks and associated records
    (cherry picked from commit 98497e92f886a3cd0fef712214674cca997d3101)

    https://bugzilla.redhat.com/show_bug.cgi?id=1618808

 app/models/miq_schedule_worker/jobs.rb | 4 +
 app/models/miq_schedule_worker/runner.rb | 5 +
 app/models/miq_task.rb | 2 +
 app/models/miq_task/purging.rb | 26 +
 config/settings.yml | 5 +
 spec/models/miq_task/purging_spec.rb | 50 +
 6 files changed, 92 insertions(+)

Comment 3 CFME Bot 2018-09-11 21:28:46 UTC
New commit detected on ManageIQ/manageiq-schema/gaprindashvili:

https://github.com/ManageIQ/manageiq-schema/commit/44ade48a30ce898b4d5a456fbcbd5f817d379133
commit 44ade48a30ce898b4d5a456fbcbd5f817d379133
Author:     Jason Frey <fryguy9>
AuthorDate: Tue Aug  7 12:26:31 2018 -0400
Commit:     Jason Frey <fryguy9>
CommitDate: Tue Aug  7 12:26:31 2018 -0400

    Merge pull request #201 from yrudman/delete-old-job-while-creating-linked-task

    Optimize migration which creates task for each job
    (cherry picked from commit 8d48f1bedf9c42661ecb83ae3ab5ab9fb2b21545)

    https://bugzilla.redhat.com/show_bug.cgi?id=1618808

 db/migrate/20170120164058_create_task_for_each_job_and_transfer_attributes.rb | 22 +
 spec/migrations/20170120164058_create_task_for_each_job_and_transfer_attributes_spec.rb | 32 +
 2 files changed, 54 insertions(+)

Comment 4 Jaroslav Henner 2018-10-23 15:10:10 UTC
With 1000 jobs the difference is negligible when migrating from 

5.7.4.3-1 to cfme-5.9.4.5-1.el7cf.x86_64

real	0m51.182s
user	0m13.961s
sys	0m2.006s

5.7.4.3-1 to cfme-5.9.5.2-1.el7cf.x86_64
real	0m40.888s
user	0m9.027s
sys	0m1.822s

Comment 5 Yuri Rudman 2018-10-23 15:32:28 UTC
yes, it is negligible with 1000, real improvement will be on millions records...

Comment 6 Jaroslav Henner 2018-10-24 11:40:02 UTC
(In reply to Yuri Rudman from comment #5)
> yes, it is negligible with 1000, real improvement will be on millions
> records...

Just creating 1M of records takes about 8 hours.

After this was done I did try the migration to older version (cfme-5.9.4.5-1.el7cf.x86_64). The db:migrate task takes more than 3min, the output stopped at:

== 20170120154842 AddStartedOnColumnToMiqTasks: migrating =====================
-- add_column(:miq_tasks, :started_on, :datetime)
   -> 0.0010s
-- add_column(:miq_tasks, :zone, :string)
   -> 0.0114s
== 20170120154842 AddStartedOnColumnToMiqTasks: migrated (0.0126s) ============

== 20170120164058 CreateTaskForEachJobAndTransferAttributes: migrating ========
-- Creating tasks associated with jobs

While the fixed verson is done in 0m 44s.

Comment 8 errata-xmlrpc 2018-11-05 13:58:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:3466