Bug 1618808 - Migrations linking jobs and miq_tasks could take long time when upgrading to 5.9
Summary: Migrations linking jobs and miq_tasks could take long time when upgrading to 5.9
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Appliance
Version: 5.9.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: GA
: 5.9.5
Assignee: Yuri Rudman
QA Contact: Jaroslav Henner
Depends On: 1614023
TreeView+ depends on / blocked
Reported: 2018-08-17 16:17 UTC by Satoe Imaishi
Modified: 2018-11-05 14:41 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1614023
Last Closed: 2018-11-05 13:58:28 UTC
Category: ---
Cloudforms Team: ---

Attachments (Terms of Use)

System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:3466 None None None 2018-11-05 13:58:57 UTC

Comment 2 CFME Bot 2018-09-11 21:26:33 UTC
New commit detected on ManageIQ/manageiq/gaprindashvili:

commit b28bb84022819b3252fb96099abead785011456a
Author:     Keenan Brock <keenan@thebrocks.net>
AuthorDate: Thu Aug  2 11:02:37 2018 -0400
Commit:     Keenan Brock <keenan@thebrocks.net>
CommitDate: Thu Aug  2 11:02:37 2018 -0400

    Merge pull request #17727 from yrudman/purge-old-jobs-and-tasks

    Purge old MiqTasks and associated records
    (cherry picked from commit 98497e92f886a3cd0fef712214674cca997d3101)


 app/models/miq_schedule_worker/jobs.rb | 4 +
 app/models/miq_schedule_worker/runner.rb | 5 +
 app/models/miq_task.rb | 2 +
 app/models/miq_task/purging.rb | 26 +
 config/settings.yml | 5 +
 spec/models/miq_task/purging_spec.rb | 50 +
 6 files changed, 92 insertions(+)

Comment 3 CFME Bot 2018-09-11 21:28:46 UTC
New commit detected on ManageIQ/manageiq-schema/gaprindashvili:

commit 44ade48a30ce898b4d5a456fbcbd5f817d379133
Author:     Jason Frey <fryguy9@gmail.com>
AuthorDate: Tue Aug  7 12:26:31 2018 -0400
Commit:     Jason Frey <fryguy9@gmail.com>
CommitDate: Tue Aug  7 12:26:31 2018 -0400

    Merge pull request #201 from yrudman/delete-old-job-while-creating-linked-task

    Optimize migration which creates task for each job
    (cherry picked from commit 8d48f1bedf9c42661ecb83ae3ab5ab9fb2b21545)


 db/migrate/20170120164058_create_task_for_each_job_and_transfer_attributes.rb | 22 +
 spec/migrations/20170120164058_create_task_for_each_job_and_transfer_attributes_spec.rb | 32 +
 2 files changed, 54 insertions(+)

Comment 4 Jaroslav Henner 2018-10-23 15:10:10 UTC
With 1000 jobs the difference is negligible when migrating from to cfme-

real	0m51.182s
user	0m13.961s
sys	0m2.006s to cfme-
real	0m40.888s
user	0m9.027s
sys	0m1.822s

Comment 5 Yuri Rudman 2018-10-23 15:32:28 UTC
yes, it is negligible with 1000, real improvement will be on millions records...

Comment 6 Jaroslav Henner 2018-10-24 11:40:02 UTC
(In reply to Yuri Rudman from comment #5)
> yes, it is negligible with 1000, real improvement will be on millions
> records...

Just creating 1M of records takes about 8 hours.

After this was done I did try the migration to older version (cfme- The db:migrate task takes more than 3min, the output stopped at:

== 20170120154842 AddStartedOnColumnToMiqTasks: migrating =====================
-- add_column(:miq_tasks, :started_on, :datetime)
   -> 0.0010s
-- add_column(:miq_tasks, :zone, :string)
   -> 0.0114s
== 20170120154842 AddStartedOnColumnToMiqTasks: migrated (0.0126s) ============

== 20170120164058 CreateTaskForEachJobAndTransferAttributes: migrating ========
-- Creating tasks associated with jobs

While the fixed verson is done in 0m 44s.

Comment 8 errata-xmlrpc 2018-11-05 13:58:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.