Bug 1592480

Proposed patch that should solve the issue:

https://github.com/ManageIQ/manageiq/pull/17615

New commits detected on ManageIQ/manageiq/master:

https://github.com/ManageIQ/manageiq/commit/0ad75c2117faa8ce3678932fc8d383429d55b4e5
commit 0ad75c2117faa8ce3678932fc8d383429d55b4e5
Author:     Nick LaMuro <nicklamuro>
AuthorDate: Tue Jun 19 20:03:05 2018 -0400
Commit:     Nick LaMuro <nicklamuro>
CommitDate: Tue Jun 19 20:03:05 2018 -0400

    Add MiqReport#all_custom_attributes_are_virtual_sql_attributes?

    https://bugzilla.redhat.com/show_bug.cgi?id=1592480

    This method is responsible for checking if the virtual_attributes in the
    Rbac query will contain all of the custom_attributes virtual_attributes.

    To be used to determine if we can safely remove:

        { :custom_attributes => {} }

    From the `include` hash on MiqReport, avoiding needing to load a lot of
    records.

 app/models/miq_report.rb | 5 +
 1 file changed, 5 insertions(+)


https://github.com/ManageIQ/manageiq/commit/e7fd484acc9813462744046dd8b464a08b443527
commit e7fd484acc9813462744046dd8b464a08b443527
Author:     Nick LaMuro <nicklamuro>
AuthorDate: Tue Jun 19 20:03:48 2018 -0400
Commit:     Nick LaMuro <nicklamuro>
CommitDate: Tue Jun 19 20:03:48 2018 -0400

    Remove custom_attributes from MiqReport includes

    https://bugzilla.redhat.com/show_bug.cgi?id=1592480

    Conditionally removes `:custom_attributes => {}` from the includes
    clause if all of the `custom_attribute` columns happen to be in the
    select statement (the `@va_sql_cols` value).

    When the `custom_attribute` `virtual_attributes` can be converted to
    Arel, this will have some value, but for now this will mostly do nothing

 app/models/miq_report/generator.rb | 7 +
 1 file changed, 7 insertions(+)


https://github.com/ManageIQ/manageiq/commit/065405d49975c80b3b852ddb21d413719bec976e
commit 065405d49975c80b3b852ddb21d413719bec976e
Author:     Nick LaMuro <nicklamuro>
AuthorDate: Tue Jun 19 20:06:35 2018 -0400
Commit:     Nick LaMuro <nicklamuro>
CommitDate: Tue Jun 19 20:06:35 2018 -0400

    Convert custom_attributes to arel virtual_attributes

    https://bugzilla.redhat.com/show_bug.cgi?id=1592480

    This change allows `custom_attributes` to be accessible via SQL.

    Most of the additions are found in the `custom_attribute_arel`, included
    by `CustomAttributeMixin`, and will simply generate the Arel equivalent
    necessary for determined the `virtual_attribute` value via SQL.

    Other notable changes:

    - The method defined for accessing the `virtual_attribute` from the
      instance will now check the attributes hash prior to executing the
      rest of the method, and return something if it has a key
    - Some shared values calculated in both the method definition and would
      be in the `custom_attribute_arel` method have been extracted out so it
      is only calculated once (minor performance benefit).

 app/models/mixins/custom_attribute_mixin.rb | 34 +-
 1 file changed, 28 insertions(+), 6 deletions(-)

Steps for replicating this bug can be found here:

https://gist.github.com/NickLaMuro/87dddcfbd549b03099f8e55f632b2b57#file-bz_1592480_db_replication_script-rb

* * *

To replicate this bug, you can run the following script from below to populate a fresh database with data to replicate this bug locally:

    $ bin/rails r bz_1588082_db_replication_script.rb


From there, the easiest way to confirm the bug is fixed or not is to run the following in a console:

    $ bin/rails c
    irb> r = MiqReport.where(:name => "BZ 1592480 Example Report").first
    irb> puts Benchmark.measure { r.queue_generate_table(:report_sync => true) }

Without the patch from https://github.com/ManageIQ/manageiq/pull/17615 (or it's backported equivalent), this will take a large amount of memory and over 5 minutes to complete.

With the patch in place, the memory stays relatively low and only takes about 1 minute to complete (depending on DB latency, CPU power, etc.)