Bug 1516401 - Refresh targets can grow unbounded causing Postgres InternalError
Summary: Refresh targets can grow unbounded causing Postgres InternalError
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Providers
Version: 5.9.0
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: GA
: 5.10.0
Assignee: Adam Grare
QA Contact: Dave Johnson
URL:
Whiteboard:
Depends On:
Blocks: 1517962
TreeView+ depends on / blocked
 
Reported: 2017-11-22 14:40 UTC by Adam Grare
Modified: 2019-08-06 20:04 UTC (History)
5 users (show)

Fixed In Version: 5.10.0.0
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1517962 (view as bug list)
Environment:
Last Closed: 2018-06-21 20:41:12 UTC
Category: Bug
Cloudforms Team: CFME Core
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Adam Grare 2017-11-22 14:40:13 UTC
Taking the uniq check out of EmsRefresh targets allows the number of targets in the args to grow unbounded to the point of running postgres out of memory for the transaction.

/opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/connection_adapters/postgresql_adapter.rb:598:in `async_exec': MIQ(MiqQueue.put_or_update) "PG::InternalError: ERROR:  invalid memory alloc request size 1073741824 (RuntimeError)
: UPDATE "miq_queue" SET "msg_data" = '\x04085b031e03375b0749221b4d616e61...

        from /opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/connection_adapters/abstract/database_statements.rb:232:in `transaction'
        from /opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/transactions.rb:211:in `transaction'
        from /opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/transactions.rb:392:in `with_transaction_returning_status'
        from /opt/rh/cfme-gemset/gems/activerecord-5.0.6/lib/active_record/persistence.rb:286:in `update!'
        from /var/www/miq/vmdb/app/models/miq_queue.rb:359:in `block in put_or_update'
        from /var/www/miq/vmdb/app/models/miq_queue.rb:337:in `loop'
        from /var/www/miq/vmdb/app/models/miq_queue.rb:337:in `put_or_update'
        from /var/www/miq/vmdb/app/models/ems_refresh.rb:173:in `queue_merge'
        from /var/www/miq/vmdb/app/models/ems_refresh.rb:60:in `block in queue_refresh'
        from /var/www/miq/vmdb/app/models/ems_refresh.rb:58:in `each'
        from /var/www/miq/vmdb/app/models/ems_refresh.rb:58:in `collect'
        from /var/www/miq/vmdb/app/models/ems_refresh.rb:58:in `queue_refresh'
        from /var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresh_worker/runner.rb:20:in `block in do_before_work_loop'
        from /var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresh_worker/runner.rb:17:in `each'
        from /var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresh_worker/runner.rb:17:in `do_before_work_loop'
        from /var/www/miq/vmdb/app/models/miq_worker/runner.rb:148:in `prepare'
        from /var/www/miq/vmdb/app/models/miq_worker/runner.rb:126:in `start'

Comment 3 Adam Grare 2017-11-22 14:44:01 UTC
This is related to (and possibly a duplicate of) https://bugzilla.redhat.com/show_bug.cgi?id=1514595

It is unclear at this point if this is the root cause or a side-effect

Comment 5 CFME Bot 2017-11-27 17:16:35 UTC
New commit detected on ManageIQ/manageiq/master:
https://github.com/ManageIQ/manageiq/commit/d100b568ecd9d759e6239051e45658baadd65afe

commit d100b568ecd9d759e6239051e45658baadd65afe
Author:     Adam Grare <agrare>
AuthorDate: Thu Nov 9 08:50:09 2017 -0500
Commit:     Adam Grare <agrare>
CommitDate: Wed Nov 22 09:59:01 2017 -0500

    Uniq ems_refresh targets if over a threshold
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1516401

 app/models/ems_refresh.rb | 1 +
 1 file changed, 1 insertion(+)


Note You need to log in before you can comment on or make changes to this bug.