Bug 1573588 - Refresh Failing for VMware VIM object is too large
Summary: Refresh Failing for VMware VIM object is too large
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Providers
Version: 5.8.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: GA
: 5.10.0
Assignee: Adam Grare
QA Contact: Kedar Kulkarni
Depends On:
Blocks: 1574154 1574155
TreeView+ depends on / blocked
Reported: 2018-05-01 18:19 UTC by myoder
Modified: 2019-02-11 13:59 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1574154 1574155 (view as bug list)
Last Closed: 2019-02-11 13:59:20 UTC
Category: Bug
Cloudforms Team: VMware
Target Upstream Version:

Attachments (Terms of Use)

Description myoder 2018-05-01 18:19:12 UTC
Description of problem:

Refreshes are failing for a VMware provider.  The logs are indicating the VIM object is too large:

[----] I, [2018-05-01T09:01:45.655061 #13690:3c5140]  INFO -- : MIQ(MiqVimBrokerWorker::Runner#on_update_event) Queueing refresh for ManageIQ::Providers::Vmware::InfraManager::Vm id: [10363], EMS id: [1] on event [VirtualMachine-update] for properties ["summary.guest.ipAddress"]

[----] I, [2018-05-01T09:01:47.555578 #13690:3c5140]  INFO -- : MIQ(MiqQueue.put_or_update) Message id: [142481706] updated with following: {:queue_name=>"ems_1", :state=>"ready", :zone=>"vmware", :class_name=>"EmsRefresh", :method_name=>"refresh", :role=>"ems_inventory", :args=>[[["ManageIQ::Providers::Vmware::InfraManager::Vm", 11808], ["ManageIQ::Providers::Vmware::InfraManager::Vm", 10363]]], :task_id=>nil, :msg_timeout=>7200, :miq_callback=>{:class_name=>"MiqTask", :method_name=>:queue_callback, :instance_id=>nil, :args=>["Finished"]}}

[----] I, [2018-05-01T09:01:47.556098 #13690:3c5140]  INFO -- : MIQ(MiqQueue.put_or_update) Message id: [142481706],  id: [], Zone: [vmware], Role: [ems_inventory], Server: [], Ident: [ems_1], Target id: [], Instance id: [], Task id: [], Command: [EmsRefresh.refresh], Timeout: [7200], Priority: [100], State: [ready], Deliver On: [], Data: [], Args: [[["ManageIQ::Providers::Vmware::InfraManager::Vm", 11808], ["ManageIQ::Providers::Vmware::InfraManager::Vm", 10363]]], Requeued

[----] E, [2018-05-01T09:02:32.692674 #13690:1a0be504] ERROR -- : <VIM> DRb packet size too large: 52946953

[----] E, [2018-05-01T09:02:44.808595 #13690:1a0be504] ERROR -- : <VIM> object: 


...incredibly long hash...


[----] E, [2018-05-01T09:01:43.123886 #13690:a2851cc] ERROR -- : <VIM> caller:


The object itself appears to be a hash of incredible size, I put the 1 log line that included the object into a file that is 30MB, and 1.2 million words:


  [root@billy log]# wc wow.txt
         1  1259726 30718416 wow.txt

  [root@billy log]# ls -lht
  -rw-r--r--. 1 root root  30M May  1 11:25 wow.txt


Restarting evmserverd has not helped.

No significant size change in provider in the last few days.  This issue started 4/29.

Versions Affected:

vSphere 6.0 with ESXi 6.0

How reproducible:

Steps to Reproduce:

Actual results:

Expected results:

Additional info:

Files have been uploaded to collab shell:


Comment 2 Adam Grare 2018-05-01 18:29:47 UTC
So the packet size says it is 52946953 which is a hair over 50MiB (50.49Mib to be exact).  The default packet size in ruby for DRb is 25 MiB, we increase this to 50MiB here: https://github.com/ManageIQ/vmware_web_service/blob/master/lib/VMwareWebService/MiqVimBroker.rb#L23

It is possible that they have a large enough inventory that they are going over this 50MiB limit.

Comment 9 CFME Bot 2018-05-02 16:04:37 UTC
New commit detected on ManageIQ/vmware_web_service/master:

commit fd469e8de93d02f6afa189ea26cb63e1291981a9
Author:     Adam Grare <agrare@redhat.com>
AuthorDate: Tue May  1 15:39:26 2018 -0400
Commit:     Adam Grare <agrare@redhat.com>
CommitDate: Tue May  1 15:39:26 2018 -0400

    Increase the DRb::DRbServer.load_limit size

    Increase the DRb::DRbServer.load_limit size to handle larger vmware


 lib/VMwareWebService/MiqVimBroker.rb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Comment 13 CFME Bot 2018-05-02 18:35:19 UTC
New commit detected on ManageIQ/manageiq-providers-vmware/master:

commit 8dfa67c0c9f5a08a150502cc5f62c6d5afb01e4d
Author:     Adam Grare <agrare@redhat.com>
AuthorDate: Wed May  2 14:03:13 2018 -0400
Commit:     Adam Grare <agrare@redhat.com>
CommitDate: Wed May  2 14:03:13 2018 -0400

    Bump version of vmware_web_service

    Require vmware_web_service 0.2.9 to pull in fix for large DRb message


 manageiq-providers-vmware.gemspec | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Comment 14 Kedar Kulkarni 2018-10-30 17:39:06 UTC
On CFME Version, this is fixed by updating settings in  /opt/rh/cfme-gemset/gems/vmware_web_service-0.3.0/lib/VMwareWebService/MiqVimBroker.rb as follows:

DRb::DRbServer.default_load_limit(100 * MB)

Note You need to log in before you can comment on or make changes to this bug.