Bug 1573588 - Refresh Failing for VMware VIM object is too large
Summary: Refresh Failing for VMware VIM object is too large
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Providers
Version: 5.8.0
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: GA
: 5.10.0
Assignee: Adam Grare
QA Contact: Kedar Kulkarni
URL:
Whiteboard:
Depends On:
Blocks: 1574154 1574155
TreeView+ depends on / blocked
 
Reported: 2018-05-01 18:19 UTC by myoder
Modified: 2019-02-11 13:59 UTC (History)
9 users (show)

Fixed In Version: 5.10.0.0
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1574154 1574155 (view as bug list)
Environment:
Last Closed: 2019-02-11 13:59:20 UTC
Category: Bug
Cloudforms Team: VMware
Target Upstream Version:


Attachments (Terms of Use)

Description myoder 2018-05-01 18:19:12 UTC
Description of problem:

Refreshes are failing for a VMware provider.  The logs are indicating the VIM object is too large:

~~~
[----] I, [2018-05-01T09:01:45.655061 #13690:3c5140]  INFO -- : MIQ(MiqVimBrokerWorker::Runner#on_update_event) Queueing refresh for ManageIQ::Providers::Vmware::InfraManager::Vm id: [10363], EMS id: [1] on event [VirtualMachine-update] for properties ["summary.guest.ipAddress"]

[----] I, [2018-05-01T09:01:47.555578 #13690:3c5140]  INFO -- : MIQ(MiqQueue.put_or_update) Message id: [142481706] updated with following: {:queue_name=>"ems_1", :state=>"ready", :zone=>"vmware", :class_name=>"EmsRefresh", :method_name=>"refresh", :role=>"ems_inventory", :args=>[[["ManageIQ::Providers::Vmware::InfraManager::Vm", 11808], ["ManageIQ::Providers::Vmware::InfraManager::Vm", 10363]]], :task_id=>nil, :msg_timeout=>7200, :miq_callback=>{:class_name=>"MiqTask", :method_name=>:queue_callback, :instance_id=>nil, :args=>["Finished"]}}

[----] I, [2018-05-01T09:01:47.556098 #13690:3c5140]  INFO -- : MIQ(MiqQueue.put_or_update) Message id: [142481706],  id: [], Zone: [vmware], Role: [ems_inventory], Server: [], Ident: [ems_1], Target id: [], Instance id: [], Task id: [], Command: [EmsRefresh.refresh], Timeout: [7200], Priority: [100], State: [ready], Deliver On: [], Data: [], Args: [[["ManageIQ::Providers::Vmware::InfraManager::Vm", 11808], ["ManageIQ::Providers::Vmware::InfraManager::Vm", 10363]]], Requeued

[----] E, [2018-05-01T09:02:32.692674 #13690:1a0be504] ERROR -- : <VIM> DRb packet size too large: 52946953

[----] E, [2018-05-01T09:02:44.808595 #13690:1a0be504] ERROR -- : <VIM> object: 

...

...incredibly long hash...

...


[----] E, [2018-05-01T09:01:43.123886 #13690:a2851cc] ERROR -- : <VIM> caller:

~~~


The object itself appears to be a hash of incredible size, I put the 1 log line that included the object into a file that is 30MB, and 1.2 million words:

~~~

  [root@billy log]# wc wow.txt
         1  1259726 30718416 wow.txt

  [root@billy log]# ls -lht
  -rw-r--r--. 1 root root  30M May  1 11:25 wow.txt

~~~

Restarting evmserverd has not helped.

No significant size change in provider in the last few days.  This issue started 4/29.

Versions Affected:

CFME 5.8.3.4
vSphere 6.0 with ESXi 6.0

How reproducible:
custom

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Files have been uploaded to collab shell:

  http://collab-shell.usersys.redhat.com/02089694/

Comment 2 Adam Grare 2018-05-01 18:29:47 UTC
So the packet size says it is 52946953 which is a hair over 50MiB (50.49Mib to be exact).  The default packet size in ruby for DRb is 25 MiB, we increase this to 50MiB here: https://github.com/ManageIQ/vmware_web_service/blob/master/lib/VMwareWebService/MiqVimBroker.rb#L23

It is possible that they have a large enough inventory that they are going over this 50MiB limit.

Comment 9 CFME Bot 2018-05-02 16:04:37 UTC
New commit detected on ManageIQ/vmware_web_service/master:

https://github.com/ManageIQ/vmware_web_service/commit/fd469e8de93d02f6afa189ea26cb63e1291981a9
commit fd469e8de93d02f6afa189ea26cb63e1291981a9
Author:     Adam Grare <agrare@redhat.com>
AuthorDate: Tue May  1 15:39:26 2018 -0400
Commit:     Adam Grare <agrare@redhat.com>
CommitDate: Tue May  1 15:39:26 2018 -0400

    Increase the DRb::DRbServer.load_limit size

    Increase the DRb::DRbServer.load_limit size to handle larger vmware
    inventories.

    https://bugzilla.redhat.com/show_bug.cgi?id=1573588

 lib/VMwareWebService/MiqVimBroker.rb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Comment 13 CFME Bot 2018-05-02 18:35:19 UTC
New commit detected on ManageIQ/manageiq-providers-vmware/master:

https://github.com/ManageIQ/manageiq-providers-vmware/commit/8dfa67c0c9f5a08a150502cc5f62c6d5afb01e4d
commit 8dfa67c0c9f5a08a150502cc5f62c6d5afb01e4d
Author:     Adam Grare <agrare@redhat.com>
AuthorDate: Wed May  2 14:03:13 2018 -0400
Commit:     Adam Grare <agrare@redhat.com>
CommitDate: Wed May  2 14:03:13 2018 -0400

    Bump version of vmware_web_service

    Require vmware_web_service 0.2.9 to pull in fix for large DRb message
    payloads.

    https://bugzilla.redhat.com/show_bug.cgi?id=1573588

 manageiq-providers-vmware.gemspec | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Comment 14 Kedar Kulkarni 2018-10-30 17:39:06 UTC
On CFME Version 5.10.0.21, this is fixed by updating settings in  /opt/rh/cfme-gemset/gems/vmware_web_service-0.3.0/lib/VMwareWebService/MiqVimBroker.rb as follows:

DRb::DRbServer.default_load_limit(100 * MB)


Note You need to log in before you can comment on or make changes to this bug.