Description of problem: Refreshes are failing for a VMware provider. The logs are indicating the VIM object is too large: ~~~ [----] I, [2018-05-01T09:01:45.655061 #13690:3c5140] INFO -- : MIQ(MiqVimBrokerWorker::Runner#on_update_event) Queueing refresh for ManageIQ::Providers::Vmware::InfraManager::Vm id: [10363], EMS id: [1] on event [VirtualMachine-update] for properties ["summary.guest.ipAddress"] [----] I, [2018-05-01T09:01:47.555578 #13690:3c5140] INFO -- : MIQ(MiqQueue.put_or_update) Message id: [142481706] updated with following: {:queue_name=>"ems_1", :state=>"ready", :zone=>"vmware", :class_name=>"EmsRefresh", :method_name=>"refresh", :role=>"ems_inventory", :args=>[[["ManageIQ::Providers::Vmware::InfraManager::Vm", 11808], ["ManageIQ::Providers::Vmware::InfraManager::Vm", 10363]]], :task_id=>nil, :msg_timeout=>7200, :miq_callback=>{:class_name=>"MiqTask", :method_name=>:queue_callback, :instance_id=>nil, :args=>["Finished"]}} [----] I, [2018-05-01T09:01:47.556098 #13690:3c5140] INFO -- : MIQ(MiqQueue.put_or_update) Message id: [142481706], id: [], Zone: [vmware], Role: [ems_inventory], Server: [], Ident: [ems_1], Target id: [], Instance id: [], Task id: [], Command: [EmsRefresh.refresh], Timeout: [7200], Priority: [100], State: [ready], Deliver On: [], Data: [], Args: [[["ManageIQ::Providers::Vmware::InfraManager::Vm", 11808], ["ManageIQ::Providers::Vmware::InfraManager::Vm", 10363]]], Requeued [----] E, [2018-05-01T09:02:32.692674 #13690:1a0be504] ERROR -- : <VIM> DRb packet size too large: 52946953 [----] E, [2018-05-01T09:02:44.808595 #13690:1a0be504] ERROR -- : <VIM> object: ... ...incredibly long hash... ... [----] E, [2018-05-01T09:01:43.123886 #13690:a2851cc] ERROR -- : <VIM> caller: ~~~ The object itself appears to be a hash of incredible size, I put the 1 log line that included the object into a file that is 30MB, and 1.2 million words: ~~~ [root@billy log]# wc wow.txt 1 1259726 30718416 wow.txt [root@billy log]# ls -lht -rw-r--r--. 1 root root 30M May 1 11:25 wow.txt ~~~ Restarting evmserverd has not helped. No significant size change in provider in the last few days. This issue started 4/29. Versions Affected: CFME 5.8.3.4 vSphere 6.0 with ESXi 6.0 How reproducible: custom Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: Files have been uploaded to collab shell: http://collab-shell.usersys.redhat.com/02089694/
So the packet size says it is 52946953 which is a hair over 50MiB (50.49Mib to be exact). The default packet size in ruby for DRb is 25 MiB, we increase this to 50MiB here: https://github.com/ManageIQ/vmware_web_service/blob/master/lib/VMwareWebService/MiqVimBroker.rb#L23 It is possible that they have a large enough inventory that they are going over this 50MiB limit.
https://github.com/ManageIQ/manageiq-providers-vmware/pull/245
https://github.com/ManageIQ/vmware_web_service/pull/38
https://github.com/ManageIQ/manageiq/pull/17371
New commit detected on ManageIQ/vmware_web_service/master: https://github.com/ManageIQ/vmware_web_service/commit/fd469e8de93d02f6afa189ea26cb63e1291981a9 commit fd469e8de93d02f6afa189ea26cb63e1291981a9 Author: Adam Grare <agrare> AuthorDate: Tue May 1 15:39:26 2018 -0400 Commit: Adam Grare <agrare> CommitDate: Tue May 1 15:39:26 2018 -0400 Increase the DRb::DRbServer.load_limit size Increase the DRb::DRbServer.load_limit size to handle larger vmware inventories. https://bugzilla.redhat.com/show_bug.cgi?id=1573588 lib/VMwareWebService/MiqVimBroker.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
New commit detected on ManageIQ/manageiq-providers-vmware/master: https://github.com/ManageIQ/manageiq-providers-vmware/commit/8dfa67c0c9f5a08a150502cc5f62c6d5afb01e4d commit 8dfa67c0c9f5a08a150502cc5f62c6d5afb01e4d Author: Adam Grare <agrare> AuthorDate: Wed May 2 14:03:13 2018 -0400 Commit: Adam Grare <agrare> CommitDate: Wed May 2 14:03:13 2018 -0400 Bump version of vmware_web_service Require vmware_web_service 0.2.9 to pull in fix for large DRb message payloads. https://bugzilla.redhat.com/show_bug.cgi?id=1573588 manageiq-providers-vmware.gemspec | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
On CFME Version 5.10.0.21, this is fixed by updating settings in /opt/rh/cfme-gemset/gems/vmware_web_service-0.3.0/lib/VMwareWebService/MiqVimBroker.rb as follows: DRb::DRbServer.default_load_limit(100 * MB)