Bug 1328336 - The host,cluster & Datastores count is shown as zero for vmware provider
Summary: The host,cluster & Datastores count is shown as zero for vmware provider
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Providers
Version: 5.6.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: GA
: 5.6.0
Assignee: Adam Grare
QA Contact: Niyaz Akhtar Ansari
URL:
Whiteboard: provider:infra:vsphere
: 1328520 1331110 (view as bug list)
Depends On:
Blocks: 1331110
TreeView+ depends on / blocked
 
Reported: 2016-04-19 07:21 UTC by Niyaz Akhtar Ansari
Modified: 2016-06-29 15:51 UTC (History)
12 users (show)

Fixed In Version: 5.6.0.5
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-06-29 15:51:10 UTC
Category: ---
Cloudforms Team: ---
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
snippet of vim.log (407.51 KB, text/plain)
2016-04-25 08:59 UTC, Niyaz Akhtar Ansari
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:1348 0 normal SHIPPED_LIVE CFME 5.6.0 bug fixes and enhancement update 2016-06-29 18:50:04 UTC

Description Niyaz Akhtar Ansari 2016-04-19 07:21:39 UTC
Created attachment 1148363 [details]
error

Description of problem:


Version-Release number of selected component (if applicable):
5.6.0.1-beta2.20160413141124_e25ac0e 

How reproducible:
100%

Steps to Reproduce:
1.Navigate to infrastructure->Providers->configuration->Add New Infrastructure Provider 
2.Add new vsphere provider and wait for few mins.
3.

Actual results:
After sometime the host,clusters, Datastores  count is shown as zero. see attached screenshot.

Expected results:
correct count should be displayed for all the objects and status should be up. 

Additional info:

Comment 2 Satyajit Bulage 2016-04-19 07:30:11 UTC
This issue is Affecting provisioning of VM from vSphere provider as "<No Choices Available>" while selecting "vLan*" Network. 
I have logged https://bugzilla.redhat.com/show_bug.cgi?id=1327601 to track this issue.

Comment 4 Satyajit Bulage 2016-04-19 11:41:45 UTC
This issue is reproducible and I am getting following error on Provider Refresh 

Here is a snippet of evm.log :

[----] I, [2016-04-19T07:05:59.575409 #13030:cef98c]  INFO -- : MIQ(EmsRefresh.save_ems_infra_inventory) EMS: [vSphere55], id: [7000        000000001] Saving EMS Inventory...Complete
 [----] I, [2016-04-19T07:05:59.576000 #13030:cef98c]  INFO -- : MIQ(EmsRefresh.link_ems_inventory) EMS: [vSphere55], id: [7000000000        001] Linking EMS Inventory...
 [----] I, [2016-04-19T07:05:59.577091 #13030:cef98c]  INFO -- : MIQ(EmsRefresh.link_ems_inventory) EMS: [vSphere55], id: [7000000000        001] Updating EMS root folder relationship.
 [----] I, [2016-04-19T07:05:59.581515 #13030:cef98c]  INFO -- : MIQ(EmsRefresh.update_relats) Updating Folders To Folders relationsh        ips.
  [----] E, [2016-04-19T07:05:59.599786 #13030:cef98c] ERROR -- : MIQ(EmsRefresh.update_relats_by_ids) An error occurred while disconn        ecting id [7000000000016]: undefined method `base_class' for NilClass:Class
   [----] E, [2016-04-19T07:05:59.600296 #13030:cef98c] ERROR -- : [NoMethodError]: undefined method `base_class' for NilClass:Class  M        ethod:[rescue in block in update_relats_by_ids]
  [----] E, [2016-04-19T07:05:59.600578 #13030:cef98c] ERROR -- : /var/www/miq/vmdb/app/models/mixins/relationship_mixin.rb:600:in `bl        ock in remove_children'
  /var/www/miq/vmdb/app/models/mixins/relationship_mixin.rb:600:in `collect'
  /var/www/miq/vmdb/app/models/mixins/relationship_mixin.rb:600:in `remove_children'
  /var/www/miq/vmdb/app/models/ems_refresh/link_inventory.rb:32:in `block (2 levels) in link_ems_inventory'
  /var/www/miq/vmdb/app/models/ems_refresh/link_inventory.rb:171:in `call'
  /var/www/miq/vmdb/app/models/ems_refresh/link_inventory.rb:171:in `block in update_relats_by_ids'
  /var/www/miq/vmdb/app/models/ems_refresh/link_inventory.rb:169:in `each'
  /var/www/miq/vmdb/app/models/ems_refresh/link_inventory.rb:169:in `update_relats_by_ids'
  /var/www/miq/vmdb/app/models/ems_refresh/link_inventory.rb:156:in `block in update_relats'
  /var/www/miq/vmdb/app/models/ems_refresh/link_inventory.rb:154:in `each'
/var/www/miq/vmdb/app/models/ems_refresh/link_inventory.rb:154:in `update_relats'
/var/www/miq/vmdb/app/models/ems_refresh/link_inventory.rb:29:in `link_ems_inventory'
/var/www/miq/vmdb/app/models/ems_refresh/save_inventory_infra.rb:78:in `save_ems_infra_inventory'
/var/www/miq/vmdb/app/models/ems_refresh/save_inventory.rb:5:in `save_ems_inventory'
/var/www/miq/vmdb/app/models/manageiq/providers/vmware/infra_manager/refresher.rb:59:in `block in save_inventory'
/var/www/miq/vmdb/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/var/www/miq/vmdb/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block'
/var/www/miq/vmdb/app/models/manageiq/providers/vmware/infra_manager/refresher.rb:56:in `save_inventory'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:86:in `block in refresh_targets_for_ems'
/var/www/miq/vmdb/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/var/www/miq/vmdb/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:86:in `refresh_targets_for_ems'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:21:in `block (2 levels) in refresh'
/var/www/miq/vmdb/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/var/www/miq/vmdb/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:21:in `block in refresh'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:11:in `each'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:11:in `refresh'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:10:in `refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:78:in `block in refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:77:in `each'
/var/www/miq/vmdb/app/models/ems_refresh.rb:77:in `refresh'
/var/www/miq/vmdb/app/models/miq_queue.rb:347:in `block in deliver'
/opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:89:in `block in timeout'
/opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:34:in `block in catch'
/opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:34:in `catch' 
/opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:34:in `catch' 
/opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:104:in `timeout'
/var/www/miq/vmdb/app/models/miq_queue.rb:343:in `deliver'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:106:in `deliver_queue_message'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:134:in `deliver_message'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:152:in `block in do_work'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:146:in `loop'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:146:in `do_work'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:334:in `block in do_work_loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:331:in `loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:331:in `loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:331:in `do_work_loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:153:in `run'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:128:in `start'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:21:in `start_worker'
/var/www/miq/vmdb/app/models/miq_worker.rb:346:in `block in start'
/opt/rh/cfme-gemset/gems/nakayoshi_fork-0.0.3/lib/nakayoshi_fork.rb:24:in `fork'
/opt/rh/cfme-gemset/gems/nakayoshi_fork-0.0.3/lib/nakayoshi_fork.rb:24:in `fork'
/var/www/miq/vmdb/app/models/miq_worker.rb:344:in `start'
/var/www/miq/vmdb/app/models/miq_worker.rb:274:in `start_worker'
/var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:68:in `start_worker_for_ems'
/var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:46:in `block in sync_workers'
/var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:45:in `each'
/var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:45:in `sync_workers'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:49:in `block in sync_workers'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:47:in `each'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:47:in `sync_workers'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:19:in `monitor_workers'
/var/www/miq/vmdb/app/models/miq_server.rb:361:in `block in monitor'
/var/www/miq/vmdb/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/var/www/miq/vmdb/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block'
/var/www/miq/vmdb/app/models/miq_server.rb:361:in `monitor'
/var/www/miq/vmdb/app/models/miq_server.rb:381:in `block (2 levels) in monitor_loop'
/var/www/miq/vmdb/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/var/www/miq/vmdb/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block'
/var/www/miq/vmdb/app/models/miq_server.rb:381:in `block in monitor_loop'
/var/www/miq/vmdb/app/models/miq_server.rb:380:in `loop'
/var/www/miq/vmdb/app/models/miq_server.rb:380:in `monitor_loop'
/var/www/miq/vmdb/app/models/miq_server.rb:265:in `start'
/var/www/miq/vmdb/lib/workers/evm_server.rb:65:in `start'
/var/www/miq/vmdb/lib/workers/evm_server.rb:92:in `start'
/var/www/miq/vmdb/lib/workers/bin/evm_server.rb:4:in `<main>'

Comment 6 Adam Grare 2016-04-21 20:36:08 UTC
Satyajit, I was able to reproduce the exception you noted as far back as 5.4 simply by removing a sub-folder and re-running an inventory refresh.  I did not notice any negative side-effects of this and I doubt this is the root cause of the issue.

I do see the following:
[2016-04-19T07:02:20.575561 #13030:cef98c] (snip) Retrieving Host inventory...Complete - Count: [5]
...
[2016-04-19T07:02:58.393027 #13030:cef98c] (snip) Retrieving Host inventory...Complete - Count: [0]

And in the vim.log we start seeing this:
[2016-04-19T07:04:09.552102 #13074:cd02f8]  WARN -- : updateProps (10.8.58.22_root): object domain-c241 not found in @clusterComputeResourcesByMor

This is about 26mins after initially adding the provider, I suspect there is some issue with the VimBroker cache.

Niyaz, could you change the log level for the VimBroker (in config "level_vim: debug") and restart your appliance?  This should give us a better idea what else is happening with the broker.

Comment 8 Niyaz Akhtar Ansari 2016-04-25 08:59:10 UTC
Created attachment 1150314 [details]
snippet of vim.log

Have attached the vim.log

Comment 11 Adam Grare 2016-04-25 12:07:11 UTC
*** Bug 1328520 has been marked as a duplicate of this bug. ***

Comment 12 CFME Bot 2016-04-26 15:40:54 UTC
New commit detected on ManageIQ/manageiq/master:
https://github.com/ManageIQ/manageiq/commit/b55d184035c5e106a408ee454574e94894635adc

commit b55d184035c5e106a408ee454574e94894635adc
Author:     Adam Grare <agrare>
AuthorDate: Fri Apr 22 17:07:23 2016 -0400
Commit:     Adam Grare <agrare>
CommitDate: Mon Apr 25 16:11:59 2016 +0000

    Collect all properties if return is truncated
    
    If the return result from RetrievePropertiesEx is truncated,
    call ContinueRetrievePropertiesEx with the token to collect
    the rest of the results.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1328336

 gems/pending/VMwareWebService/VimService.rb | 23 ++++++++++++++++++++++-
 1 file changed, 22 insertions(+), 1 deletion(-)

Comment 13 Dan Clarizio 2016-05-03 19:06:40 UTC
*** Bug 1331110 has been marked as a duplicate of this bug. ***

Comment 14 Niyaz Akhtar Ansari 2016-05-04 05:58:43 UTC
Able to see correct count for all the objects and status is up and running.

Verified in 5.6.0.5-beta2.4.20160503153816_1fb554f

Comment 17 errata-xmlrpc 2016-06-29 15:51:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1348


Note You need to log in before you can comment on or make changes to this bug.