Bug 1465761

Summary: Refresh failed for VMware Provider in Cloudforms 4.5
Product: Red Hat CloudForms Management Engine Reporter: Neha Chugh <nchugh>
Component: ProvidersAssignee: James Wong <jwong>
Status: CLOSED CURRENTRELEASE QA Contact: Leo Khomenko <lkhomenk>
Severity: high Docs Contact:
Priority: high    
Version: 5.8.0CC: agrare, bsorota, cchen, dajohnso, gblomqui, greartes, jfrey, jhardy, lkhomenk, nchugh, obarenbo, simaishi, tachoi
Target Milestone: GAKeywords: TestOnly, ZStream
Target Release: 5.9.0   
Hardware: All   
OS: Linux   
Whiteboard: vmware:ems_refresh
Fixed In Version: 5.9.0.1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1477727 1479976 (view as bug list) Environment:
Last Closed: 2018-03-06 15:11:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: VMware Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1477727, 1479976    
Attachments:
Description Flags
MOB screen shot
none
disk detail
none
mob output
none
disk1 detail
none
inconsistent hard disk none

Description Neha Chugh 2017-06-28 07:07:10 UTC
Description of problem:

While doing the manual refresh of the Provider, it failed with below exception:

[----] E, [2017-06-26T15:28:24.832907 #29096:1311130] ERROR -- : MIQ(ManageIQ::Providers::Vmware::InfraManager::Refresher#refresh) EMS: [vCenter-DR], id: [2000000000007] Unable to perform refresh for the following targets:
[----] E, [2017-06-26T15:28:24.833036 #29096:1311130] ERROR -- : MIQ(ManageIQ::Providers::Vmware::InfraManager::Refresher#refresh)  --- ManageIQ::Providers::Vmware::InfraManager [vCenter-DR] id [2000000000007]
[----] I, [2017-06-26T15:28:24.864483 #29096:1311130]  INFO -- : MIQ(ManageIQ::Providers::Vmware::InfraManager::Refresher#refresh) Refreshing all targets...Complete
[----] E, [2017-06-26T15:28:24.869871 #29096:1311130] ERROR -- : [EmsRefresh::Refreshers::EmsRefresherMixin::PartialRefreshError]: undefined method `xsiType' for nil:NilClass  Method:[rescue in deliver]
[----] E, [2017-06-26T15:28:24.872356 #29096:1311130] ERROR -- : /var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:50:in `refresh'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:9:in `refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:98:in `block in refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:97:in `each'
/var/www/miq/vmdb/app/models/ems_refresh.rb:97:in `refresh'

Version-Release number of selected component (if applicable):

5.8.0

Actual results:

Refresh is failing with exception undefined method `xsiType' for nil:NilClass  Method:[rescue in deliver]

Expected results:
Refresh should be done successfully.

Additional info:

Increased the memory threshold from 2GB to 4 GB of vim_broker_worker but still didn't work.
Seems like the target refresh is failing as full refresh is completed above it.

Comment 3 Adam Grare 2017-06-29 20:39:32 UTC
From the logs it looks like the error is:

[----] E, [2017-06-22T12:17:18.852473 #26538:1311130] ERROR -- : [NoMethodError]: undefined method `xsiType' for nil:NilClass  Method:[rescue in block in refresh]
[----] E, [2017-06-22T12:17:18.852641 #26538:1311130] ERROR -- : /opt/rh/cfme-gemset/bundler/gems/manageiq-providers-vmware-30e6243bf993/app/models/manageiq/providers/vmware/infra_manager/refresh_parser.rb:1074:
/opt/rh/cfme-gemset/bundler/gems/manageiq-providers-vmware-30e6243bf993/app/models/manageiq/providers/vmware/infra_manager/refresh_parser.rb:1062:in `each'
/opt/rh/cfme-gemset/bundler/gems/manageiq-providers-vmware-30e6243bf993/app/models/manageiq/providers/vmware/infra_manager/refresh_parser.rb:1062:in `vm_inv_to_disk_hashes'

This is coming from https://github.com/ManageIQ/manageiq-providers-vmware/blob/fine/app/models/manageiq/providers/vmware/infra_manager/refresh_parser.rb#L1074

It looks to me like there is a VM that has a disk assigned to a deleted controller but we will need to see the managed object browser to confirm.

Comment 11 James Wong 2017-07-05 12:27:11 UTC
Created attachment 1294568 [details]
MOB screen shot

Comment 13 Neha Chugh 2017-07-07 06:02:02 UTC
Created attachment 1295178 [details]
disk detail

Comment 14 Neha Chugh 2017-07-07 06:02:33 UTC
Created attachment 1295179 [details]
mob output

Comment 15 Neha Chugh 2017-07-07 06:04:16 UTC
Created attachment 1295180 [details]
disk1 detail

Comment 25 James Wong 2017-07-17 13:15:49 UTC
Hi Chen,

Don't know how such a vm is created. vSphere client doesn't seem to allow that to be created.  Can you request customer to inspect this vm and see how they can fix it, e.g. adding controller to the disks or removing the disks?  And then rerun the refresh.

We probably can't do much with such disks without controller key.  What we can do to fix this error is to skip them. 

regards,
James

Comment 26 Neha Chugh 2017-07-24 04:53:52 UTC
Hello James,

Customer is unable to remove the disk as the disk is in inconsistent state( as per attached). Let me know if you have any other suggestion so to rectify this issue.

Regards,
Neha Chugh

Comment 27 Neha Chugh 2017-07-24 04:57:22 UTC
Created attachment 1303436 [details]
inconsistent hard disk

Comment 28 James Wong 2017-07-24 12:55:10 UTC
Hi Neha,

The customer probably has to contact VMware support for fixing the vm.

If they are savvy enough, they may try to de-register the VM and then edit the vm file (in the datastore) to remove the disk.  But they should consult VMware support first.

James

Comment 30 James Wong 2017-07-31 18:07:12 UTC
Hi Neha,

Any update from the customer?  

What we are doing to the code is to inventory these disks as having 'unknown' controller type.

regards,
James

Comment 33 Neha Chugh 2017-08-01 13:09:33 UTC
Hello James,

We haven't received any response from customer till date whether he is able to remove the disk or not so we can lower down the severity.

In case if he is not able to delete the disk we can suggest the provided PR.

Thanks for all help.

Regards,
Neha Chugh

Comment 34 CFME Bot 2017-08-01 16:08:48 UTC
New commit detected on ManageIQ/manageiq-providers-vmware/master:
https://github.com/ManageIQ/manageiq-providers-vmware/commit/b5f1089beb71f98026f22e8352383ecf83a04593

commit b5f1089beb71f98026f22e8352383ecf83a04593
Author:     James Wong <jwong>
AuthorDate: Tue Jul 18 09:54:46 2017 -0400
Commit:     James Wong <jwong>
CommitDate: Tue Aug 1 11:47:55 2017 -0400

    dealing with disks which have no controller key
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1465761

 .../providers/vmware/infra_manager/refresh_parser.rb    | 17 +++++++++++------
 .../providers/vmware/infra_manager/refresher_spec.rb    | 15 +++++++++++++++
 .../vim_data/miq_vim_inventory/virtualMachinesByMor.yml | 10 +---------
 3 files changed, 27 insertions(+), 15 deletions(-)