Bug 1465761 - Refresh failed for VMware Provider in Cloudforms 4.5
Refresh failed for VMware Provider in Cloudforms 4.5
Status: POST
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Providers (Show other bugs)
5.8.0
All Linux
high Severity high
: GA
: 5.9.0
Assigned To: James Wong
Leo Khomenko
vmware:ems_refresh
: TestOnly, ZStream
Depends On:
Blocks: 1477727 1479976
  Show dependency treegraph
 
Reported: 2017-06-28 03:07 EDT by Neha Chugh
Modified: 2017-08-09 16:43 EDT (History)
12 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1477727 1479976 (view as bug list)
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: VMware


Attachments (Terms of Use)
MOB screen shot (226.75 KB, image/png)
2017-07-05 08:27 EDT, James Wong
no flags Details
disk detail (26.98 KB, image/png)
2017-07-07 02:02 EDT, Neha Chugh
no flags Details
mob output (31.58 KB, image/png)
2017-07-07 02:02 EDT, Neha Chugh
no flags Details
disk1 detail (26.97 KB, image/png)
2017-07-07 02:04 EDT, Neha Chugh
no flags Details
inconsistent hard disk (28.63 KB, image/png)
2017-07-24 00:57 EDT, Neha Chugh
no flags Details

  None (edit)
Description Neha Chugh 2017-06-28 03:07:10 EDT
Description of problem:

While doing the manual refresh of the Provider, it failed with below exception:

[----] E, [2017-06-26T15:28:24.832907 #29096:1311130] ERROR -- : MIQ(ManageIQ::Providers::Vmware::InfraManager::Refresher#refresh) EMS: [vCenter-DR], id: [2000000000007] Unable to perform refresh for the following targets:
[----] E, [2017-06-26T15:28:24.833036 #29096:1311130] ERROR -- : MIQ(ManageIQ::Providers::Vmware::InfraManager::Refresher#refresh)  --- ManageIQ::Providers::Vmware::InfraManager [vCenter-DR] id [2000000000007]
[----] I, [2017-06-26T15:28:24.864483 #29096:1311130]  INFO -- : MIQ(ManageIQ::Providers::Vmware::InfraManager::Refresher#refresh) Refreshing all targets...Complete
[----] E, [2017-06-26T15:28:24.869871 #29096:1311130] ERROR -- : [EmsRefresh::Refreshers::EmsRefresherMixin::PartialRefreshError]: undefined method `xsiType' for nil:NilClass  Method:[rescue in deliver]
[----] E, [2017-06-26T15:28:24.872356 #29096:1311130] ERROR -- : /var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:50:in `refresh'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:9:in `refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:98:in `block in refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:97:in `each'
/var/www/miq/vmdb/app/models/ems_refresh.rb:97:in `refresh'

Version-Release number of selected component (if applicable):

5.8.0

Actual results:

Refresh is failing with exception undefined method `xsiType' for nil:NilClass  Method:[rescue in deliver]

Expected results:
Refresh should be done successfully.

Additional info:

Increased the memory threshold from 2GB to 4 GB of vim_broker_worker but still didn't work.
Seems like the target refresh is failing as full refresh is completed above it.
Comment 3 Adam Grare 2017-06-29 16:39:32 EDT
From the logs it looks like the error is:

[----] E, [2017-06-22T12:17:18.852473 #26538:1311130] ERROR -- : [NoMethodError]: undefined method `xsiType' for nil:NilClass  Method:[rescue in block in refresh]
[----] E, [2017-06-22T12:17:18.852641 #26538:1311130] ERROR -- : /opt/rh/cfme-gemset/bundler/gems/manageiq-providers-vmware-30e6243bf993/app/models/manageiq/providers/vmware/infra_manager/refresh_parser.rb:1074:
/opt/rh/cfme-gemset/bundler/gems/manageiq-providers-vmware-30e6243bf993/app/models/manageiq/providers/vmware/infra_manager/refresh_parser.rb:1062:in `each'
/opt/rh/cfme-gemset/bundler/gems/manageiq-providers-vmware-30e6243bf993/app/models/manageiq/providers/vmware/infra_manager/refresh_parser.rb:1062:in `vm_inv_to_disk_hashes'

This is coming from https://github.com/ManageIQ/manageiq-providers-vmware/blob/fine/app/models/manageiq/providers/vmware/infra_manager/refresh_parser.rb#L1074

It looks to me like there is a VM that has a disk assigned to a deleted controller but we will need to see the managed object browser to confirm.
Comment 11 James Wong 2017-07-05 08:27 EDT
Created attachment 1294568 [details]
MOB screen shot
Comment 13 Neha Chugh 2017-07-07 02:02 EDT
Created attachment 1295178 [details]
disk detail
Comment 14 Neha Chugh 2017-07-07 02:02 EDT
Created attachment 1295179 [details]
mob output
Comment 15 Neha Chugh 2017-07-07 02:04 EDT
Created attachment 1295180 [details]
disk1 detail
Comment 25 James Wong 2017-07-17 09:15:49 EDT
Hi Chen,

Don't know how such a vm is created. vSphere client doesn't seem to allow that to be created.  Can you request customer to inspect this vm and see how they can fix it, e.g. adding controller to the disks or removing the disks?  And then rerun the refresh.

We probably can't do much with such disks without controller key.  What we can do to fix this error is to skip them. 

regards,
James
Comment 26 Neha Chugh 2017-07-24 00:53:52 EDT
Hello James,

Customer is unable to remove the disk as the disk is in inconsistent state( as per attached). Let me know if you have any other suggestion so to rectify this issue.

Regards,
Neha Chugh
Comment 27 Neha Chugh 2017-07-24 00:57 EDT
Created attachment 1303436 [details]
inconsistent hard disk
Comment 28 James Wong 2017-07-24 08:55:10 EDT
Hi Neha,

The customer probably has to contact VMware support for fixing the vm.

If they are savvy enough, they may try to de-register the VM and then edit the vm file (in the datastore) to remove the disk.  But they should consult VMware support first.

James
Comment 30 James Wong 2017-07-31 14:07:12 EDT
Hi Neha,

Any update from the customer?  

What we are doing to the code is to inventory these disks as having 'unknown' controller type.

regards,
James
Comment 33 Neha Chugh 2017-08-01 09:09:33 EDT
Hello James,

We haven't received any response from customer till date whether he is able to remove the disk or not so we can lower down the severity.

In case if he is not able to delete the disk we can suggest the provided PR.

Thanks for all help.

Regards,
Neha Chugh
Comment 34 CFME Bot 2017-08-01 12:08:48 EDT
New commit detected on ManageIQ/manageiq-providers-vmware/master:
https://github.com/ManageIQ/manageiq-providers-vmware/commit/b5f1089beb71f98026f22e8352383ecf83a04593

commit b5f1089beb71f98026f22e8352383ecf83a04593
Author:     James Wong <jwong@jwong-OSX.local>
AuthorDate: Tue Jul 18 09:54:46 2017 -0400
Commit:     James Wong <jwong@jwong-OSX.local>
CommitDate: Tue Aug 1 11:47:55 2017 -0400

    dealing with disks which have no controller key
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1465761

 .../providers/vmware/infra_manager/refresh_parser.rb    | 17 +++++++++++------
 .../providers/vmware/infra_manager/refresher_spec.rb    | 15 +++++++++++++++
 .../vim_data/miq_vim_inventory/virtualMachinesByMor.yml | 10 +---------
 3 files changed, 27 insertions(+), 15 deletions(-)

Note You need to log in before you can comment on or make changes to this bug.