Bug 1538741

Summary: completely broken Instances view and different errors in logs after rhos instance refresh
Product: Red Hat CloudForms Management Engine Reporter: Ievgen Zapolskyi <izapolsk>
Component: ProvidersAssignee: Marek Aufart <maufart>
Status: CLOSED CURRENTRELEASE QA Contact: Ido Ovadia <iovadia>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 5.9.0CC: bsorota, cpelland, gblomqui, iovadia, jfrey, jhardy, lsmola, obarenbo, pakotvan
Target Milestone: GAKeywords: Regression, TestOnly
Target Release: 5.10.0Flags: izapolsk: automate_bug-
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 5.10.0.0 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1539874 (view as bug list) Environment:
Last Closed: 2019-02-11 13:59:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: Bug
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: Openstack Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1539874    
Attachments:
Description Flags
backtrace 1
none
backtrace 2
none
screenshot
none
appliance logs none

Description Ievgen Zapolskyi 2018-01-25 17:23:56 UTC 
Created attachment 1386238 [details]
backtrace 1

Description of problem:
When user kicks off refresh relationships for rhos instance from Instances or Instance Details page, often logs contain
[----] E, [2018-01-25T11:50:03.545848 #14037:c7f13c] ERROR -- : [EmsRefresh::Refreshers::EmsRefresherMixin::PartialRefreshError]: undefined method `tenants' for #<OpenstackHandle::IdentityDelegate:0x00000019412670>  Method:[block in method_missing]

or 

[----] F, [2018-01-25T11:00:11.802084 #13251:12136d4] FATAL -- : Error caught: [NoMethodError] undefined method `cloud_volumes' for nil:NilClass
/opt/rh/cfme-gemset/bundler/gems/manageiq-ui-classic-5486ed75d0d7/app/helpers/application_helper/button/instance_attach.rb:3:in `disabled?'

When second error happens Instances view gets broken and doesn't allow to open Instance Details page at all. Relogin or changing browser doesn't help.
Search and Paginator disappear from page

Version-Release number of selected component (if applicable):
5.9.0.17

How reproducible:
80%. I cannot reproduce it manually but there is automated test which allows to easily reproduce it.

Steps to Reproduce:
1. add rhos11 provider to appliance and wait until cfme retrieves all provider data 
2. Add instance to provider
3. Go to Instances view find instance and try to refresh its relationships
4. Click that instance and try to refresh relationships several times
5. Click Back button and choose go to All Instances page
6. Try to get to Details page again
7. check production and evm logs.

Actual results:
either first or second error happens. When second error happens, Instances view gets broken

Additional info:
screwed up appliances:
https://10.8.197.127/vm_cloud/explorer
https://10.8.196.192/vm_cloud/explorer
steps: go to instances view and try to clik any instance
Comment 2 Ievgen Zapolskyi 2018-01-25 17:25:15 UTC 
Created attachment 1386239 [details]
backtrace 2
Comment 3 Ievgen Zapolskyi 2018-01-25 17:25:58 UTC 
Created attachment 1386240 [details]
screenshot
Comment 4 Ievgen Zapolskyi 2018-01-25 17:37:30 UTC 
Created attachment 1386242 [details]
appliance logs
Comment 5 Ievgen Zapolskyi 2018-01-26 10:13:18 UTC 
psav figured out steps which allow to 100% reproduce the issue.

1. add rhos11 provider to appliance and wait until cfme retrieves all provider data 
2. Add instance to provider
3. Go to Instances view find instance and try to refresh its relationships
4. Wait for some time ~ 5 - 10 seconds until request is queued
5. try to click that instance in order to open Instance Details page
Comment 8 CFME Bot 2018-01-29 14:42:38 UTC 
https://github.com/ManageIQ/manageiq-providers-openstack/pull/213
Comment 9 CFME Bot 2018-01-29 18:22:23 UTC 
New commit detected on ManageIQ/manageiq-providers-openstack/master:
https://github.com/ManageIQ/manageiq-providers-openstack/commit/34de6c3b00023c1a725ffcf34a6dd87e84f45a91

commit 34de6c3b00023c1a725ffcf34a6dd87e84f45a91
Author:     Marek Aufart <maufart>
AuthorDate: Mon Jan 29 15:25:47 2018 +0100
Commit:     Marek Aufart <maufart>
CommitDate: Mon Jan 29 18:51:09 2018 +0100

    Fix CloudTenant Vm targeted refresh
    
    Vm tagreted refresh queued related Cloud Tenant refresh, that could fail due to
    keystone v3 backend in fog or an issue in fog/openstack method get_project_by_id.
    
    Adding custom method to get_project derived from list_project.
    
    Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1538741

 .../providers/openstack/inventory/collector/target_collection.rb    | 4 ++--
 .../openstack/legacy/openstack_handle/identity_delegate.rb          | 6 ++++++
 2 files changed, 8 insertions(+), 2 deletions(-)
Comment 11 Marek Aufart 2018-01-30 08:44:15 UTC 
*** Bug 1539590 has been marked as a duplicate of this bug. ***
Comment 12 Ido Ovadia 2018-06-17 21:04:48 UTC 
Verified
========
5.10.0.0