Bug 1305865

Summary: Scaling out an infra provider does not refresh
Product: Red Hat CloudForms Management Engine Reporter: Ronnie Rasouli <rrasouli>
Component: ProvidersAssignee: Ladislav Smola <lsmola>
Status: CLOSED NOTABUG QA Contact: Dave Johnson <dajohnso>
Severity: high Docs Contact:
Priority: high    
Version: 5.5.0CC: gblomqui, jfrey, jhardy, mcornea, obarenbo, rrasouli
Target Milestone: GA   
Target Release: 5.6.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: openstack
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-03-29 05:25:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Ronnie Rasouli 2016-02-09 12:32:54 UTC 
Description of problem:

Scalling out infrastructure provider.


The status on 'States of Root Orchestration Stacks' is	'CREATE_COMPLETE' 

While on the openstack director

overcloud  | UPDATE_IN_PROGRES

And eventually
overcloud  | UPDATE_COMPLETE

Did not help refresh provider or web refresh
Version-Release number of selected component (if applicable):


How reproducible:
100%

Steps to Reproduce:
1. Login to the Cloudforms UI	1. After a period of time, a previously free host is provisioned with the Compute role
2. Go to the Infrastructure in the primary navigation bar and choose Providers in the secondary navigation bar	
3. Click on the Openstack Infrastructure provider virtual thumbnail	
4. Click on "Scale this Infrastructure Provider" under "Configuration" in the tertiary menu bar	
5. Select the "Compute" role and increment the node count by one	
6. commit


Actual results:

The status on 'States of Root Orchestration Stacks' is	'CREATE_COMPLETE' 

Expected results:
overcloud  | UPDATE_IN_PROGRES

And eventually
overcloud  | UPDATE_COMPLETE


Additional info:
CFME version
5.5.2.4.20160127105142_395c086
Comment 2 Greg Blomquist 2016-02-11 15:19:26 UTC 
Ronnie, can you attach the associated logs to this BZ.  Looking for fog logs and evm logs.
Comment 3 Ladislav Smola 2016-03-01 14:51:04 UTC 
I can't verify this, please Ronnie paste logs

Also, manual refresh didn't throw any error? For event based refresh you have to have OpenStack AMQP configured correctly and correct credentials on ManageIQ side
Comment 4 Ronnie Rasouli 2016-03-24 13:43:26 UTC 
I was able to reproduce it - manual scale out did not received in infra provider summary, I guess this issue is related to alerts that are not seeing in timelines, although it received in cfme vmdb, running rails r event_monitor_example.rbsaw event:


saw event: #<OpenstackAmqpEvent:0x000000055c2380 @payload={"_context_roles"=>["admin"], "_context_request_id"=>"req-2da6930b-196e-4d52-9d79-6e1b83c889b9", "event_type"=>"orchestration.stack.update.start", "_context_auth_url"=>"http://192.0.2.1:5000/v3", "timestamp"=>"2016-03-24 13:42:06.931102", "_context_region_name"=>nil, "_unique_id"=>"5877591af1ca4b2cbf4243aebd08d1bc", "_context_tenant_id"=>"c61a45f2da224533b32e0f4626a6c030", "_context_tenant"=>"admin", "_context_user_id"=>"913a4dcf2d3f414a8be1ce6240fda885", "payload"=>{"state_reason"=>"Stack UPDATE started", "user_id"=>"admin", "stack_identity"=>"778a1582-e7e8-42a9-9c43-8c0e659b8cc3", "stack_name"=>"overcloud-BlockStorageAllNodesValidationDeployment-mqgt75xhjduf", "tenant_id"=>"c61a45f2da224533b32e0f4626a6c030", "create_at"=>"2016-03-24T11:43:10", "state"=>"UPDATE_IN_PROGRESS"}, "_context_username"=>"admin", "_context_auth_token"=>"6448693cc6f344bca691425580b776f6", "_context_password"=>"3a0b41086c5f378438a1b82f77e067c8e325af33", "_context_trustor_user_id"=>nil, "_context_user_identity"=>"913a4dcf2d3f414a8be1ce6240fda885 c61a45f2da224533b32e0f4626a6c030", "_context_aws_creds"=>nil, "_context_show_deleted"=>false, "_context_trust_id"=>nil, "priority"=>"INFO", "_context_is_admin"=>true, "_context_auth_token_info"=>{"token"=>{"methods"=>["password", "token"], "roles"=>[{"id"=>"75e97a69d7374cb4b14379c8367639b1", "name"=>"admin"}], "auth_token"=>"6448693cc6f344bca691425580b776f6", "expires_at"=>"2016-03-24T17:32:50.000000Z", "project"=>{"domain"=>{"id"=>"default", "name"=>"Default"}, "id"=>"c61a45f2da224533b32e0f4626a6c030", "name"=>"admin"}, "catalog"=>[{"endpoints"=>[{"url"=>"http://192.0.2.1:8774/v2/c61a45f2da224533b32e0f4626a6c030", "interface"=>"public", "region"=>"regionOne", "region_id"=>"regionOne", "id"=>"a2b5a76fdfec4eab8740797e4c5a49c7"}, {"url"=>"http://192.0.2.1:8774/v2/c61a45f2da224533b32e0f4626a6c030", "interface"=>"admin", "region"=>"regionOne", "region_id"=>"regionOne", "id"=>"b9f6d80855a34705ba49174732ce726a"}, {"url"=>"http://192.0.2.1:8774/v2/c61a45f2da224533b32e0f4626a6c030", "interface"=>"internal", "region"=>"regionOne", "region_id"=>"regionOne", "id"=>"e540d7c1ed374ae6ba48b6f7d325672c"}], "type"=>"compute", "id"=>"07525fbf269e4529b74b4f90824ffe8f", "name"=>"nova"}, {"endpoints"=>[{"url"=>"http://192.0.2.1:9292", "interface"=>"internal", "region"=>"regionOne", "region_id"=>"regionOne", "id"=>"12272b184de44367895cdf4039cf82e9"}, {"url"=>"http://192.0.2.1:9292", "interface"=>"admin", "region"=>"regionOne", "region_id"=>"regionOne", "id"=>"854490bc4d58427da494ef8888edcae5"}, {"url"=>"http://192.0.2.1:9292", "interface"=>"public", "region"=>"regionOne", "region_id"=>"regionOne", "id"=>"8f8c99e3d5e0489b97b7fa9c7473bdbd"}], "type"=>"image", "id"=>"203c000d674943e1bf3642fb4308de19", "name"=>"glance"}, {"endpoints"=>[{"url"=>"http://192.0.2.1:8774/v3", "interface"=>"public", "region"=>"regionOne", "region_id"=>"regionOne", "id"=>"5b955f02acf74ddea73b82a67a3bbcfd"}, {"url"=>"http://192.0.2.1:8774/v3", "interface"=>"admin", "region"=>"regionOne", "region_id"=>"regionOne", "id"=>"7749ed08545048d9951a1c8180473cd1"}, {"url"=>"http://192.0.2.1:8774/v3", "interface"=>"internal", "region"=>"regionOne", "region_id"=>"regionOne", "id"=>"a090d81f049340d3892693a7277f0ad5"}], "type"=>"computev3", "id"=>"328c00fc8546449b8e7285075224947b", "name"=>"novav3"}, {"endpoints"=>[{"url"=>"http://192.0.2.1:8777", "interface"=>"internal", "region"=>"regionOne", "region_id"=>"regionOne", "id"=>"3eff4ff244c74d969305f9eaa4ad9a00"}, {"url"=>"http://192.0.2.1:8777", "interface"=>"admin", "region"=>"regionOne", "region_id"=>"regionOne", "id"=>"5e630e0c25d04352b0f6b79477d3b847"}, {"url"=>"http://192.0.2.1:8777", "interface"=>"public", "region"=>"regionOne", "region_id"=>"regionOne", "id"=>"a8a7dbe9765546b1b58fb9898efb9a6d"}], "type"=>"metering", "id"=>"4a4c6828e57c4b3e874ac07f4d70af9b", "name"=>"ceilometer"}, {"endpoints"=>[{"url"=>"http://192.0.2.1:8080/v1/AUTH_c61a45f2da224533b32e0f4626a6c030", "interface"=>"public", "region"=>"regionOne", "region_id"=>"regionOne", "id"=>"0685547b40c748c3ae5cdb7549f0e11b"}, {"url"=>"http://192.0.2.1:8080/v1/AUTH_c61a45f2da224533b32e0f4626a6c030", "interface"=>"internal", "region"=>"regionOne", "region_id"=>"regionOne", "id"=>"5defc245c6574179943a738e2d286e29"}, {"url"=>"http://192.0.2.1:8080", "interface"=>"admin", "region"=>"regionOne", "region_id"=>"regionOne", "id"=>"fe446ef20626460597d41249487aad19"}], "type"=>"object-store", "id"=>"54b76d70247d41eeaf609254168a2bde", "name"=>"swift"}, {"endpoints"=>[{"url"=>"http://192.0.2.1:8042", "interface"=>"public", "region"=>"regionOne", "region_id"=>"regionOne", "id"=>"029a2e273bfd497c8fc8c8d21cb547be"}, {"url"=>"http://192.0.2.1:8042", "interface"=>"admin", "region"=>"regionOne", "region_id"=>"regionOne", "id"=>"08df2e6a84a14811b38217a0b4aeb6e8"}, {"url"=>"http://192.0.2.1:8042", "interface"=>"internal", "region"=>"regionOne", "region_id"=>"regionOne", "id"=>"793565574d324ee08a1966a32fd5150b"}], "type"=>"alarming", "id"=>"7e1652d1ad5741a48a979151cda7d423", "name"=>"aodh"}, {"endpoints"=>[{"url"=>"http://192.0.2.1:35357/v2.0", "interface"=>"admin", "region"=>"regionOne", "region_id"=>"regionOne", "id"=>"724768ab33b448478084287dbb898f32"}, {"url"=>"http://192.0.2.1:5000/v2.0", "interface"=>"public", "region"=>"regionOne", "region_id"=>"regionOne", "id"=>"836b4399110f452fa537a785ec6799cc"}, {"url"=>"http://192.0.2.1:5000/v2.0", "interface"=>"internal", "region"=>"regionOne", "region_id"=>"regionOne", "id"=>"ae78575809524bb4acddb559a4d33be3"}], "type"=>"identity", "id"=>"a374214ac4f84af0b534c1355e0e0ce9", "name"=>"keystone"}, {"endpoints"=>[{"url"=>"http://192.0.2.1:8004/v1/c61a45f2da224533b32e0f4626a6c030", "interface"=>"admin", "region"=>"regionOne", "region_id"=>"regionOne", "id"=>"61c454391a13472c80bbf8c68b63ef63"}, {"url"=>"http://192.0.2.1:8004/v1/c61a45f2da224533b32e0f4626a6c030", "interface"=>"public", "region"=>"regionOne", "region_id"=>"regionOne", "id"=>"628adc8c1a584270a6c2cb53b6f15407"}, {"url"=>"http://192.0.2.1:8004/v1/c61a45f2da224533b32e0f4626a6c030", "interface"=>"internal", "region"=>"regionOne", "region_id"=>"regionOne", "id"=>"d4281faea5854fe4878340c780dbcc88"}], "type"=>"orchestration", "id"=>"a50c345e159b4d1ab53d53782fb9218b", "name"=>"heat"}, {"endpoints"=>[{"url"=>"http://192.0.2.1:9696", "interface"=>"internal", "region"=>"regionOne", "region_id"=>"regionOne", "id"=>"405ed202c8ba4982be53f971d6c3d4dc"}, {"url"=>"http://192.0.2.1:9696", "interface"=>"admin", "region"=>"regionOne", "region_id"=>"regionOne", "id"=>"88e0f994814c4ab08a478c969c2446e3"}, {"url"=>"http://192.0.2.1:9696", "interface"=>"public", "region"=>"regionOne", "region_id"=>"regionOne", "id"=>"f8c60d8cb8004af0ae6f47142fb5a983"}], "type"=>"network", "id"=>"a676b3dda6474f049c565f9d04fa097d", "name"=>"neutron"}, {"endpoints"=>[{"url"=>"http://192.0.2.1:6385", "interface"=>"admin", "region"=>"regionOne", "region_id"=>"regionOne", "id"=>"7bd97e0ba8f34f1e976936c12f4d13a5"}, {"url"=>"http://192.0.2.1:6385", "interface"=>"internal", "region"=>"regionOne", "region_id"=>"regionOne", "id"=>"a1ea26214e574b719da67d7e7853a734"}, {"url"=>"http://192.0.2.1:6385", "interface"=>"public", "region"=>"regionOne", "region_id"=>"regionOne", "id"=>"fb2a0c72a56944aebe4d62305a8f3962"}], "type"=>"baremetal", "id"=>"bce15df85741447082ace01d3f2bd657", "name"=>"ironic"}], "extras"=>{}, "user"=>{"domain"=>{"id"=>"default", "name"=>"Default"}, "id"=>"913a4dcf2d3f414a8be1ce6240fda885", "name"=>"admin"}, "version"=>"v3", "audit_ids"=>["9BJhWgmLQZyo7_OGbiZRFA"], "issued_at"=>"2016-03-24T13:32:50.616880"}}, "_context_user"=>"admin", "publisher_id"=>"orchestration.instack.localdomain", "message_id"=>"b731680d-a34c-44f9-bc7b-a6cf15d59b76"}, @metadata={:user_id=>nil, :priority=>nil, :content_type=>nil}>
Comment 5 Ronnie Rasouli 2016-03-29 05:25:11 UTC 
Restarting the evm resolved the issue, the alerts arrived afterwards