Description of problem: After a successful v2v migration of a single VM from VMware to RHV-4.3.1, It is not possible to view the Infrastructure mapping plans page. Not from Compute->Migrate->Infra mapping, neither from Migration plans page, by clicking the Infrastructure mapping plan, that was used by the migration plan (that was run successfully). "Error loading mappings. There was an error loading Infrastructure Mappings. Please refresh and try again." (See "error_loading_mapping.png") Refreshing the page ("F5"), opens a window with this error: " Server Error (API) URL https://<CFME_FQDN>/api/cloud_tenants?expand=resources&attributes=ext_management_system.name%2Ccloud_networks%2Ccloud_volume_types Status 500 Internal Server Error Content-Type application/json; charset=utf-8 Data {"error":{"kind":"internal_server_error","message":"undefined method `cloud_volume_types' for #<ManageIQ::Providers::Redhat::InfraManager:0x000000000f628d10>\nDid you mean? cloud_subnets","klass":"NoMethodError"}} " (See "error_loading_mapping_content.png") Version-Release number of selected component (if applicable): CFME-5.10.1.1.20190212171432_83eb777 RHV-4.3.1.1-0.1.el7
Created attachment 1538369 [details] error_loading_mapping.png
Created attachment 1538371 [details] error_loading_mapping_content.png
That API request (/api/cloud_tenants?expand=resources&attributes=ext_management_system.name%2Ccloud_networks%2Ccloud_volume_types) should never fail. I suspect this is a backend problem. The error message is particularly confusing to me because of the type "ManageIQ::Providers::Redhat::InfraManager", which is a RHV provider, and this is an OpenStack-related API request. I'm not sure why a request to /api/cloud_tenants would be looking for methods on that class, the type there should instead be "ManageIQ::Providers::Openstack::CloudManager::CloudTenant". Adding Fabien: maybe someone on your end knows what's going on here?
Ilanit, would it be possible that the mapping was created through the API with a source provider type being 'openstack' and a source provider being a RHV provider ? This could have happened if the object ids are wrong in the JSON sent to the API.
No, this is not related to adding mapping via API. It occurs even before adding mapping, when just trying to access the Infra mapping page. This is probably related to: Bug 1668720 - [RHV] CFME fail to refresh\discover RHV-4.3 When removing the RHV-4.3 provider, it is possible to navigate to the Infra mapping, with no error. Need to track if this bug is resolved, once Bug 1668720 is resolved.
This bug is a side effect of Bug 1668720. When applying this fix for Bug 1668720: https://bugzilla.redhat.com/show_bug.cgi?id=1668720#c8 This bug no longer exist. *** This bug has been marked as a duplicate of bug 1668720 ***
Please ignore comment #7. This bug is still there, even after applying this fix for Bug 1668720: https://bugzilla.redhat.com/show_bug.cgi?id=1668720#c8 It seems this bug is not related to the RHV-4.3 refresh issue, since the refresh issue is solved by this fix, but still we get same exception, when trying to access the Infra mapping page, even if there is no Infra mapping plans defined.
My understanding of what's happening. The error message in the UI gives us the API call made by UI. So I replayed it with cURL to confirm it: # curl -sk -u admin:smartvm 'https://localhost/api/cloud_tenants?expand=resources&attributes=ext_management_system.name,cloud_networks,cloud_volume_types' | python -m json.tool { "error": { "kind": "internal_server_error", "klass": "NoMethodError", "message": "undefined method `cloud_volume_types' for #<ManageIQ::Providers::Redhat::InfraManager:0x0000000015c3b738>\nDid you mean? cloud_subnets" } } So, the API call lists all cloud_tenants and asks to expose the associated ext_management_system (aka provider), cloud_networks and cloud_volume_types. The problem is that the cloud_volume_types method doesn't exist for the RHV provider. This sounds logical, but let's check what we have in the VMDB: irb> CloudTenant.all.length => 1 irb> CloudTenant.last.type => "ManageIQ::Providers::Openstack::CloudManager::CloudTenant" We can see that a CloudTenant object exist in the VMDB, of type "ManageIQ::Providers::Openstack::CloudManager::CloudTenant", even though no cloud provider is configured. irb> CloudTenant.last.ext_management_system => "rhv43" irb> CloudTenant.last.ext_management_system => "ManageIQ::Providers::Redhat::InfraManager", So, the API correctly identifies the CloudTenant object, but fails when trying to collect the associated cloud_volume_types. And if we look at the oVirt provider (/opt/rh/cfme-gemset/gems/manageiq-providers-ovirt-0.1.0), we can verify that the InfraManager class has relationships to ExtManagementSystem, CloudTenants and CloudNetwork, and that CloudNetwork class is inherit from the OpenStack provider. But it doesn't have a relationship to CloudVolumeType. So the error is normal. Looking at the manageiq-providers-ovirt code history, it seems that these relationships have been implemented 10 months ago for CFME 5.10, to support OVN networks in RHV, so it's not recent change in CFME. However, we may think that the RHV inventory didn't trigger the CloudTenant creation with RHV 4.2, while it now does with RHV 4.3. So, the data collected by CloudForms probably have changed.
Fabien can you paste the backtrace from the CFME appliance for this error? It is possible the API is trying to return cloud_volume_types without checking if they are supported.
Here is the request trace from api.log: [----] I, [2019-03-21T02:41:27.990492 #15171:13890f4] INFO -- : MIQ(Api::CloudTenantsController.log_request_initiated) [----] I, [2019-03-21T02:41:27.990742 #15171:13890f4] INFO -- : MIQ(Api::CloudTenantsController.log_request) API Request: {:requested_at=>"2019-03-21 06:41:27 UTC", :method=>"GET", :url=>"https://10.8.198.192/api/cloud_tenants?expand=resources&attributes=ext_ma nagement_system.name%2Ccloud_networks%2Ccloud_volume_types"} [----] I, [2019-03-21T02:41:27.995167 #15171:13890f4] INFO -- : MIQ(Api::CloudTenantsController.log_request) Authentication: {:type=>"token", :token=>"ce5f786bc6226ad0a170207fedff8b6c", :x_miq_group=>nil, :user=>"admin"} [----] I, [2019-03-21T02:41:27.998458 #15171:13890f4] INFO -- : MIQ(Api::CloudTenantsController.log_request) Authorization: {:user=>"admin", :group=>"EvmGroup-super_administrator", :role=>"EvmRole-super_administrator", :tenant=>"My Company"} [----] I, [2019-03-21T02:41:27.998875 #15171:13890f4] INFO -- : MIQ(Api::CloudTenantsController.log_request) Request: {:method=>:get, :action=>"read", :fullpath=>"/api/cloud_tenants?expand=resources&attributes=ext_management_system.name%2Ccloud_networks%2Cc loud_volume_types", :url=>"https://10.8.198.192/api/cloud_tenants?expand=resources&attributes=ext_management_system.name%2Ccloud_networks%2Ccloud_volume_types", :base=>"https://10.8.198.192", :path=>"/api/cloud_tenants", :prefix=>"/api", :version=>"4.0.0", :api_pre fix=>"https://10.8.198.192/api", :collection=>"cloud_tenants", :c_suffix=>nil, :collection_id=>nil, :subcollection=>nil, :subcollection_id=>nil} [----] I, [2019-03-21T02:41:27.999182 #15171:13890f4] INFO -- : MIQ(Api::CloudTenantsController.log_request) Parameters: {"expand"=>"resources", "attributes"=>"ext_management_system.name,cloud_networks,cloud_volume_types", "action"=>"index", "controller"=>"api /cloud_tenants", "format"=>"json", "body"=>{}} [----] E, [2019-03-21T02:41:28.136698 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) API Error [----] E, [2019-03-21T02:41:28.136785 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) NoMethodError: undefined method `cloud_volume_types' for #<ManageIQ::Providers::Redhat::InfraManager:0x0000000009fa9b88> [----] E, [2019-03-21T02:41:28.136840 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) Did you mean? cloud_subnets [----] E, [2019-03-21T02:41:28.137098 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) API Error [----] E, [2019-03-21T02:41:28.137226 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) [----] E, [2019-03-21T02:41:28.137249 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) [----] E, [2019-03-21T02:41:28.137266 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/activemodel-5.0.7.1/lib/active_model/attribute_methods.rb:433:in `method_missing' [----] E, [2019-03-21T02:41:28.137298 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /var/www/miq/vmdb/lib/extensions/ar_virtual.rb:93:in `cloud_volume_types' [----] E, [2019-03-21T02:41:28.137314 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/bundler/gems/cfme-api-93195015aedc/app/controllers/api/base_controller/renderer.rb:197:in `public_send' [----] E, [2019-03-21T02:41:28.137330 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/bundler/gems/cfme-api-93195015aedc/app/controllers/api/base_controller/renderer.rb:197:in `virtual_attribute_search' [----] E, [2019-03-21T02:41:28.137366 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/bundler/gems/cfme-api-93195015aedc/app/controllers/api/base_controller/renderer.rb:266:in `fetch_direct_virtual_attribute' [----] E, [2019-03-21T02:41:28.137401 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/bundler/gems/cfme-api-93195015aedc/app/controllers/api/base_controller/renderer.rb:254:in `block in expand_virtual_attributes' [----] E, [2019-03-21T02:41:28.137417 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/bundler/gems/cfme-api-93195015aedc/app/controllers/api/base_controller/renderer.rb:250:in `each' [----] E, [2019-03-21T02:41:28.137433 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/bundler/gems/cfme-api-93195015aedc/app/controllers/api/base_controller/renderer.rb:250:in `expand_virtual_attributes' [----] E, [2019-03-21T02:41:28.137459 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/bundler/gems/cfme-api-93195015aedc/app/controllers/api/base_controller/renderer.rb:79:in `resource_to_jbuilder' [----] E, [2019-03-21T02:41:28.137501 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/bundler/gems/cfme-api-93195015aedc/app/controllers/api/base_controller/renderer.rb:49:in `block (2 levels) in collection_to_jbuilder' [----] E, [2019-03-21T02:41:28.137519 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/jbuilder-2.8.0/lib/jbuilder.rb:298:in `block (2 levels) in _map_collection' [----] E, [2019-03-21T02:41:28.137536 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/jbuilder-2.8.0/lib/jbuilder.rb:305:in `_scope' [----] E, [2019-03-21T02:41:28.137551 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/jbuilder-2.8.0/lib/jbuilder.rb:298:in `block in _map_collection' [----] E, [2019-03-21T02:41:28.137567 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/jbuilder-2.8.0/lib/jbuilder.rb:297:in `collect' [----] E, [2019-03-21T02:41:28.137583 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/jbuilder-2.8.0/lib/jbuilder.rb:297:in `each' [----] E, [2019-03-21T02:41:28.137614 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/jbuilder-2.8.0/lib/jbuilder.rb:297:in `map' [----] E, [2019-03-21T02:41:28.137630 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/jbuilder-2.8.0/lib/jbuilder.rb:297:in `_map_collection' [----] E, [2019-03-21T02:41:28.137671 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/jbuilder-2.8.0/lib/jbuilder.rb:188:in `array!' [----] E, [2019-03-21T02:41:28.137696 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/jbuilder-2.8.0/lib/jbuilder.rb:34:in `block in set!' [----] E, [2019-03-21T02:41:28.137727 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/jbuilder-2.8.0/lib/jbuilder.rb:305:in `_scope' [----] E, [2019-03-21T02:41:28.137750 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/jbuilder-2.8.0/lib/jbuilder.rb:34:in `set!' [----] E, [2019-03-21T02:41:28.137774 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/jbuilder-2.8.0/lib/jbuilder.rb:66:in `method_missing' [----] E, [2019-03-21T02:41:28.137791 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/bundler/gems/cfme-api-93195015aedc/app/controllers/api/base_controller/renderer.rb:47:in `block in collection_to_jbuilder' [----] E, [2019-03-21T02:41:28.137810 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/jbuilder-2.8.0/lib/jbuilder.rb:18:in `initialize' [----] E, [2019-03-21T02:41:28.137833 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/bundler/gems/cfme-api-93195015aedc/app/controllers/api/base_controller/renderer.rb:34:in `new' [----] E, [2019-03-21T02:41:28.137857 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/bundler/gems/cfme-api-93195015aedc/app/controllers/api/base_controller/renderer.rb:34:in `collection_to_jbuilder' [----] E, [2019-03-21T02:41:28.137882 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/bundler/gems/cfme-api-93195015aedc/app/controllers/api/base_controller/renderer.rb:10:in `render_collection' [----] E, [2019-03-21T02:41:28.137918 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/bundler/gems/cfme-api-93195015aedc/app/controllers/api/base_controller.rb:56:in `index' [----] E, [2019-03-21T02:41:28.137946 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/actionpack-5.0.7.1/lib/action_controller/metal/basic_implicit_render.rb:4:in `send_action' [----] E, [2019-03-21T02:41:28.137966 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/actionpack-5.0.7.1/lib/abstract_controller/base.rb:188:in `process_action' [----] E, [2019-03-21T02:41:28.137988 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/actionpack-5.0.7.1/lib/action_controller/metal/rendering.rb:30:in `process_action' [----] E, [2019-03-21T02:41:28.138010 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/actionpack-5.0.7.1/lib/abstract_controller/callbacks.rb:20:in `block in process_action' [----] E, [2019-03-21T02:41:28.138030 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:126:in `call' [----] E, [2019-03-21T02:41:28.138055 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:506:in `block (2 levels) in compile' [----] E, [2019-03-21T02:41:28.138076 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:455:in `call' [----] E, [2019-03-21T02:41:28.138119 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:101:in `__run_callbacks__' [----] E, [2019-03-21T02:41:28.138142 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:750:in `_run_process_action_callbacks' [----] E, [2019-03-21T02:41:28.138164 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:90:in `run_callbacks' [----] E, [2019-03-21T02:41:28.138179 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/actionpack-5.0.7.1/lib/abstract_controller/callbacks.rb:19:in `process_action' [----] E, [2019-03-21T02:41:28.138201 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/actionpack-5.0.7.1/lib/action_controller/metal/rescue.rb:20:in `process_action' [----] E, [2019-03-21T02:41:28.138221 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/actionpack-5.0.7.1/lib/action_controller/metal/instrumentation.rb:32:in `block in process_action' [----] E, [2019-03-21T02:41:28.138245 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/activesupport-5.0.7.1/lib/active_support/notifications.rb:164:in `block in instrument' [----] E, [2019-03-21T02:41:28.138260 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/activesupport-5.0.7.1/lib/active_support/notifications/instrumenter.rb:21:in `instrument' [----] E, [2019-03-21T02:41:28.138286 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/activesupport-5.0.7.1/lib/active_support/notifications.rb:164:in `instrument' [----] E, [2019-03-21T02:41:28.138301 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/actionpack-5.0.7.1/lib/action_controller/metal/instrumentation.rb:30:in `process_action' [----] E, [2019-03-21T02:41:28.138322 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/actionpack-5.0.7.1/lib/action_controller/metal/params_wrapper.rb:248:in `process_action' [----] E, [2019-03-21T02:41:28.138337 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/activerecord-5.0.7.1/lib/active_record/railties/controller_runtime.rb:18:in `process_action' [----] E, [2019-03-21T02:41:28.138360 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/actionpack-5.0.7.1/lib/abstract_controller/base.rb:126:in `process' [----] E, [2019-03-21T02:41:28.138382 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/actionpack-5.0.7.1/lib/action_controller/metal.rb:190:in `dispatch' [----] E, [2019-03-21T02:41:28.138404 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/actionpack-5.0.7.1/lib/action_controller/metal.rb:262:in `dispatch' [----] E, [2019-03-21T02:41:28.138423 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/actionpack-5.0.7.1/lib/action_dispatch/routing/route_set.rb:50:in `dispatch' [----] E, [2019-03-21T02:41:28.138441 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/actionpack-5.0.7.1/lib/action_dispatch/routing/route_set.rb:32:in `serve' [----] E, [2019-03-21T02:41:28.138459 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/actionpack-5.0.7.1/lib/action_dispatch/journey/router.rb:39:in `block in serve' [----] E, [2019-03-21T02:41:28.138478 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/actionpack-5.0.7.1/lib/action_dispatch/journey/router.rb:26:in `each' [----] E, [2019-03-21T02:41:28.138501 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/actionpack-5.0.7.1/lib/action_dispatch/journey/router.rb:26:in `serve' [----] E, [2019-03-21T02:41:28.138519 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/actionpack-5.0.7.1/lib/action_dispatch/routing/route_set.rb:727:in `call' [----] E, [2019-03-21T02:41:28.138541 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/bundler/gems/cfme-graphql-b854537b42a4/lib/manageiq/graphql/rest_api_proxy.rb:18:in `call' [----] E, [2019-03-21T02:41:28.138562 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/secure_headers-3.0.3/lib/secure_headers/middleware.rb:10:in `call' [----] E, [2019-03-21T02:41:28.138581 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /var/www/miq/vmdb/lib/request_started_on_middleware.rb:12:in `call' [----] E, [2019-03-21T02:41:28.138601 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/rack-2.0.6/lib/rack/etag.rb:25:in `call' [----] E, [2019-03-21T02:41:28.138621 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/rack-2.0.6/lib/rack/conditional_get.rb:25:in `call' [----] E, [2019-03-21T02:41:28.138652 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/rack-2.0.6/lib/rack/head.rb:12:in `call' [----] E, [2019-03-21T02:41:28.138672 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:232:in `context' [----] E, [2019-03-21T02:41:28.138694 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:226:in `call' [----] E, [2019-03-21T02:41:28.138710 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/cookies.rb:613:in `call' [----] E, [2019-03-21T02:41:28.138731 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/callbacks.rb:38:in `block in call' [----] E, [2019-03-21T02:41:28.138755 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:97:in `__run_callbacks__' [----] E, [2019-03-21T02:41:28.138769 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:750:in `_run_call_callbacks' [----] E, [2019-03-21T02:41:28.138797 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:90:in `run_callbacks' [----] E, [2019-03-21T02:41:28.138818 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/callbacks.rb:36:in `call' [----] E, [2019-03-21T02:41:28.138837 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/remote_ip.rb:79:in `call' [----] E, [2019-03-21T02:41:28.138858 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/debug_exceptions.rb:49:in `call' [----] E, [2019-03-21T02:41:28.138878 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/show_exceptions.rb:31:in `call' [----] E, [2019-03-21T02:41:28.138899 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/railties-5.0.7.1/lib/rails/rack/logger.rb:36:in `call_app' [----] E, [2019-03-21T02:41:28.138919 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/railties-5.0.7.1/lib/rails/rack/logger.rb:26:in `call' [----] E, [2019-03-21T02:41:28.138940 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/request_id.rb:24:in `call' [----] E, [2019-03-21T02:41:28.138956 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/rack-2.0.6/lib/rack/method_override.rb:22:in `call' [----] E, [2019-03-21T02:41:28.138977 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/rack-2.0.6/lib/rack/runtime.rb:22:in `call' [----] E, [2019-03-21T02:41:28.138997 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/activesupport-5.0.7.1/lib/active_support/cache/strategy/local_cache_middleware.rb:28:in `call' [----] E, [2019-03-21T02:41:28.139017 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/executor.rb:12:in `call' [----] E, [2019-03-21T02:41:28.139034 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/rack-2.0.6/lib/rack/sendfile.rb:111:in `call' [----] E, [2019-03-21T02:41:28.139051 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /opt/rh/cfme-gemset/gems/railties-5.0.7.1/lib/rails/engine.rb:522:in `call' [----] E, [2019-03-21T02:41:28.139074 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /usr/share/gems/gems/puma-3.7.1/lib/puma/configuration.rb:232:in `call' [----] E, [2019-03-21T02:41:28.139094 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /usr/share/gems/gems/puma-3.7.1/lib/puma/server.rb:578:in `handle_request' [----] E, [2019-03-21T02:41:28.139118 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /usr/share/gems/gems/puma-3.7.1/lib/puma/server.rb:415:in `process_client' [----] E, [2019-03-21T02:41:28.139143 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /usr/share/gems/gems/puma-3.7.1/lib/puma/server.rb:275:in `block in run' [----] E, [2019-03-21T02:41:28.139159 #15171:13890f4] ERROR -- : MIQ(Api::CloudTenantsController.api_error) /usr/share/gems/gems/puma-3.7.1/lib/puma/thread_pool.rb:120:in `block in spawn_thread' [----] I, [2019-03-21T02:41:28.141847 #15171:13890f4] INFO -- : MIQ(Api::CloudTenantsController.log_request) Response: {:completed_at=>"2019-03-21 06:41:28 UTC", :size=>"0.224 KBytes", :time_taken=>"0.151 Seconds", :status=>500} I've quickly looked at /opt/rh/cfme-gemset/gems/manageiq-api-4.0.0/app/controllers/api/providers_controller.rb and it doesn't declare a subcollection for cloud_volume_types. It might not be related, but I'm wondering if it's normal ?
Okay so you're looking for a list of all cloud tenants and you're explicitly asking for cloud_volume_types, but since RHV creates a cloud tenant it is getting picked up. This doesn't look new at all, you said this only started happening recently? The quick fix would be to just add cloud_volume_types to the base ems or an infra manager but if we follow that path we'd have to essentially add all relationships from a cloud_manager to the base ems.
You're right the code is not new (Summer 2018), but it didn't create and cloud_tenant when connected to RHV 4.2. So, somehow, RHV 4.3 now has this tenant object and it is reflected in the VMDB. About the solution, we could also have oVirt provider implement the cloud_volume_types relationships, and always returns a empty array, as there no such object in oVirt. @mperina, do you know why RHV 4.3 now provides cloud tenants ?
(In reply to Fabien Dupont from comment #13) > You're right the code is not new (Summer 2018), but it didn't create and > cloud_tenant when connected to RHV 4.2. So, somehow, RHV 4.3 now has this > tenant object and it is reflected in the VMDB. > About the solution, we could also have oVirt provider implement the > cloud_volume_types relationships, and always returns a empty array, as there > no such object in oVirt. > > @mperina, do you know why RHV 4.3 now provides cloud tenants ? As this is network part, Dominik do you know any details about this change between RHV 4.2 and 4.3?
(In reply to Martin Perina from comment #14) > (In reply to Fabien Dupont from comment #13) > > You're right the code is not new (Summer 2018), but it didn't create and > > cloud_tenant when connected to RHV 4.2. So, somehow, RHV 4.3 now has this > > tenant object and it is reflected in the VMDB. > > About the solution, we could also have oVirt provider implement the > > cloud_volume_types relationships, and always returns a empty array, as there > > no such object in oVirt. > > > > @mperina, do you know why RHV 4.3 now provides cloud tenants ? > > As this is network part, Dominik do you know any details about this change > between RHV 4.2 and 4.3? Sorry, I have no knowledge about the CloudForms internals, but I know about the RHV APIs. If the question is if the OpenStack Identity API implementation of RHV is changed recently, the answer is No. During the last 15 month, there is only a single change which should not modify behavior: https://github.com/oVirt/ovirt-provider-ovn/commits/633b5ca1380b0f5448e3a4368f0813f95a644be7/provider/handlers/keystone_responses.py The only visible recent change on OpenStack Identity API is the implementation of security groups. Are there any hints, which information on which API, or even on which call, might trigger the bug?
Dominik, it seems that RHV now has a tenant / project to which the cloud networks belong. With RHV 4.2, the tenant is not created in CloudForms inventory, so it's probably not in RHV. With RHV 4.3, the tenant is there.
(In reply to Fabien Dupont from comment #16) > Dominik, it seems that RHV now has a tenant / project to which the cloud > networks belong. Sounds like the relevant API is OpenStack Networking API. There are recent changes in OpenStack Networking API implementation: https://github.com/oVirt/ovirt-provider-ovn/commits/master/provider/neutron/neutron_api_mappers.py > With RHV 4.2, the tenant is not created in CloudForms inventory, so it's > probably not in RHV. With RHV 4.3, the tenant is there. The changes might be already available since the last releases of RHV-4.2. Is there anything wrong having these changes? Does RHV return anything invalid? Last reported issue in the communication of ManageIQ and RHV is https://github.com/ManageIQ/manageiq/issues/18457 is this relevant here?
It's not necessarily wrong. I'm just trying to identify what has changed and can cause CloudForms to generate objects it didn't generate previously. This causes some errors, but before changing CloudForms, we need to understand if it's normal behaviour of RHV and will remain. If so, CloudForms has to adjust its internals.
RHV might complete its implementation of the OpenStack Networking API stepwise. Looks like we should improve the time to detect unexpected changes.
Fabien, what are the exact API calls to RHV that trigger the bug ? The security groups (and rules) API was added, which features an hard-coded tenant object (to comply w/ the API). The tenant object is also featured in the network / subnet / port / router entities (literally, any object we implement from the networking API). Our security group implementation also returns the 'project', and seems to be the only entity in our model that does so.
Please also update the bug with the ovirt-provider-ovn log (/var/log/ovirt-provider-ovn.log) and ovn database contents (/var/lib/openvswitch/*.db) to the bug, so I can take a look. Those can be found in the rhv manager instance.
I've done some extra tests to see if the UI could refine its API call. The following call works: /api/cloud_tenants?expand=resources&attributes=ext_management_system.name,cloud_networks,cloud_volume_types&filter[]=ext_management_system.type=ManageIQ::Providers::Openstack::CloudManager @mturley, do you think you can change the request ? What would be the impact on the UI ?
Fabien, we could change the request in the UI to include that filter, I would just need to make sure we apply that filter to every request we make to /api/cloud_tenants (which we do on the mappings page, in the mapping wizard, and in the plan wizard). Some of those are simple GET request, and some of them are POST query actions, so it will take a little effort to make sure we get that filter everywhere we need it. It shouldn't be too hard to do that, but in order to test that fix thoroughly I would like to be able to reproduce the issue in my dev environment. Is there a CloudForms appliance I can access that is presenting this issue? I would like to grab a copy of its vmdb_production database to test against.
https://github.com/ManageIQ/manageiq-v2v/pull/920
New commit detected on ManageIQ/manageiq-v2v/master: https://github.com/ManageIQ/manageiq-v2v/commit/0f6abc0be8e22eea95715f9f9825dac6aec3d0a1 commit 0f6abc0be8e22eea95715f9f9825dac6aec3d0a1 Author: Mike Turley <mturley> AuthorDate: Tue Mar 26 15:24:29 2019 -0400 Commit: Mike Turley <mturley> CommitDate: Tue Mar 26 15:24:29 2019 -0400 Add Openstack filter on /api/cloud_tenants request in Mappings and Plans pages and in Mapping wizard. https://bugzilla.redhat.com/show_bug.cgi?id=1680521 app/javascript/react/screens/App/Mappings/MappingsConstants.js | 3 +- app/javascript/react/screens/App/Overview/__test__/__snapshots__/index.test.js.snap | 2 +- app/javascript/redux/common/targetResources/targetResourcesConstants.js | 3 +- 3 files changed, 5 insertions(+), 3 deletions(-)
I've tested it on a development appliance and the Migration > Infrastructure Mappings page is correctly displayed with a RHV 4.3 provider configured. I checked and the RHV 4.3 provider has a cloud_tenant. Moving the BZ to POST.
I've tested it on a CFME 5.10.2.2 (followed [1] to recompile the assets) and I can confirm it works with a RHV 4.3 provider. [1] https://github.com/ManageIQ/ui-components/wiki/Updating-assets-on-an-appliance
Fabien, Satoe: The PR to master that fixes this bug (https://github.com/ManageIQ/manageiq-v2v/pull/920) will not sufficiently fix it if backported to hammer. We instead need to merge https://github.com/ManageIQ/manageiq-v2v/pull/921. To verify that this is truly fixed, in addition to the mapping page not being broken, we need to verify that when using the mapping wizard to create a RHV mapping, openstack tenants do not appear in the list of target RHV clusters.
Disregard my last comment; because the conflicting changes were backported first, https://github.com/ManageIQ/manageiq-v2v/pull/920 can now be backported to fix this BZ and https://github.com/ManageIQ/manageiq-v2v/pull/921 is no longer necessary.
Fixed in 5.11.0.4.20190514210444_0c91ee1