Bug 1708359 - [V2V][OSP] End to end migration not able to proceed with false "no conversion host was configured" error
Summary: [V2V][OSP] End to end migration not able to proceed with false "no conversion...
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: V2V
Version: 5.10.4
Hardware: Unspecified
OS: Unspecified
Target Milestone: GA
: 5.11.0
Assignee: Fabien Dupont
QA Contact: Yadnyawalk Tale
Red Hat CloudForms Documentation
Whiteboard: v2v
Depends On:
Blocks: 1711036
TreeView+ depends on / blocked
Reported: 2019-05-09 17:11 UTC by Yadnyawalk Tale
Modified: 2019-12-13 15:09 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1711036 (view as bug list)
Last Closed: 2019-12-13 15:09:52 UTC
Category: Bug
Cloudforms Team: V2V
Target Upstream Version:

Attachments (Terms of Use)
issue.png (57.89 KB, image/png)
2019-05-09 17:11 UTC, Yadnyawalk Tale
no flags Details

Description Yadnyawalk Tale 2019-05-09 17:11:59 UTC
Created attachment 1566293 [details]

Description of problem:
End to end migration not able to proceed with false "no conversion host was configured" error

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Configure OSP conversion host via UI
2. Run migration

Actual results:
Migration not able to proceed with following single line exception in automation log 

[----] E, [2019-05-09T13:05:46.833624 #15303:7c8f58] ERROR -- : State=<ValidateRequest> running on_entry raised exception: <Method exited with rc=MIQ_ABORT>

It is saying "conversion host was configured" even we can see conversion configured in UI as well as on /api/conversion_hosts endpoint. 

Expected results:
Migration should work normally. 

Additional info:

Comment 2 Yadnyawalk Tale 2019-05-09 17:16:14 UTC
Appliance can be accessed via

Comment 3 Fabien Dupont 2019-05-14 10:09:08 UTC
The problem is not about the conversion hosts not being there, but is due to the mapping being wrong.
The ValidateRequest method follows the relationships to verify that there is at least on conversion host available in the destination providers.
The chain is:

  -> TransformationMapping
    -> TransformationMappingItems where source_type == 'EmsCluster'
      -> EmsCluster or CloudTenant
        -> ExtManagementSystem
          -> ConversionHost

Looking into the appliance, I focused on the denied requests, i.e. those who received MIQ_ABORT.

irb(main):011:0> ServiceTemplateTransformationPlanRequest.where(approval_state: 'denied').last.validate_conversion_hosts
NoMethodError: undefined method `ext_management_system' for nil:NilClass
	from /var/www/miq/vmdb/app/models/service_template_transformation_plan_request.rb:22:in `block in validate_conversion_hosts'
	from /var/www/miq/vmdb/app/models/service_template_transformation_plan_request.rb:21:in `each'
	from /var/www/miq/vmdb/app/models/service_template_transformation_plan_request.rb:21:in `all?'
	from /var/www/miq/vmdb/app/models/service_template_transformation_plan_request.rb:21:in `validate_conversion_hosts'
	from (irb):11
	from /opt/rh/cfme-gemset/gems/railties- `start'
	from /opt/rh/cfme-gemset/gems/railties- `start'
	from /opt/rh/cfme-gemset/gems/railties- `console'
	from /opt/rh/cfme-gemset/gems/railties- `run_command!'
	from /opt/rh/cfme-gemset/gems/railties- `<top (required)>'
	from bin/rails:4:in `require'
	from bin/rails:4:in `<main>'

What this stack trace says is that the destination EmsCluster or CloudTenant is nil.
This is confirmed by the next command:

irb(main):013:0> ServiceTemplateTransformationPlanRequest.where(approval_state: 'denied').last.transformation_mapping.transformation_mapping_items.select { |i| i.source_type == 'EmsCluster' }
=> [#<TransformationMappingItem id: 48, source_id: 2, source_type: "EmsCluster", destination_id: 7, destination_type: "CloudTenant", transformation_mapping_id: 6, options: {}, created_at: "2019-05-09 18:06:40", updated_at: "2019-05-09 18:06:40">]
irb(main):014:0> CloudTenant.find(7)
ActiveRecord::RecordNotFound: Couldn't find CloudTenant with 'id'=7

I see on this same appliance that some requests went beyond the ValidateRequest method but they still failed. So, I went further into the error by checking the task:

irb(main):016:0> ServiceTemplateTransformationPlanRequest.where(approval_state: 'approved').last
=> #<ServiceTemplateTransformationPlanRequest id: 30, description: "vddk_plan", approval_state: "approved", type: "ServiceTemplateTransformationPlanRequest", created_on: "2019-05-09 19:28:43", updated_on: "2019-05-09 19:31:30", fulfilled_on: "2019-05-09 19:31:30", requester_id: 1, requester_name: "Administrator", request_type: "transformation_plan", request_state: "finished", message: "VM Transformations failed", status: "Error", options: {:dialog=>nil, :workflow_settings=>{:resource_action_id=>82}, :initiator=>nil, :src_id=>9, :request_options=>{:submit_workflow=>true, :init_defaults=>false}, :cart_state=>"ordered", :requester_group=>"EvmGroup-super_administrator", :executed_on_servers=>[1, 1], :delivered_on=>2019-05-09 19:28:59 UTC}, userid: "admin", source_id: 9, source_type: "ServiceTemplate", destination_id: nil, destination_type: nil, tenant_id: 1, service_order_id: 20, process: true, cancelation_status: nil>

irb(main):017:0> ServiceTemplateTransformationPlanTask.where(miq_request_id: 30).last.options[:virtv2v_wrapper]
=> {"wrapper_log"=>"/var/log/virt-v2v/v2v-import-20190509T153112-520-wrapper.log", "v2v_log"=>"/var/log/virt-v2v/v2v-import-20190509T153112-520.log", "state_file"=>"/tmp/v2v-import-20190509T153112-520.state", "throttling_file"=>"/tmp/v2v-import-20190509T153112-520.throttle"}

This allowed me to know the path of the logs for virt-v2v-wrapper and virt-v2v. I connect to the conversion host (using the credentials registered in CFME) and looked into the virt-v2v-wrapper log where I found:

2019-05-09 15:31:18,615:ERROR: Command exited with non-zero return code 1, output:
Failed to discover available identity versions when contacting Attempting to parse version from URL.
SSL exception connecting to [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)

So, the second issue here is that the SSL CA certificate for the OSP provider isn't present on the conversion host.

In the current implementation, we don't store the provider's CA certificate for OSP providers. The RHV provider allows setting it in the provider configuration via the UI, while it's not possible for OSP. We will file a RFE, but it won't be implemented before CFME 5.11. The conversion host enablement playbook is already able to accept a CA certificate, so we could ask the user to provide it in the wizard. The change on the backend would be minimal.

Comment 4 Fabien Dupont 2019-05-14 20:33:52 UTC
Conversion Host - Handle CA bundle from UI - https://github.com/ManageIQ/manageiq/pull/18762

Comment 5 Mike Turley 2019-05-14 21:03:28 UTC
Add OpenStack Trusted CA Certificates field to Conversion Host Wizard - https://github.com/ManageIQ/manageiq-v2v/pull/952

Comment 7 Yadnyawalk Tale 2019-06-14 10:50:42 UTC
Fixed! I can see migration can proceed when we have conversion host configured at endpoint.
Verified on:

Note You need to log in before you can comment on or make changes to this bug.