Bug 1887138 - [v2v][VMware to CNV VM import API] VM import takes 31 minutes to finish last 25% part.
Summary: [v2v][VMware to CNV VM import API] VM import takes 31 minutes to finish last ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: V2V
Version: 2.5.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 2.5.0
Assignee: Fabien Dupont
QA Contact: Ilanit Stein
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-10-11 08:24 UTC by Ilanit Stein
Modified: 2020-11-17 13:25 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-11-17 13:24:56 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
events (26.09 KB, text/plain)
2020-10-11 08:37 UTC, Ilanit Stein
no flags Details
cdi-deployment.log (31.24 KB, text/plain)
2020-10-11 08:38 UTC, Ilanit Stein
no flags Details
vm import controller log (899.11 KB, text/plain)
2020-10-11 08:38 UTC, Ilanit Stein
no flags Details
vmimport.v2v.kubevirt.log (816.38 KB, text/plain)
2020-10-13 07:14 UTC, Ilanit Stein
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2020:5127 0 None None None 2020-11-17 13:25:33 UTC

Description Ilanit Stein 2020-10-11 08:24:14 UTC
Description of problem:
VM import from VMware to CNV via API (vmio) of a RHEL-7/8 reaches 75%, once the 
image import is completed (that takes ~5 minutes for a 25GB disk)  

Version-Release number of selected component (if applicable):
CNV-2.5 - iib:19290

How reproducible:
100%
Tried 3 different VMs, with NFS & Ceph-RBD/Block.

Steps to Reproduce:
1. Add vddk-init-image

# oc -n openshift-cnv edit cm v2v-vmware
Listing:
apiVersion: v1
data:
  kubevirt-vmware-image: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/kubevirt-vmware:v2.1.0-6
  kubevirt-vmware-image-pull-policy: IfNotPresent
  v2v-conversion-image: registry-proxy.engineering.redhat.com/rh-osbs/container-native-virtualization-kubevirt-v2v-conversion:v2.1.0-10
  vddk-init-image: <server FQDN>:5000/vddk-images/vddk:v700  <==== vddk image
kind: ConfigMap
metadata:
  creationTimestamp: 2019-10-24T11:40:26Z
  name: v2v-vmware
  namespace: openshift-cnv
  resourceVersion: "3652730"
  selfLink: /api/v1/namespaces/openshift-cnv/configmaps/v2v-vmware
  uid: 1272cd5a-f653-11e9-bd31-5254000a752d

2. Add Secret:

cat <<EOF | oc create -f -
---
apiVersion: v1
kind: Secret
metadata:
 name: vmw-secret
type: Opaque
stringData:
 vmware: |-
   # API URL of the vCenter or ESXi host
   apiUrl: "https://<VMware IP address>/sdk"
   # Username provided in the format of username@domain.
   username: <username>
   password: <password>
   # The certificate thumbprint of the vCenter or ESXi host, in colon-separated hexidecimal octets.
   thumbprint: 31:...:30
EOF

3. External mapping

cat <<EOF | oc create -f -
apiVersion: v2v.kubevirt.io/v1beta1
kind: ResourceMapping
metadata:
 name: example-vmware-resourcemappings
 namespace: default
spec:
  vmware:
    networkMappings:
    - source:
        name: VM Network # map network name to network attachment definition
      target: 
        name: pod
      type: pod
    storageMappings:
    - source:
        id: datastore-12 
      target: 
        name: nfs  <== Or use Ceph-RBD/Block
EOF

4. VM import create

cat <<EOF | oc create -f -
apiVersion: v2v.kubevirt.io/v1beta1
kind: VirtualMachineImport
metadata:
  name: vmware-import-1
  namespace: default
spec:
  providerCredentialsSecret:
    name: vmw-secret
    namespace: default # optional, if not specified, use CR's namespace
  resourceMapping:
    name: example-vmware-resourcemappings
    namespace: default
  targetVmName: vmw-import-1
  startVm: false
  source:
    vmware:
      vm:
        id: 4203b421-575c-ed5e-cbad-9d390f2eb101   
EOF

Expected results:
VM import should not take so long to complete the post image import tasks - last 25% of the overall VM import process.

Comment 1 Ilanit Stein 2020-10-11 08:37:56 UTC
Created attachment 1720555 [details]
events

Comment 2 Ilanit Stein 2020-10-11 08:38:23 UTC
Created attachment 1720556 [details]
cdi-deployment.log

Comment 3 Ilanit Stein 2020-10-11 08:38:57 UTC
Created attachment 1720557 [details]
vm import controller log

Comment 4 Fabien Dupont 2020-10-12 11:46:14 UTC
31 minutes is definitely to long, but it can be due to the multiple layers of abstraction. In my lab, it usually takes 15 minutes, which is still too long.

Can you also attach the logs for the vmimport.v2v.kubevirt.io* pod that did the conversion?

Comment 5 Ilanit Stein 2020-10-13 07:14:44 UTC
Created attachment 1721114 [details]
vmimport.v2v.kubevirt.log

Comment 6 Ilanit Stein 2020-10-13 07:19:27 UTC
Comment on attachment 1721114 [details]
vmimport.v2v.kubevirt.log

This log is for a VM with 10GB.
It took ~24 minutes to do the import (cdi import ~1 min, and v2v conversion ~23 minutes).

Comment 7 Fabien Dupont 2020-10-13 10:38:22 UTC
Thanks. We have a solution, but we need to groom it a bit more to make sure it's secure enough.

Comment 13 Ilanit Stein 2020-10-26 08:58:59 UTC
Verified on CNV-2.5: iib-22696 hco-v2.5.0-396, 

VM import of a RHEL7 with 10GB disk took overall 6 minutes:
3 minutes for importer pod
3 minutes for v2v pod

Comment 16 errata-xmlrpc 2020-11-17 13:24:56 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Virtualization 2.5.0 Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:5127


Note You need to log in before you can comment on or make changes to this bug.