Bug 1887138

Summary: [v2v][VMware to CNV VM import API] VM import takes 31 minutes to finish last 25% part.
Product: Container Native Virtualization (CNV) Reporter: Ilanit Stein <istein>
Component: V2VAssignee: Fabien Dupont <fdupont>
Status: CLOSED ERRATA QA Contact: Ilanit Stein <istein>
Severity: high Docs Contact:
Priority: high    
Version: 2.5.0CC: cnv-qe-bugs, dagur, fdupont, mnadeem, nachandr
Target Milestone: ---Keywords: AutomationBlocker
Target Release: 2.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-11-17 13:24:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
events
none
cdi-deployment.log
none
vm import controller log
none
vmimport.v2v.kubevirt.log none

Description Ilanit Stein 2020-10-11 08:24:14 UTC
Description of problem:
VM import from VMware to CNV via API (vmio) of a RHEL-7/8 reaches 75%, once the 
image import is completed (that takes ~5 minutes for a 25GB disk)  

Version-Release number of selected component (if applicable):
CNV-2.5 - iib:19290

How reproducible:
100%
Tried 3 different VMs, with NFS & Ceph-RBD/Block.

Steps to Reproduce:
1. Add vddk-init-image

# oc -n openshift-cnv edit cm v2v-vmware
Listing:
apiVersion: v1
data:
  kubevirt-vmware-image: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/kubevirt-vmware:v2.1.0-6
  kubevirt-vmware-image-pull-policy: IfNotPresent
  v2v-conversion-image: registry-proxy.engineering.redhat.com/rh-osbs/container-native-virtualization-kubevirt-v2v-conversion:v2.1.0-10
  vddk-init-image: <server FQDN>:5000/vddk-images/vddk:v700  <==== vddk image
kind: ConfigMap
metadata:
  creationTimestamp: 2019-10-24T11:40:26Z
  name: v2v-vmware
  namespace: openshift-cnv
  resourceVersion: "3652730"
  selfLink: /api/v1/namespaces/openshift-cnv/configmaps/v2v-vmware
  uid: 1272cd5a-f653-11e9-bd31-5254000a752d

2. Add Secret:

cat <<EOF | oc create -f -
---
apiVersion: v1
kind: Secret
metadata:
 name: vmw-secret
type: Opaque
stringData:
 vmware: |-
   # API URL of the vCenter or ESXi host
   apiUrl: "https://<VMware IP address>/sdk"
   # Username provided in the format of username@domain.
   username: <username>
   password: <password>
   # The certificate thumbprint of the vCenter or ESXi host, in colon-separated hexidecimal octets.
   thumbprint: 31:...:30
EOF

3. External mapping

cat <<EOF | oc create -f -
apiVersion: v2v.kubevirt.io/v1beta1
kind: ResourceMapping
metadata:
 name: example-vmware-resourcemappings
 namespace: default
spec:
  vmware:
    networkMappings:
    - source:
        name: VM Network # map network name to network attachment definition
      target: 
        name: pod
      type: pod
    storageMappings:
    - source:
        id: datastore-12 
      target: 
        name: nfs  <== Or use Ceph-RBD/Block
EOF

4. VM import create

cat <<EOF | oc create -f -
apiVersion: v2v.kubevirt.io/v1beta1
kind: VirtualMachineImport
metadata:
  name: vmware-import-1
  namespace: default
spec:
  providerCredentialsSecret:
    name: vmw-secret
    namespace: default # optional, if not specified, use CR's namespace
  resourceMapping:
    name: example-vmware-resourcemappings
    namespace: default
  targetVmName: vmw-import-1
  startVm: false
  source:
    vmware:
      vm:
        id: 4203b421-575c-ed5e-cbad-9d390f2eb101   
EOF

Expected results:
VM import should not take so long to complete the post image import tasks - last 25% of the overall VM import process.

Comment 1 Ilanit Stein 2020-10-11 08:37:56 UTC
Created attachment 1720555 [details]
events

Comment 2 Ilanit Stein 2020-10-11 08:38:23 UTC
Created attachment 1720556 [details]
cdi-deployment.log

Comment 3 Ilanit Stein 2020-10-11 08:38:57 UTC
Created attachment 1720557 [details]
vm import controller log

Comment 4 Fabien Dupont 2020-10-12 11:46:14 UTC
31 minutes is definitely to long, but it can be due to the multiple layers of abstraction. In my lab, it usually takes 15 minutes, which is still too long.

Can you also attach the logs for the vmimport.v2v.kubevirt.io* pod that did the conversion?

Comment 5 Ilanit Stein 2020-10-13 07:14:44 UTC
Created attachment 1721114 [details]
vmimport.v2v.kubevirt.log

Comment 6 Ilanit Stein 2020-10-13 07:19:27 UTC
Comment on attachment 1721114 [details]
vmimport.v2v.kubevirt.log

This log is for a VM with 10GB.
It took ~24 minutes to do the import (cdi import ~1 min, and v2v conversion ~23 minutes).

Comment 7 Fabien Dupont 2020-10-13 10:38:22 UTC
Thanks. We have a solution, but we need to groom it a bit more to make sure it's secure enough.

Comment 13 Ilanit Stein 2020-10-26 08:58:59 UTC
Verified on CNV-2.5: iib-22696 hco-v2.5.0-396, 

VM import of a RHEL7 with 10GB disk took overall 6 minutes:
3 minutes for importer pod
3 minutes for v2v pod

Comment 16 errata-xmlrpc 2020-11-17 13:24:56 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Virtualization 2.5.0 Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:5127