Created attachment 1769772 [details] progress stuck Created attachment 1769772 [details] progress stuck While migrating two vSphere plans with 20 and 19 VMs respectively, a portion of the VMs had their Datavolume imports succeed, but the import did not proceed from CopyingDisks to ConvertingGuest. Editing a stuck VirtualMachineImport resource caused it to reconcile, and at that point the import resumed and completed successfully. vm-import-controller apparently failed to notice and reconcile the resources in response to the successful disk import on its own. Description of problem: all the VMS are identicals each plan use different VMs Vsphere 6.7 single esx 6.7 plan 1 - 20 VMs plan 2 - 19 VMs root@f02-h07-000-r640:~$ oc get pods -nopenshift-rhmtv NAME READY STATUS RESTARTS AGE forklift-controller-64999d5bb7-s2mqc 2/2 Running 1 5d forklift-operator-86f4fbb575-pbt4z 1/1 Running 0 5d forklift-ui-79b4757954-mppxx 1/1 Running 0 5d root@f02-h07-000-r640:~$ oc get vm -nopenshift-rhmtv NAME AGE VOLUME dc67-fc-rhel7-scale-50gb-70usage-vm-1 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-10 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-11 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-12 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-13 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-14 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-15 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-16 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-17 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-18 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-2 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-20 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-21 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-22 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-23 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-24 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-25 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-26 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-27 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-37 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-38 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-39 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-4 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-40 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-41 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-42 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-43 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-44 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-45 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-46 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-47 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-48 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-49 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-5 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-50 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-6 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-7 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-8 3h16m dc67-fc-rhel7-scale-50gb-70usage-vm-9 3h16m root@f02-h07-000-r640:~$ oc get vmimports -nopenshift-rhmtv NAME AGE plan-20210406-single-esx-19vms-50gb70util-vcenter6-7-plan1-vm-1008-6fad 3h17m plan-20210406-single-esx-19vms-50gb70util-vcenter6-7-plan1-vm-1009-6fad 3h16m plan-20210406-single-esx-19vms-50gb70util-vcenter6-7-plan1-vm-1017-6fad 3h17m plan-20210406-single-esx-19vms-50gb70util-vcenter6-7-plan1-vm-1018-6fad 3h16m plan-20210406-single-esx-19vms-50gb70util-vcenter6-7-plan1-vm-1019-6fad 3h16m plan-20210406-single-esx-19vms-50gb70util-vcenter6-7-plan1-vm-1020-6fad 3h16m plan-20210406-single-esx-19vms-50gb70util-vcenter6-7-plan1-vm-1021-6fad 3h16m plan-20210406-single-esx-19vms-50gb70util-vcenter6-7-plan1-vm-1022-6fad 3h16m plan-20210406-single-esx-19vms-50gb70util-vcenter6-7-plan1-vm-1023-6fad 3h16m plan-20210406-single-esx-19vms-50gb70util-vcenter6-7-plan1-vm-1024-6fad 3h16m plan-20210406-single-esx-19vms-50gb70util-vcenter6-7-plan1-vm-1025-6fad 3h16m plan-20210406-single-esx-19vms-50gb70util-vcenter6-7-plan1-vm-1027-6fad 3h16m plan-20210406-single-esx-19vms-50gb70util-vcenter6-7-plan1-vm-1028-6fad 3h16m plan-20210406-single-esx-19vms-50gb70util-vcenter6-7-plan1-vm-1029-6fad 3h16m plan-20210406-single-esx-19vms-50gb70util-vcenter6-7-plan1-vm-1030-6fad 3h16m plan-20210406-single-esx-19vms-50gb70util-vcenter6-7-plan1-vm-1031-6fad 3h16m plan-20210406-single-esx-19vms-50gb70util-vcenter6-7-plan1-vm-1032-6fad 3h16m plan-20210406-single-esx-19vms-50gb70util-vcenter6-7-plan1-vm-1033-6fad 3h16m plan-20210406-single-esx-19vms-50gb70util-vcenter6-7-plan1-vm-1034-6fad 3h16m plan-20210406-single-esx-20vms-50gb70util-vcenter6-7-plan2-vm-1011-6f72 3h16m plan-20210406-single-esx-20vms-50gb70util-vcenter6-7-plan2-vm-1012-6f72 3h16m plan-20210406-single-esx-20vms-50gb70util-vcenter6-7-plan2-vm-1013-6f72 3h16m plan-20210406-single-esx-20vms-50gb70util-vcenter6-7-plan2-vm-1014-6f72 3h16m plan-20210406-single-esx-20vms-50gb70util-vcenter6-7-plan2-vm-1015-6f72 3h16m plan-20210406-single-esx-20vms-50gb70util-vcenter6-7-plan2-vm-1016-6f72 3h16m plan-20210406-single-esx-20vms-50gb70util-vcenter6-7-plan2-vm-1044-6f72 3h16m plan-20210406-single-esx-20vms-50gb70util-vcenter6-7-plan2-vm-1045-6f72 3h16m plan-20210406-single-esx-20vms-50gb70util-vcenter6-7-plan2-vm-1046-6f72 3h16m plan-20210406-single-esx-20vms-50gb70util-vcenter6-7-plan2-vm-1047-6f72 3h16m plan-20210406-single-esx-20vms-50gb70util-vcenter6-7-plan2-vm-1048-6f72 3h16m plan-20210406-single-esx-20vms-50gb70util-vcenter6-7-plan2-vm-1049-6f72 3h16m plan-20210406-single-esx-20vms-50gb70util-vcenter6-7-plan2-vm-1050-6f72 3h16m plan-20210406-single-esx-20vms-50gb70util-vcenter6-7-plan2-vm-1051-6f72 3h16m plan-20210406-single-esx-20vms-50gb70util-vcenter6-7-plan2-vm-1052-6f72 3h16m plan-20210406-single-esx-20vms-50gb70util-vcenter6-7-plan2-vm-1053-6f72 3h16m plan-20210406-single-esx-20vms-50gb70util-vcenter6-7-plan2-vm-1054-6f72 3h16m plan-20210406-single-esx-20vms-50gb70util-vcenter6-7-plan2-vm-1055-6f72 3h16m plan-20210406-single-esx-20vms-50gb70util-vcenter6-7-plan2-vm-1056-6f72 3h16m plan-20210406-single-esx-20vms-50gb70util-vcenter6-7-plan2-vm-1057-6f72 3h16m Version-Release number of selected component (if applicable): MTV : version v2.0.0.beta1-28 CNV : version 2.6.1 VM size 50GB 70% usage vddk :702 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Created attachment 1769773 [details] plans stuck
The problem is located in CNV > VMIO. Updating the product/component.
https://github.com/kubevirt/vm-import-operator/pull/482
The fix should be in CNV v2.6.2-28 +
Correction: The bug fix should be in version CNV-2.6.2-31 (iib-69036) and above.
verified on cloud38 CNV: 2.6.2 MTV: 2.0.0.21 root@f02-h07-000-r640:~$ oc get plan -nopenshift-rhmtv NAME READY EXECUTING SUCCEEDED FAILED AGE 1946884-part1-20vms True 50s 1946884-part2-19vms 12s root@f02-h07-000-r640:~/mtv-autodeploy$ oc get csv -n openshift-cnv NAME DISPLAY VERSION REPLACES PHASE kubevirt-hyperconverged-operator.v2.6.2 OpenShift Virtualization 2.6.2 kubevirt-hyperconverged-operator.v2.6.1 Succeeded root@f02-h07-000-r640:~$ oc get vmimports -n openshift-rhmtv NAME AGE 1946884-part1-20vms-vm-1023-7w4sv 4m9s 1946884-part1-20vms-vm-1024-gbs4c 3m59s 1946884-part1-20vms-vm-1025-qfv5d 3m43s 1946884-part1-20vms-vm-1026-7sv5c 3m28s 1946884-part1-20vms-vm-1027-zlqk9 3m16s 1946884-part1-20vms-vm-1028-hxq6k 3m6s 1946884-part1-20vms-vm-1029-q4mtw 2m55s 1946884-part1-20vms-vm-1030-79twm 2m44s 1946884-part1-20vms-vm-1031-mwtpn 2m33s 1946884-part1-20vms-vm-1032-jmrvb 2m20s 1946884-part1-20vms-vm-1033-77kjw 2m9s 1946884-part1-20vms-vm-1034-pdm7x 116s 1946884-part1-20vms-vm-1035-gfrrf 104s 1946884-part1-20vms-vm-1036-pbkt2 92s 1946884-part1-20vms-vm-1037-c547b 78s 1946884-part1-20vms-vm-1038-tfmqk 64s 1946884-part1-20vms-vm-1039-vtq9r 50s 1946884-part1-20vms-vm-1040-6q4g7 36s 1946884-part1-20vms-vm-1041-n8tgw 21s 1946884-part1-20vms-vm-1042-xk7cg 6s 1946884-part2-19vms-vm-1008-tcvf7 4m4s 1946884-part2-19vms-vm-1009-8wh4x 2m38s 1946884-part2-19vms-vm-1010-7sqg2 2m27s 1946884-part2-19vms-vm-1012-5nwdv 58s 1946884-part2-19vms-vm-1013-ksz6l 29s 1946884-part2-19vms-vm-1014-dxt4z 13s 1946884-part2-19vms-vm-1017-zknrk 3m51s 1946884-part2-19vms-vm-1018-nxrkc 3m33s 1946884-part2-19vms-vm-1019-5vhql 3m21s 1946884-part2-19vms-vm-1020-c7gc9 3m11s 1946884-part2-19vms-vm-1021-plhdm 3m1s 1946884-part2-19vms-vm-1022-qcwh4 2m50s 1946884-part2-19vms-vm-1051-5hdt8 2m15s 1946884-part2-19vms-vm-1052-dn8xb 2m3s 1946884-part2-19vms-vm-1053-qw6nd 111s 1946884-part2-19vms-vm-1054-588d7 98s 1946884-part2-19vms-vm-1055-6x59r 84s 1946884-part2-19vms-vm-1056-qs2fq 71s 1946884-part2-19vms-vm-1057-67pdl 42s root@f02-h07-000-r640:~$ oc get vmimports -n openshift-rhmtv |grep -v NAME |wc -l 39
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Virtualization 2.6.2 Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2021:1502