Bug 2016290

Updating that AFAIK Windows2019 warm migration was OK in MTV 2.2.0-39, regarding this issue.

Maayan, can you tell us what version on CNV you're testing with?

(In reply to Fabien Dupont from comment #5)
> Maayan, can you tell us what version on CNV you're testing with?

CNV 4.9.0-249 / iib: 122549

Run warm migration on a few CNV 4.9.0-249 + MTV 2.2.0-63 (BM/PSI) with only 1 disk and the results were a little different (The DV is showing 80/90% Pause While The Plan is Successful.

Sam: 

"Very strange that the DV is showing paused with completion of only 80%, but the importer pod is done. The CDI controller is also full of strange network errors."

"In the initial report, the DVs were getting stuck at ImportInProgress even when the importer was done, so MTV never considered them as complete. The DV yaml in this case is a little different but also suffering from the status being wrong. It's at the Paused phase, but 80% complete and still has the Running condition. (edited) 
Something seems to be broken with CDI"



"It seems like there's a problem with CDI managing the status of the data volume, though the plan showing as complete is very strange unless the DV passed through a phase that would make MTV think it was completed." 

http://pastebin.test.redhat.com/1004347 (plan)
http://pastebin.test.redhat.com/1004338 (dv)
http://pastebin.test.redhat.com/1004343 (cdi)

Most of the errors in the logs in CDI can be categories into 2 groups:
1. Standard k8s concurrent access, which are normal and not concerning at all
2. Network errors. The DV controller attempts to connect to the import pod to read the progress so it can update the DV status. It is failing to connect, so either there is a network problem, or the import pod endpoint is down (aka the pod is not running). Assuming the network itself is fine, the import pod must not be running.

I also see this in the log:
I1027 15:24:49.999979       1 util.go:604] Saving VDDK annotations from pod status message: messageUnable to process data: pread: nbd_pread: poll: Interrupted system call Unable to transfer source data to target file kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessDataWithPause     /remote-source/app/pkg/importer/data-processor.go:206 kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessData         /remote-source/app/pkg/importer/data-processor.go:166 main.main         /remote-source/app/cmd/cdi-importer/importer.go:189 runtime.main        /usr/lib/golang/src/runtime/proc.go:204 runtime.goexit  /usr/lib/golang/src/runtime/asm_amd64.s:1374; VDDK: {"Version":"7.0.2","Host":""}

This is from the pod exit log, thus the pod must have died. The original error is originating from the nbdkit process it seems, but without the pod logs I cannot be sure. The import pod log might shed some light on what is actually failing.

importer log http://pastebin.test.redhat.com/1004642

@awels ^^

So the importer logs indicate a successful import, I see no failures, and a success, but the DV controller logs do not indicate this. So something else is happening.

According to the current state of investigation, the issue seems to be in CNV Storage.
Changing the product and component, respectively to CNV and Storage

So after investigating the problem came down to a new debugging feature added to CDI and subsequently used in MVT. This feature allows one to put an annotation on the DataVolume, and we will retain the pod that populates the DataVolume. Now in the case of warm migration this was enabled for MTV, this presents a problem because of the following.

 Warm migration takes one or more snapshots of the running the VM in the source system, and then copies the snapshots over in sequence until we can switch off the source and copy the final small delta. On the CDI side this means creating a new importer pod for each snapshots being copied. However due to the way the name of the pod is generated it doesn't change between snapshots. So if the DV is named `example`, then the importer pod will be named `importer-example`, and if a new snapshot is being copied, the name will be same. But if the first pod was retained due to the annotation, the second cannot be created (cannot have 2 pods with the same name). The import is reporting 100% because it is reporting the progress of a single pod, which completed successfully, its not 100% complete for the entire process, just one snapshot.

This can be fixed by disabling the annotation on warm migrations, and if we want to keep it enabled we should figure out a different mechanism for generating the pod names that is 'stable' and predictable but different for each snapshot. There is a checkpoint name we could potentially use for this.

In MTV-2.2 this workaround was added: Bug 2020297 - "Disable retaining "Importer" pods for warm migration",
to allow warm migration from VMware.

There will not be a MTV 2.2.z release to leverage a fix in CNV 4.9.z. Moving to CNV 4.10.0.

Hi, Matthew, Do you have plan to make it in 4.10.0?

I have a fix that is almost ready (see linked pull request), but I think it is unlikely to be accepted into 4.10. This can be pushed out again.

I checked, and CNV 4.10 is blockers-only at this point. I'm not sure if there's a later release that would benefit MTV near-term.

It is now possible to backport for 4.10.1, if you'd like this sooner than 4.11 (as suggested by the target release).

Yes, it would be good to have this backported to 4.10.1. Is there a process to follow for that, or can I just open a cherrypick in CDI?

A cherry pick in CDI will do it

oops, chose the version of the importer rather than CNV bundle.

The issue was reproduced in MTV warm migration with RHV RHEL8 VM with 2 disks, using Ceph-RBD as targeted storage class.
The 2nd precopy crashed, and once 'Cutover' was executed - the "Transfer disks" step was not ended (running forever)

Attachments in bug-2016290_may-15.zip file:
plan.yaml
Log of the crashed importer pod
Screenshots


Versions:
CNV 4.10.1-99 / iib: 224744
MTV STAGE 2.3.1-6 + patched ForkliftController CR


$ oc get forkliftcontroller forklift-controller -n openshift-mtv -oyaml
apiVersion: forklift.konveyor.io/v1beta1
kind: ForkliftController
metadata:
  creationTimestamp: "2022-05-15T08:27:50Z"
  generation: 3
  name: forklift-controller
  namespace: openshift-mtv
  resourceVersion: "16935517"
  uid: 73311c57-5b4e-4870-95a8-7426ed019855
spec:
  controller_image_fqin: quay.io/mrnold/forklift-controller:latest
  controller_precopy_interval: 3
  feature_must_gather_api: "true"
  feature_ui: "true"
  feature_validation: "true"
status:
  conditions:
  - ansibleResult:
      changed: 0
      completion: 2022-05-15T13:00:44.965727
      failures: 0
      ok: 27
      skipped: 9
    lastTransitionTime: "2022-05-15T08:27:50Z"
    message: Awaiting next reconciliation
    reason: Successful
    status: "True"
    type: Running

Hi, Matthew
Could we re-target it to 4.10.2?

It looks like the fix is working, since the latest screenshot shows a bunch of correctly retained importer pods. The logs show the "no space left" bug from https://bugzilla.redhat.com/show_bug.cgi?id=2087916, so it's going to be stuck trying to transfer a disk that won't fit in the scratch space. I think that bug is specific to imports from RHV, so I might consider this fixed if it works with a warm import from VMware.

Moving to VERIFIED, based on comment #29.
RHEL8 2-disks VM was successfully migrated from VMware, tested with NFS & Ceph-RBD

Versions:
CNV 4.10.1-99 / iib: 224744
OCP 4.10.12
MTV STAGE 2.3.1-7 + patched ForkliftController CR


$ oc get forkliftcontroller forklift-controller -n openshift-mtv -oyaml
apiVersion: forklift.konveyor.io/v1beta1
kind: ForkliftController
metadata:
  creationTimestamp: "2022-05-25T19:00:28Z"
  generation: 3
  name: forklift-controller
  namespace: openshift-mtv
  resourceVersion: "33615748"
  uid: ca1497f5-7a2b-48c7-bf7f-e26828a7024e
spec:
  controller_image_fqin: quay.io/mrnold/forklift-controller:latest
  controller_precopy_interval: 3
  feature_must_gather_api: "true"
  feature_ui: "true"
  feature_validation: "true"
status:
  conditions:
  - ansibleResult:
      changed: 0
      completion: 2022-05-26T07:09:13.571672
      failures: 0
      ok: 27
      skipped: 9
    lastTransitionTime: "2022-05-25T19:00:28Z"
    message: Awaiting next reconciliation
    reason: Successful
    status: "True"
    type: Running

(In reply to Maayan Hadasi from comment #30)
> Moving to VERIFIED, based on comment #29.
> RHEL8 2-disks VM was successfully migrated from VMware, tested with NFS &
> Ceph-RBD
> 
> Versions:
> CNV 4.10.1-99 / iib: 224744
> OCP 4.10.12
> MTV 2.3.1 GA + patched ForkliftController CR

Correcting that MTV 2.3.1 GA was used when testing the above

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Virtualization 4.10.2 Images security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5026