Original issue in Jira: https://issues.redhat.com/browse/CNV-34724. Moving it to bugzilla because CNV 4.14.0 we are still using bugzilla to report bugs. We will use Jira for all components starting CNV 4.15. 


Description of problem:

Live migration is no longer possible after a VMI has a hotplug volume added and removed. 

While this has nothing to do directly with hypershift/kubevirt, this issue was discovered while testing hypershift/kubevirt due to our usage of hotplug to provide volumes to the worker node VMs. Once we use kubevirt-csi to hotplug a volume and then we remove that volume, we noticed the VMIs failed to live migrate later on.

It is trivial to reproduce this error outside of hypershift/kubevirt using a fedora vm.


Version-Release number of selected component (if applicable):

CNV 4.14


How reproducible:

100%


Steps to Reproduce:

See the live-migration-failure-script.sh script attached to this issue to reproduce this easily. Below are the general steps that script performs.

This was reproduced using ODF 4.13 on OCP 4.14.0 with the latest CNV 4.14.0 pre-release

1. Create a vm and start it
2. live migrate the vmi to prove it is live migratable.
3. add a hotplug volume (RWX) vmi
4. remove a hotplug volume from the vmi
5. live migration is now permanently broken for the vmi



Actual results:

Live migration fails after adding and removing a  RMX hotplug volume



Expected results:

Live migration should continue to work after hotplug



Additional info:

The qemu log on the source pod reports the following after the migration fails.

{code:java}
#2023-10-30T19:53:14.880648Z qemu-kvm: qemu_savevm_state_complete_precopy_non_iterable: bdrv_inactivate_all() failed (-1)
#2023-10-30T19:53:15.053395Z qemu-kvm: Unable to read from socket: Bad file descriptor
#2023-10-30T19:53:15.053477Z qemu-kvm: Unable to read from socket: Bad file descriptor
#2023-10-30T19:53:15.053500Z qemu-kvm: Unable to read from socket: Bad file descriptor

The libvirt logs on the src only indicate that the migration failed due to an expected error.

{code:java}
{"component":"virt-launcher","kind":"","level":"error","msg":"Live migration failed.","name":"test-vm","namespace":"default","pos":"live-migration-source.go:718","reason":"error encountered during MigrateToURI3 libvirt api call: virError(Code=1, Domain=7, Message='internal error: client socket is closed')","timestamp":"2023-10-31T20:03:26.521711Z","uid":"388fd212-9187-488b-9989-43d2f19368f1"}