Bug 1760028

Created attachment 1623891 [details]
virsh dumpxml

Created attachment 1623893 [details]
log from source pod

Created attachment 1623895 [details]
log from target pod

DOC BZ for 2.1: https://bugzilla.redhat.com/show_bug.cgi?id=1760385

I need to see the VMI of the running VM and the respective Pod.

There is no VM.spec.cpu.model set here and kubevirt defaulted it to something. And I think the default is passthrough..

>  <cpu mode='custom' match='exact' check='full'>
>     <model fallback='forbid'>Haswell-noTSX-IBRS</model>

And this indeed looks like CPU passthrough.


Migration between different hosts will never work with CPU passthrough. And it is no just different model, just one extra flag will break it.

I checked the default kubevirt behaviour and the CPU is selected by checking these places (in order):

1) VM.spec.cpu.model
2) the default-cpu-model key from kubevirt config map
3) default host-model


And it is quite hard to check for compatibility when host-model is used. Neither the VM, VMI or Pod object actually have any idea what CPU is selected by libvirt and qemu for the started VM. We would need some kind of backpropagation mechanism for that.

The other option is to disable migration for VMs that are using host-model CPU.

>And it is quite hard to check for compatibility when host-model is used. Neither the VM, VMI or Pod object actually have any idea what CPU is selected by libvirt and qemu for the started VM. We would need some kind of backpropagation mechanism for that.

>The other option is to disable migration for VMs that are using host-model CPU.
 In this case, using of templates is useless without updating kubevirt config map. And in general, we need to be sure: customers are aware that if they will not modify cpu model parameter - migration will not work. In my opinion, this way is very strict..

It is the container way. Containers are not migratable... But you are right, I believe the virt-operator should generate the kubevirt config map and add sane default CPU there on CNV deployment. The discussion whether we should default to host-model or not is a separate topic.

Here we are discussing what happens when you try to migrate a VM that is using cpu passthrough. The VM should not crash or get stuck and so on this level we should either make sure the scheduler prevents the migration to incompatible host or we have to prevent the migration as a whole.

Per Comment #9 there are three general paths forward:

1) document this and do not change the code
2) prevent migration altogether when using host-model
3) only schedule to compatible target nodes when using host-model

Option 3 would be good, but there is a lot of complexity and corner cases involved in this. I think it would make sense to implement "disable migration altogether" in the short term. We can then decide if further refinement is worth it.

I agree with 2) for the short term. It has easy workarounds (just use a cpu model when creating VM or define one in kubevirt config map) and even RHV implemented this very recently so people are not really used to it yet :)

(In reply to Martin Sivák from comment #9)
> Here we are discussing what happens when you try to migrate a VM that is
> using cpu passthrough. The VM should not crash or get stuck and so on this
> level we should either make sure the scheduler prevents the migration to
> incompatible host or we have to prevent the migration as a whole.

There are too many different ways that host-passthrough migration can break,
I recommend strongly against it.  host-model is the live-migration-safe way of
getting all available CPU features from the host.

(In reply to Martin Sivák from comment #7)
> And it is quite hard to check for compatibility when host-model is used.
> Neither the VM, VMI or Pod object actually have any idea what CPU is
> selected by libvirt and qemu for the started VM. We would need some kind of
> backpropagation mechanism for that.

Just to make sure I understand the issue correctly: the libvirt API already propagates the information you need to the domain XML, but KubeVirt is not propagating the information back to the Kubernetes objects, correct?

Eduardo:

We are somewhat incorrectly using host passthrough terminology even though we actually mean host-model. So you are right :) But even with host-model the migration might not be possible when the hosts are not compatible enough.

And yes, I think libvirt knows the flags as I remember I was reading them back in my RHV days. Kubevirt is not reading them or exposing them just like you said.

(In reply to Martin Sivák from comment #18)
> [...] But even with host-model
> the migration might not be possible when the hosts are not compatible enough.

That's correct.  You are probably going to need virConnectGetDomainCapabilities() and virConnectBaselineHypervisorCPU() to generate a CPU configuration that can run on all nodes in a given cluster.

Maybe you can implement something equivalent using CPUNodeDiscovery features, but I don't know much about it.  Having something that is guaranteed to not lose any information from the domain capabilities XML would be ideal.

To summarize and highlight what Martin and Eduardo already said above,
host-model is better for migration than host-passthrough because migration
will not be allowed if the destination host cannot provide the same virtual
CPU which the VM is running with on the source. Its purpose is not making
migration between heterogeneous hosts easier or always working, it will just
make such migration safe by guaranteeing the virtual CPU will not change.

To make migration between heterogeneous always work, you need to gather
host-model CPUs from all hosts (virConnectGetDomainCapabilities, the returned
XML shows the CPU definition which will be used for starting a VM with
host-model CPU) and generate the best CPU definition which will work on all
hosts (virConnectBaselineHypervisorCPU).

The exact CPU configuration a VM is running with can be fetched with
virDomainGetXMLDesc and you can use virConnectCompareHypervisorCPU on other
hosts to check whether whether a VM with such CPU definition can be started
there.

Just to summarize I understand correctly what solution should be implemented: Disable the migration when host-model is used as for short term solution.

Is that right Stu?

Correct.

The hotfix is in the PR https://github.com/kubevirt/kubevirt/pull/2982
It disables the migration for host-model and cpu-passthrough.

As described previously, this is not long term solution.

Let's do it as follows:

1. passthrough blocks migraion
2. host model should not (because this is helpful in homogenous clusters where perf is needed)

I do believe the hostmodel in the current situation is still problematic.
The discussion settled on blocking with hostmodel and postponing for a proper solution.

The solution is done in a way that creates less damage.

Should I really change the existing pr?

Replied on the pr.
We must lock passthrouugh, but should leave an option for model

Back to assigned as the PR got closed and we need a conclusion.

Rhys we have W/A for this:
In 2.6: Edit configmap kubevirt-config -n openshift-cnv
https://docs.openshift.com/container-platform/4.7/virt/virt-2-6-release-notes.html#virt-2-6-known-issues
In 4.8: (use json patch)
https://bugzilla.redhat.com/show_bug.cgi?id=1968583

To verify:

create a heterogeneus cluster. attempt a migration from a node with no other of the same type. Observe that migration does not occur.

Verified on v4.10.0-486, target pod created with necessary Node-Selectors labels and waiting in Pending state because there are no any other nodes with matching CPU.

> virt-launcher-vm-fedora-nocpu-kq6vk                            2/2     Running   0             5m14s
> virt-launcher-vm-fedora-nocpu-xmts8                            0/2     Pending   0             3m1s

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Virtualization 4.10.0 Images security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0947

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days