Created attachment 1623884 [details] VM yaml file Description of problem: I have 3 schedulable nodes with different cpu models: Node1: host-172-16-0-33 - Intel Core Processor (Haswell, no TSX, IBRS) Node2: host-172-16-0-13 - Intel Core Processor (Broadwell, IBRS) Node3: host-172-16-0-18 - Intel Core Processor (Haswell, no TSX, IBRS) VM created and running on Node1: $ oc get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE virt-launcher-vm-fedora-c8mqh 2/2 Running 0 1m 10.128.0.40 host-172-16-0-33 Run migration and new POD appeared on Node2 (that has another CPU model), as result - migration failed with error: {"component":"virt-launcher","kind":"","level":"error","msg":"Live migration failed","name":"vm-fedora","namespace":"default","pos":"manager.go:473","reason":"virError(Code=9, Domain=31, Message='operation failed: guest CPU doesn't match specification: missing features: md-clear')","timestamp":"2019-10-09T15:43:59.828172Z","uid":"2ce9e37d-eaab-11e9-ba41-fa163e35f769"} $ oc get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE virt-launcher-vm-fedora-c8mqh 2/2 Running 0 9m7s 10.128.0.40 host-172-16-0-33 virt-launcher-vm-fedora-nr947 0/2 Error 0 6m32s 10.131.0.47 host-172-16-0-13 However, if new POD were created on Node3 (that has the same cpu as Node1) - migration would be successfully completed. Probably, we need to have some validation of CPU models before running of migration and choose the right node where we are able to migrate VM. Steps to Reproduce: 1. create cluster where nodes have different cpu model 2. create and run vm 3. run migration Actual results: Migration started regardless of cpu models on target node and failed with error Expected results: Need to choose appropriate node as migration target. Additional info: Enabled feature-gates: DataVolumes,SRIOV,LiveMigration,CPUManager,CPUNodeDiscovery,Sidecar Attached files: vm-fedora.yml vm-dumpxml source-pod.log target-pod.log
Created attachment 1623891 [details] virsh dumpxml
Created attachment 1623893 [details] log from source pod
Created attachment 1623895 [details] log from target pod
DOC BZ for 2.1: https://bugzilla.redhat.com/show_bug.cgi?id=1760385
I need to see the VMI of the running VM and the respective Pod. There is no VM.spec.cpu.model set here and kubevirt defaulted it to something. And I think the default is passthrough.. > <cpu mode='custom' match='exact' check='full'> > <model fallback='forbid'>Haswell-noTSX-IBRS</model> And this indeed looks like CPU passthrough. Migration between different hosts will never work with CPU passthrough. And it is no just different model, just one extra flag will break it.
I checked the default kubevirt behaviour and the CPU is selected by checking these places (in order): 1) VM.spec.cpu.model 2) the default-cpu-model key from kubevirt config map 3) default host-model And it is quite hard to check for compatibility when host-model is used. Neither the VM, VMI or Pod object actually have any idea what CPU is selected by libvirt and qemu for the started VM. We would need some kind of backpropagation mechanism for that. The other option is to disable migration for VMs that are using host-model CPU.
>And it is quite hard to check for compatibility when host-model is used. Neither the VM, VMI or Pod object actually have any idea what CPU is selected by libvirt and qemu for the started VM. We would need some kind of backpropagation mechanism for that. >The other option is to disable migration for VMs that are using host-model CPU. In this case, using of templates is useless without updating kubevirt config map. And in general, we need to be sure: customers are aware that if they will not modify cpu model parameter - migration will not work. In my opinion, this way is very strict..
It is the container way. Containers are not migratable... But you are right, I believe the virt-operator should generate the kubevirt config map and add sane default CPU there on CNV deployment. The discussion whether we should default to host-model or not is a separate topic. Here we are discussing what happens when you try to migrate a VM that is using cpu passthrough. The VM should not crash or get stuck and so on this level we should either make sure the scheduler prevents the migration to incompatible host or we have to prevent the migration as a whole.
Per Comment #9 there are three general paths forward: 1) document this and do not change the code 2) prevent migration altogether when using host-model 3) only schedule to compatible target nodes when using host-model Option 3 would be good, but there is a lot of complexity and corner cases involved in this. I think it would make sense to implement "disable migration altogether" in the short term. We can then decide if further refinement is worth it.
I agree with 2) for the short term. It has easy workarounds (just use a cpu model when creating VM or define one in kubevirt config map) and even RHV implemented this very recently so people are not really used to it yet :)
(In reply to Martin Sivák from comment #9) > Here we are discussing what happens when you try to migrate a VM that is > using cpu passthrough. The VM should not crash or get stuck and so on this > level we should either make sure the scheduler prevents the migration to > incompatible host or we have to prevent the migration as a whole. There are too many different ways that host-passthrough migration can break, I recommend strongly against it. host-model is the live-migration-safe way of getting all available CPU features from the host.
(In reply to Martin Sivák from comment #7) > And it is quite hard to check for compatibility when host-model is used. > Neither the VM, VMI or Pod object actually have any idea what CPU is > selected by libvirt and qemu for the started VM. We would need some kind of > backpropagation mechanism for that. Just to make sure I understand the issue correctly: the libvirt API already propagates the information you need to the domain XML, but KubeVirt is not propagating the information back to the Kubernetes objects, correct?
Eduardo: We are somewhat incorrectly using host passthrough terminology even though we actually mean host-model. So you are right :) But even with host-model the migration might not be possible when the hosts are not compatible enough. And yes, I think libvirt knows the flags as I remember I was reading them back in my RHV days. Kubevirt is not reading them or exposing them just like you said.
(In reply to Martin Sivák from comment #18) > [...] But even with host-model > the migration might not be possible when the hosts are not compatible enough. That's correct. You are probably going to need virConnectGetDomainCapabilities() and virConnectBaselineHypervisorCPU() to generate a CPU configuration that can run on all nodes in a given cluster. Maybe you can implement something equivalent using CPUNodeDiscovery features, but I don't know much about it. Having something that is guaranteed to not lose any information from the domain capabilities XML would be ideal.
To summarize and highlight what Martin and Eduardo already said above, host-model is better for migration than host-passthrough because migration will not be allowed if the destination host cannot provide the same virtual CPU which the VM is running with on the source. Its purpose is not making migration between heterogeneous hosts easier or always working, it will just make such migration safe by guaranteeing the virtual CPU will not change. To make migration between heterogeneous always work, you need to gather host-model CPUs from all hosts (virConnectGetDomainCapabilities, the returned XML shows the CPU definition which will be used for starting a VM with host-model CPU) and generate the best CPU definition which will work on all hosts (virConnectBaselineHypervisorCPU). The exact CPU configuration a VM is running with can be fetched with virDomainGetXMLDesc and you can use virConnectCompareHypervisorCPU on other hosts to check whether whether a VM with such CPU definition can be started there.
Just to summarize I understand correctly what solution should be implemented: Disable the migration when host-model is used as for short term solution. Is that right Stu?
Correct.
The hotfix is in the PR https://github.com/kubevirt/kubevirt/pull/2982 It disables the migration for host-model and cpu-passthrough. As described previously, this is not long term solution.
Let's do it as follows: 1. passthrough blocks migraion 2. host model should not (because this is helpful in homogenous clusters where perf is needed)
I do believe the hostmodel in the current situation is still problematic. The discussion settled on blocking with hostmodel and postponing for a proper solution. The solution is done in a way that creates less damage. Should I really change the existing pr?
Replied on the pr. We must lock passthrouugh, but should leave an option for model
Back to assigned as the PR got closed and we need a conclusion.
Rhys we have W/A for this: In 2.6: Edit configmap kubevirt-config -n openshift-cnv https://docs.openshift.com/container-platform/4.7/virt/virt-2-6-release-notes.html#virt-2-6-known-issues In 4.8: (use json patch) https://bugzilla.redhat.com/show_bug.cgi?id=1968583
To verify: create a heterogeneus cluster. attempt a migration from a node with no other of the same type. Observe that migration does not occur.
Verified on v4.10.0-486, target pod created with necessary Node-Selectors labels and waiting in Pending state because there are no any other nodes with matching CPU. > virt-launcher-vm-fedora-nocpu-kq6vk 2/2 Running 0 5m14s > virt-launcher-vm-fedora-nocpu-xmts8 0/2 Pending 0 3m1s
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Virtualization 4.10.0 Images security and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0947
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days