Bug 1760028 - CPU compatibility is not checked when migrating host-model VMs
Summary: CPU compatibility is not checked when migrating host-model VMs
Keywords:
Status: POST
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: Virtualization
Version: 2.1.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 2.5.0
Assignee: sgott
QA Contact: vsibirsk
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-10-09 16:41 UTC by Denys Shchedrivyi
Modified: 2020-06-26 13:18 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)
VM yaml file (1.54 KB, text/plain)
2019-10-09 16:41 UTC, Denys Shchedrivyi
no flags Details
virsh dumpxml (6.93 KB, text/plain)
2019-10-09 16:42 UTC, Denys Shchedrivyi
no flags Details
log from source pod (18.39 KB, text/plain)
2019-10-09 16:43 UTC, Denys Shchedrivyi
no flags Details
log from target pod (16.87 KB, text/plain)
2019-10-09 16:44 UTC, Denys Shchedrivyi
no flags Details

Description Denys Shchedrivyi 2019-10-09 16:41:02 UTC
Created attachment 1623884 [details]
VM yaml file

Description of problem:
 I have 3 schedulable nodes with different cpu models:

Node1: host-172-16-0-33 - Intel Core Processor (Haswell, no TSX, IBRS)
Node2: host-172-16-0-13 - Intel Core Processor (Broadwell, IBRS)
Node3: host-172-16-0-18 - Intel Core Processor (Haswell, no TSX, IBRS)


VM created and running on Node1:

$ oc get pod -o wide
NAME                            READY     STATUS    RESTARTS   AGE       IP            NODE               
virt-launcher-vm-fedora-c8mqh   2/2       Running   0          1m        10.128.0.40   host-172-16-0-33


Run migration and new POD appeared on Node2 (that has another CPU model), as result - migration failed with error: 

{"component":"virt-launcher","kind":"","level":"error","msg":"Live migration failed","name":"vm-fedora","namespace":"default","pos":"manager.go:473","reason":"virError(Code=9, Domain=31, Message='operation failed: guest CPU doesn't match specification: missing features: md-clear')","timestamp":"2019-10-09T15:43:59.828172Z","uid":"2ce9e37d-eaab-11e9-ba41-fa163e35f769"}

$ oc get pod -o wide
NAME                            READY     STATUS    RESTARTS   AGE       IP            NODE            
virt-launcher-vm-fedora-c8mqh   2/2       Running   0          9m7s      10.128.0.40   host-172-16-0-33
virt-launcher-vm-fedora-nr947   0/2       Error     0          6m32s     10.131.0.47   host-172-16-0-13


However, if new POD were created on Node3 (that has the same cpu as Node1) - migration would be successfully completed.

Probably, we need to have some validation of CPU models before running of migration and choose the right node where we are able to migrate VM. 


Steps to Reproduce:
1. create cluster where nodes have different cpu model
2. create and run vm
3. run migration 


Actual results:
 Migration started regardless of cpu models on target node and failed with error

Expected results:
 Need to choose appropriate node as migration target.   



Additional info:

Enabled feature-gates:   DataVolumes,SRIOV,LiveMigration,CPUManager,CPUNodeDiscovery,Sidecar


Attached files:
vm-fedora.yml
vm-dumpxml
source-pod.log
target-pod.log

Comment 1 Denys Shchedrivyi 2019-10-09 16:42:15 UTC
Created attachment 1623891 [details]
virsh dumpxml

Comment 2 Denys Shchedrivyi 2019-10-09 16:43:27 UTC
Created attachment 1623893 [details]
log from source pod

Comment 3 Denys Shchedrivyi 2019-10-09 16:44:14 UTC
Created attachment 1623895 [details]
log from target pod

Comment 5 Israel Pinto 2019-10-10 12:23:12 UTC
DOC BZ for 2.1: https://bugzilla.redhat.com/show_bug.cgi?id=1760385

Comment 6 Martin Sivák 2019-10-15 12:25:53 UTC
I need to see the VMI of the running VM and the respective Pod.

There is no VM.spec.cpu.model set here and kubevirt defaulted it to something. And I think the default is passthrough..

>  <cpu mode='custom' match='exact' check='full'>
>     <model fallback='forbid'>Haswell-noTSX-IBRS</model>

And this indeed looks like CPU passthrough.


Migration between different hosts will never work with CPU passthrough. And it is no just different model, just one extra flag will break it.

Comment 7 Martin Sivák 2019-10-15 12:51:06 UTC
I checked the default kubevirt behaviour and the CPU is selected by checking these places (in order):

1) VM.spec.cpu.model
2) the default-cpu-model key from kubevirt config map
3) default host-model


And it is quite hard to check for compatibility when host-model is used. Neither the VM, VMI or Pod object actually have any idea what CPU is selected by libvirt and qemu for the started VM. We would need some kind of backpropagation mechanism for that.

The other option is to disable migration for VMs that are using host-model CPU.

Comment 8 Denys Shchedrivyi 2019-10-17 22:15:30 UTC
>And it is quite hard to check for compatibility when host-model is used. Neither the VM, VMI or Pod object actually have any idea what CPU is selected by libvirt and qemu for the started VM. We would need some kind of backpropagation mechanism for that.

>The other option is to disable migration for VMs that are using host-model CPU.
 In this case, using of templates is useless without updating kubevirt config map. And in general, we need to be sure: customers are aware that if they will not modify cpu model parameter - migration will not work. In my opinion, this way is very strict..

Comment 9 Martin Sivák 2019-10-18 06:04:35 UTC
It is the container way. Containers are not migratable... But you are right, I believe the virt-operator should generate the kubevirt config map and add sane default CPU there on CNV deployment. The discussion whether we should default to host-model or not is a separate topic.

Here we are discussing what happens when you try to migrate a VM that is using cpu passthrough. The VM should not crash or get stuck and so on this level we should either make sure the scheduler prevents the migration to incompatible host or we have to prevent the migration as a whole.

Comment 11 sgott 2019-11-05 19:54:39 UTC
Per Comment #9 there are three general paths forward:

1) document this and do not change the code
2) prevent migration altogether when using host-model
3) only schedule to compatible target nodes when using host-model

Option 3 would be good, but there is a lot of complexity and corner cases involved in this. I think it would make sense to implement "disable migration altogether" in the short term. We can then decide if further refinement is worth it.

Comment 12 Martin Sivák 2019-11-05 20:09:40 UTC
I agree with 2) for the short term. It has easy workarounds (just use a cpu model when creating VM or define one in kubevirt config map) and even RHV implemented this very recently so people are not really used to it yet :)

Comment 16 Eduardo Habkost 2019-11-18 18:21:38 UTC
(In reply to Martin Sivák from comment #9)
> Here we are discussing what happens when you try to migrate a VM that is
> using cpu passthrough. The VM should not crash or get stuck and so on this
> level we should either make sure the scheduler prevents the migration to
> incompatible host or we have to prevent the migration as a whole.

There are too many different ways that host-passthrough migration can break,
I recommend strongly against it.  host-model is the live-migration-safe way of
getting all available CPU features from the host.

Comment 17 Eduardo Habkost 2019-11-18 18:30:00 UTC
(In reply to Martin Sivák from comment #7)
> And it is quite hard to check for compatibility when host-model is used.
> Neither the VM, VMI or Pod object actually have any idea what CPU is
> selected by libvirt and qemu for the started VM. We would need some kind of
> backpropagation mechanism for that.

Just to make sure I understand the issue correctly: the libvirt API already propagates the information you need to the domain XML, but KubeVirt is not propagating the information back to the Kubernetes objects, correct?

Comment 18 Martin Sivák 2019-11-18 19:49:12 UTC
Eduardo:

We are somewhat incorrectly using host passthrough terminology even though we actually mean host-model. So you are right :) But even with host-model the migration might not be possible when the hosts are not compatible enough.

And yes, I think libvirt knows the flags as I remember I was reading them back in my RHV days. Kubevirt is not reading them or exposing them just like you said.

Comment 19 Eduardo Habkost 2019-11-18 20:22:01 UTC
(In reply to Martin Sivák from comment #18)
> [...] But even with host-model
> the migration might not be possible when the hosts are not compatible enough.

That's correct.  You are probably going to need virConnectGetDomainCapabilities() and virConnectBaselineHypervisorCPU() to generate a CPU configuration that can run on all nodes in a given cluster.

Maybe you can implement something equivalent using CPUNodeDiscovery features, but I don't know much about it.  Having something that is guaranteed to not lose any information from the domain capabilities XML would be ideal.

Comment 20 Jiri Denemark 2019-11-19 09:18:41 UTC
To summarize and highlight what Martin and Eduardo already said above,
host-model is better for migration than host-passthrough because migration
will not be allowed if the destination host cannot provide the same virtual
CPU which the VM is running with on the source. Its purpose is not making
migration between heterogeneous hosts easier or always working, it will just
make such migration safe by guaranteeing the virtual CPU will not change.

To make migration between heterogeneous always work, you need to gather
host-model CPUs from all hosts (virConnectGetDomainCapabilities, the returned
XML shows the CPU definition which will be used for starting a VM with
host-model CPU) and generate the best CPU definition which will work on all
hosts (virConnectBaselineHypervisorCPU).

The exact CPU configuration a VM is running with can be fetched with
virDomainGetXMLDesc and you can use virConnectCompareHypervisorCPU on other
hosts to check whether whether a VM with such CPU definition can be started
there.

Comment 21 Petr Kotas 2020-01-07 10:59:43 UTC
Just to summarize I understand correctly what solution should be implemented: Disable the migration when host-model is used as for short term solution.

Is that right Stu?

Comment 22 sgott 2020-01-07 19:29:45 UTC
Correct.

Comment 23 Petr Kotas 2020-01-13 17:15:59 UTC
The hotfix is in the PR https://github.com/kubevirt/kubevirt/pull/2982
It disables the migration for host-model and cpu-passthrough.

As described previously, this is not long term solution.

Comment 24 Fabian Deutsch 2020-03-13 13:48:10 UTC
Let's do it as follows:

1. passthrough blocks migraion
2. host model should not (because this is helpful in homogenous clusters where perf is needed)

Comment 25 Petr Kotas 2020-03-17 05:41:14 UTC
I do believe the hostmodel in the current situation is still problematic.
The discussion settled on blocking with hostmodel and postponing for a proper solution.

The solution is done in a way that creates less damage.

Should I really change the existing pr?

Comment 26 Fabian Deutsch 2020-03-17 08:17:43 UTC
Replied on the pr.
We must lock passthrouugh, but should leave an option for model


Note You need to log in before you can comment on or make changes to this bug.