Bug 1645674 - Migration failed if CPU type is not specified in VMI
Summary: Migration failed if CPU type is not specified in VMI
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: Virtualization
Version: 1.3
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 1.4
Assignee: Martin Sivák
QA Contact: Denys Shchedrivyi
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-11-02 19:47 UTC by Denys Shchedrivyi
Modified: 2019-02-26 13:24 UTC (History)
7 users (show)

Fixed In Version: kubevirt-0.13.0-1.g7b9cb66.f6a440c virt-api-container-v1.4.0-12 virt-handler-container-v1.4.0-12 virt-launcher-container-v1.4.0-12 virt-operator-container-v1.4.0-4 virt-api-container-v1.4.0-12
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-02-26 13:24:16 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github kubevirt/kubevirt/1721 0 None None None 2020-06-05 03:42:02 UTC
Red Hat Product Errata RHEA-2019:0417 0 None None None 2019-02-26 13:24:20 UTC

Description Denys Shchedrivyi 2018-11-02 19:47:24 UTC
Description of problem:

 VM can't be migrated if CPU model was not specified.

Version-Release number of selected component (if applicable):

CNV-1.3

How reproducible:

Steps to Reproduce:
1.  Create VM without "cpu.model" tag in yaml file
2.  Run migration 
3.

Actual results:

Migration fails, new pod has an "error" status:
# oc get pod virt-launcher-vmi-migratable-46vcp -o wide
NAME                                 READY     STATUS    RESTARTS   AGE       IP            NODE
virt-launcher-vmi-migratable-46vcp   1/2       Error     0          7m        10.130.0.86   cnv-executor-ipinto-node2.example.com


There is an error message in pod logs:
{"component":"virt-launcher","level":"error","msg":"operation failed: guest CPU doesn't match specification: missing features: mpx,avx512f,avx512dq,clflushopt,clwb,avx512cd,avx512bw,avx512vl,pku,spec-ctrl,xsavec,xgetbv1","pos":"virCPUx86UpdateLive:2755","subcomponent":"libvirt","timestamp":"2018-11-02T19:32:41.765000Z"}


Expected results:
 Migration completed, no CPU error messages in logs


Additional info:
 Workaround - specify the CPU model in VMI, for example:

spec:
  domain:
    cpu:
      model: Nehalem

 In this case new pod created well and migration completed.

Comment 2 Fabian Deutsch 2018-11-05 10:59:36 UTC
Please provide kubectl get -o yaml vmi …

Comment 4 Denys Shchedrivyi 2018-11-05 17:08:28 UTC
# kubectl get -o yaml vmi vmi-migratable
apiVersion: kubevirt.io/v1alpha2
kind: VirtualMachineInstance
metadata:
  creationTimestamp: 2018-11-05T16:20:16Z
  finalizers:
  - foregroundDeleteVirtualMachine
  generation: 1
  labels:
    kubevirt.io/migrationTargetNodeName: cnv-executor-ipinto-node2.example.com
    kubevirt.io/nodeName: cnv-executor-ipinto-node1.example.com
    special: vmi-migratable
  name: vmi-migratable
  namespace: default
  resourceVersion: "4079601"
  selfLink: /apis/kubevirt.io/v1alpha2/namespaces/default/virtualmachineinstances/vmi-migratable
  uid: ae839663-e116-11e8-99f9-fa163ecc89c7
spec:
  domain:
    devices:
      disks:
      - disk:
          bus: virtio
        name: registrydisk
        volumeName: registryvolume
      interfaces:
      - bridge: {}
        name: default
    features:
      acpi:
        enabled: true
    firmware:
      uuid: 6e3b0e5b-4c9d-4dcb-a21a-89f05824b316
    machine:
      type: q35
    resources:
      requests:
        memory: 64M
  networks:
  - name: default
    pod: {}
  terminationGracePeriodSeconds: 0
  volumes:
  - name: registryvolume
    registryDisk:
      image: kubevirt/cirros-registry-disk-demo:latest
status:
  interfaces:
  - ipAddress: 10.129.0.185
  migrationState:
    migrationUid: e858c716-e117-11e8-99f9-fa163ecc89c7
    sourceNode: cnv-executor-ipinto-node1.example.com
    startTimestamp: 2018-11-05T16:29:18Z
    targetNode: cnv-executor-ipinto-node2.example.com
    targetNodeAddress: 10.130.0.48:43208
  nodeName: cnv-executor-ipinto-node1.example.com
  phase: Running

Comment 5 Martin Sivák 2018-11-27 13:29:08 UTC
Can you please attach the domxml of the VM too? Just so we know what CPU model was set by kubevirt as it might have defaulted to host-model for example.

Comment 6 Denys Shchedrivyi 2018-11-28 16:42:48 UTC
# cat /etc/libvirt/qemu/default_vmi-migratable.xml
<domain type='kvm'>
  <name>default_vmi-migratable</name>
  <uuid>0bcbe651-2a5f-4c91-8a74-11c037c23840</uuid>
  <metadata>
    <kubevirt xmlns="http://kubevirt.io">
      <uid>e81913cb-f32b-11e8-8e67-fa163e46d324</uid>
      <graceperiod>
        <deletionGracePeriodSeconds>0</deletionGracePeriodSeconds>
      </graceperiod>
    </kubevirt>
  </metadata>
  <memory unit='KiB'>62500</memory>
  <currentMemory unit='KiB'>62500</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <iothreads>1</iothreads>
  <sysinfo type='smbios'>
    <system>
      <entry name='uuid'>0bcbe651-2a5f-4c91-8a74-11c037c23840</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-q35-2.12'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
  </features>
  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/var/run/kubevirt-ephemeral-disks/registry-disk-data/default/vmi-migratable/disk_registryvolume/disk-image.raw'/>
      <target dev='vda' bus='virtio'/>
      <alias name='ua-registrydisk'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='none'/>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <interface type='bridge'>
      <mac address='0a:58:0a:81:00:2f'/>
      <source bridge='k6t-eth0'/>
      <model type='virtio'/>
      <alias name='ua-default'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='unix'>
      <source mode='bind' path='/var/run/kubevirt-private/e81913cb-f32b-11e8-8e67-fa163e46d324/virt-serial0'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='unix'>
      <source mode='bind' path='/var/run/kubevirt-private/e81913cb-f32b-11e8-8e67-fa163e46d324/virt-serial0'/>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' socket='/var/run/kubevirt-private/e81913cb-f32b-11e8-8e67-fa163e46d324/virt-vnc'>
      <listen type='socket' socket='/var/run/kubevirt-private/e81913cb-f32b-11e8-8e67-fa163e46d324/virt-vnc'/>
    </graphics>
    <video>
      <model type='vga' vram='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <memballoon model='none'/>
  </devices>
</domain>

Comment 7 Denys Shchedrivyi 2018-11-28 16:59:22 UTC
Quick research showed, that adding CPU model "Nehalem" to yaml file resolved the problem:

spec:
  domain:
    cpu:
      model: Nehalem

Comment 8 Martin Sivák 2018-11-28 23:29:56 UTC
So the default cpu mode='host-model' means the VM uses most of the features of the node it was started at. Migrating the VM to any lesser host will end up like in your report.

Model Nehalem solves that for you only because it is compatible with both of your hosts.

We are working on adding support for CPU model compatibility based scheduling and expect to have it soon: https://github.com/kubevirt/kubevirt/pull/1721

Comment 9 sgott 2019-01-23 15:37:51 UTC
Moving this to MODIFIED as PR 1721 was merged upstream before Releases 0.12 and 0.13 were branched.

Comment 10 Denys Shchedrivyi 2019-01-23 18:57:31 UTC
Verified on latest d/s build, fixed

Comment 13 errata-xmlrpc 2019-02-26 13:24:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:0417


Note You need to log in before you can comment on or make changes to this bug.