Description of problem: On bare metal environment VM can't be scheduled nodes are missing all cpu models labels. Karel finding: Node labeller is trying to run virt-launcher to get all supported cpus, it runs this command inside virt-launcher: libvirtd -d; chmod o+rw /dev/kvm; virsh domcapabilities --machine q35 --arch x86_64 --virttype kvm > /etc/kubernetes/node-feature-discovery/source.d/virsh_domcapabilities.xml; But command : virsh domcapabilities --machine q35 --arch x86_64 --virttype kvm fails with error: error: failed to get emulator capabilities error: invalid argument: unable to find any emulator to serve 'x86_64' architecture Version-Release number of selected component (if applicable): 2.4.0 Kubevirt: oc get kv -n openshift-cnv -o yaml | grep operatorVersion operatorVersion: v0.29.2 How reproducible: 100% SVVP setup Steps to Reproduce: Create VM and run it More info: Intel Node CPU capabilities: $ lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 256 On-line CPU(s) list: 0-255 Thread(s) per core: 2 Core(s) per socket: 16 Socket(s): 8 NUMA node(s): 8 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Platinum 8153 CPU @ 2.00GHz Stepping: 4 CPU MHz: 1000.010 BogoMIPS: 4000.00 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 1024K L3 cache: 22528K NUMA node0 CPU(s): 0-15,128-143 NUMA node1 CPU(s): 16-31,144-159 NUMA node2 CPU(s): 32-47,160-175 NUMA node3 CPU(s): 48-63,176-191 NUMA node4 CPU(s): 64-79,192-207 NUMA node5 CPU(s): 80-95,208-223 NUMA node6 CPU(s): 96-111,224-239 NUMA node7 CPU(s): 112-127,240-255 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp_epp pku ospke md_clear flush_l1d ===================== # oc describe vmi win2019-sut-intel Name: win2019-sut-intel Namespace: default Labels: kubevirt.io/nodeName=lenovo-sr950-01.lab.eng.pek2.redhat.com Annotations: kubevirt.io/latest-observed-api-version: v1alpha3 kubevirt.io/storage-observed-api-version: v1alpha3 API Version: kubevirt.io/v1alpha3 Kind: VirtualMachineInstance Metadata: Creation Timestamp: 2020-06-02T09:11:07Z Finalizers: foregroundDeleteVirtualMachine Generate Name: win2019-sut-intel Generation: 253 Managed Fields: API Version: kubevirt.io/v1alpha3 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:kubevirt.io/latest-observed-api-version: f:kubevirt.io/storage-observed-api-version: f:generateName: f:labels: .: f:kubevirt.io/nodeName: f:ownerReferences: f:spec: .: f:domain: .: f:clock: .: f:timer: .: f:hpet: .: f:present: f:hyperv: f:pit: .: f:tickPolicy: f:rtc: .: f:tickPolicy: f:utc: f:cpu: .: f:cores: f:sockets: f:threads: f:devices: .: f:disks: f:interfaces: f:features: .: f:acpi: f:apic: f:hyperv: .: f:relaxed: f:spinlocks: .: f:spinlocks: f:vapic: f:firmware: .: f:uuid: f:machine: .: f:type: f:resources: .: f:requests: .: f:memory: f:evictionStrategy: f:networks: f:nodeSelector: .: f:kubernetes.io/hostname: f:terminationGracePeriodSeconds: f:volumes: f:status: .: f:activePods: .: f:389c5d52-7238-4478-bcb1-8f256bbfd40f: f:conditions: f:guestOSInfo: f:nodeName: f:phase: f:qosClass: Manager: virt-controller Operation: Update Time: 2020-06-02T09:12:25Z API Version: kubevirt.io/v1alpha3 Fields Type: FieldsV1 fieldsV1: f:status: f:interfaces: f:migrationMethod: Manager: virt-handler Operation: Update Time: 2020-06-02T09:12:25Z Owner References: API Version: kubevirt.io/v1alpha3 Block Owner Deletion: true Controller: true Kind: VirtualMachine Name: win2019-sut-intel UID: c94850b0-ca90-4299-87bb-8688b91b8fe4 Resource Version: 924915 Self Link: /apis/kubevirt.io/v1alpha3/namespaces/default/virtualmachineinstances/win2019-sut-intel UID: 267ea34a-f290-45a3-9b60-632efb763561 Spec: Domain: Clock: Timer: Hpet: Present: false Hyperv: Present: true Pit: Present: true Tick Policy: delay Rtc: Present: true Tick Policy: catchup Utc: Cpu: Cores: 4 Sockets: 1 Threads: 1 Devices: Disks: Boot Order: 1 Disk: Bus: virtio Name: rootdisk Interfaces: Bridge: Model: virtio Name: pub-nic Bridge: Model: virtio Name: private-network Features: Acpi: Enabled: true Apic: Enabled: true Hyperv: Relaxed: Enabled: true Spinlocks: Enabled: true Spinlocks: 8191 Vapic: Enabled: true Firmware: Uuid: 1f41841f-f3f0-5c2c-b539-b294d33c33f1 Machine: Type: pc-q35-rhel8.2.0 Resources: Requests: Cpu: 100m Memory: 4Gi Eviction Strategy: LiveMigrate Networks: Name: pub-nic Pod: Multus: Network Name: private-network Name: private-network Node Selector: kubernetes.io/hostname: lenovo-sr950-01.lab.eng.pek2.redhat.com Termination Grace Period Seconds: 0 Volumes: Name: rootdisk Persistent Volume Claim: Claim Name: dv-sut-intel Status: Active Pods: 389c5d52-7238-4478-bcb1-8f256bbfd40f: lenovo-sr950-01.lab.eng.pek2.redhat.com Conditions: Last Probe Time: <nil> Last Transition Time: <nil> Message: cannot migrate VMI with non-shared PVCs Reason: DisksNotLiveMigratable Status: False Type: LiveMigratable Last Probe Time: <nil> Last Transition Time: <nil> Message: cannot migrate VMI which does not use masquerade to connect to the pod network Reason: InterfaceNotLiveMigratable Status: False Type: LiveMigratable Guest OS Info: Interfaces: Ip Address: 10.128.4.55 Mac: 52:54:00:b2:e1:15 Name: pub-nic Mac: 52:54:00:69:ea:b7 Name: private-network Migration Method: BlockMigration Node Name: lenovo-sr950-01.lab.eng.pek2.redhat.com Phase: Scheduled Qos Class: Burstable Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulCreate 78s disruptionbudget-controller Created PodDisruptionBudget kubevirt-disruption-budget-rfthq Normal SuccessfulCreate 78s virtualmachine-controller Created virtual machine pod virt-launcher-win2019-sut-intel-trm8j Warning SyncFailed 68s virt-handler, lenovo-sr950-01.lab.eng.pek2.redhat.com server error. command SyncVMI failed: "LibvirtError(Code=1, Domain=10, Message='internal error: qemu unexpectedly closed the monitor: 2020-06-02T09:11:17.212388Z qemu-kvm: error: failed to set MSR 0x48e to 0xfff9fffe04006172\nqemu-kvm: /builddir/build/BUILD/qemu-4.2.0/target/i386/kvm.c:2695: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.')" Warning SyncFailed 67s virt-handler, lenovo-sr950-01.lab.eng.pek2.redhat.com server error. command SyncVMI failed: "LibvirtError(Code=1, Domain=10, Message='internal error: qemu unexpectedly closed the monitor: 2020-06-02T09:11:18.238703Z qemu-kvm: error: failed to set MSR 0x48e to 0xfff9fffe04006172\nqemu-kvm: /builddir/build/BUILD/qemu-4.2.0/target/i386/kvm.c:2695: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.')" Warning SyncFailed 66s virt-handler, lenovo-sr950-01.lab.eng.pek2.redhat.com server error. command SyncVMI failed: "LibvirtError(Code=1, Domain=10, Message='internal error: qemu unexpectedly closed the monitor: 2020-06-02T09:11:19.233513Z qemu-kvm: error: failed to set MSR 0x48e to 0xfff9fffe04006172\nqemu-kvm: /builddir/build/BUILD/qemu-4.2.0/target/i386/kvm.c:2695: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.')" Warning SyncFailed 65s virt-handler, lenovo-sr950-01.lab.eng.pek2.redhat.com server error. command SyncVMI failed: "LibvirtError(Code=1, Domain=10, Message='internal error: qemu unexpectedly closed the monitor: 2020-06-02T09:11:20.188028Z qemu-kvm: error: failed to set MSR 0x48e to 0xfff9fffe04006172\nqemu-kvm: /builddir/build/BUILD/qemu-4.2.0/target/i386/kvm.c:2695: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.')" Warning SyncFailed 64s virt-handler, lenovo-sr950-01.lab.eng.pek2.redhat.com server error. command SyncVMI failed: "LibvirtError(Code=1, Domain=10, Message='internal error: qemu unexpectedly closed the monitor: 2020-06-02T09:11:21.117444Z qemu-kvm: error: failed to set MSR 0x48e to 0xfff9fffe04006172\nqemu-kvm: /builddir/build/BUILD/qemu-4.2.0/target/i386/kvm.c:2695: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.')" Warning SyncFailed 63s virt-handler, lenovo-sr950-01.lab.eng.pek2.redhat.com server error. command SyncVMI failed: "LibvirtError(Code=1, Domain=10, Message='internal error: qemu unexpectedly closed the monitor: 2020-06-02T09:11:22.100088Z qemu-kvm: error: failed to set MSR 0x48e to 0xfff9fffe04006172\nqemu-kvm: /builddir/build/BUILD/qemu-4.2.0/target/i386/kvm.c:2695: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.')" Warning SyncFailed 62s virt-handler, lenovo-sr950-01.lab.eng.pek2.redhat.com server error. command SyncVMI failed: "LibvirtError(Code=1, Domain=10, Message='internal error: qemu unexpectedly closed the monitor: 2020-06-02T09:11:23.145973Z qemu-kvm: error: failed to set MSR 0x48e to 0xfff9fffe04006172\nqemu-kvm: /builddir/build/BUILD/qemu-4.2.0/target/i386/kvm.c:2695: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.')" Warning SyncFailed 61s virt-handler, lenovo-sr950-01.lab.eng.pek2.redhat.com server error. command SyncVMI failed: "LibvirtError(Code=1, Domain=10, Message='internal error: qemu unexpectedly closed the monitor: 2020-06-02T09:11:24.116708Z qemu-kvm: error: failed to set MSR 0x48e to 0xfff9fffe04006172\nqemu-kvm: /builddir/build/BUILD/qemu-4.2.0/target/i386/kvm.c:2695: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.')" Warning SyncFailed 60s virt-handler, lenovo-sr950-01.lab.eng.pek2.redhat.com server error. command SyncVMI failed: "LibvirtError(Code=1, Domain=10, Message='internal error: qemu unexpectedly closed the monitor: 2020-06-02T09:11:25.073723Z qemu-kvm: error: failed to set MSR 0x48e to 0xfff9fffe04006172\nqemu-kvm: /builddir/build/BUILD/qemu-4.2.0/target/i386/kvm.c:2695: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.')" Warning SyncFailed 44s (x16 over 59s) virt-handler, lenovo-sr950-01.lab.eng.pek2.redhat.com (combined from similar events): server error. command SyncVMI failed: "LibvirtError(Code=1, Domain=10, Message='internal error: qemu unexpectedly closed the monitor: 2020-06-02T09:11:40.645250Z qemu-kvm: error: failed to set MSR 0x48e to 0xfff9fffe04006172\nqemu-kvm: /builddir/build/BUILD/qemu-4.2.0/target/i386/kvm.c:2695: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.')"
I still see that the issue with # oc get kv -n openshift-cnv -o yaml | grep operatorVersion f:operatorVersion: {} operatorVersion: v0.30.0 Warning SyncFailed 19s (x16 over 33s) virt-handler, XXXXXX.redhat.com (combined from similar events): server error. command SyncVMI failed: "LibvirtErro r(Code=1, Domain=10, Message='internal error: process exited while connecting to monitor: 2020-06-12T02:56:34.619962Z qemu-kvm: error: failed to set MSR 0x48e to 0xfff9fffe04006172\nqemu-kvm : /builddir/build/BUILD/qemu-4.2.0/target/i386/kvm.c:2695: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.')" The libvirt did not update on the virt-launcher still 6.0.0-17, [root@vm-simple-rhel7-2cpu /]# rpm -qa | grep libvirt libvirt-libs-6.0.0-17.3.module+el8.2.0+6907+6abdb1b6.x86_64 libvirt-bash-completion-6.0.0-17.3.module+el8.2.0+6907+6abdb1b6.x86_64 libvirt-daemon-driver-qemu-6.0.0-17.3.module+el8.2.0+6907+6abdb1b6.x86_64 libvirt-daemon-6.0.0-17.3.module+el8.2.0+6907+6abdb1b6.x86_64 libvirt-client-6.0.0-17.3.module+el8.2.0+6907+6abdb1b6.x86_64 libvirt-daemon-driver-storage-core-6.0.0-17.3.module+el8.2.0+6907+6abdb1b6.x86_64 Failed QA
Recheck the labels on node i see that we have node labeller working now. From oc describe node Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux cpumanager=false feature.node.kubernetes.io/cpu-feature-3dnowprefetch=true feature.node.kubernetes.io/cpu-feature-abm=true feature.node.kubernetes.io/cpu-feature-adx=true feature.node.kubernetes.io/cpu-feature-aes=true feature.node.kubernetes.io/cpu-feature-arat=true feature.node.kubernetes.io/cpu-feature-avx=true feature.node.kubernetes.io/cpu-feature-avx2=true feature.node.kubernetes.io/cpu-feature-avx512bw=true feature.node.kubernetes.io/cpu-feature-avx512cd=true feature.node.kubernetes.io/cpu-feature-avx512dq=true feature.node.kubernetes.io/cpu-feature-avx512f=true feature.node.kubernetes.io/cpu-feature-avx512vl=true feature.node.kubernetes.io/cpu-feature-avx512vnni=true feature.node.kubernetes.io/cpu-feature-bmi1=true feature.node.kubernetes.io/cpu-feature-bmi2=true feature.node.kubernetes.io/cpu-feature-clflushopt=true feature.node.kubernetes.io/cpu-feature-clwb=true feature.node.kubernetes.io/cpu-feature-erms=true feature.node.kubernetes.io/cpu-feature-f16c=true feature.node.kubernetes.io/cpu-feature-fma=true feature.node.kubernetes.io/cpu-feature-fsgsbase=true feature.node.kubernetes.io/cpu-feature-hle=true feature.node.kubernetes.io/cpu-feature-invpcid=true feature.node.kubernetes.io/cpu-feature-movbe=true feature.node.kubernetes.io/cpu-feature-mpx=true feature.node.kubernetes.io/cpu-feature-pcid=true feature.node.kubernetes.io/cpu-feature-pclmuldq=true feature.node.kubernetes.io/cpu-feature-pdpe1gb=true feature.node.kubernetes.io/cpu-feature-popcnt=true feature.node.kubernetes.io/cpu-feature-rdrand=true feature.node.kubernetes.io/cpu-feature-rdseed=true feature.node.kubernetes.io/cpu-feature-rdtscp=true feature.node.kubernetes.io/cpu-feature-rtm=true feature.node.kubernetes.io/cpu-feature-smap=true feature.node.kubernetes.io/cpu-feature-smep=true feature.node.kubernetes.io/cpu-feature-spec-ctrl=true feature.node.kubernetes.io/cpu-feature-ssbd=true feature.node.kubernetes.io/cpu-feature-sse4.2=true feature.node.kubernetes.io/cpu-feature-svm=true feature.node.kubernetes.io/cpu-feature-tsc-deadline=true feature.node.kubernetes.io/cpu-feature-vme=true feature.node.kubernetes.io/cpu-feature-x2apic=true feature.node.kubernetes.io/cpu-feature-xgetbv1=true feature.node.kubernetes.io/cpu-feature-xsave=true feature.node.kubernetes.io/cpu-feature-xsavec=true feature.node.kubernetes.io/cpu-feature-xsaveopt=true feature.node.kubernetes.io/cpu-model-Broadwell=true feature.node.kubernetes.io/cpu-model-Broadwell-IBRS=true feature.node.kubernetes.io/cpu-model-Broadwell-noTSX=true feature.node.kubernetes.io/cpu-model-Broadwell-noTSX-IBRS=true feature.node.kubernetes.io/cpu-model-Cascadelake-Server=true feature.node.kubernetes.io/cpu-model-Haswell=true feature.node.kubernetes.io/cpu-model-Haswell-IBRS=true feature.node.kubernetes.io/cpu-model-Haswell-noTSX=true feature.node.kubernetes.io/cpu-model-Haswell-noTSX-IBRS=true feature.node.kubernetes.io/cpu-model-IvyBridge=true feature.node.kubernetes.io/cpu-model-IvyBridge-IBRS=true feature.node.kubernetes.io/cpu-model-Nehalem=true feature.node.kubernetes.io/cpu-model-Nehalem-IBRS=true feature.node.kubernetes.io/cpu-model-Opteron_G1=true feature.node.kubernetes.io/cpu-model-Opteron_G2=true feature.node.kubernetes.io/cpu-model-Penryn=true feature.node.kubernetes.io/cpu-model-SandyBridge=true feature.node.kubernetes.io/cpu-model-SandyBridge-IBRS=true feature.node.kubernetes.io/cpu-model-Skylake-Client=true feature.node.kubernetes.io/cpu-model-Skylake-Client-IBRS=true feature.node.kubernetes.io/cpu-model-Skylake-Server=true feature.node.kubernetes.io/cpu-model-Skylake-Server-IBRS=true feature.node.kubernetes.io/cpu-model-Westmere=true feature.node.kubernetes.io/cpu-model-Westmere-IBRS=true feature.node.kubernetes.io/cpu-model-kvm32=true feature.node.kubernetes.io/cpu-model-kvm64=true feature.node.kubernetes.io/cpu-model-qemu32=true feature.node.kubernetes.io/cpu-model-qemu64=true feature.node.kubernetes.io/kvm-info-cap-hyperv-base=true feature.node.kubernetes.io/kvm-info-cap-hyperv-frequencies=true feature.node.kubernetes.io/kvm-info-cap-hyperv-ipi=true feature.node.kubernetes.io/kvm-info-cap-hyperv-reenlightenment=true feature.node.kubernetes.io/kvm-info-cap-hyperv-reset=true feature.node.kubernetes.io/kvm-info-cap-hyperv-runtime=true feature.node.kubernetes.io/kvm-info-cap-hyperv-synic=true feature.node.kubernetes.io/kvm-info-cap-hyperv-synic2=true feature.node.kubernetes.io/kvm-info-cap-hyperv-synictimer=true feature.node.kubernetes.io/kvm-info-cap-hyperv-time=true feature.node.kubernetes.io/kvm-info-cap-hyperv-tlbflush=true feature.node.kubernetes.io/kvm-info-cap-hyperv-vpindex=true kubernetes.io/arch=amd64 kubernetes.io/hostname=cnvqe-11.lab.eng.tlv2.redhat.com kubernetes.io/os=linux kubevirt.io/schedulable=true node-role.kubernetes.io/worker= node.openshift.io/os_id=rhcos Annotations: kubevirt.io/heartbeat: 2020-06-15T12:14:35Z machine.openshift.io/machine: openshift-machine-api/cnvcl2-worker-0-t5vfk machineconfiguration.openshift.io/currentConfig: rendered-worker-fbc789a04d142cce741e1a0cc33e167a machineconfiguration.openshift.io/desiredConfig: rendered-worker-fbc789a04d142cce741e1a0cc33e167a machineconfiguration.openshift.io/reason: machineconfiguration.openshift.io/state: Done node-labeller-feature.node.kubernetes.io/cpu-feature-3dnowprefetch: true node-labeller-feature.node.kubernetes.io/cpu-feature-abm: true node-labeller-feature.node.kubernetes.io/cpu-feature-adx: true node-labeller-feature.node.kubernetes.io/cpu-feature-aes: true node-labeller-feature.node.kubernetes.io/cpu-feature-arat: true node-labeller-feature.node.kubernetes.io/cpu-feature-avx: true node-labeller-feature.node.kubernetes.io/cpu-feature-avx2: true node-labeller-feature.node.kubernetes.io/cpu-feature-avx512bw: true node-labeller-feature.node.kubernetes.io/cpu-feature-avx512cd: true node-labeller-feature.node.kubernetes.io/cpu-feature-avx512dq: true node-labeller-feature.node.kubernetes.io/cpu-feature-avx512f: true node-labeller-feature.node.kubernetes.io/cpu-feature-avx512vl: true node-labeller-feature.node.kubernetes.io/cpu-feature-avx512vnni: true node-labeller-feature.node.kubernetes.io/cpu-feature-bmi1: true node-labeller-feature.node.kubernetes.io/cpu-feature-bmi2: true node-labeller-feature.node.kubernetes.io/cpu-feature-clflushopt: true node-labeller-feature.node.kubernetes.io/cpu-feature-clwb: true node-labeller-feature.node.kubernetes.io/cpu-feature-erms: true node-labeller-feature.node.kubernetes.io/cpu-feature-f16c: true node-labeller-feature.node.kubernetes.io/cpu-feature-fma: true node-labeller-feature.node.kubernetes.io/cpu-feature-fsgsbase: true node-labeller-feature.node.kubernetes.io/cpu-feature-hle: true node-labeller-feature.node.kubernetes.io/cpu-feature-invpcid: true node-labeller-feature.node.kubernetes.io/cpu-feature-movbe: true node-labeller-feature.node.kubernetes.io/cpu-feature-mpx: true node-labeller-feature.node.kubernetes.io/cpu-feature-pcid: true node-labeller-feature.node.kubernetes.io/cpu-feature-pclmuldq: true node-labeller-feature.node.kubernetes.io/cpu-feature-pdpe1gb: true node-labeller-feature.node.kubernetes.io/cpu-feature-popcnt: true node-labeller-feature.node.kubernetes.io/cpu-feature-rdrand: true node-labeller-feature.node.kubernetes.io/cpu-feature-rdseed: true node-labeller-feature.node.kubernetes.io/cpu-feature-rdtscp: true node-labeller-feature.node.kubernetes.io/cpu-feature-rtm: true node-labeller-feature.node.kubernetes.io/cpu-feature-smap: true node-labeller-feature.node.kubernetes.io/cpu-feature-smep: true node-labeller-feature.node.kubernetes.io/cpu-feature-spec-ctrl: true node-labeller-feature.node.kubernetes.io/cpu-feature-ssbd: true node-labeller-feature.node.kubernetes.io/cpu-feature-sse4.2: true node-labeller-feature.node.kubernetes.io/cpu-feature-svm: true node-labeller-feature.node.kubernetes.io/cpu-feature-tsc-deadline: true node-labeller-feature.node.kubernetes.io/cpu-feature-vme: true node-labeller-feature.node.kubernetes.io/cpu-feature-x2apic: true node-labeller-feature.node.kubernetes.io/cpu-feature-xgetbv1: true node-labeller-feature.node.kubernetes.io/cpu-feature-xsave: true node-labeller-feature.node.kubernetes.io/cpu-feature-xsavec: true node-labeller-feature.node.kubernetes.io/cpu-feature-xsaveopt: true node-labeller-feature.node.kubernetes.io/cpu-model-Broadwell: true node-labeller-feature.node.kubernetes.io/cpu-model-Broadwell-IBRS: true node-labeller-feature.node.kubernetes.io/cpu-model-Broadwell-noTSX: true node-labeller-feature.node.kubernetes.io/cpu-model-Broadwell-noTSX-IBRS: true node-labeller-feature.node.kubernetes.io/cpu-model-Cascadelake-Server: true node-labeller-feature.node.kubernetes.io/cpu-model-Haswell: true node-labeller-feature.node.kubernetes.io/cpu-model-Haswell-IBRS: true node-labeller-feature.node.kubernetes.io/cpu-model-Haswell-noTSX: true node-labeller-feature.node.kubernetes.io/cpu-model-Haswell-noTSX-IBRS: true node-labeller-feature.node.kubernetes.io/cpu-model-IvyBridge: true node-labeller-feature.node.kubernetes.io/cpu-model-IvyBridge-IBRS: true node-labeller-feature.node.kubernetes.io/cpu-model-Nehalem: true node-labeller-feature.node.kubernetes.io/cpu-model-Nehalem-IBRS: true node-labeller-feature.node.kubernetes.io/cpu-model-Opteron_G1: true node-labeller-feature.node.kubernetes.io/cpu-model-Opteron_G2: true node-labeller-feature.node.kubernetes.io/cpu-model-Penryn: true node-labeller-feature.node.kubernetes.io/cpu-model-SandyBridge: true node-labeller-feature.node.kubernetes.io/cpu-model-SandyBridge-IBRS: true node-labeller-feature.node.kubernetes.io/cpu-model-Skylake-Client: true node-labeller-feature.node.kubernetes.io/cpu-model-Skylake-Client-IBRS: true node-labeller-feature.node.kubernetes.io/cpu-model-Skylake-Server: true node-labeller-feature.node.kubernetes.io/cpu-model-Skylake-Server-IBRS: true node-labeller-feature.node.kubernetes.io/cpu-model-Westmere: true node-labeller-feature.node.kubernetes.io/cpu-model-Westmere-IBRS: true node-labeller-feature.node.kubernetes.io/cpu-model-kvm32: true node-labeller-feature.node.kubernetes.io/cpu-model-kvm64: true node-labeller-feature.node.kubernetes.io/cpu-model-qemu32: true node-labeller-feature.node.kubernetes.io/cpu-model-qemu64: true node-labeller-feature.node.kubernetes.io/kvm-info-cap-hyperv-base: true node-labeller-feature.node.kubernetes.io/kvm-info-cap-hyperv-frequencies: true node-labeller-feature.node.kubernetes.io/kvm-info-cap-hyperv-ipi: true node-labeller-feature.node.kubernetes.io/kvm-info-cap-hyperv-reenlightenment: true node-labeller-feature.node.kubernetes.io/kvm-info-cap-hyperv-reset: true node-labeller-feature.node.kubernetes.io/kvm-info-cap-hyperv-runtime: true node-labeller-feature.node.kubernetes.io/kvm-info-cap-hyperv-synic: true node-labeller-feature.node.kubernetes.io/kvm-info-cap-hyperv-synic2: true node-labeller-feature.node.kubernetes.io/kvm-info-cap-hyperv-synictimer: true node-labeller-feature.node.kubernetes.io/kvm-info-cap-hyperv-time: true node-labeller-feature.node.kubernetes.io/kvm-info-cap-hyperv-tlbflush: true node-labeller-feature.node.kubernetes.io/kvm-info-cap-hyperv-vpindex: true volumes.kubernetes.io/controller-managed-attach-detach: true Moving back to on_qe , need to test with VM on BM.
It still does not work on my svvp env.(BM) After specify cpu mode, vmi always in Scheduling status: # oc describe node lenovo-sr950-01.lab.eng.pek2.redhat.com Name: lenovo-sr950-01.lab.eng.pek2.redhat.com Roles: worker Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux cpumanager=false kubernetes.io/arch=amd64 kubernetes.io/hostname=lenovo-sr950-01.lab.eng.pek2.redhat.com kubernetes.io/os=linux kubevirt.io/schedulable=true node-role.kubernetes.io/worker= node.openshift.io/os_id=rhel Annotations: kubevirt.io/heartbeat: 2020-06-17T05:39:20Z machineconfiguration.openshift.io/currentConfig: rendered-worker-0037cd49eb88c67f4dbf5c584dfc90b3 machineconfiguration.openshift.io/desiredConfig: rendered-worker-0037cd49eb88c67f4dbf5c584dfc90b3 machineconfiguration.openshift.io/reason: machineconfiguration.openshift.io/ssh: accessed machineconfiguration.openshift.io/state: Done volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Wed, 17 Jun 2020 00:06:23 -0400 ... # oc describe pod virt-launcher-win2019-sut-intel-pssd9 ... Node-Selectors: feature.node.kubernetes.io/cpu-model-Skylake-Server-IBRS=true kubernetes.io/hostname=lenovo-sr950-01.lab.eng.pek2.redhat.com kubevirt.io/schedulable=true Tolerations: node.kubernetes.io/memory-pressure:NoSchedule node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 85s (x18 over 25m) default-scheduler 0/6 nodes are available: 1 node(s) didn't match node selector, 3 Insufficient bridge.network.kubevirt.io/switch1, 5 Insufficient memory. # oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.5.0-0.nightly-2020-06-16-045437 True False 123m Cluster version is 4.5.0-0.nightly-2020-06-16-045437 # oc get csv -A NAMESPACE NAME DISPLAY VERSION REPLACES PHASE openshift-cnv kubevirt-hyperconverged-operator.v2.4.0 OpenShift virtualization 2.4.0 Succeeded Remove cpu model from vm yaml files, vmi can be started correly
No node-labeller pod was running: # oc get pods --all-namespaces | grep labeller # oc logs -n openshift-cnv kubevirt-ssp-operator-6957bf4d48-h8629 ... {"level":"info","ts":1592368618.081518,"logger":"cmd","msg":"Could not generate and serve custom resource metrics","Namespace":"","error":"discovering resource information failed for KubevirtNodeLabellerBundle in ssp.kubevirt.io/v1: unable to retrieve the complete list of server APIs: upload.cdi.kubevirt.io/v1alpha1: the server is currently unable to handle the request"} After follow Karel's suggestion: delete ssp pod and rebuild, it works normally, vmi is running. # oc delete pod kubevirt-ssp-operator-6957bf4d48-h8629 -n openshift-cnv # oc get pods -n openshift-cnv | grep labeller kubevirt-node-labeller-2ntvd 1/1 Running 0 11m kubevirt-node-labeller-8xs9m 1/1 Running 0 11m kubevirt-node-labeller-phkzw 1/1 Running 0 11m kubevirt-node-labeller-rzsjj 1/1 Running 0 11m kubevirt-node-labeller-tcdvb 1/1 Running 0 11m kubevirt-node-labeller-xn4mr 1/1 Running 0 11m # oc describe node lenovo-sr950-01.lab.eng.pek2.redhat.com Name: lenovo-sr950-01.lab.eng.pek2.redhat.com Roles: worker Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux cpumanager=false feature.node.kubernetes.io/cpu-feature-3dnowprefetch=true feature.node.kubernetes.io/cpu-feature-abm=true feature.node.kubernetes.io/cpu-feature-adx=true feature.node.kubernetes.io/cpu-feature-aes=true feature.node.kubernetes.io/cpu-feature-arat=true feature.node.kubernetes.io/cpu-feature-avx=true feature.node.kubernetes.io/cpu-feature-avx2=true feature.node.kubernetes.io/cpu-feature-avx512bw=true feature.node.kubernetes.io/cpu-feature-avx512cd=true feature.node.kubernetes.io/cpu-feature-avx512dq=true feature.node.kubernetes.io/cpu-feature-avx512f=true feature.node.kubernetes.io/cpu-feature-avx512vl=true feature.node.kubernetes.io/cpu-feature-bmi1=true feature.node.kubernetes.io/cpu-feature-bmi2=true feature.node.kubernetes.io/cpu-feature-clwb=true feature.node.kubernetes.io/cpu-feature-erms=true feature.node.kubernetes.io/cpu-feature-f16c=true feature.node.kubernetes.io/cpu-feature-fma=true feature.node.kubernetes.io/cpu-feature-fsgsbase=true feature.node.kubernetes.io/cpu-feature-hle=true feature.node.kubernetes.io/cpu-feature-invpcid=true feature.node.kubernetes.io/cpu-feature-movbe=true feature.node.kubernetes.io/cpu-feature-mpx=true feature.node.kubernetes.io/cpu-feature-pcid=true feature.node.kubernetes.io/cpu-feature-pclmuldq=true feature.node.kubernetes.io/cpu-feature-pdpe1gb=true feature.node.kubernetes.io/cpu-feature-popcnt=true feature.node.kubernetes.io/cpu-feature-rdrand=true feature.node.kubernetes.io/cpu-feature-rdseed=true feature.node.kubernetes.io/cpu-feature-rdtscp=true feature.node.kubernetes.io/cpu-feature-rtm=true feature.node.kubernetes.io/cpu-feature-smap=true feature.node.kubernetes.io/cpu-feature-smep=true feature.node.kubernetes.io/cpu-feature-spec-ctrl=true feature.node.kubernetes.io/cpu-feature-sse4.2=true feature.node.kubernetes.io/cpu-feature-svm=true feature.node.kubernetes.io/cpu-feature-tsc-deadline=true feature.node.kubernetes.io/cpu-feature-vme=true feature.node.kubernetes.io/cpu-feature-x2apic=true feature.node.kubernetes.io/cpu-feature-xgetbv1=true feature.node.kubernetes.io/cpu-feature-xsave=true feature.node.kubernetes.io/cpu-feature-xsavec=true feature.node.kubernetes.io/cpu-feature-xsaveopt=true feature.node.kubernetes.io/cpu-model-Broadwell=true feature.node.kubernetes.io/cpu-model-Broadwell-IBRS=true feature.node.kubernetes.io/cpu-model-Broadwell-noTSX=true feature.node.kubernetes.io/cpu-model-Broadwell-noTSX-IBRS=true feature.node.kubernetes.io/cpu-model-Haswell=true feature.node.kubernetes.io/cpu-model-Haswell-IBRS=true feature.node.kubernetes.io/cpu-model-Haswell-noTSX=true feature.node.kubernetes.io/cpu-model-Haswell-noTSX-IBRS=true feature.node.kubernetes.io/cpu-model-IvyBridge=true feature.node.kubernetes.io/cpu-model-IvyBridge-IBRS=true feature.node.kubernetes.io/cpu-model-Nehalem=true feature.node.kubernetes.io/cpu-model-Nehalem-IBRS=true feature.node.kubernetes.io/cpu-model-Opteron_G1=true feature.node.kubernetes.io/cpu-model-Opteron_G2=true feature.node.kubernetes.io/cpu-model-Penryn=true feature.node.kubernetes.io/cpu-model-SandyBridge=true feature.node.kubernetes.io/cpu-model-SandyBridge-IBRS=true feature.node.kubernetes.io/cpu-model-Skylake-Client=true feature.node.kubernetes.io/cpu-model-Skylake-Client-IBRS=true feature.node.kubernetes.io/cpu-model-Skylake-Server=true feature.node.kubernetes.io/cpu-model-Skylake-Server-IBRS=true feature.node.kubernetes.io/cpu-model-Westmere=true feature.node.kubernetes.io/cpu-model-Westmere-IBRS=true feature.node.kubernetes.io/cpu-model-kvm32=true feature.node.kubernetes.io/cpu-model-kvm64=true feature.node.kubernetes.io/cpu-model-qemu32=true feature.node.kubernetes.io/cpu-model-qemu64=true feature.node.kubernetes.io/kvm-info-cap-hyperv-base=true feature.node.kubernetes.io/kvm-info-cap-hyperv-frequencies=true feature.node.kubernetes.io/kvm-info-cap-hyperv-reset=true feature.node.kubernetes.io/kvm-info-cap-hyperv-runtime=true feature.node.kubernetes.io/kvm-info-cap-hyperv-synic=true feature.node.kubernetes.io/kvm-info-cap-hyperv-synic2=true feature.node.kubernetes.io/kvm-info-cap-hyperv-synictimer=true feature.node.kubernetes.io/kvm-info-cap-hyperv-time=true feature.node.kubernetes.io/kvm-info-cap-hyperv-vpindex=true kubernetes.io/arch=amd64 kubernetes.io/hostname=lenovo-sr950-01.lab.eng.pek2.redhat.com kubernetes.io/os=linux kubevirt.io/schedulable=true node-role.kubernetes.io/worker= node.openshift.io/os_id=rhel # oc get vmi NAME AGE PHASE IP NODENAME win2019-sut-intel 35m Running 10.129.2.26 lenovo-sr950-01.lab.eng.pek2.redhat.com Thanks Karel
Verify with: OCP 4.5 RC2 OS: Red Hat Enterprise Linux CoreOS 45.82.202006190229-0 (Ootpa) Kernel: 4.18.0-193.9.1.el8_2.x86_64 CRI-O 1.18.1-13.dev.rhaos4.5.git6d00f64.el8 Create VM VM is running.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:3194