Bug 822148

Summary: libvirt should enforce guest CPU configuration
Product: Red Hat Enterprise Linux 7 Reporter: Eduardo Habkost <ehabkost>
Component: libvirtAssignee: Jiri Denemark <jdenemar>
Status: CLOSED ERRATA QA Contact: Luyao Huang <lhuang>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.0CC: cwei, dyuan, gsun, jdenemar, jmiao, juzhang, knoel, lhuang, mzhan, shuang, tburke, xfu, xuzhang
Target Milestone: rcKeywords: Upstream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-3.2.0-1.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 822613 (view as bug list) Environment:
Last Closed: 2017-08-01 17:06:41 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 822616, 824987, 824989, 1055532    
Bug Blocks: 1179458, 1199452, 1288337    
Attachments:
Description Flags
libvirtd log of S4
none
libvirtd log for S6 for debuging none

Description Eduardo Habkost 2012-05-16 13:47:08 UTC 
Description of problem:
To make sure libvirt is always getting exactly the CPU flags it asked qemu for, it has to use "-cpu <model>,enforce", so qemu won't start if the requested flags can't be exposed.

This may be a problem in case users have existing VMs that are working perfectly for them, but would fail to start if "enforce" is used. Maybe the new behavior could be enabled only for new machine-types?
Comment 1 Martin Kletzander 2012-08-09 05:47:45 UTC 
*** Bug 822613 has been marked as a duplicate of this bug. ***
Comment 2 Eduardo Habkost 2013-01-16 19:26:03 UTC 
Important bugfix for "-cpu enforce" was included on qemu.git master today. See commit 07ca59450c9a0c5df65665ce46aa8487af59a1dd and other patches from the same series.

commit 07ca59450c9a0c5df65665ce46aa8487af59a1dd
Author: Eduardo Habkost <ehabkost>
Date:   Mon Jan 7 16:20:48 2013 -0200

    target-i386: check/enforce: Check all feature words

    This adds the following feature words to the list of flags to be checked
    by kvm_check_features_against_host():

     - cpuid_7_0_ebx_features
     - ext4_features
     - kvm_features
     - svm_features

    This will ensure the "enforce" flag works as it should: it won't allow
    QEMU to be started unless every flag that was requested by the user or
    defined in the CPU model is supported by the host.

    This patch may cause existing configurations where "enforce" wasn't
    preventing QEMU from being started to abort QEMU. But that's exactly the
    point of this patch: if a flag was not supported by the host and QEMU
    wasn't aborting, it was a bug in the "enforce" code.
Comment 3 Eduardo Habkost 2013-04-10 12:32:26 UTC 
See RFC at <http://article.gmane.org/gmane.comp.emulators.qemu/204336>. We need feedback from the libvirt team, so we can proceed on a solution that works for libvirt.
Comment 4 Eduardo Habkost 2013-05-14 17:16:55 UTC 
Reference for how libvirt could use the feature from bug 824987:
http://wiki.qemu.org/Features/CPUModels#Interfaces.2Frequirements_for_libvirt
Comment 5 Wayne Sun 2013-06-04 07:01:30 UTC 
pkgs:
libvirt-1.0.6-1.el7.x86_64
qemu-kvm-1.5.0-2.el7.x86_64

1. steps
# lscpu |grep Model\ name
Model name:            AMD Opteron(tm) Processor 6128

prepare a domain with following cpu model part:
...
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Opteron_G3</model>
    <feature policy='require' name='popcnt'/>
    <feature policy='require' name='sse4a'/>
    <feature policy='require' name='abm'/>
    <feature policy='optional' name='monitor'/>
    <feature policy='optional' name='rdtscp'/>
  </cpu>
...
2. start domain
# virsh start rhel7
Domain rhel7 started

# ps aux|grep qemu
qemu     15493 20.1  1.1 1851552 290696 ?      Sl   14:55   0:50 /usr/libexec/qemu-kvm -name rhel7 -S -machine pc-i440fx-1.4,accel=kvm,usb=off -cpu Opteron_G3 -m 1024 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid d45afe60-be16-4e88-a6f2-e40cdcea9136 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/rhel7.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -no-acpi -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x7 -drive file=/var/lib/libvirt/images/rhel72.img,if=none,id=drive-ide0-0-0,format=raw -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=26,id=hostnet0,vhost=on,vhostfd=27 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:48:88:34,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -spice port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on -vga qxl -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=67108864 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5

no 'enforce' added.

3. tested with qemu command
# /usr/libexec/qemu-kvm -name rhel7 -S -machine pc-i440fx-1.4,accel=kvm,usb=off -cpu Opteron_G3,-rdtscp,-monitor,enforce -m 1024 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1
VNC server running on `::1:5900'

it works
Comment 10 Luyao Huang 2015-08-11 01:26:09 UTC 
A test case from bug 1178850 comment 13:

1. test with host-passthrough cpu mode

  <cpu mode='host-passthrough'>
    <feature policy='require' name='tsc-deadline'/>
  </cpu>

# ps axu|grep qemu
...-cpu host,+tsc-deadline...

Howerver my host not support tsc-deadline:

# cpuid -1 |grep "time stamp counter"
time stamp counter deadline             = false

and this is not right from doc:

require
    Guest creation will fail unless the feature is supported by host CPU.

Test if if change the mode to 'host-model' will get error:

# virsh start test3
error: Failed to start domain test3
error: unsupported configuration: guest and host CPU are not compatible: Host CPU does not provide required features: tsc-deadline

And we won't check if the feature name is valid:

# virsh dumpxml test4 |grep -A 5 "cpu mode"
  <cpu mode='host-passthrough'>
    <feature policy='require' name='tsc-deadlineaaa'/>
    <numa>
      <cell id='0' cpus='0-1' memory='1024000' unit='KiB'/>
    </numa>
  </cpu>

# virsh start test4
error: Failed to start domain test4
error: internal error: process exited while connecting to monitor: 2015-08-11T01:23:31.988928Z qemu-kvm: warning: CPU(s) not present in any NUMA nodes: 2 3
2015-08-11T01:23:31.989130Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config
2015-08-11T01:23:31.990121Z qemu-kvm: CPU feature tsc-deadlineaaa not found

# virsh dumpxml test4 |grep -A 5 "cpu mode"
  <cpu mode='host-passthrough'>
    <feature policy='require' name='tsc-deadline,+invtsc'/>
    <numa>
      <cell id='0' cpus='0-1' memory='1024000' unit='KiB'/>
    </numa>
  </cpu>

# virsh start test4
Domain test4 started

And Martin suggest add a comment to this bug and we think this could be fixed (or easy to fix) in this bug.

Thanks,
Luyao
Comment 11 Radim Krčmář 2015-09-16 17:06:56 UTC 
*** Bug 1177204 has been marked as a duplicate of this bug. ***
Comment 14 Jiri Denemark 2017-03-14 16:59:43 UTC 
Patches sent upstream for review: https://www.redhat.com/archives/libvir-list/2017-March/msg00578.html
Comment 15 Jiri Denemark 2017-03-17 11:02:24 UTC 
Finally fixed upstream by:

commit cab2c1af96c4b446e7f0fb72809402302a2560be
Refs: v3.1.0-182-gcab2c1af9
Author:     Jiri Denemark <jdenemar>
AuthorDate: Wed Mar 1 14:54:53 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 17 11:50:47 2017 +0100

    tests: Switch to sparse initialization of virCPUDef

    Signed-off-by: Jiri Denemark <jdenemar>

commit 38567e94dea56eeafa487d754875cdc6a5e776d9
Refs: v3.1.0-183-g38567e94d
Author:     Jiri Denemark <jdenemar>
AuthorDate: Thu Mar 2 14:53:18 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 17 11:50:48 2017 +0100

    docs: Clarify /domain/cpu/@match description

    Signed-off-by: Jiri Denemark <jdenemar>

commit 641b8c721e01dd2a7d0cae7b7f3b7a06e43f9da2
Refs: v3.1.0-184-g641b8c721
Author:     Jiri Denemark <jdenemar>
AuthorDate: Wed Mar 1 15:18:22 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 17 11:50:48 2017 +0100

    Introduce /domain/cpu/@check XML attribute

    The attribute can be used to request a specific way of checking whether
    the virtual CPU matches created by the hypervisor matches the
    specification in domain XML.

    Signed-off-by: Jiri Denemark <jdenemar>

commit fcd56ce86640bf8bc230025a3791593a5bf10507
Refs: v3.1.0-185-gfcd56ce86
Author:     Jiri Denemark <jdenemar>
AuthorDate: Wed Mar 1 16:12:07 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 17 11:50:48 2017 +0100

    qemu: Set default values for CPU check attribute

    Signed-off-by: Jiri Denemark <jdenemar>

commit e9dbe70110ac2d6ad8264c1c404e5ca4069e0e7b
Refs: v3.1.0-186-ge9dbe7011
Author:     Jiri Denemark <jdenemar>
AuthorDate: Fri Mar 10 23:55:59 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 17 11:50:48 2017 +0100

    qemu: Refactor Hyper-V features check

    The checks are now in a dedicated qemuProcessVerifyHypervFeatures
    function.

    In addition to moving the code this patch also fixes a few bugs: the
    original code was leaking cpuFeature and the return value of
    virCPUDataCheckFeature was not checked properly.

    Signed-off-by: Jiri Denemark <jdenemar>

commit d5f47d7d75eefe6afd941f963515cafe865c7ab0
Refs: v3.1.0-187-gd5f47d7d7
Author:     Jiri Denemark <jdenemar>
AuthorDate: Sat Mar 11 00:10:56 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 17 11:50:48 2017 +0100

    qemu: Refactor KVM features check

    The checks are now in a dedicated qemuProcessVerifyKVMFeatures function.

    Signed-off-by: Jiri Denemark <jdenemar>

commit af1ca855454e026395ac1e6f515381ed2f5aa0c4
Refs: v3.1.0-188-gaf1ca8554
Author:     Jiri Denemark <jdenemar>
AuthorDate: Sat Mar 11 00:36:55 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 17 11:50:48 2017 +0100

    qemu: Refactor CPU features check

    The checks are now in a dedicated qemuProcessVerifyCPUFeatures function.

    Signed-off-by: Jiri Denemark <jdenemar>

commit cfeee3373bf2afefe7bbf958af6c00e672756f20
Refs: v3.1.0-189-gcfeee3373
Author:     Jiri Denemark <jdenemar>
AuthorDate: Sat Mar 11 00:45:37 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 17 11:50:48 2017 +0100

    qemu: Refactor qemuProcessVerifyGuestCPU

    Signed-off-by: Jiri Denemark <jdenemar>

commit 253db85e2d3da7e997dc4860c0009bdb7afbef4c
Refs: v3.1.0-190-g253db85e2
Author:     Jiri Denemark <jdenemar>
AuthorDate: Mon Mar 13 10:23:37 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 17 11:50:48 2017 +0100

    qemu: Use ARCH_IS_X86 in qemuMonitorJSONGetGuestCPU

    Signed-off-by: Jiri Denemark <jdenemar>

commit 77c9c4f127e983f67561fafcb257fb4c440bd28f
Refs: v3.1.0-191-g77c9c4f12
Author:     Jiri Denemark <jdenemar>
AuthorDate: Mon Mar 13 11:00:48 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 17 11:50:48 2017 +0100

    qemu: Ask QEMU for filtered CPU features

    qemuMonitorGetGuestCPU can now optionally create CPU data from
    filtered-features in addition to feature-words.

    Signed-off-by: Jiri Denemark <jdenemar>

commit def9401acb6f1b6a4eefbb8b5c84629811bc309d
Refs: v3.1.0-192-gdef9401ac
Author:     Jiri Denemark <jdenemar>
AuthorDate: Mon Mar 13 12:32:02 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 17 11:50:48 2017 +0100

    qemu: Update CPU definition according to QEMU

    When starting a domain with custom guest CPU specification QEMU may add
    or remove some CPU features. There are several reasons for this, e.g.,
    QEMU/KVM does not support some requested features or the definition of
    the requested CPU model in libvirt's cpu_map.xml differs from the one
    QEMU is using. We can't really avoid this because CPU models are allowed
    to change with machine types and libvirt doesn't know (and probably
    doesn't even want to know) about such changes.

    Thus when we want to make sure guest ABI doesn't change when a domain
    gets migrated to another host, we need to update our live CPU definition
    according to the CPU QEMU created. Once updated, we will change CPU
    checking to VIR_CPU_CHECK_FULL to make sure the virtual CPU created
    after migration exactly matches the one on the source.

    https://bugzilla.redhat.com/show_bug.cgi?id=822148
    https://bugzilla.redhat.com/show_bug.cgi?id=824989

    Signed-off-by: Jiri Denemark <jdenemar>

commit 77ca2f6d8f6221f2991c95d74740c8607cd48baf
Refs: v3.1.0-193-g77ca2f6d8
Author:     Jiri Denemark <jdenemar>
AuthorDate: Tue Mar 14 15:05:02 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 17 11:50:48 2017 +0100

    qemu: Enforce guest CPU specification

    When guest CPU definition uses VIR_CPU_CHECK_FULL checks, we need to
    make sure QEMU does not add or remove any features.

    https://bugzilla.redhat.com/show_bug.cgi?id=822148
    https://bugzilla.redhat.com/show_bug.cgi?id=824989

    Signed-off-by: Jiri Denemark <jdenemar>
Comment 16 Jiri Denemark 2017-03-20 18:09:55 UTC 
One more patch is needed to fix a possible crash:

commit c74207cb182f0af7ea626579f5be3be48fc9c2d4
Refs: v3.1.0-201-gc74207cb1
Author:     Jiri Denemark <jdenemar>
AuthorDate: Mon Mar 20 09:03:58 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Mon Mar 20 09:03:58 2017 +0100

    qemu: Don't try to update undefined guest CPU

    Calling virCPUUpdateLive on a domain with no guest CPU configuration
    does not make sense. Especially when doing so would crash libvirtd.

    Signed-off-by: Jiri Denemark <jdenemar>
Comment 18 Luyao Huang 2017-05-25 03:51:19 UTC 
Test with libvirt-3.2.0-6.el7.x86_64:

S1: Use domcapabilities check guest vcpu flags cap

1.

# virsh domcapabilities
<domainCapabilities>
...
  <cpu>
    <mode name='host-passthrough' supported='yes'/>
    <mode name='host-model' supported='yes'>
      <model fallback='forbid'>Opteron_G5</model>
      <vendor>AMD</vendor>
      <feature policy='require' name='vme'/>
      <feature policy='require' name='x2apic'/>
      <feature policy='require' name='tsc-deadline'/>
      <feature policy='require' name='hypervisor'/>
      <feature policy='require' name='arat'/>
      <feature policy='require' name='tsc_adjust'/>
      <feature policy='require' name='bmi1'/>
      <feature policy='require' name='mmxext'/>
      <feature policy='require' name='fxsr_opt'/>
      <feature policy='require' name='cmp_legacy'/>
      <feature policy='require' name='cr8legacy'/>
      <feature policy='require' name='osvw'/>
      <feature policy='require' name='invtsc'/>
      <feature policy='disable' name='rdtscp'/>
      <feature policy='disable' name='svm'/>
    </mode>
    <mode name='custom' supported='yes'>
      <model usable='yes'>qemu64</model>
      <model usable='yes'>qemu32</model>
      <model usable='no'>phenom</model>
      <model usable='yes'>pentium3</model>
      <model usable='yes'>pentium2</model>
      <model usable='yes'>pentium</model>
      <model usable='no'>n270</model>
      <model usable='yes'>kvm64</model>
      <model usable='yes'>kvm32</model>
      <model usable='yes'>cpu64-rhel6</model>
      <model usable='no'>coreduo</model>
      <model usable='no'>core2duo</model>
      <model usable='no'>athlon</model>
      <model usable='yes'>Westmere</model>
      <model usable='no'>Skylake-Client</model>
      <model usable='no'>SandyBridge</model>
      <model usable='yes'>Penryn</model>
      <model usable='yes'>Opteron_G5</model>
      <model usable='yes'>Opteron_G4</model>
      <model usable='yes'>Opteron_G3</model>
      <model usable='yes'>Opteron_G2</model>
      <model usable='yes'>Opteron_G1</model>
      <model usable='yes'>Nehalem</model>
      <model usable='no'>IvyBridge</model>
      <model usable='no'>Haswell</model>
      <model usable='no'>Haswell-noTSX</model>
      <model usable='yes'>Conroe</model>
      <model usable='no'>Broadwell</model>
      <model usable='no'>Broadwell-noTSX</model>
      <model usable='yes'>486</model>
    </mode>
  </cpu>
...

2. define and start a guest with cpu model xml in domcapabilities:

# virsh dumpxml r7
  <cpu mode='host-model' check='full'>
    <model fallback='forbid'>Opteron_G5</model>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='tsc-deadline'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='arat'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='bmi1'/>
    <feature policy='require' name='mmxext'/>
    <feature policy='require' name='fxsr_opt'/>
    <feature policy='require' name='cmp_legacy'/>
    <feature policy='require' name='cr8legacy'/>
    <feature policy='require' name='osvw'/>
    <feature policy='require' name='invtsc'/>
    <feature policy='disable' name='rdtscp'/>
    <feature policy='disable' name='svm'/>
    <numa>
      <cell id='0' cpus='0-2' memory='524288' unit='KiB'/>
      <cell id='1' cpus='3-5' memory='524288' unit='KiB'/>
    </numa>
  </cpu>

# virsh start r7
Domain r7 started

3. recheck xml:

# virsh dumpxml r7
...
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Opteron_G5</model>
    <vendor>AMD</vendor>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='tsc-deadline'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='arat'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='bmi1'/>
    <feature policy='require' name='mmxext'/>
    <feature policy='require' name='fxsr_opt'/>
    <feature policy='require' name='cmp_legacy'/>
    <feature policy='require' name='cr8legacy'/>
    <feature policy='require' name='osvw'/>
    <feature policy='disable' name='rdtscp'/>
    <feature policy='disable' name='svm'/>
    <feature policy='require' name='invtsc'/>
    <numa>
      <cell id='0' cpus='0-2' memory='524288' unit='KiB'/>
      <cell id='1' cpus='3-5' memory='524288' unit='KiB'/>
    </numa>
  </cpu>
...


S2: start a guest with host-model:

1. guest xml

  <cpu mode='host-model' check='full'>
    <model fallback='allow'/>
    <numa>
      <cell id='0' cpus='0-2' memory='524288' unit='KiB'/>
      <cell id='1' cpus='3-5' memory='524288' unit='KiB'/>
    </numa>
  </cpu>

2. start guest

# virsh start r7
Domain r7 started

3. recheck xml:

# virsh dumpxml r7
...
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Opteron_G5</model>
    <vendor>AMD</vendor>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='tsc-deadline'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='arat'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='bmi1'/>
    <feature policy='require' name='mmxext'/>
    <feature policy='require' name='fxsr_opt'/>
    <feature policy='require' name='cmp_legacy'/>
    <feature policy='require' name='cr8legacy'/>
    <feature policy='require' name='osvw'/>
    <feature policy='disable' name='rdtscp'/>
    <feature policy='disable' name='svm'/>
    <numa>
      <cell id='0' cpus='0-2' memory='524288' unit='KiB'/>
      <cell id='1' cpus='3-5' memory='524288' unit='KiB'/>
    </numa>
  </cpu>
...

4. check qemu command line
# ps aux|grep qemu
qemu     17388 20.3  1.4 2029600 460316 ?      Sl   23:26   0:25 .... -cpu Opteron_G5,vme=on,x2apic=on,tsc-deadline=on,hypervisor=on,arat=on,tsc_adjust=on,bmi1=on,mmxext=on,fxsr_opt=on,cmp_legacy=on,cr8legacy=on,osvw=on,rdtscp=off,svm=off

5. login guest check guest cpu flags:

# lscpu |grep Flags
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt pdpe1gb lm art rep_good nopl extd_apicid pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw xop fma4 tbm tsc_adjust bmi1 arat


S3: start a guest with host-model and a guest unsupported flags:

1. check is none (no check from libvirt side)

# virsh dumpxml r7 --inactive |grep -A5 host-model
  <cpu mode='host-model' check='none'>
    <model fallback='allow'/>
    <feature policy='require' name='svm'/>
    <numa>
      <cell id='0' cpus='0-2' memory='524288' unit='KiB'/>
      <cell id='1' cpus='3-5' memory='524288' unit='KiB'/>

# virsh start r7
Domain r7 started

# virsh dumpxml r7 |grep -A20 custom
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Opteron_G5</model>
    <vendor>AMD</vendor>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='tsc-deadline'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='arat'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='bmi1'/>
    <feature policy='require' name='mmxext'/>
    <feature policy='require' name='fxsr_opt'/>
    <feature policy='require' name='cmp_legacy'/>
    <feature policy='require' name='cr8legacy'/>
    <feature policy='require' name='osvw'/>
    <feature policy='disable' name='rdtscp'/>
    <feature policy='disable' name='svm'/>        <--- move to disable
    <numa>
      <cell id='0' cpus='0-2' memory='524288' unit='KiB'/>
      <cell id='1' cpus='3-5' memory='524288' unit='KiB'/>
    </numa>

# ps aux|grep qemu
qemu     17954 13.7  1.2 2021404 404260 ?      Sl   23:32   0:25 .... -cpu Opteron_G5,vme=on,x2apic=on,tsc-deadline=on,hypervisor=on,arat=on,tsc_adjust=on,bmi1=on,mmxext=on,fxsr_opt=on,cmp_legacy=on,cr8legacy=on,osvw=on,rdtscp=off,svm=on


2. check is partial (libvirt will check if feature name is supported by host):

# virsh dumpxml r7 |grep -A3 host-model
  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
    <feature policy='require' name='mpx'/>
    <numa>

# virsh capabilities |grep mpx

# virsh start r7
error: Failed to start domain r7
error: the CPU is incompatible with host CPU: Host CPU does not provide required features: mpx


3. check is full (guest fail to start if the vcpu flags not match xml):

# virsh dumpxml r7 |grep -A3 host-model
  <cpu mode='host-model' check='full'>
    <model fallback='allow'/>
    <feature policy='require' name='svm'/>
    <numa>

# virsh start r7
error: Failed to start domain r7
error: operation failed: guest CPU doesn't match specification: missing features: svm
Comment 19 Luyao Huang 2017-05-25 05:59:09 UTC 
Test with libvirt-3.2.0-6.el7.x86_64:

S4: host-passthrough + (check=full)

# virsh dumpxml r7 |grep -A3 host-
  <cpu mode='host-passthrough' check='full'>
    <feature policy='require' name='svms'/>   <---- not exist
    <numa>
      <cell id='0' cpus='0-2' memory='524288' unit='KiB'/>

# virsh start r7
error: Failed to start domain r7
error: internal error: qemu unexpectedly closed the monitor: 2017-05-25T03:58:44.618947Z qemu-kvm: -chardev pty,id=charserial0: char device redirected to /dev/pts/2 (label charserial0)
2017-05-25T03:58:44.619426Z qemu-kvm: -chardev pty,id=charredir0: char device redirected to /dev/pts/3 (label charredir0)
2017-05-25T03:58:44.632332Z qemu-kvm: warning: CPU(s) not present in any NUMA nodes: 6 7 8 9
2017-05-25T03:58:44.632354Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config
2017-05-25T03:58:44.633322Z qemu-kvm: can't apply global host-x86_64-cpu.svms=on: Property '.svms' not found


Libvirt didn't give any check for this



S5: Tcg + host-model + check=full

# virsh dumpxml r7
<domain type='qemu'>
...
  <cpu mode='host-model' check='full'>
    <model fallback='allow'/>
    <feature policy='require' name='rtm'/>
    <numa>
      <cell id='0' cpus='0-2' memory='524288' unit='KiB'/>
      <cell id='1' cpus='3-5' memory='524288' unit='KiB'/>
    </numa>
  </cpu>
...

# virsh start r7
error: Failed to start domain r7
error: operation failed: guest CPU doesn't match specification: missing features: rtm


# virsh dumpxml r7
<domain type='qemu'>
...
  <cpu mode='host-model' check='full'>
    <model fallback='allow'/>
    <feature policy='require' name='svm'/>
    <numa>
      <cell id='0' cpus='0-2' memory='524288' unit='KiB'/>
      <cell id='1' cpus='3-5' memory='524288' unit='KiB'/>
    </numa>
  </cpu>
...

# virsh start r7
Domain r7 started

# virsh dumpxml r7
...
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Opteron_G4</model>
    <vendor>AMD</vendor>
    <feature policy='require' name='acpi'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='monitor'/>
    <feature policy='require' name='movbe'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='arat'/>
    <feature policy='require' name='fsgsbase'/>
    <feature policy='require' name='bmi1'/>
    <feature policy='require' name='smep'/>
    <feature policy='require' name='bmi2'/>
    <feature policy='require' name='erms'/>
    <feature policy='require' name='mpx'/>
    <feature policy='require' name='adx'/>
    <feature policy='require' name='smap'/>
    <feature policy='require' name='clflushopt'/>
    <feature policy='require' name='pku'/>
    <feature policy='require' name='ospke'/>
    <feature policy='require' name='xsaveopt'/>
    <feature policy='require' name='xgetbv1'/>
    <feature policy='require' name='mmxext'/>
    <feature policy='require' name='3dnowext'/>
    <feature policy='require' name='3dnow'/>
    <feature policy='require' name='cr8legacy'/>
    <feature policy='disable' name='avx'/>
    <feature policy='disable' name='misalignsse'/>
    <feature policy='disable' name='3dnowprefetch'/>
    <feature policy='disable' name='xop'/>
    <feature policy='disable' name='fma4'/>
    <feature policy='require' name='svm'/>
    <numa>
      <cell id='0' cpus='0-2' memory='524288' unit='KiB'/>
      <cell id='1' cpus='3-5' memory='524288' unit='KiB'/>
    </numa>
  </cpu>
...

# ps aux|grep qemu
qemu     20331  101  1.2 2378380 402496 ?      Sl   00:12   0:29 ... -cpu Opteron_G4,acpi=on,ss=on,monitor=on,movbe=on,hypervisor=on,arat=on,fsgsbase=on,bmi1=on,smep=on,bmi2=on,erms=on,mpx=on,adx=on,smap=on,clflushopt=on,pku=on,ospke=on,xsaveopt=on,xgetbv1=on,mmxext=on,3dnowext=on,3dnow=on,cr8legacy=on,avx=off,misalignsse=off,3dnowprefetch=off,xop=off,fma4=off,svm=on 

IN GUEST:

Flags:                 fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush acpi mmx fxsr sse sse2 ss syscall nx mmxext pdpe1gb lm 3dnowext 3dnow art rep_good nopl extd_apicid eagerfpu pni pclmulqdq monitor ssse3 cx16 sse4_1 sse4_2 movbe popcnt aes xsave hypervisor lahf_lm svm cr8_legacy abm sse4a fsgsbase bmi1 smep bmi2 erms mpx adx smap clflushopt xsaveopt xgetbv1 arat


S6: TCG + host-model + invalid flags

# virsh dumpxml r7
<domain type='qemu'>
...
  <cpu mode='host-model' check='full'>
    <model fallback='allow'/>
    <feature policy='require' name='svmss'/>
...

# virsh start r7
error: Failed to start domain r7
error: internal error: qemu unexpectedly closed the monitor: 2017-05-25T05:53:12.120536Z qemu-kvm: -chardev pty,id=charserial0: char device redirected to /dev/pts/2 (label charserial0)
2017-05-25T05:53:12.120993Z qemu-kvm: -chardev pty,id=charredir0: char device redirected to /dev/pts/3 (label charredir0)
2017-05-25T05:53:12.132015Z qemu-kvm: warning: CPU(s) not present in any NUMA nodes: 6 7 8 9
2017-05-25T05:53:12.132034Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config
2017-05-25T05:53:12.132811Z qemu-kvm: can't apply global Opteron_G4-x86_64-cpu.svmss=on: Property '.svmss' not found


Libvirt didn't check if this flags is valid
Comment 20 Luyao Huang 2017-05-25 07:51:53 UTC 
Test migration between 2 host(both install libvirt-3.2.0-6.el7.x86_64):

S7: live migrate + host-model + kvm

guest cpu xml

  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
    <numa>
      <cell id='0' cpus='0-2' memory='524288' unit='KiB'/>
      <cell id='1' cpus='3-5' memory='524288' unit='KiB'/>
    </numa>
  </cpu>

# virsh dumpxml r7
...
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Opteron_G5</model>
    <vendor>AMD</vendor>
    <feature policy='require' name='vme'/>
    <feature policy='disable' name='ht'/>
    <feature policy='disable' name='monitor'/>
    <feature policy='disable' name='osxsave'/>
    <feature policy='require' name='bmi1'/>
    <feature policy='require' name='mmxext'/>
    <feature policy='require' name='fxsr_opt'/>
    <feature policy='require' name='cmp_legacy'/>
    <feature policy='disable' name='extapic'/>
    <feature policy='require' name='cr8legacy'/>
    <feature policy='require' name='osvw'/>
    <feature policy='disable' name='ibs'/>
    <feature policy='disable' name='skinit'/>
    <feature policy='disable' name='wdt'/>
    <feature policy='disable' name='lwp'/>
    <feature policy='disable' name='tce'/>
    <feature policy='disable' name='nodeid_msr'/>
    <feature policy='disable' name='topoext'/>
    <feature policy='disable' name='perfctr_core'/>
    <feature policy='disable' name='perfctr_nb'/>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='hypervisor'/>
    <numa>
      <cell id='0' cpus='0-2' memory='524288' unit='KiB'/>
      <cell id='1' cpus='3-5' memory='524288' unit='KiB'/>
    </numa>
  </cpu>

# virsh migrate r7 qemu+ssh://target/system --live

In Target host:

# virsh dumpxml r7
...
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Opteron_G5</model>
    <vendor>AMD</vendor>
    <feature policy='require' name='vme'/>
    <feature policy='disable' name='ht'/>
    <feature policy='disable' name='monitor'/>
    <feature policy='disable' name='osxsave'/>
    <feature policy='require' name='bmi1'/>
    <feature policy='require' name='mmxext'/>
    <feature policy='require' name='fxsr_opt'/>
    <feature policy='require' name='cmp_legacy'/>
    <feature policy='disable' name='extapic'/>
    <feature policy='require' name='cr8legacy'/>
    <feature policy='require' name='osvw'/>
    <feature policy='disable' name='ibs'/>
    <feature policy='disable' name='skinit'/>
    <feature policy='disable' name='wdt'/>
    <feature policy='disable' name='lwp'/>
    <feature policy='disable' name='tce'/>
    <feature policy='disable' name='nodeid_msr'/>
    <feature policy='disable' name='topoext'/>
    <feature policy='disable' name='perfctr_core'/>
    <feature policy='disable' name='perfctr_nb'/>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='hypervisor'/>
    <numa>
      <cell id='0' cpus='0-2' memory='524288' unit='KiB'/>
      <cell id='1' cpus='3-5' memory='524288' unit='KiB'/>
    </numa>
  </cpu>
...


S8: live migrate + custom + tcg

  <cpu mode='custom' match='exact' check='partial'>
    <model fallback='forbid'>Opteron_G5</model>
    <numa>
      <cell id='0' cpus='0-2' memory='524288' unit='KiB'/>
      <cell id='1' cpus='3-5' memory='524288' unit='KiB'/>
    </numa>
  </cpu>

# virsh start r7
Domain r7 started

# virsh dumpxml r7
...
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Opteron_G5</model>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='hypervisor'/>
    <numa>
      <cell id='0' cpus='0-2' memory='524288' unit='KiB'/>
      <cell id='1' cpus='3-5' memory='524288' unit='KiB'/>
    </numa>
  </cpu>
...

# virsh migrate r7 qemu+ssh://target/system --live

In Target host:

# virsh dumpxml r7
...
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Opteron_G5</model>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='hypervisor'/>
    <numa>
      <cell id='0' cpus='0-2' memory='524288' unit='KiB'/>
      <cell id='1' cpus='3-5' memory='524288' unit='KiB'/>
    </numa>
  </cpu>
...


S9: live migratation + TCG + host-model
pass

S10: live migratation + TCG + custom
pass
Comment 21 Luyao Huang 2017-05-25 08:29:00 UTC 
Test cross migration (libvirt-2.0.0-10 <-> libvirt-3.2.0-6):

S11: from a old libvirt to a new libvirt + kvm guest + host-model

  <cpu mode='host-model'>
    <model fallback='allow'/>
    <numa>
      <cell id='0' cpus='0-2' memory='524288' unit='KiB'/>
      <cell id='1' cpus='3-5' memory='524288' unit='KiB'/>
    </numa>
  </cpu>

# virsh start r7
Domain r7 started

# virsh dumpxml r7 --update-cpu
...
  <cpu mode='host-model' match='exact'>
    <model fallback='allow'>Opteron_G5</model>
    <vendor>AMD</vendor>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='ht'/>
    <feature policy='require' name='monitor'/>
    <feature policy='require' name='osxsave'/>
    <feature policy='require' name='bmi1'/>
    <feature policy='require' name='mmxext'/>
    <feature policy='require' name='fxsr_opt'/>
    <feature policy='require' name='cmp_legacy'/>
    <feature policy='require' name='extapic'/>
    <feature policy='require' name='cr8legacy'/>
    <feature policy='require' name='osvw'/>
    <feature policy='require' name='ibs'/>
    <feature policy='require' name='skinit'/>
    <feature policy='require' name='wdt'/>
    <feature policy='require' name='lwp'/>
    <feature policy='require' name='tce'/>
    <feature policy='require' name='nodeid_msr'/>
    <feature policy='require' name='topoext'/>
    <feature policy='require' name='perfctr_core'/>
    <feature policy='require' name='perfctr_nb'/>
...

# virsh migrate r7 qemu+ssh://target/system --live

In Target host:

# virsh dumpxml r7
...
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Opteron_G5</model>
    <vendor>AMD</vendor>
    <feature policy='require' name='vme'/>
    <feature policy='disable' name='ht'/>
    <feature policy='disable' name='monitor'/>
    <feature policy='disable' name='osxsave'/>
    <feature policy='require' name='bmi1'/>
    <feature policy='require' name='mmxext'/>
    <feature policy='require' name='fxsr_opt'/>
    <feature policy='require' name='cmp_legacy'/>
    <feature policy='disable' name='extapic'/>
    <feature policy='require' name='cr8legacy'/>
    <feature policy='require' name='osvw'/>
    <feature policy='disable' name='ibs'/>
    <feature policy='disable' name='skinit'/>
    <feature policy='disable' name='wdt'/>
    <feature policy='disable' name='lwp'/>
    <feature policy='disable' name='tce'/>
    <feature policy='disable' name='nodeid_msr'/>
    <feature policy='disable' name='topoext'/>
    <feature policy='disable' name='perfctr_core'/>
    <feature policy='disable' name='perfctr_nb'/>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='hypervisor'/>
...

Migrate back

# virsh migrate r7 qemu+ssh://lhuang/system --live
root@lhuang's password: 
error: unsupported configuration: guest and host CPU are not compatible: Host CPU does not provide required features: x2apic, hypervisor


S12: from a new libvirt to a old libvirt + kvm guest + host-model

# virsh migrate r7-mig qemu+ssh://lhuang/system --live
error: unsupported configuration: guest and host CPU are not compatible: Host CPU does not provide required features: x2apic, tsc-deadline, hypervisor, arat, tsc_adjust


S13: from a old libvirt to a new libvirt + kvm guest + custom

like S11 migrate success but fail to migrate back


S14: from a new libvirt to a old libvirt + kvm guest + custom

like S12 fail to migrate
Comment 22 Jiri Denemark 2017-05-25 08:41:02 UTC 
BTW, these cross migration issues are tracked in bug 1441662.
Comment 23 Luyao Huang 2017-05-25 08:48:13 UTC 
S15: from a old libvirt to a new libvirt + tcg guest + host-model

Guest xml:

# virsh dumpxml r7-mig
<domain type='qemu'>
...
  <cpu mode='host-model'>
    <model fallback='allow'/>
...

# virsh start r7-mig
Domain r7-mig started

# virsh migrate r7-mig qemu+ssh://target/system --live
error: the CPU is incompatible with host CPU: Host CPU does not provide required features: vme, ht, fma, osxsave, avx, f16c, fxsr_opt, cmp_legacy, extapic, misalignsse, 3dnowprefetch, osvw, ibs, xop, skinit, wdt, lwp, fma4, tce, nodeid_msr, tbm, topoext, perfctr_core, perfctr_nb

S16: from a new libvirt to a old libvirt + tcg guest + host-model

Migrate pass and can migrate back

S17: from a old libvirt to a new libvirt + tcg guest + custom

Migrate pass and can migrate back

S18: from a new libvirt to a old libvirt + tcg guest + custom

Migrate pass and can migrate back
Comment 24 Luyao Huang 2017-05-25 09:03:48 UTC 
Hi Jirka,

(In reply to Jiri Denemark from comment #22)
> BTW, these cross migration issues are tracked in bug 1441662.

Thanks for your point out, hope it won't be too noisy that i add 5 comments in a short time.

And comment 18-21,23 is some my test result for this patches, the problem list in comment 21 and comment 23 will be tracked in bug 1441662.

And i have a question about libvirt didn't valid the guest cpu flags name when the cpu mode is host-passthrough or the guest is a tcg guest (see S6 + S4). Could you please check S6 and S4 is expected ? Thanks in advance for your answer.
Comment 25 Luyao Huang 2017-05-26 02:35:15 UTC 
Test with qemu-kvm-1.5.3-139.el7.x86_64 and libvirt-3.2.0-6.el7.x86_64, the
virsh domcapabilities will show unknown in model usable and show the support feature which from cpu_map.xml.

virsh # domcapabilities 
...
  <cpu>
    <mode name='host-passthrough' supported='yes'/>
    <mode name='host-model' supported='yes'>
      <model fallback='allow'>Opteron_G5</model>
      <vendor>AMD</vendor>
      <feature policy='require' name='vme'/>
      <feature policy='require' name='ht'/>
      <feature policy='require' name='monitor'/>
      <feature policy='require' name='osxsave'/>
      <feature policy='require' name='bmi1'/>
      <feature policy='require' name='mmxext'/>
      <feature policy='require' name='fxsr_opt'/>
      <feature policy='require' name='cmp_legacy'/>
      <feature policy='require' name='extapic'/>
      <feature policy='require' name='cr8legacy'/>
      <feature policy='require' name='osvw'/>
      <feature policy='require' name='ibs'/>
      <feature policy='require' name='skinit'/>
      <feature policy='require' name='wdt'/>
      <feature policy='require' name='lwp'/>
      <feature policy='require' name='tce'/>
      <feature policy='require' name='nodeid_msr'/>
      <feature policy='require' name='topoext'/>
      <feature policy='require' name='perfctr_core'/>
      <feature policy='require' name='perfctr_nb'/>
      <feature policy='require' name='invtsc'/>
    </mode>
    <mode name='custom' supported='yes'>
      <model usable='unknown'>Opteron_G5</model>
      <model usable='unknown'>Opteron_G4</model>
      <model usable='unknown'>Opteron_G3</model>
      <model usable='unknown'>Opteron_G2</model>
      <model usable='unknown'>Opteron_G1</model>
      <model usable='unknown'>Skylake-Client</model>
      <model usable='unknown'>Broadwell</model>
      <model usable='unknown'>Haswell</model>
      <model usable='unknown'>IvyBridge</model>
      <model usable='unknown'>SandyBridge</model>
      <model usable='unknown'>Westmere</model>
      <model usable='unknown'>Nehalem</model>
      <model usable='unknown'>Penryn</model>
      <model usable='unknown'>Conroe</model>
      <model usable='unknown'>cpu64-rhel6</model>
      <model usable='unknown'>n270</model>
      <model usable='unknown'>athlon</model>
      <model usable='unknown'>pentium3</model>
      <model usable='unknown'>pentium2</model>
      <model usable='unknown'>pentium</model>
      <model usable='unknown'>486</model>
      <model usable='unknown'>coreduo</model>
      <model usable='unknown'>kvm32</model>
      <model usable='unknown'>qemu32</model>
      <model usable='unknown'>kvm64</model>
      <model usable='unknown'>core2duo</model>
      <model usable='unknown'>phenom</model>
      <model usable='unknown'>qemu64</model>
    </mode>
  </cpu>
...

And Retest from S1 - S14,

S1: Fail to start guest, since the cpu xml in domcapabilities output not match guest

S2: Fail to start guest
# virsh start r7
error: Failed to start domain r7
error: operation failed: guest CPU doesn't match specification: extra features: x2apic,hypervisor, missing features: ht,monitor,osxsave,svm,extapic,ibs,skinit,wdt,lwp,tce,nodeid_msr,topoext,perfctr_core,perfctr_nb

S3: The same

S4: libvirt check the cpu flags

# virsh start r7
error: Failed to start domain r7
error: internal error: Unknown CPU feature svmss

S5: Libvirt report a another flags instead of rtm

# virsh start r7
error: Failed to start domain r7
error: operation failed: guest CPU doesn't match specification: extra features: hypervisor

S6: libvirt check the cpu flags

# virsh start r7
error: Failed to start domain r7
error: internal error: Unknown CPU feature svmm

S7-S10: the same

S11-S14: the same

S15: Migrate success

S16-S17: The same
Comment 26 Luyao Huang 2017-05-26 03:34:21 UTC 
Hi Jirka,

One more question:

I have test with a old qemu (qemu-kvm-1.5.3-139.el7.x86_64) and found some test result is different with 2.9.0 qemu test result and list them in comment 25. Could you please help to check the difference is expected in comment 25 ?

Thanks in advance for your answer.
Comment 27 Jiri Denemark 2017-06-08 13:04:38 UTC 
(In reply to Luyao Huang from comment #24)
> And i have a question about libvirt didn't valid the guest cpu flags name
> when the cpu mode is host-passthrough or the guest is a tcg guest (see S6 +
> S4). Could you please check S6 and S4 is expected ?

Yeah, that's expected, libvirt is not supposed to cross check the guest CPU with host CPU for TCG guests or when the guest CPU uses host-passthrough mode. However, I think we should still validate that the XML uses valid feature names from our cpu_map.xml. In other words, we should complain about svmss since it is unknown to libvirt. Please, file a new bz for this.
Comment 28 Luyao Huang 2017-06-09 02:38:38 UTC 
Retest with libvirt-3.2.0-9.el7.x86_64, qemu-kvm-rhev-2.9.0-9.el7.x86_64 and qemu-kvm-1.5.3-140.el7 with the same steps (except migration) in comment 18-23, get the same result.

Migration test will been checked in bug 1441662. And the left issues according to comment 27 will been tracked in a new bug. Move this bug to verify.
Comment 29 Luyao Huang 2017-06-09 03:30:11 UTC 
(In reply to Jiri Denemark from comment #27)
> (In reply to Luyao Huang from comment #24)
> > And i have a question about libvirt didn't valid the guest cpu flags name
> > when the cpu mode is host-passthrough or the guest is a tcg guest (see S6 +
> > S4). Could you please check S6 and S4 is expected ?
> 
> Yeah, that's expected, libvirt is not supposed to cross check the guest CPU
> with host CPU for TCG guests or when the guest CPU uses host-passthrough
> mode. However, I think we should still validate that the XML uses valid
> feature names from our cpu_map.xml. In other words, we should complain about
> svmss since it is unknown to libvirt. Please, file a new bz for this.

Thanks a lot for your reply, i have filed a new bug 1460086 to track this problem.
Comment 30 Jiri Denemark 2017-06-09 10:44:40 UTC 
(In reply to Luyao Huang from comment #25)
> Test with qemu-kvm-1.5.3-139.el7.x86_64 and libvirt-3.2.0-6.el7.x86_64,
> the virsh domcapabilities will show unknown in model usable

Correct, 1.5.3 is too old to provide the usability info for each model.

> S1: Fail to start guest, since the cpu xml in domcapabilities output not match guest
> S2: Fail to start guest

Oh yeah, this is expected. Using check='full' with a CPU model which didn't
come from QEMU (i.e., the CPU is reported with fallback='allow' in
domcapabilities) will always fail. You should be able to start the domain with
check='partial' though.

> S4: libvirt check the cpu flags

Hmm, I'm not sure why this would be any different. Could you provide more
details (such as domain XML and libvirtd logs)?

> S5: Libvirt report a another flags instead of rtm

That's expected, as I said check='full' it not going to play well with
host-model on old QEMU.

> S6: libvirt check the cpu flags

The same as S4 above, more details please.
Comment 31 Luyao Huang 2017-06-12 03:34:04 UTC 
(In reply to Jiri Denemark from comment #30)
> (In reply to Luyao Huang from comment #25)
> > Test with qemu-kvm-1.5.3-139.el7.x86_64 and libvirt-3.2.0-6.el7.x86_64,
> > the virsh domcapabilities will show unknown in model usable
> 
> Correct, 1.5.3 is too old to provide the usability info for each model.
> 
> > S1: Fail to start guest, since the cpu xml in domcapabilities output not match guest
> > S2: Fail to start guest
> 
> Oh yeah, this is expected. Using check='full' with a CPU model which didn't
> come from QEMU (i.e., the CPU is reported with fallback='allow' in
> domcapabilities) will always fail. You should be able to start the domain
> with
> check='partial' though.

Got it

> 
> > S4: libvirt check the cpu flags
> 
> Hmm, I'm not sure why this would be any different. Could you provide more
> details (such as domain XML and libvirtd logs)?
> 

I will add the log and xml in another comment.

> > S5: Libvirt report a another flags instead of rtm
> 
> That's expected, as I said check='full' it not going to play well with
> host-model on old QEMU.
> 

I see

> > S6: libvirt check the cpu flags
> 
> The same as S4 above, more details please.

Thanks a lot for your reply
Comment 32 Luyao Huang 2017-06-12 03:36:19 UTC 
Created attachment 1286917 [details]
libvirtd log of S4

and xml:

<domain type='kvm'>
  <name>r7</name>
  <uuid>f816b705-eeac-41b9-a871-b809044dd4a1</uuid>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <vcpu placement='static' current='5'>10</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu mode='host-passthrough' check='full'>
    <feature policy='require' name='svmss'/>
    <numa>
      <cell id='0' cpus='0-2' memory='524288' unit='KiB'/>
      <cell id='1' cpus='3-5' memory='524288' unit='KiB'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='yes'/>
    <suspend-to-disk enabled='yes'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/RHEL-7.4-x86_64-latest.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:af:19:fb'/>
      <source network='default'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='pci-serial' port='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/123.sock'/>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes' listen='127.0.0.1'>
      <listen type='address' address='127.0.0.1'/>
    </graphics>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <redirdev bus='usb' type='pty'>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </memballoon>
  </devices>
</domain>
Comment 33 Luyao Huang 2017-06-12 03:38:13 UTC 
Created attachment 1286918 [details]
libvirtd log for S6 for debuging

and xml:

<domain type='qemu'>
  <name>r7</name>
  <uuid>f816b705-eeac-41b9-a871-b809044dd4a1</uuid>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <vcpu placement='static' current='5'>10</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu mode='host-model' check='full'>
    <model fallback='allow'/>
    <feature policy='require' name='svmss'/>
    <numa>
      <cell id='0' cpus='0-2' memory='524288' unit='KiB'/>
      <cell id='1' cpus='3-5' memory='524288' unit='KiB'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='yes'/>
    <suspend-to-disk enabled='yes'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/RHEL-7.4-x86_64-latest.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:af:19:fb'/>
      <source network='default'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='pci-serial' port='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/123.sock'/>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes' listen='127.0.0.1'>
      <listen type='address' address='127.0.0.1'/>
    </graphics>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <redirdev bus='usb' type='pty'>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </memballoon>
  </devices>
</domain>
Comment 34 errata-xmlrpc 2017-08-01 17:06:41 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846
Comment 35 errata-xmlrpc 2017-08-01 23:48:45 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846
Comment 36 errata-xmlrpc 2017-08-02 01:25:04 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846