Bug 1365500

Summary: CPU feature cmt not found with 2.0.0-1
Product: Red Hat Enterprise Linux 7 Reporter: Jiri Denemark <jdenemar>
Component: libvirtAssignee: Jiri Denemark <jdenemar>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.3CC: chhu, dyuan, jdenemar, lhuang, libvirt-maint, rbalakri, yafu
Target Milestone: rcKeywords: Regression
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: libvirt-2.0.0-8.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1355857 Environment:
Last Closed: 2016-11-03 18:52:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1355857    
Bug Blocks:    

Description Jiri Denemark 2016-08-09 11:53:35 UTC 
+++ This bug was initially created as a clone of Bug #1355857 +++

Description of problem:

after upgrading libvirt-daemon from 1.3.5-1 to 2.0.0-1 I am unable to 
start a virtual machine which is configured with <cpu mode='host-model'>. With libvirt-daemon 1.3.5-1 this worked just fine. Now you receive an error like that one below when turning a machine (which was live migrated to the updated host) off and on again.

Starting the same machine with these settings works as expected:
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Haswell-noTSX</model>
  </cpu>

I think that this patch is causing the error for our CPUs (see below): 
http://libvirt.org/git/?p=libvirt.git;a=commit;h=f294b83ee632a6330f3a3045fbb5bcb9d9951c03

Commenting this out in /usr/share/libvirt/cpu_map.xml solves the issue 
as a workaround:

<!--    <feature name='cmt'>
      <cpuid eax_in='0x07' ebx='0x00001000'/>
    </feature> -->

Version-Release number of selected component (if applicable):
2.0.0-1

How reproducible:
100%


Steps to Reproduce:
1. Create a virtual machine with host-model CPU:

...
  <cpu mode='host-model'>
    <model fallback='allow'/>
  </cpu>
...

2. Start virtual machine


Actual results:
Start fails with this error:
"qemu-system-x86_64: CPU feature cmt not found"

Expected results:
Virtual machine should start without errors

Additional info:
virsh capabilities XML shows 'cmt' feature as supported by host CPU. The bug does not reproduce on hosts where 'cmt' feature is not advertised.
Comment 2 Jiri Denemark 2016-08-10 12:37:41 UTC 
This bug is now fixed upstream by

commit dbb14bb0f1316d92c89541fca816c91dce0dc8fb
Refs: v2.1.0-108-gdbb14bb
Author:     Jiri Denemark <jdenemar>
AuthorDate: Tue Jun 28 10:51:41 2016 +0200
Commit:     Jiri Denemark <jdenemar>
CommitDate: Wed Aug 10 14:25:24 2016 +0200

    cpu_x86: Introduce x86FeatureIsMigratable

    Signed-off-by: Jiri Denemark <jdenemar>
---
 src/cpu/cpu_x86.c | 37 +++++++++++++++++++++++--------------
 1 file changed, 23 insertions(+), 14 deletions(-)

commit 1ac897a15da11d1bfca2642bce3b0beaad32bcf1
Refs: v2.1.0-109-g1ac897a
Author:     Jiri Denemark <jdenemar>
AuthorDate: Tue Jun 28 11:12:41 2016 +0200
Commit:     Jiri Denemark <jdenemar>
CommitDate: Wed Aug 10 14:25:24 2016 +0200

    cpu_x86: Properly drop non-migratable features

    By removing a non-migratable feature in a for loop we would fail to drop
    every second non-migratable feature if the features array contained
    several of them in a row.

    Signed-off-by: Jiri Denemark <jdenemar>
---
 src/cpu/cpu_x86.c | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

commit 58ba240df89a7dd7bff16ebb65d9aadd49fb8bbb
Refs: v2.1.0-110-g58ba240
Author:     Jiri Denemark <jdenemar>
AuthorDate: Tue Aug 9 15:03:20 2016 +0200
Commit:     Jiri Denemark <jdenemar>
CommitDate: Wed Aug 10 14:25:24 2016 +0200

    tests: Add a test for host-model CPU with CMT feature

    The generated command line wouldn't work since QEMU doesn't know what
    'cmt' is. The following patch will fix this issue.

    https://bugzilla.redhat.com/show_bug.cgi?id=1355857

    Signed-off-by: Jiri Denemark <jdenemar>
---
 tests/qemuxml2argvdata/qemuxml2argv-cpu-host-model-cmt.args | 22 ++++++++++++++++++++++
 tests/qemuxml2argvdata/qemuxml2argv-cpu-host-model-cmt.xml  | 19 +++++++++++++++++++
 tests/qemuxml2argvtest.c                                    |  1 +
 tests/testutilsqemu.c                                       |  1 +
 4 files changed, 43 insertions(+)

commit 300f668c665f1ec0f834917fe8a58b5991322441
Refs: v2.1.0-111-g300f668
Author:     Jiri Denemark <jdenemar>
AuthorDate: Tue Aug 9 15:15:20 2016 +0200
Commit:     Jiri Denemark <jdenemar>
CommitDate: Wed Aug 10 14:25:24 2016 +0200

    cpu_x86: Fix host-model CPUs on hosts with CMT

    Since the introduction of CMT features (commit v1.3.5-461-gf294b83)
    starting a domain with host-model CPU on a host which supports CMT fails
    because QEMU complains about unknown 'cmt' feature:

        qemu-system-x86_64: CPU feature cmt not found

    https://bugzilla.redhat.com/show_bug.cgi?id=1355857

    Signed-off-by: Jiri Denemark <jdenemar>
---
 src/cpu/cpu_x86.c                                           | 8 ++++++--
 tests/qemuxml2argvdata/qemuxml2argv-cpu-host-model-cmt.args | 2 +-
 2 files changed, 7 insertions(+), 3 deletions(-)
Comment 5 chhu 2016-08-18 09:22:07 UTC 
Reproduced with package: 
libvirt-2.0.0-4.el7.x86_64, 
qemu-kvm-rhev-2.6.0-20.el7.x86_64, 
kernel:  3.10.0-492.el7.x86_64

Tested with libvirt-2.0.0-5.el7.x86_64

Test with xml cpu mode='host-model' pass, but test with xml: cpu mode='custom' match='minimum' failed with the same error.

Steps:
1. Start VM with xml: cpu mode='host-model': PASS
1) model fallback='allow'
# virsh dumpxml r7.3-raw | grep "<cpu" -A 5
  <cpu mode='host-model'>
    <model fallback='allow'/>
  </cpu>

# virsh start r7.3-raw
Domain r7.3-raw started

# virsh list --all
 Id    Name                           State
----------------------------------------------------
 8     r7.3-raw                       running

2) model fallback='forbid'
# virsh list --all
 Id    Name                           State
----------------------------------------------------
 -     r7.3-raw                       shut off

# virsh start r7.3-raw
Domain r7.3-raw started

# virsh list --all
 Id    Name                           State
----------------------------------------------------
 9     r7.3-raw                       running

# virsh dumpxml r7.3-raw | grep "<cpu" -A 5
  <cpu mode='host-model'>
    <model fallback='forbid'/>
  </cpu>


2. Start VM with xml: cpu mode='custom' match='minimum': Failed

# virsh dumpxml r7.3-raw | grep "<cpu" -A 5
  <cpu mode='custom' match='minimum'>
    <model fallback='allow'>Haswell-noTSX</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='ds'/>
  </cpu>

# virsh list --all
 Id    Name                           State
----------------------------------------------------
 -     r7.3-raw                       shut off

# virsh start r7.3-raw
error: Failed to start domain r7.3-raw
error: internal error: qemu unexpectedly closed the monitor: 2016-08-18T08:55:33.155930Z qemu-kvm: CPU feature cmt not found
Comment 6 chhu 2016-08-18 09:52:13 UTC 
Hi, Jiri

I tested with xml cpu mode='host-model', VM started successfully, but when tested with xml: cpu mode='custom' match='minimum', VM failed to start and got the same error. Please see more details in Comment5, do you think this need to be fixed ? Could you please help to confirm? 

Thanks,
chhu
Comment 7 Jiri Denemark 2016-08-22 19:24:49 UTC 
Ah yeah, the upstream series should take care of this already, I'll try to look at ways to fix this part downstream without having to backport all the downstream patches.
Comment 8 chhu 2016-09-01 03:38:02 UTC 
Did more test for this bug:

Tested with packages:
libvirt-2.0.0-6.el7.x86_64
qemu-kvm-rhev-2.6.0-22.el7.x86_64


1. Start the VM, check the dumpxml，did operations: Start VM, managed save, migration(7.3<=>7.3)：PASS

<cpu mode='host-model'>
  <model fallback='allow'/>
</cpu>


2. Start the VM, check the dumpxml and the qemu cmdline.

1） mode=custom, match=minimum: failed, updated in comment 5.
<cpu mode='custom' match='minimum'>   
  <model fallback='allow'>Haswell-noTSX</model>
</cpu>

2) mode=custom, match= strict/exact: start VM with xml below:
<cpu mode='custom' match='strict'>  
  <model fallback='allow'>Haswell-noTSX</model>
  <vendor>Intel</vendor>
  <feature policy='require' name='cmt'/>
  ......
</cpu>
<cpu mode='custom' match='exact'>
  <model fallback='allow'>Haswell-noTSX</model>
  <vendor>Intel</vendor>
  <feature policy='require' name='cmt'/>  
</cpu>

Met the same error message: 
" qemu-kvm: CPU feature cmt not found" 
As the cmt in not known by qemu-kvm command line, so the error message is reasonable. 

Changed to <feature policy='disable' name='cmt'/>, start VM  successfully, qemu command line is without "-cmt", this is related to Bug 1371039 - CPU features with policy 'disable' should not be supported in the guest.


3. Start the VM, with cpu mode: host-model/customer, with different policy, check the dumpxml and the qemu cmdline.

- Feature policy: force/ require/ optional/ disable/ forbid
<cpu mode='host-model'>
  <model fallback='allow'/>
  <feature policy='disable' name='cmt'/>
</cpu>

<cpu mode='custom' match='exact'>
  <model fallback='allow'>Haswell-noTSX</model>
  <vendor>Intel</vendor>
  <feature policy='disable' name='cmt'/>
</cpu>


1) policy=forbid: met "error: unsupported configuration: guest and host CPU are not compatible: Host CPU provides forbidden features: cmt" => this is as design.

2) policy=require/ force/optional: met error message: "qemu-kvm: CPU feature cmt not found" =>  as the cmt in not known by qemu-kvm command line, so the error message is reasonable.

3) policy=disable: started the vm successfully and the qemu command line is without "-cmt". Normally, the qemu command line should include '-cmt'. 
This is related to Bug 1371039 - CPU features with policy 'disable' should not be supported in the guest.


4. Tried to start a VM with cpu mode: host-passthrough with different policy, met the same error: "qemu-kvm: CPU feature cmt not found".

-policy: disable/require/optional/force/forbid

  <cpu mode='host-passthrough'>
    <feature policy='disable' name='cmt'/>
  </cpu>

# virsh start r7t
error: Failed to start domain r7t
error: internal error: qemu unexpectedly closed the monitor: 2016-08-26T07:40:24.934930Z qemu-kvm: CPU feature cmt not found

Issues:
1) policy=require/ force/optional: met error message: "qemu-kvm: CPU feature cmt not found" =>  as the cmt in not known by qemu-kvm command line, so the error message is reasonable.

2) For policy='disable', the VM should be started successfully, now met error:"qemu-kvm: CPU feature cmt not found" 
  
3) For policy='forbid', should met error message as below:
"error: unsupported configuration: guest and host CPU are not compatible: Host CPU provides forbidden features: cmt"
Now met error:"qemu-kvm: CPU feature cmt not found"

@Jiri:
As QEMU complains about unknown 'cmt' feature, not sure if above Issues: 2)3) should be fixed. Could you please help to confirm?  Thank you!
Comment 9 chhu 2016-09-05 10:41:36 UTC 
Updated another test issue which may related to this bug here:

Imporve the error message:
Migrate with 'invtsc' get qemu error:
"unable to execute QEMU command 'migrate': State blocked by non-migratable device 'cpu'" 
The error message should be reported by libvirt as:
"error: Requested operation is not valid: domain has CPU feature: invtsc"

How reproducible:
100%

Version-Release number of selected component (if applicable):
qemu-kvm-rhev-2.6.0-22.el7.x86_64
libvirt-2.0.0-6.el7.x86_64
kernel: kernel-3.10.0-495.el7.x86_64

Steps to reproduce:
1. Start a VM with xml below successfully.
# virsh start r7-s1
Domain r7-s1 started

# virsh dumpxml r7-s1|grep "<cpu" -A 6
  <cpu mode='custom' match='minimum'>
    <model fallback='forbid'>Skylake-Client</model>
  </cpu>

# ps -ef|grep r7-s1
qemu     17490     1 11 22:59 ?        00:00:12 /usr/libexec/qemu-kvm -name guest=r7-s1...... -cpu Skylake-Client,+ds,+acpi,+ss,+ht,+tm,+pbe,+dtes64,+monitor,+ds_cpl,+vmx,+smx,+est,+tm2,+xtpr,+pdcm,+osxsave,+tsc_adjust,+clflushopt,+xsaves,+pdpe1gb,+invtsc -m 1024

2. Try to do migration test: met error: "unable to execute QEMU command 'migrate': State blocked by non-migratable device 'cpu'"

# virsh migrate r7-s1 qemu+ssh://10.19.176.166/system --live --verbose
root.176.166's password: 
error: internal error: unable to execute QEMU command 'migrate': State blocked by non-migratable device 'cpu'


Hi, Jiri

Will the issue in this and above comments be fixed? As it's rhel7.3 snapshot-2 now, and this bug is with rhel-7.3.0 +, could you please help to provide the fixed patches for us to test? Many thanks!
Comment 10 Jiri Denemark 2016-09-05 11:06:21 UTC 
CPU with match='minimum' is a lot like host-model, but it wasn't treated as such :-( and my upstream series fixes this. However, it's not something we can backport. Anyway, this part of the bug (invtsc added for minimum match CPUs) should not be a regression and as such, I don't think it's worth another downstream only hack.
Comment 11 chhu 2016-09-07 05:45:48 UTC 
(In reply to chhu from comment #8)
> Did more test for this bug:
> 
> Tested with packages:
> libvirt-2.0.0-6.el7.x86_64
> qemu-kvm-rhev-2.6.0-22.el7.x86_64
> 
> 
> 1. Start the VM, check the dumpxml，did operations: Start VM, managed save,
> migration(7.3<=>7.3)：PASS
> 
> <cpu mode='host-model'>
>   <model fallback='allow'/>
> </cpu>
> 
> 
> 2. Start the VM, check the dumpxml and the qemu cmdline.
> 
> 1） mode=custom, match=minimum: failed, updated in comment 5.
> <cpu mode='custom' match='minimum'>   
>   <model fallback='allow'>Haswell-noTSX</model>
> </cpu>
> 
> 2) mode=custom, match= strict/exact: start VM with xml below:
> <cpu mode='custom' match='strict'>  
>   <model fallback='allow'>Haswell-noTSX</model>
>   <vendor>Intel</vendor>
>   <feature policy='require' name='cmt'/>
>   ......
> </cpu>
> <cpu mode='custom' match='exact'>
>   <model fallback='allow'>Haswell-noTSX</model>
>   <vendor>Intel</vendor>
>   <feature policy='require' name='cmt'/>  
> </cpu>
> 
> Met the same error message: 
> " qemu-kvm: CPU feature cmt not found" 
> As the cmt in not known by qemu-kvm command line, so the error message is
> reasonable. 
> 
> Changed to <feature policy='disable' name='cmt'/>, start VM  successfully,
> qemu command line is without "-cmt", this is related to Bug 1371039 - CPU
> features with policy 'disable' should not be supported in the guest.
> 
> 
> 3. Start the VM, with cpu mode: host-model/customer, with different policy,
> check the dumpxml and the qemu cmdline.
> 
> - Feature policy: force/ require/ optional/ disable/ forbid
> <cpu mode='host-model'>
>   <model fallback='allow'/>
>   <feature policy='disable' name='cmt'/>
> </cpu>
> 
> <cpu mode='custom' match='exact'>
>   <model fallback='allow'>Haswell-noTSX</model>
>   <vendor>Intel</vendor>
>   <feature policy='disable' name='cmt'/>
> </cpu>
> 
> 
> 1) policy=forbid: met "error: unsupported configuration: guest and host CPU
> are not compatible: Host CPU provides forbidden features: cmt" => this is as
> design.
> 
> 2) policy=require/ force/optional: met error message: "qemu-kvm: CPU feature
> cmt not found" =>  as the cmt in not known by qemu-kvm command line, so the
> error message is reasonable.
> 
> 3) policy=disable: started the vm successfully and the qemu command line is
> without "-cmt". Normally, the qemu command line should include '-cmt'. 
> This is related to Bug 1371039 - CPU features with policy 'disable' should
> not be supported in the guest.
> 
> 
> 4. Tried to start a VM with cpu mode: host-passthrough with different
> policy, met the same error: "qemu-kvm: CPU feature cmt not found".
> 
> -policy: disable/require/optional/force/forbid
> 
>   <cpu mode='host-passthrough'>
>     <feature policy='disable' name='cmt'/>
>   </cpu>
> 
> # virsh start r7t
> error: Failed to start domain r7t
> error: internal error: qemu unexpectedly closed the monitor:
> 2016-08-26T07:40:24.934930Z qemu-kvm: CPU feature cmt not found
> 
> Issues:
> 1) policy=require/ force/optional: met error message: "qemu-kvm: CPU feature
> cmt not found" =>  as the cmt in not known by qemu-kvm command line, so the
> error message is reasonable.
> 
> 2) For policy='disable', the VM should be started successfully, now met
> error:"qemu-kvm: CPU feature cmt not found" 
>   
> 3) For policy='forbid', should met error message as below:
> "error: unsupported configuration: guest and host CPU are not compatible:
> Host CPU provides forbidden features: cmt"
> Now met error:"qemu-kvm: CPU feature cmt not found"
> 
> @Jiri:
> As QEMU complains about unknown 'cmt' feature, not sure if above Issues:
> 2)3) should be fixed. Could you please help to confirm?  Thank you!

Tested with mbm_local/mbm_total on Broadwell met the similar results as test cmt on Skylake-Client, will retest mbm_local/mbm_total after this bug is fixed.
Comment 12 chhu 2016-09-07 05:57:11 UTC 
(In reply to chhu from comment #9)
> Updated another test issue which may related to this bug here:
> 
> Imporve the error message:
> Migrate with 'invtsc' get qemu error:
> "unable to execute QEMU command 'migrate': State blocked by non-migratable
> device 'cpu'" 
> The error message should be reported by libvirt as:
> "error: Requested operation is not valid: domain has CPU feature: invtsc"
> 
> How reproducible:
> 100%
> 
> Version-Release number of selected component (if applicable):
> qemu-kvm-rhev-2.6.0-22.el7.x86_64
> libvirt-2.0.0-6.el7.x86_64
> kernel: kernel-3.10.0-495.el7.x86_64
> 
> Steps to reproduce:
> 1. Start a VM with xml below successfully.
> # virsh start r7-s1
> Domain r7-s1 started
> 
> # virsh dumpxml r7-s1|grep "<cpu" -A 6
>   <cpu mode='custom' match='minimum'>
>     <model fallback='forbid'>Skylake-Client</model>
>   </cpu>
> 
> # ps -ef|grep r7-s1
> qemu     17490     1 11 22:59 ?        00:00:12 /usr/libexec/qemu-kvm -name
> guest=r7-s1...... -cpu
> Skylake-Client,+ds,+acpi,+ss,+ht,+tm,+pbe,+dtes64,+monitor,+ds_cpl,+vmx,+smx,
> +est,+tm2,+xtpr,+pdcm,+osxsave,+tsc_adjust,+clflushopt,+xsaves,+pdpe1gb,
> +invtsc -m 1024
> 
> 2. Try to do migration test: met error: "unable to execute QEMU command
> 'migrate': State blocked by non-migratable device 'cpu'"
> 
> # virsh migrate r7-s1 qemu+ssh://10.19.176.166/system --live --verbose
> root.176.166's password: 
> error: internal error: unable to execute QEMU command 'migrate': State
> blocked by non-migratable device 'cpu'
> 
> 
> Hi, Jiri
> 
> Will the issue in this and above comments be fixed? As it's rhel7.3
> snapshot-2 now, and this bug is with rhel-7.3.0 +, could you please help to
> provide the fixed patches for us to test? Many thanks!

According to Jiri's reply:"Anyway, this part of the bug (invtsc added for minimum match CPUs) should not be a regression and as such, I don't think it's worth another downstream only hack."

I add the issues in this comment to bug1372581.
Comment 13 chhu 2016-09-07 09:35:19 UTC 
(In reply to chhu from comment #11)
> (In reply to chhu from comment #8)
> > Did more test for this bug:
> > 
> > Tested with packages:
> > libvirt-2.0.0-6.el7.x86_64
> > qemu-kvm-rhev-2.6.0-22.el7.x86_64
> > 
> > 
> > 1. Start the VM, check the dumpxml，did operations: Start VM, managed save,
> > migration(7.3<=>7.3)：PASS
> > 
> > <cpu mode='host-model'>
> >   <model fallback='allow'/>
> > </cpu>
> > 
> > 
> > 2. Start the VM, check the dumpxml and the qemu cmdline.
> > 
> > 1） mode=custom, match=minimum: failed, updated in comment 5.
> > <cpu mode='custom' match='minimum'>   
> >   <model fallback='allow'>Haswell-noTSX</model>
> > </cpu>
> > 
> > 2) mode=custom, match= strict/exact: start VM with xml below:
> > <cpu mode='custom' match='strict'>  
> >   <model fallback='allow'>Haswell-noTSX</model>
> >   <vendor>Intel</vendor>
> >   <feature policy='require' name='cmt'/>
> >   ......
> > </cpu>
> > <cpu mode='custom' match='exact'>
> >   <model fallback='allow'>Haswell-noTSX</model>
> >   <vendor>Intel</vendor>
> >   <feature policy='require' name='cmt'/>  
> > </cpu>
> > 
> > Met the same error message: 
> > " qemu-kvm: CPU feature cmt not found" 
> > As the cmt in not known by qemu-kvm command line, so the error message is
> > reasonable. 
> > 
> > Changed to <feature policy='disable' name='cmt'/>, start VM  successfully,
> > qemu command line is without "-cmt", this is related to Bug 1371039 - CPU
> > features with policy 'disable' should not be supported in the guest.
> > 
> > 
> > 3. Start the VM, with cpu mode: host-model/customer, with different policy,
> > check the dumpxml and the qemu cmdline.
> > 
> > - Feature policy: force/ require/ optional/ disable/ forbid
> > <cpu mode='host-model'>
> >   <model fallback='allow'/>
> >   <feature policy='disable' name='cmt'/>
> > </cpu>
> > 
> > <cpu mode='custom' match='exact'>
> >   <model fallback='allow'>Haswell-noTSX</model>
> >   <vendor>Intel</vendor>
> >   <feature policy='disable' name='cmt'/>
> > </cpu>
> > 
> > 
> > 1) policy=forbid: met "error: unsupported configuration: guest and host CPU
> > are not compatible: Host CPU provides forbidden features: cmt" => this is as
> > design.
> > 
> > 2) policy=require/ force/optional: met error message: "qemu-kvm: CPU feature
> > cmt not found" =>  as the cmt in not known by qemu-kvm command line, so the
> > error message is reasonable.
> > 
> > 3) policy=disable: started the vm successfully and the qemu command line is
> > without "-cmt". Normally, the qemu command line should include '-cmt'. 
> > This is related to Bug 1371039 - CPU features with policy 'disable' should
> > not be supported in the guest.
> > 
> > 
> > 4. Tried to start a VM with cpu mode: host-passthrough with different
> > policy, met the same error: "qemu-kvm: CPU feature cmt not found".
> > 
> > -policy: disable/require/optional/force/forbid
> > 
> >   <cpu mode='host-passthrough'>
> >     <feature policy='disable' name='cmt'/>
> >   </cpu>
> > 
> > # virsh start r7t
> > error: Failed to start domain r7t
> > error: internal error: qemu unexpectedly closed the monitor:
> > 2016-08-26T07:40:24.934930Z qemu-kvm: CPU feature cmt not found
> > 
> > Issues:
> > 1) policy=require/ force/optional: met error message: "qemu-kvm: CPU feature
> > cmt not found" =>  as the cmt in not known by qemu-kvm command line, so the
> > error message is reasonable.
> > 
> > 2) For policy='disable', the VM should be started successfully, now met
> > error:"qemu-kvm: CPU feature cmt not found" 
> >   
> > 3) For policy='forbid', should met error message as below:
> > "error: unsupported configuration: guest and host CPU are not compatible:
> > Host CPU provides forbidden features: cmt"
> > Now met error:"qemu-kvm: CPU feature cmt not found"
> > 
> > @Jiri:
> > As QEMU complains about unknown 'cmt' feature, not sure if above Issues:
> > 2)3) should be fixed. Could you please help to confirm?  Thank you!
> 
> Tested with mbm_local/mbm_total on Broadwell met the similar results as test
> cmt on Skylake-Client, will retest mbm_local/mbm_total after this bug is
> fixed.

According to Jiri's reply:"CPU with match='minimum' is a lot like host-model, but it wasn't treated as such :-( and my upstream series fixes this. However, it's not something we can backport."

File a new Bug 1373849 to record the issue.
Comment 14 chhu 2016-09-07 09:49:00 UTC 
According to the comment 12,13, the bug is verified.
Comment 15 Jiri Denemark 2016-09-07 14:06:41 UTC 
Oh, you got me wrong, the cmt/mbm_local/mbm_total thing should be fixed as it is a regression. The invtsc issue with minimum match CPUs is not worth a RHEL-only hack since it's been broken ever since we introduced invtsc.
Comment 16 Jiri Denemark 2016-09-07 14:06:53 UTC 
*** Bug 1373849 has been marked as a duplicate of this bug. ***
Comment 17 Jiri Denemark 2016-09-07 14:07:45 UTC 
The additional patch: http://post-office.corp.redhat.com/archives/rhvirt-patches/2016-September/msg00204.html
Comment 19 Jiri Denemark 2016-09-09 09:27:52 UTC 
So the additional patch fixes just one of the paths where custom/minimum CPUs are updated according to a host CPU and unfortunately it was not the path which is used when building a command line for QEMU. The patch only fixes "virsh dumpxml --update-cpu".

In other words, domains custom/minimum CPUs will still fail to start on hosts with CMT. I'm reopening bug 1373849 to track this.
Comment 20 chhu 2016-09-12 02:25:47 UTC 
Tested for the patch in Comment18: 

Reproduced on package:
libvirt-2.0.0-6.el7.x86_64

Verified on package:
libvirt-2.0.0-8.el7.x86_64

Test steps:
1. Try to start a VM with cpu mode custom and match=minimum on host with CMT: Failed

# virsh dumpxml r7| grep "<cpu" -A 4
  <cpu mode='custom' match='minimum'>
    <model fallback='allow'>Broadwell</model>
  </cpu>
# virsh start r7
error: Failed to start domain r7
error: internal error: qemu unexpectedly closed the monitor: 2016-09-12T02:02:06.688799Z qemu-kvm: CPU feature cmt not found


2. Check the "virsh dumpxml --update-cpu": PASS

1) On libvirt-2.0.0-6.el7.x86_64: there are cmt, mbm_total, mbm_local listed.
# virsh dumpxml --update-cpu r7
<domain type='kvm'>
  <name>r7</name>
.....
  <cpu mode='custom' match='exact'>
    <model fallback='forbid'>Broadwell</model>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='ds'/>
    <feature policy='require' name='acpi'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='ht'/>
    <feature policy='require' name='tm'/>
    <feature policy='require' name='pbe'/>
    <feature policy='require' name='dtes64'/>
    <feature policy='require' name='monitor'/>
    <feature policy='require' name='ds_cpl'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='smx'/>
    <feature policy='require' name='est'/>
    <feature policy='require' name='tm2'/>
    <feature policy='require' name='xtpr'/>
    <feature policy='require' name='pdcm'/>
    <feature policy='require' name='dca'/>
    <feature policy='require' name='osxsave'/>
    <feature policy='require' name='f16c'/>
    <feature policy='require' name='rdrand'/>
    <feature policy='require' name='arat'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='cmt'/>
    <feature policy='require' name='xsaveopt'/>
    <feature policy='require' name='mbm_total'/>
    <feature policy='require' name='mbm_local'/>
    <feature policy='require' name='pdpe1gb'/>
    <feature policy='require' name='abm'/>
    <feature policy='require' name='invtsc'/>
  </cpu>

2) on libvirt-2.0.0-8.el7.x86_64, the cmt, mbm_total, mbm_local are moved.

# virsh dumpxml --update-cpu r7
<domain type='kvm'>
  <name>r7</name>
......
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Broadwell</model>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='ds'/>
    <feature policy='require' name='acpi'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='ht'/>
    <feature policy='require' name='tm'/>
    <feature policy='require' name='pbe'/>
    <feature policy='require' name='dtes64'/>
    <feature policy='require' name='monitor'/>
    <feature policy='require' name='ds_cpl'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='smx'/>
    <feature policy='require' name='est'/>
    <feature policy='require' name='tm2'/>
    <feature policy='require' name='xtpr'/>
    <feature policy='require' name='pdcm'/>
    <feature policy='require' name='dca'/>
    <feature policy='require' name='osxsave'/>
    <feature policy='require' name='f16c'/>
    <feature policy='require' name='rdrand'/>
    <feature policy='require' name='arat'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='xsaveopt'/>
    <feature policy='require' name='pdpe1gb'/>
    <feature policy='require' name='abm'/>
    <feature policy='require' name='invtsc'/>
  </cpu>

According to comment 12, 19, this bug is verified.
Comment 22 errata-xmlrpc 2016-11-03 18:52:02 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2577.html