Bug 1406827

Summary: Blacklist TSX feature from specific Intel CPU models
Product: Red Hat Enterprise Linux 7 Reporter: Eduardo Habkost <ehabkost>
Component: qemu-kvm-rhevAssignee: Eduardo Habkost <ehabkost>
Status: CLOSED ERRATA QA Contact: Guo, Zhiyi <zhguo>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.4CC: berrange, chayang, fweimer, jdenemar, juzhang, knoel, mrezanin, rbalakri, virt-bugs, virt-maint
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qemu-kvm-rhev-2.9.0-1.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1406791 Environment:
Last Closed: 2017-08-01 23:42:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Eduardo Habkost 2016-12-21 15:11:39 UTC
Cloning bug for qemu-kvm-rhev: we need the same rules implemented in the "host" CPU model in QEMU, so libvirt won't need to keep the workaround once it starts using the new query-cpu-model-* QMP commands.

+++ This bug was initially created as a clone of Bug #1406791 +++

Description of problem:
Relying on users having installed the microcode updates is found to be insufficiently robust to prevent use of the broken TSX feature in certain Intel CPUs.

We need to follow glibc's lead in checking CPUID and using code in libvirt to explicitly blacklist the TSX feature

https://sourceware.org/git/?p=glibc.git;a=commit;h=2702856bf45c82cf8e69f2064f5aa15c0ceb6359

Specific logic is:

+           case 0x3f:
+             /* Xeon E7 v3 with stepping >= 4 has working TSX.  */
+             if (stepping >= 4)
+               break;
+           case 0x3c:
+           case 0x45:
+           case 0x46:
+             /* Disable Intel TSX on Haswell processors (except Xeon E7 v3
+                with stepping >= 4) to avoid TSX on kernels that weren't
+                updated with the latest microcode package (which disables
+                broken feature by default).  */
+             cpu_features->cpuid[COMMON_CPUID_INDEX_7].ebx &= ~(bit_cpu_RTM);
+             break;


When run, we need to block exposing of TSX in the host CPU model reported in virsh capabilities, and prevent TSX being used in KVM (but not QEMU) guest CPUs.

Version-Release number of selected component (if applicable):
libvirt-2.0.0-10.el7

Comment 1 Daniel Berrangé 2016-12-21 15:26:46 UTC
(In reply to Eduardo Habkost from comment #0)
> Cloning bug for qemu-kvm-rhev: we need the same rules implemented in the
> "host" CPU model in QEMU, so libvirt won't need to keep the workaround once
> it starts using the new query-cpu-model-* QMP commands.

NB Libvirt will need its own support for this for years, since libvirt doesn't drop support for older QEMU versions when it starts using a new QMP command.

Comment 3 Eduardo Habkost 2017-02-22 22:08:23 UTC
Upstream v2 submitted:

From: Eduardo Habkost <ehabkost>
To: qemu-devel
Subject: [PATCH v2 0/3] Use non-blacklisted family/model/stepping for Haswell CPU model
Date: Wed, 22 Feb 2017 19:07:06 -0300
Message-Id: <20170222220709.3707-1-ehabkost>

Comment 4 Eduardo Habkost 2017-03-10 18:47:46 UTC
Included in upstream pull request:

Subject: [PULL 0/3] x86: Haswell TSX blacklist fix for 2.9
Date: Fri, 10 Mar 2017 15:45:49 -0300
Message-Id: <20170310184552.1481-1-ehabkost>

Comment 5 Eduardo Habkost 2017-03-14 17:45:55 UTC
Merged upstream:

commit f962709c69a05183bf314d3d8c69802d1e3f139c
Merge: b1616fe0e2 ec56a4a7b0
Author: Peter Maydell <peter.maydell>
Date:   Mon Mar 13 13:16:35 2017 +0000

    Merge remote-tracking branch 'remotes/ehabkost/tags/x86-pull-request' into staging
    
    x86: Haswell TSX blacklist fix for 2.9
    
    # gpg: Signature made Fri 10 Mar 2017 18:45:08 GMT
    # gpg:                using RSA key 0x2807936F984DC5A6
    # gpg: Good signature from "Eduardo Habkost <ehabkost>"
    # Primary key fingerprint: 5A32 2FD5 ABC4 D3DB ACCF  D1AA 2807 936F 984D C5A6
    
    * remotes/ehabkost/tags/x86-pull-request:
      i386: Change stepping of Haswell to non-blacklisted value
      i386/kvm: Blacklist TSX on known broken hosts
      i386: host_vendor_fms() helper function
    
    Signed-off-by: Peter Maydell <peter.maydell>

Comment 7 Guo, Zhiyi 2017-05-17 01:54:23 UTC
Test against qemu-kvm-rhev-2.8.0-6.el7.x86_64 on TSX broken haswell host, host info:
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                144
On-line CPU(s) list:   0-143
Thread(s) per core:    2
Core(s) per socket:    18
Socket(s):             4
NUMA node(s):          4
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 63
Model name:            Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz
Stepping:              3

Boot rhel7.4 guest witch cli:
/usr/libexec/qemu-kvm -name intel74 -m 32G \
        -cpu host,enforce \
        -smp 1 \
        -monitor stdio \
        -qmp tcp:0:4444,server,nowait \
        -vga std \
        -vnc :0 \
        -serial unix:/tmp/console,server,nowait \
        -drive file=sea74-0428.qcow2,if=none,id=drive-scsi-disk0,format=qcow2,ca
che=none,werror=stop,rerror=stop -device virtio-scsi-pci,id=scsi0,addr=04 -devic
e scsi-hd,drive=drive-scsi-disk0,bus=scsi0.0,scsi-id=0,lun=0,id=scsi-disk0,booti
ndex=1 \
        -netdev tap,id=idinWyYp,vhost=on -device virtio-net-pci,mac=42:ce:a9:d2:
4d:d7,id=idlbq7eA,netdev=idinWyYp \


Check flags inside guest, can find hle and rtm:
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 63
Model name:            Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz
Stepping:              3
CPU MHz:               2494.012
BogoMIPS:              4988.02
Virtualization:        VT-x
Hypervisor vendor:     KVM
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              4096K
L3 cache:              16384K
NUMA node0 CPU(s):     0
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constan
t_tsc arch_perfmon rep_good nopl xtopology eagerfpu pni pclmulqdq vmx ssse3 fma 
cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16
c rdrand hypervisor lahf_lm abm tpr_shadow vnmi flexpriority ept vpid fsgsbase t
sc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm xsaveopt arat

Change qemu cpu model to Haswell, check cpu info, stepping is 1 and can find hle and rtm:
On-line CPU(s) list:   0
Thread(s) per core:    1
Core(s) per socket:    1
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 60
Model name:            Intel Core Processor (Haswell)
Stepping:              1
CPU MHz:               2494.012
BogoMIPS:              4988.02
Hypervisor vendor:     KVM
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              4096K
L3 cache:              16384K
NUMA node0 CPU(s):     0
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_g
ood nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2ap
ic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm 
abm fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm xsaveopt arat

For qemu Haswell-noTSX cpu model, stepping in cpu info is:
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                1
On-line CPU(s) list:   0
Thread(s) per core:    1
Core(s) per socket:    1
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 60
Model name:            Intel Core Processor (Haswell, no TSX)
Stepping:              1

Test against qemu-kvm-rhev-2.9.0-5.el7.x86_64, boot rhel7.4 guest and check flags inside guest, cannot find hle and rtm.

Try to boot guest with Haswell,enforce, qemu cannot launch with warning prompt:
QEMU 2.9.0 monitor - type 'help' for more information
(qemu) warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4
]
warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]
qemu-kvm: Host doesn't support requested features

Try to boot guest with Haswell-noTSX,enforce, guest can boot and cpu info remain the same

query unavailable cpu flags from qmp, it show lacking of hle and rtm correctly:
{ "execute": "query-cpu-definitions"}
...
{"name": "Haswell", "typename": "Haswell-x86_
64-cpu", "unavailable-features": ["hle", "rtm"], "static": false, "migration-saf
e": true}
...
{"name": "Broadwell", "typename": "Broadwell-x86_64
-cpu", "unavailable-features": ["hle", "rtm", "rdseed", "adx", "smap", "3dnowpre
fetch"], "static": false, "migration-safe": true}
...
{"name": "Skylake-Client", "typename": "Skylake-Clie
nt-x86_64-cpu", "unavailable-features": ["hle", "rtm", "mpx", "rdseed", "adx", "
smap", "3dnowprefetch", "xsavec", "xgetbv1", "mpx", "mpx"], "static": false, "mi
gration-safe": true}
...

Test against qemu-kvm-rhev-2.9.0-5.el7.x86_64 and host has workable TSX, host info:
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                144
On-line CPU(s) list:   0-143
Thread(s) per core:    2
Core(s) per socket:    18
Socket(s):             4
NUMA node(s):          4
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 63
Model name:            Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz
Stepping:              4
....

Boot guest with Haswell,enforce, can find hle and rtm inside guest and stepping is 4:
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 60
Model name:            Intel Core Processor (Haswell)
Stepping:              4
CPU MHz:               2493.988
BogoMIPS:              4987.97
Hypervisor vendor:     KVM
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              4096K
L3 cache:              16384K
NUMA node0 CPU(s):     0
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_g
ood nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2ap
ic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm 
abm fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm xsaveopt arat

Boot guest with host,enforce, can also find hle and rtm inside guest

query unavailable cpu flags from qmp, it show host has workable hle and rtm:
{ "execute": "query-cpu-definitions"}
...
{"name": "Haswell", "typename": "Haswell-x86_64-cpu", "unav
ailable-features": [], "static": false, "migration-safe": true}
...
{"name": "Broadwell", "typename": "Broadwell-x86_64-cpu", "unavailable-featur
es": ["rdseed", "adx", "smap", "3dnowprefetch"], "static": false, "migration-saf
e": true}
...
{"name": "Skylake-Client", "typename": "Skylake-Clie
nt-x86_64-cpu", "unavailable-features": ["mpx", "rdseed", "adx", "smap", "3dnowp
refetch", "xsavec", "xgetbv1", "mpx", "mpx"], "static": false, "migration-safe":
 true}
...

Comment 8 Guo, Zhiyi 2017-05-17 01:56:06 UTC
Verified per comment 7

Comment 10 errata-xmlrpc 2017-08-01 23:42:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392

Comment 11 errata-xmlrpc 2017-08-02 01:19:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392

Comment 12 errata-xmlrpc 2017-08-02 02:11:53 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392

Comment 13 errata-xmlrpc 2017-08-02 02:52:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392

Comment 14 errata-xmlrpc 2017-08-02 03:17:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392