Bug 1384124 - cpu flag nonstop_tsc is not present in guest with host-passthrough and feature policy require invtsc
Summary: cpu flag nonstop_tsc is not present in guest with host-passthrough and featur...
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev   
(Show other bugs)
Version: 7.2
Hardware: x86_64
OS: Linux
high
high
Target Milestone: rc
: 7.2
Assignee: Eduardo Habkost
QA Contact: Guo, Zhiyi
URL:
Whiteboard:
Keywords: ZStream
Depends On:
Blocks: 1018952 1446211 1413897
TreeView+ depends on / blocked
 
Reported: 2016-10-12 15:21 UTC by djdumas
Modified: 2017-08-02 03:32 UTC (History)
21 users (show)

Fixed In Version: QEMU 2.8.0
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1413897 (view as bug list)
Environment:
Last Closed: 2017-08-01 23:37:14 UTC
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
host passthrough /proc/cpuinfo from host and guest, qemu ps (deleted)
2016-10-12 15:21 UTC, djdumas
no flags Details
host passthrough /proc/cpuinfo from host and guest, qemu ps (5.35 KB, text/plain)
2016-10-12 15:24 UTC, djdumas
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:2392 normal SHIPPED_LIVE Important: qemu-kvm-rhev security, bug fix, and enhancement update 2017-08-01 20:04:36 UTC

Description djdumas 2016-10-12 15:21:26 UTC
Description of problem:
Starting a guest with <cpu mode='host-passthrough' and feature policy='require' name='invtsc' does not provide cpu flag nonstop_tsc in the guest.  

Using <cpu mode='custom' match='exact'> and requiring invtsc on the same system does give nonstop_tsc in the guest.

SAP HANA tests for this flag in the guest and if not found, reverts to use of system calls and performance is impacted.

Version-Release number of selected component (if applicable):
qemu-kvm-rhev-2.3.0-31.el7_2.21.x86_64
qemu-kvm-common-rhev-2.3.0-31.el7_2.21.x86_64
qemu-kvm-tools-rhev-2.3.0-31.el7_2.21.x86_64
qemu-img-rhev-2.3.0-31.el7_2.21.x86_64


How reproducible:
always

Steps to Reproduce:
1.Create a RHEL7.2 guest on a host that has CPU flag nonstop_tsc
2.Use host passthrough and require invtsc as described above
3.Start guest, check cpu flags for nonstop_tsc

Actual results:
no nonstop_tsc cpu flag in guest

Expected results:
nonstop_tsc cpu flag in guest

Additional info:

Comment 1 djdumas 2016-10-12 15:24 UTC
Created attachment 1209659 [details]
host passthrough /proc/cpuinfo from host and guest, qemu ps

Comment 3 Bandan Das 2016-10-12 18:43:46 UTC
(In reply to djdumas from comment #0)
> Description of problem:
> Starting a guest with <cpu mode='host-passthrough' and feature
> policy='require' name='invtsc' does not provide cpu flag nonstop_tsc in the
> guest.  
> 
I know we spoke a little bit about this before. On my 7.3 system, my guest sees nonstop_tsc if I specify -cpu host,+invtsc,migratable=no. I think this is a matter of the right XML syntax or missing support in libvirt[1]. [Ccing Eduardo to confirm]

[1] https://www.redhat.com/archives/libvir-list/2014-September/msg01680.html

Comment 4 djdumas 2016-10-13 15:09:49 UTC
(In reply to Bandan Das from comment #3)
> (In reply to djdumas from comment #0)
> > Description of problem:
> > Starting a guest with <cpu mode='host-passthrough' and feature
> > policy='require' name='invtsc' does not provide cpu flag nonstop_tsc in the
> > guest.  
> > 
> I know we spoke a little bit about this before. On my 7.3 system, my guest
> sees nonstop_tsc if I specify -cpu host,+invtsc,migratable=no. I think this
> is a matter of the right XML syntax or missing support in libvirt[1]. [Ccing
> Eduardo to confirm]
> 
> [1] https://www.redhat.com/archives/libvir-list/2014-September/msg01680.html

One piece of information I didn't initially supply is that for this workload, host-passthrough performs significantly better than 'exact' even with nonstop_tsc missing.  I've accounted for other cpu flag differences and still cannot match host-passthrough performance. The hope is that passthrough plus nonstop_tsc will result in even better performance since HANA will no longer fall back to system calls.

Comment 5 Karen Noel 2016-10-13 17:04:01 UTC
(In reply to djdumas from comment #4)
> (In reply to Bandan Das from comment #3)
> > (In reply to djdumas from comment #0)
> > > Description of problem:
> > > Starting a guest with <cpu mode='host-passthrough' and feature
> > > policy='require' name='invtsc' does not provide cpu flag nonstop_tsc in the
> > > guest.  
> > > 

Dave, Can you check the qemu command line produced by libvirt?

> > I know we spoke a little bit about this before. On my 7.3 system, my guest
> > sees nonstop_tsc if I specify -cpu host,+invtsc,migratable=no. 

I thought migratable=no was automatic with invtsc.

> > I think this
> > is a matter of the right XML syntax or missing support in libvirt[1]. [Ccing
> > Eduardo to confirm]
> > 
> > [1] https://www.redhat.com/archives/libvir-list/2014-September/msg01680.html
> 
> One piece of information I didn't initially supply is that for this
> workload, host-passthrough performs significantly better than 'exact' even
> with nonstop_tsc missing.  I've accounted for other cpu flag differences and
> still cannot match host-passthrough performance. The hope is that
> passthrough plus nonstop_tsc will result in even better performance since
> HANA will no longer fall back to system calls.

Hold that thought. It might be a different problem. 

Adding jtomko who added the libvirt support. I thought -cpu host was required for invtsc, but maybe it's the other way around -  not allowed? Jan?

Comment 6 djdumas 2016-10-13 17:19:19 UTC
(In reply to Karen Noel from comment #5)
> 
> Dave, Can you check the qemu command line produced by libvirt?
> 

Hi Karen, see attachment

Comment 7 Ján Tomko 2016-10-14 07:51:12 UTC
(In reply to Karen Noel from comment #5)
> (In reply to djdumas from comment #4)
> > (In reply to Bandan Das from comment #3)
> > > I know we spoke a little bit about this before. On my 7.3 system, my guest
> > > sees nonstop_tsc if I specify -cpu host,+invtsc,migratable=no. 
> 
> I thought migratable=no was automatic with invtsc.
> 

I would also expect that. Why would the user need to say it twice?

Libvirt does not touch the migratable= option at all.
(And when the invtsc feature was added, it did not support
customizing the features for model='host-passthrough')

> > > I think this
> > > is a matter of the right XML syntax or missing support in libvirt[1]. [Ccing
> > > Eduardo to confirm]
> > > 
> > > [1] https://www.redhat.com/archives/libvir-list/2014-September/msg01680.html
> > 

That patch was to remove it from the automatically-generated feature list shown in the live XML.
Libvirt does not strip the flag if it was specified in the XML, even the attachment has:
-cpu host,+invtsc,+vme,...

> 
> Hold that thought. It might be a different problem. 
> 
> Adding jtomko who added the libvirt support. I thought -cpu host was
> required for invtsc, but maybe it's the other way around -  not allowed? Jan?

I do not know of any reason why libvirt would need to block it.

Comment 8 Bandan Das 2016-10-14 17:17:41 UTC
(In reply to Ján Tomko from comment #7)
> (In reply to Karen Noel from comment #5)
> > (In reply to djdumas from comment #4)
> > > (In reply to Bandan Das from comment #3)
> > > > I know we spoke a little bit about this before. On my 7.3 system, my guest
> > > > sees nonstop_tsc if I specify -cpu host,+invtsc,migratable=no. 
> > 
> > I thought migratable=no was automatic with invtsc.
> > 
> 
> I would also expect that. Why would the user need to say it twice?

"-cpu host" runs with migratable=on by default.

> Libvirt does not touch the migratable= option at all.
> (And when the invtsc feature was added, it did not support
> customizing the features for model='host-passthrough')
> 
> > > > I think this
> > > > is a matter of the right XML syntax or missing support in libvirt[1]. [Ccing
> > > > Eduardo to confirm]
> > > > 
> > > > [1] https://www.redhat.com/archives/libvir-list/2014-September/msg01680.html
> > > 
> 
> That patch was to remove it from the automatically-generated feature list
> shown in the live XML.
> Libvirt does not strip the flag if it was specified in the XML, even the
> attachment has:
> -cpu host,+invtsc,+vme,...
> 
> > 
> > Hold that thought. It might be a different problem. 
> > 
> > Adding jtomko who added the libvirt support. I thought -cpu host was
> > required for invtsc, but maybe it's the other way around -  not allowed? Jan?
> 
> I do not know of any reason why libvirt would need to block it.

Libvirt doesn't,  but is there a way to specify migratable=no in the guest XML ? I think that is what we want here.

Comment 9 Eduardo Habkost 2016-10-14 18:40:54 UTC
(In reply to Bandan Das from comment #8)
> (In reply to Ján Tomko from comment #7)
> > (In reply to Karen Noel from comment #5)
> > > (In reply to djdumas from comment #4)
> > > > (In reply to Bandan Das from comment #3)
> > > > > I know we spoke a little bit about this before. On my 7.3 system, my guest
> > > > > sees nonstop_tsc if I specify -cpu host,+invtsc,migratable=no. 
> > > 
> > > I thought migratable=no was automatic with invtsc.
> > > 
> > 
> > I would also expect that. Why would the user need to say it twice?
> 
> "-cpu host" runs with migratable=on by default.
> 
> > Libvirt does not touch the migratable= option at all.
> > (And when the invtsc feature was added, it did not support
> > customizing the features for model='host-passthrough')
> > 
> > > > > I think this
> > > > > is a matter of the right XML syntax or missing support in libvirt[1]. [Ccing
> > > > > Eduardo to confirm]
> > > > > 
> > > > > [1] https://www.redhat.com/archives/libvir-list/2014-September/msg01680.html
> > > > 
> > 
> > That patch was to remove it from the automatically-generated feature list
> > shown in the live XML.
> > Libvirt does not strip the flag if it was specified in the XML, even the
> > attachment has:
> > -cpu host,+invtsc,+vme,...
> > 
> > > 
> > > Hold that thought. It might be a different problem. 
> > > 
> > > Adding jtomko who added the libvirt support. I thought -cpu host was
> > > required for invtsc, but maybe it's the other way around -  not allowed? Jan?
> > 
> > I do not know of any reason why libvirt would need to block it.
> 
> Libvirt doesn't,  but is there a way to specify migratable=no in the guest
> XML ? I think that is what we want here.

That's true, invtsc is available only in migratable=off mode. But requiring explicit migratable=on would be different from the behavior of all other CPU models ("-cpu Haswell,+invtsc" works, for example). "-cpu host" could use the "migratable" property to choose which features will be enabled by default on "-cpu host", but not when filtering out features that were explicitly enabled by the user. We can probably address that on QEMU without changing libvirt.

Comment 10 Eduardo Habkost 2016-10-14 19:29:56 UTC
Upstream fix submitted:

From: Eduardo Habkost <ehabkost@redhat.com>
To: qemu-devel@nongnu.org
Subject: [PATCH] target-i386: Don't cpu->migratable field when filtering features
Date: Fri, 14 Oct 2016 16:28:14 -0300
Message-Id: <1476473294-11052-1-git-send-email-ehabkost@redhat.com>

Comment 11 Karen Noel 2016-12-16 17:03:47 UTC
Patch in 2.8.0. Move to POST.

Comment 16 Guo, Zhiyi 2017-03-16 08:47:18 UTC
Reproduce this issue against qemu-kvm-rhev-2.6.0-29.el7.x86_64.

Steps:
1.Boot rhel7.4 guest with qemu options: -cpu host,+invtsc and check whether nonstop_tsc present in guest.

Results:
Cannot find nonstop_tsc from guest by cat /proc/cpuinfo and qemu prompt:
QEMU 2.6.0 monitor - type 'help' for more information
(qemu) warning: host doesn't support requested feature: CPUID.80000007H:EDX.invtsc [bit 8]
warning: host doesn't support requested feature: CPUID.80000007H:EDX.invtsc [bit 8]
warning: host doesn't support requested feature: CPUID.80000007H:EDX.invtsc [bit 8]
warning: host doesn't support requested feature: CPUID.80000007H:EDX.invtsc [bit 8]

Verify against qemu-kvm-rhev-2.8.0-6.el7.x86_64, nothing prompt from qemu and nonstop_tsc can be found inside guest

Comment 17 Guo, Zhiyi 2017-03-16 08:48:14 UTC
Verified per comment 16

Comment 19 errata-xmlrpc 2017-08-01 23:37:14 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392

Comment 20 errata-xmlrpc 2017-08-02 01:14:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392

Comment 21 errata-xmlrpc 2017-08-02 02:06:52 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392

Comment 22 errata-xmlrpc 2017-08-02 02:47:39 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392

Comment 23 errata-xmlrpc 2017-08-02 03:12:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392

Comment 24 errata-xmlrpc 2017-08-02 03:32:30 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392


Note You need to log in before you can comment on or make changes to this bug.