Bug 1259200 - old machine types in qemu-kvm-rhev should not expose vPMU to KVM guest
old machine types in qemu-kvm-rhev should not expose vPMU to KVM guest
Status: CLOSED CANTFIX
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev (Show other bugs)
7.2
x86_64 All
high Severity high
: rc
: ---
Assigned To: Wei Huang
Virtualization Bugs
Jiri Herrmann
: FutureFeature
Depends On:
Blocks: 1259210
  Show dependency treegraph
 
Reported: 2015-09-02 03:46 EDT by Xiaoqing Wei
Modified: 2016-09-27 10:02 EDT (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Known Issue
Doc Text:
Cause: New KVM code allows KVM hosts to emulate a PMU on AMD CPUs, but migration support for the PMU is not available. Consequence: KVM guests will detect a PMU when running on RHEL-7.2 hosts, and will be able to use it. However, the PMU state won't be migrated if the VM is migrated to another host. So the PMU is not fully functional if the VM is migrated. Workaround (if any): Using the PMU in a KVM guest is still not supported, so users are advised to not rely on performance counters being fully functional. Result: No PMU-related errors if the PMU is not used. See Also: Bug 1076010 - failed to access perfctr error shown after VM clean install (without -cpu host)
Story Points: ---
Clone Of:
: 1259210 (view as bug list)
Environment:
Last Closed: 2016-09-27 09:58:50 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Xiaoqing Wei 2015-09-02 03:46:08 EDT
Description of problem:

Recently, with the backport of PMU series into RHEL 7,

75f2bf3 [x86] kvm: Enable PMU handling for AMD PERFCTRn and EVNTSELn MSRs
65be410 [x86] kvm: Implement AMD vPMU code for KVM
c7c4aa0 [x86] kvm: Define kvm_pmu_ops to support vPMU function dispatch
9d57867 [x86] kvm: vpmu: introduce kvm_pmu_msr_idx_to_pmc
148c080 [x86] kvm: vpmu: reorder PMU functions
6e0cbda [x86] kvm: vpmu: whitespace and stylistic adjustments in PMU code
74212f8 [x86] kvm: vpmu: use the new macros to go between PMC, PMU and VCPU
56b426d [x86] kvm: vpmu: introduce pmu.h header
e13fb99 [x86] kvm: vpmu: rename a few PMU functions

SVM hosts would expose K7 level PMU feature to KVM guests, without a chance to turn on or off.

boot vm the -M pc-i440fx-rhel7.0.0 machine type still sees PMU in guest dmesg

Since machine type could not control this guest awared cpuid change, it is likely to broke migration.

Version-Release number of selected component (if applicable):
kernel-3.10.0-290.el7.x86_64 or later (the kernel on host, which 290 has patches above)
qemu-kvm-rhev-2.3.0-21.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. boot a VM on AMD host(host with kernel >= 290)
    -machine pc-i440fx-rhel7.0.0 \
    -cpu 'Opteron_G5'

2. in guest: dmesg|less to check whether PMU is detected
3.

Actual results:
PMU detected

Expected results:
PMU shouldn't detected with old machine type.
7.0.0 GA does not able to expose such feature to guest.

Additional info:

Since the PMU emulation codes are in host kernel, it's likely qemu-kvm (rhel) would be affected too, I'll try and clone(if it affected) later.
Comment 1 Xiaoqing Wei 2015-09-02 03:49:33 EDT
Hello Wei,

Could you please help to have a look at this bug ?

Thank you.
Comment 3 Xiaoqing Wei 2015-09-02 04:08:25 EDT
thinking use machine type to control would be better then introduce new option, thus change the bug summary.
Comment 4 Andrew Jones 2015-09-04 09:22:52 EDT
Here are the scenarios I can think of

              to-version
probed            vs.                          result
              from-version
-------------------------------------------------------
before        >=                               OK
after         >=                               OK
before        <                                output "unimplemented msr" on use
after         <                                OK, PMU not used

Note, the '<' scenarios are probably not supported.

So, unless we can come up with a supported, potentially problematic scenario to test (and find that it is indeed problematic by testing), then I think this bug should be closed as not-a-bug.
Comment 5 Karen Noel 2015-10-03 07:59:23 EDT
Can someone confirm that live migration is not broken between RHEL 7.1 <--> RHEL 7.2 hosts with 6.5 and 7.1 machine types on AMD? Thanks.
Comment 6 Wei Huang 2015-10-06 00:45:37 EDT
(In reply to Karen Noel from comment #5)
> Can someone confirm that live migration is not broken between RHEL 7.1 <-->
> RHEL 7.2 hosts with 6.5 and 7.1 machine types on AMD? Thanks.

I tried migration between RHEL 7.1 and 7.2 hosts. The guest VM was a fresh copy installed from RHEL 7.1 ISO and I tested two VM machine types: RHEL6.5.0 and pc-i440fx-rhel7.0.0 (default). The migration didn't break based on what I saw. 

-Wei
Comment 7 juzhang 2015-10-08 02:49:45 EDT
(In reply to Karen Noel from comment #5)
> Can someone confirm that live migration is not broken between RHEL 7.1 <-->
> RHEL 7.2 hosts with 6.5 and 7.1 machine types on AMD? Thanks.

Hi Xiaoqing,

Could you give a update?

Best Regards,
Junyi
Comment 8 Xiaoqing Wei 2015-10-08 05:16:04 EDT
(In reply to juzhang from comment #7)
> (In reply to Karen Noel from comment #5)
> > Can someone confirm that live migration is not broken between RHEL 7.1 <-->
> > RHEL 7.2 hosts with 6.5 and 7.1 machine types on AMD? Thanks.
> 
> Hi Xiaoqing,
> 
> Could you give a update?
> 

Sure,

the test result posted by Wei in C#6 are identical to mine.
the VM is working after migration, but the underlying PMU bit has changed.
I filled this bug as I thought we should no bring hidden change to old machine types, that could be a potential break.
Fix me if I am wrong.

Regards,
Xiaoqing.
Comment 9 Paolo Bonzini 2015-10-19 05:57:14 EDT
Xiaoqing is right and I think this would ordinarily be a problem, but isn't PMU support technology preview?  In this case I think it's enough to add something to the release notes.
Comment 10 Eduardo Habkost 2015-10-19 11:25:13 EDT
PMU reporting on CPUID is disabled since qemu-kvm-1.5.2-3.el7 to avoid this kind of problem (see bug 853101). Can you clarify which CPUID bits are changing on migration, and what are the PMU messages seen on guest dmesg?
Comment 11 Xiaoqing Wei 2015-10-19 23:30:47 EDT
(In reply to Eduardo Habkost from comment #10)
> PMU reporting on CPUID is disabled since qemu-kvm-1.5.2-3.el7 to avoid this
> kind of problem (see bug 853101). Can you clarify which CPUID bits are
> changing on migration, and what are the PMU messages seen on guest dmesg?

Eduardo,

I dont have two AMD host on hand, could not test it right now.
Will try to find another rig for cross host live migration.

My guess is this bit: CPUID_EXT3_PERFNB

I found this change by booting VM on host with kernels with 7.1 GA, and 7.2(> 290), and check each VM's /proc/cpuinfo.
Comment 12 Eduardo Habkost 2015-10-20 15:38:56 EDT
By looking at the guest Linux code, it looks like it will enable the AMD PMU driver unconditionally if CPU vendor is AMD and CPUID family >= 6. I'm not sure there's anything we can do on QEMU side.
Comment 13 Xiaoqing Wei 2015-10-20 22:58:19 EDT
(In reply to Eduardo Habkost from comment #12)
> By looking at the guest Linux code, it looks like it will enable the AMD PMU
> driver unconditionally if CPU vendor is AMD and CPUID family >= 6.


Yes, that's exactly what I mean. unconditionally expose to guest.
but isn't it on host kernel side ?



 I'm not
> sure there's anything we can do on QEMU side.

I dont know the detail, just rem that vPMU for Intel could be controlled by machine type ?
Comment 14 Eduardo Habkost 2015-10-21 10:19:58 EDT
(In reply to Xiaoqing Wei from comment #13)
> (In reply to Eduardo Habkost from comment #12)
> > By looking at the guest Linux code, it looks like it will enable the AMD PMU
> > driver unconditionally if CPU vendor is AMD and CPUID family >= 6.
> 
> 
> Yes, that's exactly what I mean. unconditionally expose to guest.
> but isn't it on host kernel side ?

That's on the guest side. If CPU vendor is AMD and family >= 6 you will see "Performance Events: AMD PMU driver" in guest dmesg, it doesn't matter what's the host behavior (you would see that message even if there were absolutely no PMU emulation code implemented on the host side).

>  I'm not
> > sure there's anything we can do on QEMU side.
> 
> I dont know the detail, just rem that vPMU for Intel could be controlled by
> machine type ?

We disable reporting of architectural perfomance counters on CPUID[0xA], but we can't prevent the guest from making assumptions about the PMU based on family/model values.


I would like to clarify this from comment #11:

> I found this change by booting VM on host with kernels with 7.1 GA, and 7.2(> 290), and check each VM's /proc/cpuinfo.

You shouldn't see any difference on /proc/cpuinfo when changing the host version. Could you clarify what exactly looks different on the guest-side depending on the host version?
Comment 15 Xiaoqing Wei 2015-10-21 23:02:27 EDT
(In reply to Eduardo Habkost from comment #14)

> 
> I would like to clarify this from comment #11:
> 
> > I found this change by booting VM on host with kernels with 7.1 GA, and 7.2(> 290), and check each VM's /proc/cpuinfo.
> 
> You shouldn't see any difference on /proc/cpuinfo when changing the host
> version. Could you clarify what exactly looks different on the guest-side
> depending on the host version?

Oops, my fault, it's in dmesg

as C#0, step2
2. in guest: dmesg|less to check whether PMU is detected
Comment 16 Eduardo Habkost 2015-10-23 14:12:44 EDT
(In reply to Xiaoqing Wei from comment #15)
> > I would like to clarify this from comment #11:
> > 
> > > I found this change by booting VM on host with kernels with 7.1 GA, and 7.2(> 290), and check each VM's /proc/cpuinfo.
> > 
> > You shouldn't see any difference on /proc/cpuinfo when changing the host
> > version. Could you clarify what exactly looks different on the guest-side
> > depending on the host version?
> 
> Oops, my fault, it's in dmesg
> 
> as C#0, step2
> 2. in guest: dmesg|less to check whether PMU is detected

So, did you find actual changes in dmesg, or did you only noticed that the guest unconditionally assumes a PMU is available on all cases? Like Drew suggested on comment #4, I suggest we close this as NOTABUG unless there's something that actually breaks when upgrading the host.
Comment 17 Xiaoqing Wei 2015-10-25 05:17:52 EDT
(In reply to Eduardo Habkost from comment #16)
> (In reply to Xiaoqing Wei from comment #15)
> > > I would like to clarify this from comment #11:
> > > 
> > > > I found this change by booting VM on host with kernels with 7.1 GA, and 7.2(> 290), and check each VM's /proc/cpuinfo.
> > > 
> > > You shouldn't see any difference on /proc/cpuinfo when changing the host
> > > version. Could you clarify what exactly looks different on the guest-side
> > > depending on the host version?
> > 
> > Oops, my fault, it's in dmesg
> > 
> > as C#0, step2
> > 2. in guest: dmesg|less to check whether PMU is detected
> 
> So, did you find actual changes in dmesg, or did you only noticed that the
> guest unconditionally assumes a PMU is available on all cases? Like Drew
> suggested on comment #4, I suggest we close this as NOTABUG unless there's
> something that actually breaks when upgrading the host.

Yes, dmesg does changes, booting a vm on host with kernels before and after vPMU patches(guest img is same one, same kernel version)

on kernel-290+
Actual results:
PMU detected in guest dmesg

on old kernel, eg: kernel-123
Expected results:
PMU does not detected in guest dmesg
Comment 18 Eduardo Habkost 2015-10-26 15:31:13 EDT
I confirm that when running kernel-120 on the host, I see this:
"Broken PMU hardware detected, using software events only."

That's because the guest kernel checks if the PMU MSRs really work, before enabling PMU usage. But I don't think we can do anything to avoid that, if the kernel doesn't provide an interface to disable PMU emulation.
Comment 21 Paolo Bonzini 2016-09-27 09:58:50 EDT
There is no CPUID bit for the AMD MSRs 0xC0010000..0xC0010007.

PERFNB refers to MSRs 0xC0010200..0xC001020B

Note You need to log in before you can comment on or make changes to this bug.