Bug 1675030 - Setting kvm_amd avic=1 requires x2apic (intel) on AMD Processors
Summary: Setting kvm_amd avic=1 requires x2apic (intel) on AMD Processors
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm
Version: 7.6
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: rc
: ---
Assignee: Wei Huang (AMD)
QA Contact: Guo, Zhiyi
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-02-11 18:35 UTC by Ryan
Modified: 2019-08-21 06:43 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-02-27 22:58:44 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Ryan 2019-02-11 18:35:43 UTC
Description:
Not sure if this libvirt or qemu

This is verified by checking the libvirt qemu capabilities cache (/var/cache/libvirt/qemu/capabilities/)

If you are running an AMD CPU and enable avic=1, qemu will show AMD CPU models as usable='no' with <blocker name='x2apic'/>.

As far as I can tell 'x2apic' is an Intel only cpu flag. This should probably be checking for the AMD 'avic' flag.

Reproducibility:
Always

Steps to reproduce:
On an AMD host.
1. Start libvirtd
2. Check qemu capabilities cache. Applicable AMD cpu types will be shown as useable='yes'
3. Set 'options kvm_amd avic=1' in /etc/modprobe.d/kvm.conf
4. Unload and reload the kvm_amd module
5. Verify avic is set 'cat /sys/module/kvm_amd/parameters/avic' should be 1
6. Delete the qemu capabilities cache to ensure it gets re-detected. ' rm -f /var/cache/libvirt/qemu/capabilities/*.xml
7. Restart or start libvirtd
8. Check the capabilities cache. less /var/cache/libvirt/qemu/capabilities/*.xml
Applicable AMD CPUs will be shown with usable='no' and <blocker name='x2apic'/>

Additional Information:
Testing was done an AMD EPYC CPU.

I think qemu will silently drop unsupported flags when running a VM, so even though we had avic=1 enabled we could run VMs without error.

I don't know if enabling avic actually does anything if the x2apic flag is not present.

This caused issues for us during an upgrade to oVirt. Mailing list thread:
https://lists.ovirt.org/archives/list/users@ovirt.org/thread/4Y4X7UGDEYSB5JK45TLDERNM7IMTHIYY/

OS Info:
CentOS Linux release 7.6.1810

cpuinfo:
head -n26 /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 23
model : 1
model name : AMD EPYC 7551P 32-Core Processor
stepping : 2
microcode : 0x8001227
cpu MHz : 2000.000
cache size : 512 KB
physical id : 0
siblings : 64
core id : 0
cpu cores : 32
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate sme retpoline_amd ssbd ibpb vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 clzero irperf xsaveerptr arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif overflow_recov succor smca
bogomips : 3992.33
TLB size : 2560 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 43 bits physical, 48 bits virtual
power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]

dmesg output showing vapic support:
dmesg | grep -i AMD-Vi
[ 1.997765] AMD-Vi: IOMMU performance counters supported
[ 1.997824] AMD-Vi: IOMMU performance counters supported
[ 1.997854] AMD-Vi: IOMMU performance counters supported
[ 1.997897] AMD-Vi: IOMMU performance counters supported
[ 2.021815] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
[ 2.021816] AMD-Vi: Extended features (0xf77ef22294ada):
[ 2.021821] AMD-Vi: Found IOMMU at 0000:20:00.2 cap 0x40
[ 2.021822] AMD-Vi: Extended features (0xf77ef22294ada):
[ 2.021825] AMD-Vi: Found IOMMU at 0000:40:00.2 cap 0x40
[ 2.021827] AMD-Vi: Extended features (0xf77ef22294ada):
[ 2.021830] AMD-Vi: Found IOMMU at 0000:60:00.2 cap 0x40
[ 2.021831] AMD-Vi: Extended features (0xf77ef22294ada):
[ 2.021833] AMD-Vi: Interrupt remapping enabled
[ 2.021834] AMD-Vi: virtual APIC enabled
[ 2.022218] AMD-Vi: Lazy IO/TLB flushing enabled

I originally opened this on the CentOS bug tracker, but was told to open it upstream:
https://bugs.centos.org/view.php?id=15814

Comment 2 Wei Huang (AMD) 2019-02-21 16:41:03 UTC
AVIC doesn't virtualize x2APIC. So based on the description above, I think libvirt sees the conflict between AVIC and x2APIC. The confusing part about this BZ is where this CPU is not x2APIC-capable (you have to wait for Rome). So where did this detected x2APIC feature come from?

I tried RHEL 7.6 update 20190220.0 on "EPYC 7601" and can't re-produce it. So is this CentOS-specific problem? I will try CentOS in the next phase.

-Wei

Comment 3 Ryan 2019-02-21 17:30:27 UTC
These are oVirt hosts, although I don't know if oVirt makes any changes to libvirt that would affect this.

This is the bug report on the oVirt side:
https://bugzilla.redhat.com/show_bug.cgi?id=1674265

If you need an additional information on the systems please let me know.

Comment 4 Wei Huang (AMD) 2019-02-22 05:41:27 UTC
(In reply to Ryan from comment #3)
> These are oVirt hosts, although I don't know if oVirt makes any changes to
> libvirt that would affect this.
> 
> This is the bug report on the oVirt side:
> https://bugzilla.redhat.com/show_bug.cgi?id=1674265
> 
> If you need an additional information on the systems please let me know.

Right now the EPYC machine I borrowed can't install CentOS nor oVirt. But RHEL 7.6 actually works fine. Could you give me the info of these the following packages on the failed machine?

* rpm -qa | grep qemu
* rpm -qa | grep libvirt

Comment 5 Ryan 2019-02-22 06:03:58 UTC
rpm -qa | grep qemu
qemu-img-ev-2.12.0-18.el7_6.3.1.x86_64
qemu-kvm-common-ev-2.12.0-18.el7_6.3.1.x86_64
ipxe-roms-qemu-20170123-1.git4e85b27.el7_4.1.noarch
libvirt-daemon-driver-qemu-4.5.0-10.el7_6.4.x86_64
qemu-kvm-ev-2.12.0-18.el7_6.3.1.x86_64

rpm -qa | grep libvirt
libvirt-bash-completion-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-driver-nwfilter-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-driver-storage-logical-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-driver-storage-iscsi-4.5.0-10.el7_6.4.x86_64
libvirt-client-4.5.0-10.el7_6.4.x86_64
libvirt-python-4.5.0-1.el7.x86_64
libvirt-daemon-config-nwfilter-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-driver-interface-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-driver-storage-disk-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-driver-storage-gluster-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-kvm-4.5.0-10.el7_6.4.x86_64
libvirt-libs-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-driver-network-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-driver-secret-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-driver-lxc-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-driver-storage-rbd-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-driver-storage-4.5.0-10.el7_6.4.x86_64
libvirt-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-driver-storage-core-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-config-network-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-driver-storage-scsi-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-driver-qemu-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-driver-nodedev-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-driver-storage-mpath-4.5.0-10.el7_6.4.x86_64
libvirt-lock-sanlock-4.5.0-10.el7_6.4.x86_64

Looks like qemu-*-ev packages are being used instead of the base qemu-kvm. These are coming from the oVirt repo (http://mirror.centos.org/centos/7/virt/x86_64/kvm-common/). Wonder if that is causing the problem?

Comment 6 Wei Huang (AMD) 2019-02-25 22:41:03 UTC
So I can reproduce the problem described in Description after replacing qemu-kvm packages with -ev versions. EPYC CPU models are marked as useable='no' in /var/cache/libvirt/qemu/capabilities/ after AVIC is enabled. But the guest installation (RHEL8) actually was successful after selecting EPYC-IBPB cpu model. In other words, the results in /var/cache/libvirt/qemu/capabilities/ didn't prevent the installation from completion. Maybe oVirt follows the capability file more strictly than RHEL virt.

With that said, we still need to figure out why x2APIC can become a blocker for EPYC when AVIC=1.

Comment 7 Wei Huang (AMD) 2019-02-27 22:58:02 UTC
Here are the findings for this problem:

1. x2apic isn't support by AVIC in current design. You can find a confirmation message from https://lwn.net/Articles/675382/ which indicate that x2apic will be turned off when AVIC is enabled.

2. In KVM code, if you look at svm.c file, it clears x2apic CPUID bit when avic is turned on:

static void svm_set_supported_cpuid(u32 func, struct kvm_cpuid_entry2 *entry)
{
        switch (func) {
	case 0x1:
                if (avic)
                        entry->ecx &= ~bit(X86_FEATURE_X2APIC);
 <...>
}

3. QEMU adds x2apic as a default feature for all CPU models when KVM is enabled (search kvm_default_props target/i386/cpu.c file). If this feature is not support by KVM, it actually send out a warning message. Actually you can check the constraint yourslef: "/usr/libexec/qemu-kvm -cpu EPYC,check".

4. libvirt probes QEMU regarding the supported capabilities. Such info is stored in /var/cache/libvirt/qemu/capabilities/. That is why you saw EPYC cpu is _not_ usable.

One way to solve this problem to remove the requirement of x2apic. In a manual command, you can fix this problem with something like "/usr/libexec/qemu-kvm -cpu EPYC,-x2apic". You can also do it with libvirt guest config XML file, something like:

    <model fallback='allow'>EPYC</model>
    <vendor>AMD</vendor>
    //blah blah
    <feature policy='disable' name='x2apic'/>

Comment 8 Wei Huang (AMD) 2019-02-27 22:58:44 UTC
Since this problem is benign and has a solution, I will go ahead to close this BZ.


Note You need to log in before you can comment on or make changes to this bug.