AVIC doesn't virtualize x2APIC. So based on the description above, I think libvirt sees the conflict between AVIC and x2APIC. The confusing part about this BZ is where this CPU is not x2APIC-capable (you have to wait for Rome). So where did this detected x2APIC feature come from?

I tried RHEL 7.6 update 20190220.0 on "EPYC 7601" and can't re-produce it. So is this CentOS-specific problem? I will try CentOS in the next phase.

-Wei

These are oVirt hosts, although I don't know if oVirt makes any changes to libvirt that would affect this.

This is the bug report on the oVirt side:
https://bugzilla.redhat.com/show_bug.cgi?id=1674265

If you need an additional information on the systems please let me know.

(In reply to Ryan from comment #3)
> These are oVirt hosts, although I don't know if oVirt makes any changes to
> libvirt that would affect this.
> 
> This is the bug report on the oVirt side:
> https://bugzilla.redhat.com/show_bug.cgi?id=1674265
> 
> If you need an additional information on the systems please let me know.

Right now the EPYC machine I borrowed can't install CentOS nor oVirt. But RHEL 7.6 actually works fine. Could you give me the info of these the following packages on the failed machine?

* rpm -qa | grep qemu
* rpm -qa | grep libvirt

rpm -qa | grep qemu
qemu-img-ev-2.12.0-18.el7_6.3.1.x86_64
qemu-kvm-common-ev-2.12.0-18.el7_6.3.1.x86_64
ipxe-roms-qemu-20170123-1.git4e85b27.el7_4.1.noarch
libvirt-daemon-driver-qemu-4.5.0-10.el7_6.4.x86_64
qemu-kvm-ev-2.12.0-18.el7_6.3.1.x86_64

rpm -qa | grep libvirt
libvirt-bash-completion-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-driver-nwfilter-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-driver-storage-logical-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-driver-storage-iscsi-4.5.0-10.el7_6.4.x86_64
libvirt-client-4.5.0-10.el7_6.4.x86_64
libvirt-python-4.5.0-1.el7.x86_64
libvirt-daemon-config-nwfilter-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-driver-interface-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-driver-storage-disk-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-driver-storage-gluster-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-kvm-4.5.0-10.el7_6.4.x86_64
libvirt-libs-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-driver-network-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-driver-secret-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-driver-lxc-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-driver-storage-rbd-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-driver-storage-4.5.0-10.el7_6.4.x86_64
libvirt-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-driver-storage-core-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-config-network-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-driver-storage-scsi-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-driver-qemu-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-driver-nodedev-4.5.0-10.el7_6.4.x86_64
libvirt-daemon-driver-storage-mpath-4.5.0-10.el7_6.4.x86_64
libvirt-lock-sanlock-4.5.0-10.el7_6.4.x86_64

Looks like qemu-*-ev packages are being used instead of the base qemu-kvm. These are coming from the oVirt repo (http://mirror.centos.org/centos/7/virt/x86_64/kvm-common/). Wonder if that is causing the problem?

So I can reproduce the problem described in Description after replacing qemu-kvm packages with -ev versions. EPYC CPU models are marked as useable='no' in /var/cache/libvirt/qemu/capabilities/ after AVIC is enabled. But the guest installation (RHEL8) actually was successful after selecting EPYC-IBPB cpu model. In other words, the results in /var/cache/libvirt/qemu/capabilities/ didn't prevent the installation from completion. Maybe oVirt follows the capability file more strictly than RHEL virt.

With that said, we still need to figure out why x2APIC can become a blocker for EPYC when AVIC=1.

Here are the findings for this problem:

1. x2apic isn't support by AVIC in current design. You can find a confirmation message from https://lwn.net/Articles/675382/ which indicate that x2apic will be turned off when AVIC is enabled.

2. In KVM code, if you look at svm.c file, it clears x2apic CPUID bit when avic is turned on:

static void svm_set_supported_cpuid(u32 func, struct kvm_cpuid_entry2 *entry)
{
        switch (func) {
	case 0x1:
                if (avic)
                        entry->ecx &= ~bit(X86_FEATURE_X2APIC);
 <...>
}

3. QEMU adds x2apic as a default feature for all CPU models when KVM is enabled (search kvm_default_props target/i386/cpu.c file). If this feature is not support by KVM, it actually send out a warning message. Actually you can check the constraint yourslef: "/usr/libexec/qemu-kvm -cpu EPYC,check".

4. libvirt probes QEMU regarding the supported capabilities. Such info is stored in /var/cache/libvirt/qemu/capabilities/. That is why you saw EPYC cpu is _not_ usable.

One way to solve this problem to remove the requirement of x2apic. In a manual command, you can fix this problem with something like "/usr/libexec/qemu-kvm -cpu EPYC,-x2apic". You can also do it with libvirt guest config XML file, something like:

    <model fallback='allow'>EPYC</model>
    <vendor>AMD</vendor>
    //blah blah
    <feature policy='disable' name='x2apic'/>

Since this problem is benign and has a solution, I will go ahead to close this BZ.