This bug has been migrated to another issue tracking site. It has been closed here and may no longer be being monitored.

If you would like to get updates for this issue, or to participate in it, you may do so at Red Hat Issue Tracker .
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2184282 - [AMD] CPU flags don't match between QEMU cmdline and Libvirt xml(dumpxml) on some AMD hosts
Summary: [AMD] CPU flags don't match between QEMU cmdline and Libvirt xml(dumpxml) on ...
Keywords:
Status: CLOSED MIGRATED
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: qemu-kvm
Version: 9.3
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: rc
: ---
Assignee: Bandan Das
QA Contact: liunana
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-04-04 06:15 UTC by liunana
Modified: 2023-09-27 17:13 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-09-27 17:13:03 UTC
Type: ---
Target Upstream Version:
Embargoed:
pm-rhel: mirror+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker   RHEL-10090 0 None Migrated None 2023-09-27 17:12:27 UTC
Red Hat Issue Tracker RHELPLAN-153961 0 None None None 2023-04-04 06:17:56 UTC

Description liunana 2023-04-04 06:15:48 UTC
Description of problem:
CPU flags don't match between QEMU cmdline and Libvirt xml(dumpxml) on some AMD hosts


Version-Release number of selected component (if applicable):
Test Env:
5.14.0-252.el9.x86_64
qemu-kvm-7.2.0-6.el9.x86_64
libvirt-9.0.0-3.el9.x86_64
Model name:            AMD EPYC 7313 16-Core Processor


How reproducible: 100%


Steps to Reproduce:
1.  define one basic xml file
 # cat test.xml
     <domain type='kvm'>
      <name>test</name>
      <memory>1</memory>
      <os>
        <type arch='x86_64' machine='q35'>hvm</type>
      </os>
      <cpu mode='host-model'/>
    </domain>

2.  start libvirt with above xml configuration. Then check the cpu element by 'virsh dumpxml test' when it's running.

  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>EPYC-Milan</model>
    <vendor>AMD</vendor>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='tsc-deadline'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='vaes'/>
    <feature policy='require' name='vpclmulqdq'/>
    <feature policy='require' name='spec-ctrl'/>
    <feature policy='require' name='stibp'/>
    <feature policy='require' name='arch-capabilities'/>
    <feature policy='require' name='ssbd'/>
    <feature policy='require' name='cmp_legacy'/>
    <feature policy='require' name='virt-ssbd'/>
    <feature policy='disable' name='lbrv'/>
    <feature policy='disable' name='tsc-scale'/>
    <feature policy='disable' name='vmcb-clean'/>
    <feature policy='disable' name='pause-filter'/>
    <feature policy='disable' name='pfthreshold'/>
    <feature policy='disable' name='v-vmsave-vmload'/>
    <feature policy='disable' name='vgif'/>
    <feature policy='require' name='rdctl-no'/>
    <feature policy='require' name='skip-l1dfl-vmentry'/>
    <feature policy='require' name='mds-no'/>
    <feature policy='require' name='pschange-mc-no'/>
    <feature policy='disable' name='svm'/>
    <feature policy='require' name='topoext'/>
    <feature policy='disable' name='npt'/>
    <feature policy='disable' name='nrip-save'/>
    <feature policy='disable' name='svme-addr-chk'/>
  </cpu>

3. Check the QEMU cpu command lines with 'ps -aux | grep qemu'

-cpu EPYC-Milan,x2apic=on,tsc-deadline=on,hypervisor=on,tsc-adjust=on,vaes=on,vpclmulqdq=on,spec-ctrl=on,stibp=on,arch-capabilities=on,ssbd=on,cmp-legacy=on,virt-ssbd=on,lbrv=on,tsc-scale=on,vmcb-clean=on,pause-filter=on,pfthreshold=on,v-vmsave-vmload=on,vgif=on,rdctl-no=on,skip-l1dfl-vmentry=on,mds-no=on,pschange-mc-no=on


Actual results:
1. we can see libvirt disables lots of features but in qemu command line, these features are still 'on'.

2. Boot a separate qemu process with above qemu command line, I can get lots of warnings.
# /usr/libexec/qemu-kvm -qmp unix:/tmp/mm,server=on,wait=off -M q35 -cpu EPYC-Milan,x2apic=on,tsc-deadline=on,hypervisor=on,tsc-adjust=on,vaes=on,vpclmulqdq=on,spec-ctrl=on,stibp=on,arch-capabilities=on,ssbd=on,cmp-legacy=on,virt-ssbd=on,lbrv=on,tsc-scale=on,vmcb-clean=on,pause-filter=on,pfthreshold=on,v-vmsave-vmload=on,vgif=on,rdctl-no=on,skip-l1dfl-vmentry=on,mds-no=on,pschange-mc-no=on -monitor stdio
QEMU 7.2.0 monitor - type 'help' for more information
(qemu) qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.lbrv [bit 1]
qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.tsc-scale [bit 4]
qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.vmcb-clean [bit 5]
qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.pause-filter [bit 10]
qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.pfthreshold [bit 12]
qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.v-vmsave-vmload [bit 15]
qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.vgif [bit 16]
VNC server running on ::1:5900

3. Run `query-cpu-model-expansion` with model name `host`, these features all show 'True'.
{ "execute":"query-cpu-model-expansion","arguments":{"type":"full","model":{"name":"host"}}}

Expected results:
They should be the same

Additional info:
Such situation also happens on AMD Rome and AMD Geona,

Comment 3 Bandan Das 2023-04-06 17:54:20 UTC
I think the first part is a bug in that the value in XML should be reflected in the Qemu command line.
But since all of these are virt related features(?), they are all meaningless since svm=off. Jiri, any thoughts ?

Coming to:
(qemu) qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.lbrv [bit 1]
qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.tsc-scale [bit 4]
qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.vmcb-clean [bit 5]
qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.pause-filter [bit 10]
qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.pfthreshold [bit 12]
qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.v-vmsave-vmload [bit 15]
qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.vgif [bit 16]

I just picked up one random bit from the list above which is vGIF. This feature allows STGI and CLGI in guest mode and 
so even if it's available, it's not meaningful unless svm is enabled. I think that's what the warning message says above.

I would think the only harm this bug does is print unexpected information.

Comment 4 Jiri Denemark 2023-04-12 08:21:19 UTC
Well, the problem here is that <cpu mode='host-model'/> does not really work
as expected. We probe QEMU what features would be enabled with -cpu host and
transform that into a CPU model and a list of enabled/disabled features so
that the domain can safely be migrated as long as the destination host is able
to provide the same vCPU.

The probing is done using CPU model expansion on "host" model, which reports
all these features including "svm" as enabled. But, we know (or rather think)
the EPYC-Milan CPU model already includes svm so we do not ask for it
explicitly on the command line. Even looking at QEMU definition of the CPU
model confirms this. Unfortunately when running QEMU with -cpu EPYC-Milan,
"svm" is not actually enabled because of some runtime magic that disables it
despite it being specified in the CPU model definition. And because we
explicitly ask for some additional features which are not part of EPYC-Milan
and depend on "svm", QEMU logically complains about them and refuses to enable
them. This is all correctly reflected in the domain XML once QEMU is running.
So this part works OK.

If "svm" is explicitly requested on the command line, everything works as
expected and QEMU does not complain about anything. But this is not what
happens with host-model. So the question is why "svm" gets disabled when it's
not explicitly requested even though EPYC-Milan is defined as

        .name = "EPYC-Milan",
        .level = 0xd,
        .vendor = CPUID_VENDOR_AMD,
        .family = 25,
        .model = 1,
        .stepping = 1,
        ...
        .features[FEAT_8000_0001_ECX] =
            CPUID_EXT3_OSVW | CPUID_EXT3_3DNOWPREFETCH |
            CPUID_EXT3_MISALIGNSSE | CPUID_EXT3_SSE4A | CPUID_EXT3_ABM |
            CPUID_EXT3_CR8LEG | CPUID_EXT3_SVM | CPUID_EXT3_LAHF_LM |     /* <-- SVM enabled here */
            CPUID_EXT3_TOPOEXT | CPUID_EXT3_PERFCORE,
        ...

Theoretically we could be able to fix this on libvirt side, but our options
are quite limited:

    - Ideally we would ask QEMU what features would be enabled for EPYC-Milan
      in runtime (because just reporting the static definition is not enough
      in this case), but we would need to do so at the time we probe QEMU
      capabilities, i.e., when QEMU is started with "-machine none". Knowing
      the exact definition, we could explicitly request all feature enabled
      for -cpu host but not included in the model definition. AFAIK this is
      not supported by QEMU

    - We could explicitly ask for all enabled features, even those included in
      a given CPU model. This would result in a terribly long command line.

    - We could selectively request some features explicitly even though they
      are included in a given CPU model. But the question is which features
      would need such treatment. Just "svm" or more features? And do we need
      to do so for all CPU models or just some of them?

I don't see a clear winner here. Except for the first option, which is not
doable without non-trivial (I expect) work in QEMU.

Comment 5 liunana 2023-05-23 10:02:01 UTC
Hi Bandan,


Could you please help to check Comment 4?
Do we have a solution for this bug? Thanks.


Best regards
Nana

Comment 6 Bandan Das 2023-06-28 16:03:12 UTC
(In reply to liunana from comment #5)
> Hi Bandan,
> 
> 
> Could you please help to check Comment 4?
> Do we have a solution for this bug? Thanks.
> 
> 
> Best regards
> Nana

Nana, unfortunately, I have to keep this in the backlog for now because 
I am not sure what the right approach for upstream is. I will keep track of it
and update if something changes.

Comment 9 RHEL Program Management 2023-09-27 17:07:58 UTC
Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.

Comment 10 RHEL Program Management 2023-09-27 17:13:03 UTC
This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.


Note You need to log in before you can comment on or make changes to this bug.