RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1915229 - Guest with host-model cpu type can not migrate from rhel-av 8.2.1 to rhel-av 8.3.1 as "missing features: ssbd,amd-ssbd"
Summary: Guest with host-model cpu type can not migrate from rhel-av 8.2.1 to rhel-av ...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: qemu-kvm
Version: 9.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Dr. David Alan Gilbert
QA Contact: Min Deng
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-01-12 09:45 UTC by yalzhang@redhat.com
Modified: 2023-02-28 01:40 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-07-12 07:27:27 UTC
Type: Feature Request
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
content of /var/log/libvirt/qemu/test.log on target host (8.34 KB, text/plain)
2021-01-12 14:43 UTC, yalzhang@redhat.com 		no flags Details
libvirtd log on source host (2.87 MB, text/plain)
2021-02-10 15:28 UTC, yalzhang@redhat.com 		no flags Details
libvirtd log on source host (2.87 MB, text/plain)
2021-02-10 15:29 UTC, yalzhang@redhat.com 		no flags Details
cpuinfo: last processor of the target (1.36 KB, text/plain)
2021-02-10 15:33 UTC, yalzhang@redhat.com 		no flags Details
Show Obsolete (1) View All

Description yalzhang@redhat.com 2021-01-12 09:45:16 UTC 
Description of problem:
Guest with host-model cpu type can not migrate from rhel-av 8.2.1 to rhel-av 8.3.1 as "missing features: ssbd,amd-ssbd"

Version-Release number of selected component (if applicable):
source host:
# rpm -q libvirt qemu-kvm
libvirt-6.0.0-25.5.module+el8.2.1+8680+ea98947b.x86_64
qemu-kvm-4.2.0-29.module+el8.2.1+8442+7a3eadf7.5.x86_64

target host:
# rpm -q libvirt qemu-kvm
libvirt-6.6.0-11.module+el8.3.1+9196+74a80ca4.x86_64
qemu-kvm-5.1.0-17.module+el8.3.1+9213+7ace09c3.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Prepare 2 hosts with the same CPU as "AMD EPYC 7251 8-Core Processor", and prepare the migration environment, one host is installed with rhel 8.2, another is rhel 8.3.

2. Start a  vm on the host with rhel-av 8.2.1 as below:
# virsh dumpxml test
<domain type='kvm'>
  <name>test</name>
...
<cpu mode='host-model' check='partial'/>
...
# virsh start test
# virsh dumpxml test
<cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>EPYC-IBPB</model>
    <vendor>AMD</vendor>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='tsc-deadline'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='arch-capabilities'/>
    <feature policy='require' name='ssbd'/>
    <feature policy='require' name='cmp_legacy'/>
    <feature policy='require' name='perfctr_core'/>
    <feature policy='require' name='clzero'/>
    <feature policy='require' name='amd-ssbd'/>
    <feature policy='require' name='virt-ssbd'/>
    <feature policy='require' name='rdctl-no'/>
    <feature policy='require' name='skip-l1dfl-vmentry'/>
    <feature policy='require' name='mds-no'/>
    <feature policy='require' name='pschange-mc-no'/>
    <feature policy='disable' name='monitor'/>
    <feature policy='disable' name='svm'/>
    <feature policy='require' name='topoext'/>
  </cpu>

3. Migrate the vm to the host with rhel-av 8.3.1
# virsh migrate test qemu+ssh://xxx/system --live --verbose  
root@xxx's password:
error: operation failed: guest CPU doesn't match specification: missing features: ssbd,amd-ssbd

Actual results:
The migration fails when migrate vm to the host with same cpu model

Expected results:
The migration should succeed.

Additional info:
Part of the virsh domcapabilities output on the 2 systems are list as below, the difference are highlighted:
on source host with rhel-av 8.2.1:
<mode name='host-model' supported='yes'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <feature policy='require' name='x2apic'/>
      <feature policy='require' name='tsc-deadline'/>
      <feature policy='require' name='hypervisor'/>
      <feature policy='require' name='tsc_adjust'/>
      <feature policy='require' name='arch-capabilities'/>
     *** <feature policy='require' name='ssbd'/> ***
      <feature policy='require' name='cmp_legacy'/>
      <feature policy='require' name='perfctr_core'/>
      <feature policy='require' name='invtsc'/>
      <feature policy='require' name='clzero'/>
     *** <feature policy='require' name='amd-ssbd'/> ***
      <feature policy='require' name='virt-ssbd'/>
      <feature policy='require' name='rdctl-no'/>
      <feature policy='require' name='skip-l1dfl-vmentry'/>
      <feature policy='require' name='mds-no'/>
      <feature policy='require' name='pschange-mc-no'/>
      <feature policy='disable' name='monitor'/>
      <feature policy='disable' name='svm'/>
    </mode>

on target host with rhel-av 8.3.1:
<mode name='host-model' supported='yes'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <feature policy='require' name='x2apic'/>
      <feature policy='require' name='tsc-deadline'/>
      <feature policy='require' name='hypervisor'/>
      <feature policy='require' name='tsc_adjust'/>
      <feature policy='require' name='arch-capabilities'/>
   ****   <feature policy='require' name='xsaves'/> *****
      <feature policy='require' name='cmp_legacy'/>
      <feature policy='require' name='perfctr_core'/>
      <feature policy='require' name='invtsc'/>
      <feature policy='require' name='clzero'/>
 ******     <feature policy='require' name='xsaveerptr'/> ******
      <feature policy='require' name='virt-ssbd'/>
      <feature policy='require' name='rdctl-no'/>
      <feature policy='require' name='skip-l1dfl-vmentry'/>
      <feature policy='require' name='mds-no'/>
      <feature policy='require' name='pschange-mc-no'/>
      <feature policy='disable' name='svm'/>
      <feature policy='disable' name='monitor'/>
    </mode>

The target host did support "ssbd":
# lscpu |grep ssbd
Flags:               ...hw_pstate ssbd ibpb...
Comment 1 Jiri Denemark 2021-01-12 10:36:23 UTC 
Could you please attach qemu log from the destination host?
Comment 2 yalzhang@redhat.com 2021-01-12 14:43:20 UTC 
Created attachment 1746671 [details]
content of /var/log/libvirt/qemu/test.log on target host
Comment 3 Jiri Denemark 2021-01-12 15:17:02 UTC 
I don't know what changed in QEMU/KVM or whether there's a host-side
configuration that could influence this, but QEMU 5.1.0 on the target host
complains:

    qemu-kvm: warning: host doesn't support requested feature: CPUID.07H:EDX.ssbd [bit 31]
    qemu-kvm: warning: host doesn't support requested feature: CPUID.80000008H:EBX.amd-ssbd [bit 24]

which is why libvirt reports these two features are missing and forbids
migration because it would result in different guest ABI.

Moving to QEMU for further investigation.
Comment 6 yalzhang@redhat.com 2021-01-13 04:01:04 UTC 
BTW, migrate the same vm with cpu as "host-model" from rhel-av 8.2.1 to rhel-av 8.2.1 succeed. The src host and dst host are the same as in comment 0, the cpu of them are the same as "AMD EPYC 7251 8-Core Processor".
Comment 7 John Ferlan 2021-01-22 13:25:50 UTC 
Amnon - this would seem like something for either Eduardo or David
Comment 9 yalzhang@redhat.com 2021-01-25 03:54:46 UTC 
Same issue happened when migrate vm with host-model from rhel-av 8.2.1 to rhel-av 8.3.0
source host, rhel-av 8.2.1 package version:
libvirt-6.0.0-25.5.module+el8.2.1+8680+ea98947b.x86_64
qemu-kvm-4.2.0-29.module+el8.2.1+8442+7a3eadf7.5.x86_64

target host, rhel-av 8.3.0 package version:
libvirt-6.6.0-7.3.module+el8.3.0+9547+7d548490.x86_64
qemu-kvm-5.1.0-14.module+el8.3.0+8790+80f9c6d8.1.x86_64
Comment 10 Dr. David Alan Gilbert 2021-02-10 12:30:46 UTC 
I also get that warning off my 1st gen EPYC host:

[root@amd-speedway-05 x86_64]# /usr/libexec/qemu-kvm  -cpu EPYC-IBPB,ssbd=on,amd-ssbd=on
qemu-kvm: warning: host doesn't support requested feature: CPUID.07H:EDX.ssbd [bit 31]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000008H:EBX.amd-ssbd [bit 24]

qemu doesn't quit on me though unless I specify enforce=on.

I'm thinking that ssbd/amd-ssbd is one of the security fixes that came along later; I'm not sure if that involved newer microcode
or something else.

 yalzhang: 
 Please include: 
    a) The /var/log/libvirt/ log file from the source host as well
    b) the /proc/cpuinfo (NOT lscpu) from the source and target machine.
    c) kernel and microcode_ctl package versions of source and target.
Comment 11 Dr. David Alan Gilbert 2021-02-10 14:24:16 UTC 
I'm suspecting this is over-reporting of amd-ssbd on older kernels, as fixed by upstream:
  8715f05269bfbc6453e25e80825d781a82902f8e - kvm: x86: Host feature SSBD doesn't imply guest feature AMD_SSBD

that was downstreamed in 8.3 as 03243d4f06e2fd0892d0f8cdaf68ba84eecd2eac part of bz 1813987

note that the virt-ssbd flag still works.
Comment 12 Dr. David Alan Gilbert 2021-02-10 14:28:12 UTC 
Yes, if I roll the host kernel back to RHEL 8.2's 4.18.0-193.44.1.el8_2.x86_64 then :

# /usr/libexec/qemu-kvm  -cpu EPYC-IBPB,ssbd=on,amd-ssbd=on
VNC server running on ::1:5900

is happy.
Comment 13 yalzhang@redhat.com 2021-02-10 15:22:29 UTC 
(In reply to Dr. David Alan Gilbert from comment #12)
> Yes, if I roll the host kernel back to RHEL 8.2's
> 4.18.0-193.44.1.el8_2.x86_64 then :
> 
> # /usr/libexec/qemu-kvm  -cpu EPYC-IBPB,ssbd=on,amd-ssbd=on
> VNC server running on ::1:5900
> 
> is happy.

Yes, on 8.2, it works well for 4.18.0-193.44.1.el8_2.x86_64 and older kernel-4.18.0-193.19.1.el8_2.x86_64.
But on 8.3 with kernel 4.18.0-240.10.1.el8_3.x86_64, it reports error:

# /usr/libexec/qemu-kvm  -cpu EPYC-IBPB,ssbd=on,amd-ssbd=on
qemu-kvm: warning: host doesn't support requested feature: CPUID.07H:EDX.ssbd [bit 31]
qemu-kvm: warning: host doesn't support requested feature: CPUID.80000008H:EBX.amd-ssbd [bit 24]
VNC server running on ::1:5900
^Cqemu-kvm: terminating on signal 2

# rpm -q kernel
kernel-4.18.0-240.10.1.el8_3.x86_64
Comment 14 yalzhang@redhat.com 2021-02-10 15:28:32 UTC 
Created attachment 1756230 [details]
libvirtd log on source host

I have checked the libvirtd log on source host, there is no error or fail, attached the log for reference.
Comment 15 yalzhang@redhat.com 2021-02-10 15:29:11 UTC 
Created attachment 1756231 [details]
libvirtd log on source host

I have checked the libvirtd log on source host, there is no error or fail, attached the log for reference.
Comment 16 yalzhang@redhat.com 2021-02-10 15:33:28 UTC 
Created attachment 1756233 [details]
cpuinfo: last processor of the target

I have compared the cpu info by "cat /proc/cpuinfo", the only difference is the frequency(only compared 1 of the processors):

# diff last_processor_of_source last_processor_of_target
8c8
< cpu MHz		: 2890.947
---
> cpu MHz		: 2460.013
Comment 17 Dr. David Alan Gilbert 2021-02-10 17:02:05 UTC 
Thanks; note I wanted the /var/log/libvirt/qemu/SOMETHING.log  from the source showing the commandline, not the libvirtd.log - still, I think we understand what's going on now.

Thanks.
Comment 18 Dr. David Alan Gilbert 2021-02-11 13:48:58 UTC 
jdenemar:
  We think the kernel + qemu is now doing the 'right' thing - the problem here is really 8.2.x kernel over-promising flags that it shouldn't
be advertising.
  When the guest tries to write the MSR that was advertised it gets a backtrace anyway (bz 1808996 ).

  Could we fudge this at the libvirt level at all?
  a) By dropping these flags during the migrate (the guest's idea of that MSR is screwy enough anyway),
    give the VM virt-ssbd and it should fix itself after a reboot.
  b) Plus as a fix for 8.2 so that new machines get created with virt-ssbd rather than these flags.

  You'd want that to only happen on EPYC 7xx1 series hosts; (b) *on* those hosts,  (a) migrating
*from* one of those hosts.
Comment 19 John Ferlan 2021-09-08 21:43:22 UTC 
Move RHEL-AV bugs to RHEL9. If necessary to resolve in RHEL8, then clone to the current RHEL8 release.
Comment 22 RHEL Program Management 2022-07-12 07:27:27 UTC 
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.
Note You need to log in before you can comment on or make changes to this bug.