2031795 – KVM Fedora 35 guest x86 programs randomly crash in signal handler

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2031795 - KVM Fedora 35 guest x86 programs randomly crash in signal handler

Summary: KVM Fedora 35 guest x86 programs randomly crash in signal handler

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	None
Product:	Red Hat Enterprise Linux 8
Classification:	Red Hat
Component:	qemu-kvm
Sub Component:
Version:	8.5
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	high
Target Milestone:	rc
Target Release:	---
Assignee:	Dr. David Alan Gilbert
QA Contact:	liunana
Docs Contact:
URL:
Whiteboard:
Duplicates (1):	2065230 (view as bug list)
Depends On:
Blocks:	2043121 2043122 2052198 2052199 2065230 2065235 2065239
TreeView+	depends on / blocked

Reported:	2021-12-13 13:13 UTC by Miroslav Lichvar
Modified:	2022-03-17 14:22 UTC (History)
CC List:	13 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Clones:	2043121 2043122 2065230 (view as bug list)
Environment:
Last Closed:	2022-02-08 20:27:21 UTC
Type:	Bug
Target Upstream Version:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	RHELPLAN-105529	0	None	None	None	2021-12-13 13:26:28 UTC

Description Miroslav Lichvar 2021-12-13 13:13:47 UTC

Description of problem:
I'm hitting an odd problem with i386 programs randomly crashing in their signal handler on Fedora 35 x86_64 guest in RHEL8.5 KVM on a Ryzen 5950x CPU.

I'm not sure where exactly is the problem. I suspect is the host kernel, maybe something with the AMD CPU support, as I don't see the crash when running Fedora on real hardware or as a guest on a different CPU. Let me know if you need more details.

Version-Release number of selected component (if applicable):
kernel-4.18.0-348.2.1.el8_5.x86_64
libvirt-daemon-6.0.0-37.module+el8.5.0+12162+40884dd2.x86_64
gcc-11.2.1-7.fc35.x86_64
glibc-2.34-11.fc35.i686
kernel-5.15.6-200.fc35.x86_64

How reproducible:
reliably

Steps to Reproduce:
1. install Fedora 35 as a KVM guest:
virt-install --name fedora --ram 4096 --vcpus 2 \
--disk path=/mnt/os/fedora-test.img,size=20 \
--os-variant fedora34 --os-type linux --network bridge=virbr0 \
--graphics none --console pty,target_type=serial \
--location 'https://ftp.linux.cz/pub/linux/fedora/linux/releases/35/Server/x86_64/os/' \
--extra-args 'console=ttyS0,115200n8 serial'

2. in the guest compile the following program with gcc -m32:

#include <stdio.h>
#include <signal.h>
#include <unistd.h>
static int x = 0;
static void handler(int s) {
        x = 1;
}
int main() {
        struct sigaction sa;
        sa.sa_handler = handler;
        sa.sa_flags = SA_RESTART;
        if (sigemptyset(&sa.sa_mask) < 0)
                perror("sigemptyset");
        if (sigaction(SIGINT, &sa, NULL) < 0)
                perror("sigaction");
        kill(getpid(), SIGINT);
        printf("x = %d\n", x);
        return 0;
}

3. run the program in a loop
4. observe any crashes

Actual results:
$ while ./a.out ; do true; done
x = 1
x = 1
x = 1
x = 1
x = 1
x = 1
Segmentation fault (core dumped)

Expected results:
no crash

Additional info:
No messages in the host dmesg.
Guest dmesg has:
[ 3037.459177] a.out[3882]: segfault at ffaa896c ip 00000000f7f59559 sp 00000000ffaa896c error 25
[ 3037.459548] Code: Unable to access opcode bytes at RIP 0xf7f5952f.

Comment 1 Miroslav Lichvar 2021-12-14 15:54:00 UTC

After some tests with different libvirt configurations it seems this is related to the "xsaves" feature of the CPU. The default CPU with the "host-model" cpu is EPYC Milan, but if I change it to EPYC Rome, or just disable the xsaves feature in the libvirt xml, the crashes stop.

I'm still not sure in what component this needs to be fixed. Please reassign as appropriate.

Comment 2 Nitesh Narayan Lal 2021-12-16 16:21:23 UTC

Amnon, assigning this to you. 
Maybe Dave can help us with the triaging?

Thanks

Comment 3 John Ferlan 2021-12-20 16:19:47 UTC

Maxim - anything about this look familiar to you? Since you've debugged AMD CPU issues previous...

Comment 4 Nitesh Narayan Lal 2021-12-21 17:28:29 UTC

Moving the BZ back to virt-maint so that this keeps showing as a BZ that is not triaged yet.
Thanks, John, for adding Maxim.
Also, clearing needinfo from Amnon.

Comment 5 liunana 2022-01-06 08:15:50 UTC

I can reproduce this bug on RHEL.8.5 host with three kind of guests.

Test Env:
    dell-per7525-10.lab.eng.pek2.redhat.com
    4.18.0-348.7.1.el8_5.x86_64
    qemu-kvm-6.0.0-33.module+el8.5.0+13740+349232b6.2
    Model name:          AMD EPYC 7313 16-Core Processor

Guest: 
    Fedora 35:  5.15.12-200.fc35.x86_64
    RHEL.8.6 :  4.18.0-357.el8.x86_64
    RHEL.8.5 :  4.18.0-348.7.1.el8_5.x86_64


And I can't rerpoduce this bug with combination (RHEL.8.5 host + qemu-kvm-6.2.0-2.module+el8.6.0+13738+17338784.x86_64) with three kind of guests.


CPU configuration xml:

  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>EPYC-Milan</model>
    <vendor>AMD</vendor>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='tsc-deadline'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='vaes'/>
    <feature policy='require' name='vpclmulqdq'/>
    <feature policy='require' name='spec-ctrl'/>
    <feature policy='require' name='stibp'/>
    <feature policy='require' name='arch-capabilities'/>
    <feature policy='require' name='ssbd'/>
    <feature policy='require' name='cmp_legacy'/>
    <feature policy='require' name='virt-ssbd'/>
    <feature policy='require' name='rdctl-no'/>
    <feature policy='require' name='skip-l1dfl-vmentry'/>
    <feature policy='require' name='mds-no'/>
    <feature policy='require' name='pschange-mc-no'/>
    <feature policy='disable' name='erms'/>
    <feature policy='disable' name='fsrm'/>
    <feature policy='require' name='xsaves'/>
    <feature policy='disable' name='svm'/>
    <feature policy='require' name='topoext'/>
    <feature policy='disable' name='npt'/>
    <feature policy='disable' name='nrip-save'/>
    <feature policy='disable' name='svme-addr-chk'/>
  </cpu>


Seems it is a qemu related bz, Could you please try the qemu6.2 on rhel8.5 host to see if it works?
please help to check this, thanks.



Best regards
Liu Nana

Comment 6 Miroslav Lichvar 2022-01-06 08:59:47 UTC

Yes, upgrading to qemu-kvm-6.2.0 fixes the issue for me. Reassigning the bug to the qemu-kvm component. Thanks.

Comment 7 Nitesh Narayan Lal 2022-01-06 20:54:03 UTC

Hi Connie, 

Since this has now been moved to qemu-kvm, do you think this is something that you might be able to look into it?
If not, I think Igor should be able to help.

Also adding Dave to the loop.

Thanks

Comment 8 Cornelia Huck 2022-01-10 09:45:49 UTC

(In reply to Nitesh Narayan Lal from comment #7)
> Hi Connie, 
> 
> Since this has now been moved to qemu-kvm, do you think this is something
> that you might be able to look into it?
> If not, I think Igor should be able to help.
> 
> Also adding Dave to the loop.
> 
> Thanks

I'm afraid I'm not really familiar with the cpu models here; it seems that it has been fixed at some time before 6.2?

7bde6b18575d ("target/i386: Add CPU model versions supporting 'xsaves'") might be a candidate to look at, just going by that it mentiones 'xsaves'.

Comment 9 Dr. David Alan Gilbert 2022-01-10 12:00:31 UTC

Maybe; although if I'm reading that right it's adding xsaves to other cpu models; but Milan already has xsaves which is happening on the reporters host.
We might have to bisect between ~@6.0 and 6.2 to find out when it got fixed.

Comment 11 liunana 2022-01-16 13:53:00 UTC

(In reply to Dr. David Alan Gilbert from comment #9)
> Maybe; although if I'm reading that right it's adding xsaves to other cpu
> models; but Milan already has xsaves which is happening on the reporters
> host.
> We might have to bisect between ~@6.0 and 6.2 to find out when it got fixed.

Hi,


I test three kinds of qemu version, seems it is fixed in the early qemu6.1 version.

Test Env:
   4.18.0-348.7.1.el8_5.x86_64
   amd-milan-11.khw1.lab.eng.bos.redhat.com

Guest: 4.18.0-348.el8.x86_64

qemu versions and test results:

        Version                                                    results
------------------------------------------------------------------------------
qemu-kvm-6.0.0-33.module+el8.5.0+13740+349232b6.2              |    crash    
qemu-kvm-6.0.0-29.module+el8.6.0+12490+ec3e565c                |    crash
qemu-kvm-6.1.0-1.rc0.scrmod+el8.5.0+11915+5290fd16.wrb210721   |   No crash
-------------------------------------------------------------------------------


Could you help to check this? Thanks.


Best regards
Liu Nana

Comment 12 Dr. David Alan Gilbert 2022-01-17 11:11:06 UTC

Thanks! I'll try and bisect it further.

Comment 13 Dr. David Alan Gilbert 2022-01-17 13:09:23 UTC

I can confirm this is OK on a Rome machine; going to find a Milan to try it on

Comment 15 Dr. David Alan Gilbert 2022-01-17 16:41:30 UTC

bisected:
6.0.0 fails
1500 0c7af1a778d036402ec0829783afee1ce6ea942c bad
750     9bef7ea9d93ee6b6297a5be6cb5a557f7d1764c9 bad
743     dd52af17ec947332dfe45bd5f098c94c6ec0baa3 bad!
738     5aa10ab1a08e4123dee214a2f854909efb07b45b bad!
737     3568987f78faff90829ea6c885bbdd5b083dc86c bad!
736     fea4500841024195ec701713e05b92ebf667f192 good
734     f08b65b651bca2eac543de694f866049e48fb242 good
700     af4ba0ec8f017c402c239f2888ef62f63770ba8b good
650     2044969f0b27fa67f2b69bc710eaef45998cb6fb good
550     f6b12dfd80f3b0d6fbaf982718946e5ad72a543e good
375     2e3e3da3c2ad559d1255a9a3bf3df0782c2cf231 good!
6.1.0-rc1 good

This is a series by David Edmondson 'Derive XSAVE state component offsets from CPUID leaf 0xd where possible' and the cover letter has the following smoking gun:

                             |The offset of XSAVE state components within the XSAVE state area is
                             |currently hard-coded via reference to the X86XSaveArea structure. This
                             |structure is accurate for Intel systems at the time of writing, but
                             |incorrect for newer AMD systems, as the state component for protection
                             |keys is located differently (offset 0x980 rather than offset 0xa80).

Comment 16 Dr. David Alan Gilbert 2022-01-17 17:59:24 UTC

I'm having a crack at backporting that to 8.5.0; it's not quite trivial because there's a rearrangement of the cpu structures by Claudio in between.

Still, now that we know whtat the fix is, and that it's in 8.6, we should close this as 'fixed current version' and ask for z? for the backport.

Comment 17 Dr. David Alan Gilbert 2022-01-17 18:01:38 UTC

Asking for Z stream because corrupt state makes me nervous; maybe there's other less obscure cases that crash; we should fix it.

Comment 18 Dr. David Alan Gilbert 2022-01-17 19:51:24 UTC

[My first attempt at backporting that set isn't that succesful; I get a reliable crash in the test rather than an intermittent one!)

Comment 19 Dr. David Alan Gilbert 2022-01-18 15:26:49 UTC

OK, I seem to have a working world.

Note, this also triggers on 8.4.0 - so I think we might have to Z this further back.

Comment 21 Dr. David Alan Gilbert 2022-01-18 15:40:20 UTC

Copying in Wei for information.
(Clearing needinfo on Maxim since we know the problem)

Comment 22 Dr. David Alan Gilbert 2022-01-18 16:18:02 UTC

Yash: Anything else I need to do to z-streamify this?

Comment 40 Yash Mankad 2022-03-17 14:22:55 UTC

*** Bug 2065230 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.