Bug 2076978 - 'perf top' segfaults
Summary: 'perf top' segfaults
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: 36
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
Assignee: Kernel Maintainer List
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 2078409 2086870 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-04-20 10:56 UTC by Dr. David Alan Gilbert
Modified: 2022-05-28 01:32 UTC (History)
23 users (show)

Fixed In Version: kernel-5.17.11-300.fc36 kernel-5.17.11-200.fc35 kernel-5.17.11-100.fc34
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-05-28 01:15:05 UTC
Type: Bug


Attachments (Terms of Use)

Description Dr. David Alan Gilbert 2022-04-20 10:56:03 UTC
1. Please describe the problem:
sudo perf top   segfaults in what looks like an endless recursion

2. What is the Version-Release number of the kernel:
perf-5.17.0-300.fc36.x86_64
kernel-5.17.2-300.fc36.x86_64
libbpf-0.5.0-2.fc36.x86_64

3. Did it work previously in Fedora? If so, what kernel version did the issue
   *first* appear?  Old kernels are available for download at
   https://koji.fedoraproject.org/koji/packageinfo?packageID=8 :
Im pretty sure it was OK in f35

4. Can you reproduce this issue? If so, please provide the steps to reproduce
   the issue below:
  sudo perf top



5. Does this problem occur with the latest Rawhide kernel? To install the
   Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by
   ``sudo dnf update --enablerepo=rawhide kernel``:
not sure

6. Are you running any modules that not shipped with directly Fedora's kernel?:
no

7. Please attach the kernel logs. You can get the complete kernel log
   for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the
   issue occurred on a previous boot, use the journalctl ``-b`` flag.

Here's some debug from coredump debug
#0  0x00007f2d64893d7c in btf__get_from_id (id=51, btf=btf@entry=0x7ffd4b90b020) at btf.c:1410
        res = <optimized out>
        err = <optimized out>

1410		*btf = NULL;
1411		res = btf__load_from_kernel_by_id(id);
1412		err = libbpf_get_error(res);

#1  0x00005587fadda8a4 in btf__load_from_kernel_by_id (id=<optimized out>) at util/bpf-event.c:30
        btf = 0x0
        err = <optimized out>

(gdb) list
25	struct btf * __weak  (__u32 id)
26	{
27	       struct btf *btf;
28	#pragma GCC diagnostic push
29	#pragma GCC diagnostic ignored "-Wdeprecated-declarations"
30	       int err = btf__get_from_id(id, &btf);
31	#pragma GCC diagnostic pop
32	
33	       return err ? ERR_PTR(err) : btf;
34	}

#2  0x00007f2d64893d8d in btf__get_from_id (id=<optimized out>, btf=btf@entry=0x7ffd4b90b070) at btf.c:1411
        res = <optimized out>
        err = <optimized out>
#3  0x00005587fadda8a4 in btf__load_from_kernel_by_id (id=<optimized out>) at util/bpf-event.c:30
        btf = 0x0
        err = <optimized out>

it looks like it's got stuck in an infinite recursion.
I'm suspicious this is something to do with the '__weak' in the decl of btf__load_from_kernel_by_id  because my looking at the source wasn't finding that version of that function.

Comment 1 Richard W.M. Jones 2022-04-20 11:03:39 UTC
I am able to reproduce this with the unusual combination of:
perf-5.17.0-300.fc36.x86_64
running on kernel 5.14.0-0.rc4.20210804gitd5ad8ec3cfb5.36.fc35.x86_64
(I cannot reboot this machine because it's running large fuzzing jobs)

Stack trace:

#0  0x000055e11bb43895 in btf__load_from_kernel_by_id (id=37)
    at util/bpf-event.c:26
#1  0x00007fb25400bd8d in btf__get_from_id (id=<optimized out>, 
    btf=btf@entry=0x7ffccb43c040) at btf.c:1411
#2  0x000055e11bb438a4 in btf__load_from_kernel_by_id (id=<optimized out>)
    at util/bpf-event.c:30
#3  0x00007fb25400bd8d in btf__get_from_id (id=<optimized out>, 
    btf=btf@entry=0x7ffccb43c090) at btf.c:1411
#4  0x000055e11bb438a4 in btf__load_from_kernel_by_id (id=<optimized out>)
    at util/bpf-event.c:30
#5  0x00007fb25400bd8d in btf__get_from_id (id=<optimized out>, 
    btf=btf@entry=0x7ffccb43c0e0) at btf.c:1411
#6  0x000055e11bb438a4 in btf__load_from_kernel_by_id (id=<optimized out>)
    at util/bpf-event.c:30

repeating for at least 200,000(!) frames before I killed gdb.

Comment 2 Rob Bradford 2022-04-25 14:04:06 UTC
I was able to reproduce (same back trace) the issue with the versions in Fedora 36 of perf and the kernel. Upgrading to the Rawhide versions resolve the issue:

kernel-5.18.0-0.rc3.20220422gitd569e86915b7f2f.31.fc37.x86_64
perf-5.18.0-0.rc3.git0.1.fc37.x86_64

When running on F36 the issues is not limited to perf top, but perf record too.

Comment 3 Justin M. Forbes 2022-05-12 15:01:04 UTC
*** Bug 2078409 has been marked as a duplicate of this bug. ***

Comment 4 Justin M. Forbes 2022-05-25 20:42:08 UTC
*** Bug 2086870 has been marked as a duplicate of this bug. ***

Comment 5 Fedora Update System 2022-05-25 20:42:35 UTC
FEDORA-2022-b2cde267d9 has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2022-b2cde267d9

Comment 6 Fedora Update System 2022-05-25 20:42:39 UTC
FEDORA-2022-014c3a24d9 has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2022-014c3a24d9

Comment 7 Fedora Update System 2022-05-25 20:42:43 UTC
FEDORA-2022-8095b23575 has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-8095b23575

Comment 8 Fedora Update System 2022-05-26 02:33:05 UTC
FEDORA-2022-8095b23575 has been pushed to the Fedora 36 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-8095b23575`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-8095b23575

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 9 Fedora Update System 2022-05-26 02:33:53 UTC
FEDORA-2022-b2cde267d9 has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-b2cde267d9`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-b2cde267d9

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 10 Fedora Update System 2022-05-26 03:21:33 UTC
FEDORA-2022-014c3a24d9 has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-014c3a24d9`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-014c3a24d9

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 11 Geraldo Simião 2022-05-26 13:02:31 UTC
This update fixed the bug, no more perf crashes.
perf-5.17.11-300.fc36.x86_64

Comment 12 Fedora Update System 2022-05-28 01:15:05 UTC
FEDORA-2022-8095b23575 has been pushed to the Fedora 36 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 13 Fedora Update System 2022-05-28 01:21:54 UTC
FEDORA-2022-b2cde267d9 has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 14 Fedora Update System 2022-05-28 01:32:27 UTC
FEDORA-2022-014c3a24d9 has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.