1. Please describe the problem: I'm running the following command, which used to work for me for ages: 2. What is the Version-Release number of the kernel: [ykaul@ykaul glusterfs]$ rpm -q --whatprovides `which perf` perf-5.17.4-300.fc36.x86_64 [ykaul@ykaul glusterfs]$ rpm -q kernel kernel-5.17.1-300.fc36.x86_64 kernel-5.17.2-300.fc36.x86_64 kernel-5.17.4-300.fc36.x86_64 3. Did it work previously in Fedora? If so, what kernel version did the issue *first* appear? Old kernels are available for download at https://koji.fedoraproject.org/koji/packageinfo?packageID=8 : 36 beta 4. Can you reproduce this issue? If so, please provide the steps to reproduce the issue below: Yes, run above command. 5. Does this problem occur with the latest Rawhide kernel? To install the Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by ``sudo dnf update --enablerepo=rawhide kernel``: Did not try! 6. Are you running any modules that not shipped with directly Fedora's kernel?: No 7. Please attach the kernel logs. You can get the complete kernel log for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the issue occurred on a previous boot, use the journalctl ``-b`` flag. Apr 25 11:52:28 ykaul.tlv.redhat.com systemd-coredump[34357]: Process 34354 (perf) of user 0 dumped core. Module linux-vdso.so.1 with build-id 04b708c7c91e947d7413a24109a43e4cc798e340 Module libpcre2-8.so.0 with build-id 2827cb8b86a0a697fa6e6646690d1b01de9b6fba Metadata for module libpcre2-8.so.0 owned by FDO found: { "type" : "rpm", "name" : "pcre2", "version" : "10.39-1.fc36.1", "architecture" : "x86_64", "osCpe" : "cpe:/o:fedoraproject:fedora:36" } Module libselinux.so.1 with build-id 9ee47929aa5779a0fe6469346be26bfa6426a516 Metadata for module libselinux.so.1 owned by FDO found: { "type" : "rpm", "name" : "libselinux", "version" : "3.3-4.fc36", "architecture" : "x86_64", "osCpe" : "cpe:/o:fedoraproject:fedora:36" } Module libbrotlicommon.so.1 with build-id e2790c03a5c688b7e75e89676cdd2b5fcf247a6f Metadata for module libbrotlicommon.so.1 owned by FDO found: { "type" : "rpm", "name" : "brotli", "version" : "1.0.9-7.fc36", "architecture" : "x86_64", "osCpe" : "cpe:/o:fedoraproject:fedora:36" } Module libsasl2.so.3 with build-id db4b190f8f8dc222aebeb2b42905d93204f8aeb8 Metadata for module libsasl2.so.3 owned by FDO found: { "type" : "rpm", "name" : "cyrus-sasl", "version" : "2.1.27-18.fc36", "architecture" : "x86_64", "osCpe" : "cpe:/o:fedoraproject:fedora:36" } Module libevent-2.1.so.7 with build-id f6e4e70b6ca6ab522b59e089c865e1a33d3760bf Metadata for module libevent-2.1.so.7 owned by FDO found: { "type" : "rpm", "name" : "libevent", "version" : "2.1.12-6.fc36", "architecture" : "x86_64", "osCpe" : "cpe:/o:fedoraproject:fedora:36" } Module libresolv.so.2 with build-id 6dc6c19e2b9bd4bd27754cc88263cef3fd50379c Module libkeyutils.so.1 with build-id da9c7ac09e992ba7c560d8fda4279f6d29a62022 Metadata for module libkeyutils.so.1 owned by FDO found: { "type" : "rpm", "name" : "keyutils", "version" : "1.6.1-4.fc36", "architecture" : "x86_64", "osCpe" : "cpe:/o:fedoraproject:fedora:36" } Module libkrb5support.so.0 with build-id 37d009e344667548ae99e641f2bc86648a756162 Metadata for module libkrb5support.so.0 owned by FDO found: { "type" : "rpm", "name" : "krb5", "version" : "1.19.2-9.fc36", "architecture" : "x86_64", "osCpe" : "cpe:/o:fedoraproject:fedora:36" } Module libcom_err.so.2 with build-id c980b4303c51332b16b98f799a158445eae81aa0 Stack trace of thread 34354: Stack trace of thread 34354: #0 0x00007f7ef6a07d88 btf__get_from_id (libbpf.so.0 + 0x10d88) #1 0x000055d2e8710f94 btf__load_from_kernel_by_id (perf + 0x40ff94) #2 0x00007f7ef6a07d8d btf__get_from_id (libbpf.so.0 + 0x10d8d) #3 0x000055d2e8710f94 btf__load_from_kernel_by_id (perf + 0x40ff94) #4 0x00007f7ef6a07d8d btf__get_from_id (libbpf.so.0 + 0x10d8d) #5 0x000055d2e8710f94 btf__load_from_kernel_by_id (perf + 0x40ff94) #6 0x00007f7ef6a07d8d btf__get_from_id (libbpf.so.0 + 0x10d8d) #7 0x000055d2e8710f94 btf__load_from_kernel_by_id (perf + 0x40ff94) #8 0x00007f7ef6a07d8d btf__get_from_id (libbpf.so.0 + 0x10d8d) #9 0x000055d2e8710f94 btf__load_from_kernel_by_id (perf + 0x40ff94) #10 0x00007f7ef6a07d8d btf__get_from_id (libbpf.so.0 + 0x10d8d) #11 0x000055d2e8710f94 btf__load_from_kernel_by_id (perf + 0x40ff94) #12 0x00007f7ef6a07d8d btf__get_from_id (libbpf.so.0 + 0x10d8d) #13 0x000055d2e8710f94 btf__load_from_kernel_by_id (perf + 0x40ff94) #14 0x00007f7ef6a07d8d btf__get_from_id (libbpf.so.0 + 0x10d8d) #15 0x000055d2e8710f94 btf__load_from_kernel_by_id (perf + 0x40ff94) #16 0x00007f7ef6a07d8d btf__get_from_id (libbpf.so.0 + 0x10d8d) #17 0x000055d2e8710f94 btf__load_from_kernel_by_id (perf + 0x40ff94) #18 0x00007f7ef6a07d8d btf__get_from_id (libbpf.so.0 + 0x10d8d) #19 0x000055d2e8710f94 btf__load_from_kernel_by_id (perf + 0x40ff94) #20 0x00007f7ef6a07d8d btf__get_from_id (libbpf.so.0 + 0x10d8d) #21 0x000055d2e8710f94 btf__load_from_kernel_by_id (perf + 0x40ff94) #22 0x00007f7ef6a07d8d btf__get_from_id (libbpf.so.0 + 0x10d8d) #23 0x000055d2e8710f94 btf__load_from_kernel_by_id (perf + 0x40ff94) #24 0x00007f7ef6a07d8d btf__get_from_id (libbpf.so.0 + 0x10d8d) #25 0x000055d2e8710f94 btf__load_from_kernel_by_id (perf + 0x40ff94) #26 0x00007f7ef6a07d8d btf__get_from_id (libbpf.so.0 + 0x10d8d) #27 0x000055d2e8710f94 btf__load_from_kernel_by_id (perf + 0x40ff94) #28 0x00007f7ef6a07d8d btf__get_from_id (libbpf.so.0 + 0x10d8d) #29 0x000055d2e8710f94 btf__load_from_kernel_by_id (perf + 0x40ff94) #30 0x00007f7ef6a07d8d btf__get_from_id (libbpf.so.0 + 0x10d8d) #31 0x000055d2e8710f94 btf__load_from_kernel_by_id (perf + 0x40ff94) #32 0x00007f7ef6a07d8d btf__get_from_id (libbpf.so.0 + 0x10d8d) #33 0x000055d2e8710f94 btf__load_from_kernel_by_id (perf + 0x40ff94) #34 0x00007f7ef6a07d8d btf__get_from_id (libbpf.so.0 + 0x10d8d) #35 0x000055d2e8710f94 btf__load_from_kernel_by_id (perf + 0x40ff94) #36 0x00007f7ef6a07d8d btf__get_from_id (libbpf.so.0 + 0x10d8d) #37 0x000055d2e8710f94 btf__load_from_kernel_by_id (perf + 0x40ff94) #38 0x00007f7ef6a07d8d btf__get_from_id (libbpf.so.0 + 0x10d8d) #39 0x000055d2e8710f94 btf__load_from_kernel_by_id (perf + 0x40ff94) #40 0x00007f7ef6a07d8d btf__get_from_id (libbpf.so.0 + 0x10d8d) #41 0x000055d2e8710f94 btf__load_from_kernel_by_id (perf + 0x40ff94) #42 0x00007f7ef6a07d8d btf__get_from_id (libbpf.so.0 + 0x10d8d) #43 0x000055d2e8710f94 btf__load_from_kernel_by_id (perf + 0x40ff94) #44 0x00007f7ef6a07d8d btf__get_from_id (libbpf.so.0 + 0x10d8d) #45 0x000055d2e8710f94 btf__load_from_kernel_by_id (perf + 0x40ff94) #46 0x00007f7ef6a07d8d btf__get_from_id (libbpf.so.0 + 0x10d8d) #47 0x000055d2e8710f94 btf__load_from_kernel_by_id (perf + 0x40ff94) #48 0x00007f7ef6a07d8d btf__get_from_id (libbpf.so.0 + 0x10d8d) #49 0x000055d2e8710f94 btf__load_from_kernel_by_id (perf + 0x40ff94) #50 0x00007f7ef6a07d8d btf__get_from_id (libbpf.so.0 + 0x10d8d) #51 0x000055d2e8710f94 btf__load_from_kernel_by_id (perf + 0x40ff94) #52 0x00007f7ef6a07d8d btf__get_from_id (libbpf.so.0 + 0x10d8d) #53 0x000055d2e8710f94 btf__load_from_kernel_by_id (perf + 0x40ff94) #54 0x00007f7ef6a07d8d btf__get_from_id (libbpf.so.0 + 0x10d8d) #55 0x000055d2e8710f94 btf__load_from_kernel_by_id (perf + 0x40ff94) #56 0x00007f7ef6a07d8d btf__get_from_id (libbpf.so.0 + 0x10d8d) #57 0x000055d2e8710f94 btf__load_from_kernel_by_id (perf + 0x40ff94) #58 0x00007f7ef6a07d8d btf__get_from_id (libbpf.so.0 + 0x10d8d) #59 0x000055d2e8710f94 btf__load_from_kernel_by_id (perf + 0x40ff94) #60 0x00007f7ef6a07d8d btf__get_from_id (libbpf.so.0 + 0x10d8d) #61 0x000055d2e8710f94 btf__load_from_kernel_by_id (perf + 0x40ff94) #62 0x00007f7ef6a07d8d btf__get_from_id (libbpf.so.0 + 0x10d8d) #63 0x000055d2e8710f94 btf__load_from_kernel_by_id (perf + 0x40ff94) Stack trace of thread 34355: #0 0x00007f7ef628a64f __poll (libc.so.6 + 0x10964f) #1 0x000055d2e861eac0 perf_evlist__poll_thread (perf + 0x31dac0) #2 0x00007f7ef6211cc7 start_thread (libc.so.6 + 0x90cc7) #3 0x00007f7ef6296e00 __clone3 (libc.so.6 + 0x115e00) ELF object binary architecture: AMD x86-64 Apr 25 11:52:28 ykaul.tlv.redhat.com [34358]: [164B blob data] Apr 25 11:52:28 ykaul.tlv.redhat.com audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-coredump@1-34356-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Apr 25 11:52:28 ykaul.tlv.redhat.com audit: BPF prog-id=93 op=LOAD Apr 25 11:52:28 ykaul.tlv.redhat.com audit: BPF prog-id=92 op=LOAD Apr 25 11:52:28 ykaul.tlv.redhat.com systemd[1]: Started systemd-coredump - Process Core Dump (PID 34356/UID 0). Apr 25 11:52:28 ykaul.tlv.redhat.com audit: BPF prog-id=91 op=LOAD Apr 25 11:52:28 ykaul.tlv.redhat.com audit[34354]: ANOM_ABEND auid=13610 uid=0 gid=0 ses=1 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 pid=34354 comm="perf" exe="/usr/bin/perf" sig=11 res=1 Apr 25 11:52:28 ykaul.tlv.redhat.com audit[34353]: USER_START pid=34353 uid=13610 auid=13610 ses=1 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success' Apr 25 11:52:28 ykaul.tlv.redhat.com sudo[34353]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=13610) Apr 25 11:52:28 ykaul.tlv.redhat.com audit[34353]: CRED_REFR pid=34353 uid=13610 auid=13610 ses=1 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='op=PAM:setcred grantors=pam_env,pam_localuser,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success' Apr 25 11:52:28 ykaul.tlv.redhat.com sudo[34353]: ykaul : TTY=pts/1 ; PWD=/dev/shm ; USER=root ; COMMAND=/usr/bin/perf record --call-graph dwarf -j any --buildid-all --all-user -p 33915,34011,34014,34082,34114 -ag Apr 25 11:52:28 ykaul.tlv.redhat.com audit[34353]: USER_CMD pid=34353 uid=13610 auid=13610 ses=1 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='cwd="/dev/shm" cmd=70657266207265636F7264202D2D63616C6C2D6772617068206477617266202D6A20616E79202D2D6275696C6469642D616C6C202D2D616C6C2D75736572202D702033333931352C33343031312C33343031342C33343038322C3334313134202D6167 exe="/usr/bin/sudo" terminal=pts/1 res=success' Apr 25 11:52:28 ykaul.tlv.redhat.com audit[34353]: USER_ACCT pid=34353 uid=13610 auid=13610 ses=1 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='op=PAM:accounting grantors=pam_access,pam_unix,pam_localuser acct="ykaul" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success' Apr 25 11:51:35 ykaul.tlv.redhat.com audit[34285]: CRED_DISP pid=34285 uid=13610 auid=13610 ses=1 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='op=PAM:setcred grantors=pam_env,pam_localuser,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success' Apr 25 11:51:35 ykaul.tlv.redhat.com audit[34285]: USER_END pid=34285 uid=13610 auid=13610 ses=1 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='op=PAM:session_close grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
Of course, the 'perf' command is: sudo perf record --call-graph dwarf -j any --buildid-all --all-user -p `pgrep -d\, glusterfs` -F 2000 -ag
More background from IRC: │14:50:42 @jforbes | bkircher: there is a bug on the issue, 5.18 perf seems to work though, and no one has bisected to figure out exactly │ | which patch fixed it
*** This bug has been marked as a duplicate of bug 2076978 ***