Version-Release number of selected component: pcp-pmda-bpftrace-5.2.1-1.fc33 Additional info: reporter: libreport-2.14.0 cgroup: 0::/system.slice/pmcd.service cmdline: python3 /var/lib/pcp/pmdas/bpftrace/pmdabpftrace.python crash_function: run exception_type: subprocess.CalledProcessError executable: /var/lib/pcp/pmdas/bpftrace/pmdabpftrace.python interpreter: python3-3.9.0-1.fc33.x86_64 kernel: 5.8.16-300.fc33.x86_64 runlevel: unknown type: Python3 uid: 0 Truncated backtrace: subprocess.py:524:run:subprocess.CalledProcessError: Command '['bpftrace', '--version']' returned non-zero exit status 127. Traceback (most recent call last): File "/var/lib/pcp/pmdas/bpftrace/pmdabpftrace.python", line 21, in <module> BPFtracePMDA('bpftrace', 151).run() File "/usr/libexec/pcp/pmdas/bpftrace/bpftrace/pmda.py", line 53, in __init__ self.bpftrace_service.start_daemon() File "/usr/libexec/pcp/pmdas/bpftrace/bpftrace/service.py", line 47, in start_daemon runtime_info = self.gather_runtime_info() File "/usr/libexec/pcp/pmdas/bpftrace/bpftrace/service.py", line 32, in gather_runtime_info get_bpftrace_version(self.config.bpftrace_path) File "/usr/libexec/pcp/pmdas/bpftrace/bpftrace/utils.py", line 9, in get_bpftrace_version version_str = subprocess.check_output([bpftrace_path, '--version'], encoding='utf8').strip() File "/usr/lib64/python3.9/subprocess.py", line 420, in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, File "/usr/lib64/python3.9/subprocess.py", line 524, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['bpftrace', '--version']' returned non-zero exit status 127. Local variables in innermost frame: input: None capture_output: False timeout: None check: True popenargs: (['bpftrace', '--version'],) kwargs: {'stdout': -1, 'encoding': 'utf8'} process: <Popen: returncode: 127 args: ['bpftrace', '--version']> stdout: '' stderr: None retcode: 127
Created attachment 1725724 [details] File: backtrace
Created attachment 1725725 [details] File: cpuinfo
Created attachment 1725726 [details] File: environ
Created attachment 1725727 [details] File: mountinfo
Created attachment 1725728 [details] File: namespaces
Created attachment 1725729 [details] File: open_fds
edwin@storm-broadband:~ % bpftrace bpftrace: symbol lookup error: /lib64/libbcc-no-libbpf.so.0: undefined symbol: _ZN4llvm13IRBuilderBase18CreateGlobalStringENS_9StringRefERKNS_5TwineEj, version LLVM_11
Looks like bpftrace is completely broken due to the symbol lookup error. I have a fully updated Fedora 33. I found this in my upgrade history, perhaps related? Begin time : Mon 26 Oct 2020 20:24:19 G Upgrade llvm-11.0.0-1.fc33.x86_64 @updates Upgraded llvm-11.0.0-0.3.rc1.fc33.x86_64 @@System Looks like I previously started bpftrace on Oct 25, and restarted it on Oct 31. With that upgrade on Oct 26 inbetween: [Sun Oct 25 17:46:02] pmdabpftrace(1747) Info: manager: started pmdabpftrace process manager [Sun Oct 25 17:46:02] pmdabpftrace(1747) Info: manager: using bpftrace v0.11.0 [Sun Oct 25 17:46:02] pmdabpftrace(1651) Info: registering script from file /var/lib/pcp/pmdas/bpftrace/autostart/runqlat.bt... [Sun Oct 25 17:46:02] pmdabpftrace(1747) Info: script: starting script s6b24888363434b118ffa0cb3b5597b44 (name=runqlat, user=root)... [Sun Oct 25 17:46:02] pmdabpftrace(1651) Info: registering script from file /var/lib/pcp/pmdas/bpftrace/autostart/biolatency.bt... [Sun Oct 25 17:46:02] pmdabpftrace(1747) Info: script: starting script s635ff7ead03c4ecf9bacbe910c6bdb67 (name=biolatency, user=root)... [Sun Oct 25 17:46:02] pmdabpftrace(1747) Info: script: started script s6b24888363434b118ffa0cb3b5597b44 (name=runqlat, user=root, PID=1753) [Sun Oct 25 17:46:02] pmdabpftrace(1747) Info: script: started script s635ff7ead03c4ecf9bacbe910c6bdb67 (name=biolatency, user=root, PID=1756) [Sat Oct 31 19:35:07] pmdabpftrace(1747) Info: manager: shutting down pmdabpftrace process manager... [Sat Oct 31 19:35:07] pmdabpftrace(1747) Info: script: stopping script s6b24888363434b118ffa0cb3b5597b44 (name=runqlat, user=root, PID=1753)... [Sat Oct 31 19:35:07] pmdabpftrace(1747) Info: script: stopped script s6b24888363434b118ffa0cb3b5597b44 (name=runqlat, user=root, PID=1753) [Sat Oct 31 19:35:07] pmdabpftrace(1747) Info: script: deregistered script s6b24888363434b118ffa0cb3b5597b44 (name=runqlat, user=root, PID=1753) [Sat Oct 31 19:35:07] pmdabpftrace(1747) Info: script: stopping script s635ff7ead03c4ecf9bacbe910c6bdb67 (name=biolatency, user=root, PID=1756)... [Sat Oct 31 19:35:08] pmdabpftrace(1747) Info: script: stopped script s635ff7ead03c4ecf9bacbe910c6bdb67 (name=biolatency, user=root, PID=1756) [Sat Oct 31 19:35:08] pmdabpftrace(1747) Info: script: deregistered script s635ff7ead03c4ecf9bacbe910c6bdb67 (name=biolatency, user=root, PID=1756) [Sat Oct 31 19:35:08] pmdabpftrace(1747) Info: manager: waiting 10 secs for running tasks to stop...
In fact clang got upgraded too: Upgrade clang-11.0.0-1.fc33.x86_64 @updates Upgraded clang-11.0.0-0.2.rc1.fc33.x86_64 @@System Upgrade clang-devel-11.0.0-1.fc33.x86_64 @updates Upgraded clang-devel-11.0.0-0.2.rc1.fc33.x86_64 @@System Upgrade clang-libs-11.0.0-1.fc33.x86_64 @updates Upgraded clang-libs-11.0.0-0.2.rc1.fc33.x86_64 @@System Upgrade clang-tools-extra-11.0.0-1.fc33.x86_64 @updates Upgraded clang-tools-extra-11.0.0-0.2.rc1.fc33.x86_64 @@System If I install back the old version then bpftrace works: sudo dnf install llvm-11.0.0-0.3.rc1.fc33.x86_64 clang-libs-11.0.0-0.2.rc1.fc33.x86_64
Yep, some symbol change in LLVM affected libbcc which is used by bpftrace. Can you check which bcc version you have installed? I rebuilt bcc and a new package just hit stable, it should fix the problem.
bpftrace links against /usr/lib64/llvm10/lib/libclang.so.10, and /lib64/libclang-cpp.so.11 (weird combination? why doesn't is use llvm 11 too? anyway not related to this bug). clang-libs-11.0.0-0.2.rc1.fc33.x86_64 with llvm-libs-11.0.0-0.3.rc1.fc33.x86_64: nm -D /lib64/libclang-cpp.so.11|grep _ZN4llvm13IRBuilderBase18CreateGlobalStringENS_9StringRefERKNS_5TwineEj U _ZN4llvm13IRBuilderBase18CreateGlobalStringENS_9StringRefERKNS_5TwineEj@@LLVM_11 nm -D /lib64/libLLVM-11.so| grep _ZN4llvm13IRBuilderBase18CreateGlobalStringENS_9StringRefERKNS_5TwineEj@@LLVM_11 0000000000b67c40 T _ZN4llvm13IRBuilderBase18CreateGlobalStringENS_9StringRefERKNS_5TwineEj@@LLVM_11 After upgrading to llvm-libs-11.0.0-1.fc33.x86_64 the symbol has disappeared: edwin@storm-broadband:~ % nm -D /lib64/libLLVM-11.so| grep _ZN4llvm13IRBuilderBase18CreateGlobalStringENS_9StringRefERKNS_5TwineEj@@LLVM_11 edwin@storm-broadband:~ % nm -D /lib64/libclang-cpp.so.11|grep _ZN4llvm13IRBuilderBase18CreateGlobalStringENS_9StringRefERKNS_5TwineEj U _ZN4llvm13IRBuilderBase18CreateGlobalStringENS_9StringRefERKNS_5TwineEjPNS_6ModuleE@@LLVM_11 But the soname wasn't changed, so other binaries (such as bpftrace) linked against the previous (rc1) libclang now fail to run. I think that the soname needs to be bumped, or rpm made aware in some other way that this latest verison of llvm-libs is not installable with other binaries linked against the old llvm-libs. In fact bpftrace says it requires only .so.10, so why it ends up loading .so.11 not sure? edwin@storm-broadband:~ % rpm -q --requires bpftrace|grep -i clang libclang.so.10()(64bit) libclang.so.10(LLVM_10)(64bit) edwin@storm-broadband:~ % rpm -q --requires bpftrace|grep -i llvm libLLVM-10.so()(64bit) libLLVM-10.so(LLVM_10)(64bit) libclang.so.10(LLVM_10)(64bit) I have bcc-0.16.0-1.fc33.x86_64
I think it does because of libbcc: $: rpm -q --requires bpftrace | grep -i bcc libbcc-no-libbpf.so.0()(64bit) $: rpm -q --requires bcc | grep -i clang libclang-cpp.so.11()(64bit)
This bcc https://koji.fedoraproject.org/koji/buildinfo?buildID=1634131 was built against the new LLVM so it should work (although I agree there is still an issue with LLVM changing symbols without a soname bump).
related to https://bugzilla.redhat.com/show_bug.cgi?id=1890823. Unfortunately LLVM 11 release candidate upstream introduced an ABI regression, patch here https://src.fedoraproject.org/rpms/clang/pull-request/85# but we still have a regression to double-check.
I tested the new BCC and bpftrace works now with latest llvm/clang, thanks! https://bodhi.fedoraproject.org/updates/FEDORA-2020-b102ce4fd6
FEDORA-2020-7987e5f6e1 has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2020-7987e5f6e1
FEDORA-2020-7987e5f6e1 has been pushed to the Fedora 33 testing repository. In short time you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-7987e5f6e1` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-7987e5f6e1 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2020-7987e5f6e1 has been pushed to the Fedora 33 stable repository. If problem still persists, please make note of it in this bug report.