Bug 1893665

Summary:

[abrt] bpftrace: symbol lookup error: /lib64/libbcc-no-libbpf.so.0: undefined symbol: _ZN4llvm13IRBuilderBase18CreateGlobalStringENS_9StringRefERKNS_5TwineEj, version LLVM_11. Symbol disappeared without changing soname

Product:

[Fedora] Fedora

Reporter:

Török Edwin <edwin+bugs>

Component:

clang

Assignee:

Tom Stellard <tstellar>

Status:

CLOSED ERRATA

QA Contact:

Fedora Extras Quality Assurance <extras-qa>

Severity:

urgent

Docs Contact:

Priority:

urgent

Version:

CC:

acaringi, agerstmayr, airlied, jmarchan, jolsa, mgoodwin, nathans, rdossant, sbergman, sguelton, siddharth.kde, skozina, tstellar

Target Milestone:

---

Target Release:

---

Hardware:

x86_64

OS:

Unspecified

URL:

https://retrace.fedoraproject.org/faf/reports/bthash/602c8b6aca5ec47ed1975bda5dee58baca537d15

Whiteboard:

abrt_hash:8065e7c2f3bfbf8350963025e8ad6b2d716465ac;VARIANT_ID=workstation;

Fixed In Version:

clang-11.0.0-2.fc33

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2020-11-12 03:06:22 UTC

Type:

---

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
File: backtrace	none
File: cpuinfo	none
File: environ	none
File: mountinfo	none
File: namespaces	none
File: open_fds	none

Description Török Edwin 2020-11-02 10:29:11 UTC

Version-Release number of selected component:
pcp-pmda-bpftrace-5.2.1-1.fc33

Additional info:
reporter:       libreport-2.14.0
cgroup:         0::/system.slice/pmcd.service
cmdline:        python3 /var/lib/pcp/pmdas/bpftrace/pmdabpftrace.python
crash_function: run
exception_type: subprocess.CalledProcessError
executable:     /var/lib/pcp/pmdas/bpftrace/pmdabpftrace.python
interpreter:    python3-3.9.0-1.fc33.x86_64
kernel:         5.8.16-300.fc33.x86_64
runlevel:       unknown
type:           Python3
uid:            0

Truncated backtrace:
subprocess.py:524:run:subprocess.CalledProcessError: Command '['bpftrace', '--version']' returned non-zero exit status 127.

Traceback (most recent call last):
  File "/var/lib/pcp/pmdas/bpftrace/pmdabpftrace.python", line 21, in <module>
    BPFtracePMDA('bpftrace', 151).run()
  File "/usr/libexec/pcp/pmdas/bpftrace/bpftrace/pmda.py", line 53, in __init__
    self.bpftrace_service.start_daemon()
  File "/usr/libexec/pcp/pmdas/bpftrace/bpftrace/service.py", line 47, in start_daemon
    runtime_info = self.gather_runtime_info()
  File "/usr/libexec/pcp/pmdas/bpftrace/bpftrace/service.py", line 32, in gather_runtime_info
    get_bpftrace_version(self.config.bpftrace_path)
  File "/usr/libexec/pcp/pmdas/bpftrace/bpftrace/utils.py", line 9, in get_bpftrace_version
    version_str = subprocess.check_output([bpftrace_path, '--version'], encoding='utf8').strip()
  File "/usr/lib64/python3.9/subprocess.py", line 420, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib64/python3.9/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['bpftrace', '--version']' returned non-zero exit status 127.

Local variables in innermost frame:
input: None
capture_output: False
timeout: None
check: True
popenargs: (['bpftrace', '--version'],)
kwargs: {'stdout': -1, 'encoding': 'utf8'}
process: <Popen: returncode: 127 args: ['bpftrace', '--version']>
stdout: ''
stderr: None
retcode: 127

Comment 1 Török Edwin 2020-11-02 10:29:15 UTC

Created attachment 1725724 [details]
File: backtrace

Comment 2 Török Edwin 2020-11-02 10:29:17 UTC

Created attachment 1725725 [details]
File: cpuinfo

Comment 3 Török Edwin 2020-11-02 10:29:19 UTC

Created attachment 1725726 [details]
File: environ

Comment 4 Török Edwin 2020-11-02 10:29:21 UTC

Created attachment 1725727 [details]
File: mountinfo

Comment 5 Török Edwin 2020-11-02 10:29:23 UTC

Created attachment 1725728 [details]
File: namespaces

Comment 6 Török Edwin 2020-11-02 10:29:24 UTC

Created attachment 1725729 [details]
File: open_fds

Comment 7 Török Edwin 2020-11-02 10:30:26 UTC

edwin@storm-broadband:~ % bpftrace
bpftrace: symbol lookup error: /lib64/libbcc-no-libbpf.so.0: undefined symbol: _ZN4llvm13IRBuilderBase18CreateGlobalStringENS_9StringRefERKNS_5TwineEj, version LLVM_11

Comment 8 Török Edwin 2020-11-02 10:38:39 UTC

Looks like bpftrace is completely broken due to the symbol lookup error.
I have a fully updated Fedora 33.

I found this in my upgrade history, perhaps related?
Begin time     : Mon 26 Oct 2020 20:24:19 G
    Upgrade  llvm-11.0.0-1.fc33.x86_64                                      @updates
    Upgraded llvm-11.0.0-0.3.rc1.fc33.x86_64                                @@System

Looks like I previously started bpftrace on Oct 25, and restarted it on Oct 31. With that upgrade on Oct 26 inbetween:
[Sun Oct 25 17:46:02] pmdabpftrace(1747) Info: manager: started pmdabpftrace process manager
[Sun Oct 25 17:46:02] pmdabpftrace(1747) Info: manager: using bpftrace v0.11.0
[Sun Oct 25 17:46:02] pmdabpftrace(1651) Info: registering script from file /var/lib/pcp/pmdas/bpftrace/autostart/runqlat.bt...
[Sun Oct 25 17:46:02] pmdabpftrace(1747) Info: script: starting script s6b24888363434b118ffa0cb3b5597b44 (name=runqlat, user=root)...
[Sun Oct 25 17:46:02] pmdabpftrace(1651) Info: registering script from file /var/lib/pcp/pmdas/bpftrace/autostart/biolatency.bt...
[Sun Oct 25 17:46:02] pmdabpftrace(1747) Info: script: starting script s635ff7ead03c4ecf9bacbe910c6bdb67 (name=biolatency, user=root)...
[Sun Oct 25 17:46:02] pmdabpftrace(1747) Info: script: started script s6b24888363434b118ffa0cb3b5597b44 (name=runqlat, user=root, PID=1753)
[Sun Oct 25 17:46:02] pmdabpftrace(1747) Info: script: started script s635ff7ead03c4ecf9bacbe910c6bdb67 (name=biolatency, user=root, PID=1756)
[Sat Oct 31 19:35:07] pmdabpftrace(1747) Info: manager: shutting down pmdabpftrace process manager...
[Sat Oct 31 19:35:07] pmdabpftrace(1747) Info: script: stopping script s6b24888363434b118ffa0cb3b5597b44 (name=runqlat, user=root, PID=1753)...
[Sat Oct 31 19:35:07] pmdabpftrace(1747) Info: script: stopped script s6b24888363434b118ffa0cb3b5597b44 (name=runqlat, user=root, PID=1753)
[Sat Oct 31 19:35:07] pmdabpftrace(1747) Info: script: deregistered script s6b24888363434b118ffa0cb3b5597b44 (name=runqlat, user=root, PID=1753)
[Sat Oct 31 19:35:07] pmdabpftrace(1747) Info: script: stopping script s635ff7ead03c4ecf9bacbe910c6bdb67 (name=biolatency, user=root, PID=1756)...
[Sat Oct 31 19:35:08] pmdabpftrace(1747) Info: script: stopped script s635ff7ead03c4ecf9bacbe910c6bdb67 (name=biolatency, user=root, PID=1756)
[Sat Oct 31 19:35:08] pmdabpftrace(1747) Info: script: deregistered script s635ff7ead03c4ecf9bacbe910c6bdb67 (name=biolatency, user=root, PID=1756)
[Sat Oct 31 19:35:08] pmdabpftrace(1747) Info: manager: waiting 10 secs for running tasks to stop...

Comment 9 Török Edwin 2020-11-02 10:42:31 UTC

In fact clang got upgraded too:
 Upgrade  clang-11.0.0-1.fc33.x86_64                                     @updates
    Upgraded clang-11.0.0-0.2.rc1.fc33.x86_64                               @@System
    Upgrade  clang-devel-11.0.0-1.fc33.x86_64                               @updates
    Upgraded clang-devel-11.0.0-0.2.rc1.fc33.x86_64                         @@System
    Upgrade  clang-libs-11.0.0-1.fc33.x86_64                                @updates
    Upgraded clang-libs-11.0.0-0.2.rc1.fc33.x86_64                          @@System
    Upgrade  clang-tools-extra-11.0.0-1.fc33.x86_64                         @updates
    Upgraded clang-tools-extra-11.0.0-0.2.rc1.fc33.x86_64                   @@System

If I install back the old version then bpftrace works:
sudo dnf install llvm-11.0.0-0.3.rc1.fc33.x86_64 clang-libs-11.0.0-0.2.rc1.fc33.x86_64

Comment 10 Rafael Fonseca 2020-11-02 10:53:18 UTC

Yep, some symbol change in LLVM affected libbcc which is used by bpftrace. Can you check which bcc version you have installed? I rebuilt bcc and a new package just hit stable, it should fix the problem.

Comment 11 Török Edwin 2020-11-02 10:57:09 UTC

bpftrace links against /usr/lib64/llvm10/lib/libclang.so.10, and /lib64/libclang-cpp.so.11 (weird combination? why doesn't is use llvm 11 too? anyway not related to this bug).

clang-libs-11.0.0-0.2.rc1.fc33.x86_64 with llvm-libs-11.0.0-0.3.rc1.fc33.x86_64:
nm -D /lib64/libclang-cpp.so.11|grep _ZN4llvm13IRBuilderBase18CreateGlobalStringENS_9StringRefERKNS_5TwineEj
                 U _ZN4llvm13IRBuilderBase18CreateGlobalStringENS_9StringRefERKNS_5TwineEj@@LLVM_11
nm -D /lib64/libLLVM-11.so| grep _ZN4llvm13IRBuilderBase18CreateGlobalStringENS_9StringRefERKNS_5TwineEj@@LLVM_11
0000000000b67c40 T _ZN4llvm13IRBuilderBase18CreateGlobalStringENS_9StringRefERKNS_5TwineEj@@LLVM_11

After upgrading to llvm-libs-11.0.0-1.fc33.x86_64 the symbol has disappeared:
edwin@storm-broadband:~ % nm -D /lib64/libLLVM-11.so| grep _ZN4llvm13IRBuilderBase18CreateGlobalStringENS_9StringRefERKNS_5TwineEj@@LLVM_11
edwin@storm-broadband:~ % nm -D /lib64/libclang-cpp.so.11|grep _ZN4llvm13IRBuilderBase18CreateGlobalStringENS_9StringRefERKNS_5TwineEj
                 U _ZN4llvm13IRBuilderBase18CreateGlobalStringENS_9StringRefERKNS_5TwineEjPNS_6ModuleE@@LLVM_11

But the soname wasn't changed, so other binaries (such as bpftrace) linked against the previous (rc1) libclang now fail to run.
I think that the soname needs to be bumped, or rpm made aware in some other way that this latest verison of llvm-libs is not installable with other binaries linked against the old llvm-libs.

In fact bpftrace says it requires only .so.10, so why it ends up loading .so.11 not sure?
edwin@storm-broadband:~ % rpm -q --requires bpftrace|grep -i clang
libclang.so.10()(64bit)
libclang.so.10(LLVM_10)(64bit)
edwin@storm-broadband:~ % rpm -q --requires bpftrace|grep -i llvm
libLLVM-10.so()(64bit)
libLLVM-10.so(LLVM_10)(64bit)
libclang.so.10(LLVM_10)(64bit)


I have bcc-0.16.0-1.fc33.x86_64

Comment 12 Rafael Fonseca 2020-11-02 11:07:16 UTC

I think it does because of libbcc:

$: rpm -q --requires bpftrace | grep -i bcc
libbcc-no-libbpf.so.0()(64bit)

$: rpm -q --requires bcc | grep -i clang
libclang-cpp.so.11()(64bit)

Comment 13 Rafael Fonseca 2020-11-02 11:10:09 UTC

This bcc https://koji.fedoraproject.org/koji/buildinfo?buildID=1634131 was built against the new LLVM so it should work (although I agree there is still an issue with LLVM changing symbols without a soname bump).

Comment 14 serge_sans_paille 2020-11-02 13:55:22 UTC

related to https://bugzilla.redhat.com/show_bug.cgi?id=1890823. Unfortunately LLVM 11 release candidate upstream introduced an ABI regression, patch here https://src.fedoraproject.org/rpms/clang/pull-request/85# but we still have a regression to double-check.

Comment 15 Török Edwin 2020-11-02 20:34:43 UTC

I tested the new BCC and bpftrace works now with latest llvm/clang, thanks! https://bodhi.fedoraproject.org/updates/FEDORA-2020-b102ce4fd6

Comment 16 Fedora Update System 2020-11-03 14:28:48 UTC

FEDORA-2020-7987e5f6e1 has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2020-7987e5f6e1

Comment 17 Fedora Update System 2020-11-04 03:00:55 UTC

FEDORA-2020-7987e5f6e1 has been pushed to the Fedora 33 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-7987e5f6e1`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-7987e5f6e1

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 18 Fedora Update System 2020-11-12 03:06:22 UTC

FEDORA-2020-7987e5f6e1 has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.