Bug 2232047
| Summary: | Build-id mismatch task_finder mmap inode-uprobes callback for task XXXX failed | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | XinhuaLi <xili> |
| Component: | systemtap | Assignee: | Frank Ch. Eigler <fche> |
| systemtap sub component: | system-version | QA Contact: | qe-baseos-tools-bugs |
| Status: | CLOSED MIGRATED | Docs Contact: | |
| Severity: | unspecified | ||
| Priority: | unspecified | CC: | lberk, mcermak, mjw, wcohen |
| Version: | 8.7 | Keywords: | MigratedToJIRA, Triaged |
| Target Milestone: | rc | Flags: | pm-rhel:
mirror+
|
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-09-22 16:01:42 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
This sort of thing can happen if the installed and running copies of the binaries are different, such as during a dnf upgrade, or perhaps during mismatches between containerized and host software versions. Perhaps the change occurred across a stap module compilation / run cycle. In case it is the last problem, try: # stap --poison-cache [...] By the way, can you share the contents of the /root/.systemtap/cache/7e/stap_7edbe3badfbf0f26279da1ce235*hash.log file? Hi Frank,
Thanks for your update and info as listed below.
~~~
# cat /root/.systemtap/cache/7e/stap_7edbe3badfbf0f26279da1ce235*hash.log
[Tue Aug 15 10:08:02 2023] script_hash:
Systemtap version: 4.7/0.187/0.186, rpm 4.7-1.el8
Kernel Release: 4.18.0-425.13.1.el8_7.x86_64
Kernel Build Tree Path: /lib/modules/4.18.0-425.13.1.el8_7.x86_64/build
Kernel Build Tree Size: 4096
Kernel Build Tree Timestamp: 1677588474
Architecture: x86_64
Kernel Build Tree .config Path: /lib/modules/4.18.0-425.13.1.el8_7.x86_64/build/.config
Kernel Build Tree .config Size: 198608
Kernel Build Tree .config Timestamp: 1675361720
Kernel Build Tree .version Path: /lib/modules/4.18.0-425.13.1.el8_7.x86_64/build/.version
Kernel Build Tree .version Size: -1
Kernel Build Tree .version Timestamp: -1
Kernel Build Tree compile.h Path: /lib/modules/4.18.0-425.13.1.el8_7.x86_64/build/include/linux/compile.h
Kernel Build Tree compile.h Size: -1
Kernel Build Tree compile.h Timestamp: -1
Kernel Build Tree version.h Path: /lib/modules/4.18.0-425.13.1.el8_7.x86_64/build/include/linux/version.h
Kernel Build Tree version.h Size: -1
Kernel Build Tree version.h Timestamp: -1
Kernel Build Tree utsrelease.h Path: /lib/modules/4.18.0-425.13.1.el8_7.x86_64/build/include/linux/utsrelease.h
Kernel Build Tree utsrelease.h Size: -1
Kernel Build Tree utsrelease.h Timestamp: -1
Guru mode 1
Runtime Path: /usr/share/systemtap/runtime
Runtime Size: 4096
Runtime Timestamp: 1668568962
Runtime transport Path: /usr/share/systemtap/runtime/transport
Runtime transport Size: 192
Runtime transport Timestamp: 1668568962
Runtime unwind Path: /usr/share/systemtap/runtime/unwind
Runtime unwind Size: 139
Runtime unwind Timestamp: 1668568962
Runtime sub Path: /usr/share/systemtap/runtime/linux
Runtime sub Size: 8192
Runtime sub Timestamp: 1668568962
Compiler Path: /usr/bin/gcc
Compiler Size: 1266664
Compiler Timestamp: 1658333104
Systemtap Path: /usr/bin/stap
Systemtap Size: 3860344
Systemtap Timestamp: 1651514709
UID: 0
Bulk Mode (-b): 0
Timing (-t): 0
Skip Badvars (--skip-badvars): 0
Privilege (--privilege): 8
Compatible (--compatible): 4.7
Error suppression (--suppress-handler-errors): 0
Suppress Time Limits (--suppress-time-limits): 0
Prologue Searching (--prologue-searching[=WHEN]): 1
Unwindsym Modules Path: /lib/modules/4.18.0-425.13.1.el8_7.x86_64/vdso/vdso32.so
Unwindsym Modules Size: 5492
Unwindsym Modules Timestamp: 1675362002
Unwindsym Modules Path: /lib/modules/4.18.0-425.13.1.el8_7.x86_64/vdso/vdso64.so
Unwindsym Modules Size: 5232
Unwindsym Modules Timestamp: 1675362003
Unwindsym Modules Path: /usr/lib64/libsystemd.so.0.23.0
Unwindsym Modules Size: 1383536
Unwindsym Modules Timestamp: 1664264120
Build ID: 2cc229fa40c8781cf10c5d5f692e5bfba709fa64
Script:
# functions
exit:unknown ()
{ %{ /* unprivileged */
atomic_set (session_state(), STAP_SESSION_STOPPING);
_stp_exit ();
%} }
print_ubacktrace:unknown ()
{ %{ /* pragma:unwind */ /* pragma:symbols */
/* myproc-unprivileged */ /* pragma:uprobes */ /* pragma:vma */
_stp_stack_user_print(CONTEXT, _STP_SYM_FULL);
%} }
# probes
process("/usr/lib64/libsystemd.so.0.23.0").function("sd_bus_open_system@../src/libsystemd/sd-bus/sd-bus.c:1293") /* pc=.dynamic+0x52a50 */ /* <- process("/usr/lib64/libsystemd.so.0.23.0").function("sd_bus_open_system@../src/libsystemd/sd-bus/sd-bus.c:1293") */
{ {
print_ubacktrace();
exit();
} }
result:7edbe3badfbf0f26279da1ce235c1724_1672
~~~
Regards
Sam
The build-id for that particular version of systemd-libs matches up with the 2cc229fa40c8781cf10c5d5f692e5bfba709fa64 reported as the hexstring_theory. It appears that the checking might be getting some bogus build-id value for the hexstring_practice. This results in the following printing the warning message: https://sourceware.org/git/?p=systemtap.git;a=blob;f=runtime/sym.c;h=f5c52d6d4bb21b3d8ce9e8edcd348ac9596f643a;hb=HEAD#l729 However, I haven't been able to replicate the issue despite using identical kernel and systemd* rpms. The only difference between the *_hash.log in the comment above and the one generated during the experiments are the lines with "*Timestap: *" and the "results:*" last line. Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug. This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there. Due to differences in account names between systems, some fields were not replicated. Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information. To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer. You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like: "Bugzilla Bug" = 1234567 In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information. |
Description of problem: When we try to execute stap command as below, we can see WARNING as listed below. ~~~ # stap -vge 'probe process("/usr/lib64/libsystemd.so.0.23.0").function("sd_bus_open_system") {print_ubacktrace() exit()}' Pass 1: parsed user script and 506 library scripts using 356700virt/154692res/17156shr/139544data kb, in 500usr/100sys/617real ms. Pass 2: analyzed script: 1 probe, 2 functions, 0 embeds, 0 globals using 374716virt/173776res/18208shr/157560data kb, in 120usr/20sys/131real ms. Pass 3: using cached /root/.systemtap/cache/7e/stap_7edbe3badfbf0f26279da1ce235c1724_1672.c Pass 4: using cached /root/.systemtap/cache/7e/stap_7edbe3badfbf0f26279da1ce235c1724_1672.ko Pass 5: starting run. WARNING: Build-id mismatch [man warning::buildid]: "/usr/lib64/libsystemd.so.0.23.0" pid 4951 address 0x7fde382a3280, expected 2cc229fa40c8781cf10c5d5f692e5bfba709fa64 actual 01002501f8954c47fd3c0e45f3272230b17c64d9 WARNING: task_finder mmap inode-uprobes callback for task 4951 failed: 1 WARNING: Build-id mismatch [man warning::buildid]: "/usr/lib64/libsystemd.so.0.23.0" pid 4827 address 0x7f4b5019d280, expected 2cc229fa40c8781cf10c5d5f692e5bfba709fa64 actual 01002501f8954c47fd3c0e45f3272230b17c64d9 WARNING: task_finder mmap inode-uprobes callback for task 4827 failed: 1 WARNING: Build-id mismatch [man warning::buildid]: "/usr/lib64/libsystemd.so.0.23.0" pid 4949 address 0x7fde382a3280, expected 2cc229fa40c8781cf10c5d5f692e5bfba709fa64 actual 01002501f8954c47fd3c0e45f3272230b17c64d9 ...... ~~~ Version-Release number of selected component (if applicable): ~~~ # rpm -qa |grep systemtap systemtap-client-4.7-1.el8.x86_64 systemtap-debugsource-4.7-1.el8.x86_64 systemtap-4.7-1.el8.x86_64 systemtap-runtime-4.7-1.el8.x86_64 systemtap-devel-4.7-1.el8.x86_64 systemtap-runtime-debuginfo-4.7-1.el8.x86_64 systemtap-sdt-devel-4.7-1.el8.x86_64 systemtap-runtime-virtguest-4.7-1.el8.x86_64 systemtap-debuginfo-4.7-1.el8.x86_64 # rpm -qa |grep systemd systemd-udev-239-68.el8.x86_64 systemd-debuginfo-239-68.el8.x86_64 systemd-container-239-68.el8.x86_64 systemd-debuginfo-239-68.el8.i686 rpm-plugin-systemd-inhibit-4.14.3-24.el8_7.x86_64 systemd-libs-239-68.el8.i686 systemd-pam-debuginfo-239-68.el8.x86_64 systemd-libs-debuginfo-239-68.el8.i686 systemd-pam-239-68.el8.x86_64 systemd-239-68.el8.x86_64 systemd-devel-239-68.el8.x86_64 systemd-libs-239-68.el8.x86_64 systemd-debugsource-239-68.el8.i686 systemd-debugsource-239-68.el8.x86_64 python3-systemd-234-8.el8.x86_64 systemd-libs-debuginfo-239-68.el8.x86_64 kernel - 4.18.0-425.13.1.el8_7.x86_64 RH - Red Hat Enterprise Linux release 8.7 (Ootpa) ~~~ How reproducible: execute command as below ~~~ # stap -vge 'probe process("/usr/lib64/libsystemd.so.0.23.0").function("sd_bus_open_system") {print_ubacktrace() exit()}' ~~~ Steps to Reproduce: 1. 2. 3. Actual results: WARNING keep showing Expected results: No warning Additional info: Regards Sam