Bug 2232047 - Build-id mismatch task_finder mmap inode-uprobes callback for task XXXX failed
Summary: Build-id mismatch task_finder mmap inode-uprobes callback for task XXXX failed
Keywords:
Status: NEW
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: systemtap
Version: 8.7
Hardware: x86_64
OS: Linux
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Frank Ch. Eigler
QA Contact: qe-baseos-tools-bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-08-15 02:27 UTC by XinhuaLi
Modified: 2023-08-16 05:41 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-165808 0 None None None 2023-08-15 02:29:14 UTC

Description XinhuaLi 2023-08-15 02:27:18 UTC
Description of problem:

When we try to execute stap command as below, we can see WARNING as listed below.
~~~
 # stap -vge 'probe process("/usr/lib64/libsystemd.so.0.23.0").function("sd_bus_open_system") {print_ubacktrace() exit()}'
Pass 1: parsed user script and 506 library scripts using 356700virt/154692res/17156shr/139544data kb, in 500usr/100sys/617real ms.
Pass 2: analyzed script: 1 probe, 2 functions, 0 embeds, 0 globals using 374716virt/173776res/18208shr/157560data kb, in 120usr/20sys/131real ms.
Pass 3: using cached /root/.systemtap/cache/7e/stap_7edbe3badfbf0f26279da1ce235c1724_1672.c
Pass 4: using cached /root/.systemtap/cache/7e/stap_7edbe3badfbf0f26279da1ce235c1724_1672.ko
Pass 5: starting run.
WARNING: Build-id mismatch [man warning::buildid]: "/usr/lib64/libsystemd.so.0.23.0" pid 4951 address 0x7fde382a3280, expected 2cc229fa40c8781cf10c5d5f692e5bfba709fa64 actual 01002501f8954c47fd3c0e45f3272230b17c64d9
WARNING: task_finder mmap inode-uprobes callback for task 4951 failed: 1
WARNING: Build-id mismatch [man warning::buildid]: "/usr/lib64/libsystemd.so.0.23.0" pid 4827 address 0x7f4b5019d280, expected 2cc229fa40c8781cf10c5d5f692e5bfba709fa64 actual 01002501f8954c47fd3c0e45f3272230b17c64d9
WARNING: task_finder mmap inode-uprobes callback for task 4827 failed: 1
WARNING: Build-id mismatch [man warning::buildid]: "/usr/lib64/libsystemd.so.0.23.0" pid 4949 address 0x7fde382a3280, expected 2cc229fa40c8781cf10c5d5f692e5bfba709fa64 actual 01002501f8954c47fd3c0e45f3272230b17c64d9
......
~~~

Version-Release number of selected component (if applicable):
~~~
# rpm -qa |grep systemtap
systemtap-client-4.7-1.el8.x86_64
systemtap-debugsource-4.7-1.el8.x86_64
systemtap-4.7-1.el8.x86_64
systemtap-runtime-4.7-1.el8.x86_64
systemtap-devel-4.7-1.el8.x86_64
systemtap-runtime-debuginfo-4.7-1.el8.x86_64
systemtap-sdt-devel-4.7-1.el8.x86_64
systemtap-runtime-virtguest-4.7-1.el8.x86_64
systemtap-debuginfo-4.7-1.el8.x86_64

# rpm -qa |grep systemd
systemd-udev-239-68.el8.x86_64
systemd-debuginfo-239-68.el8.x86_64
systemd-container-239-68.el8.x86_64
systemd-debuginfo-239-68.el8.i686
rpm-plugin-systemd-inhibit-4.14.3-24.el8_7.x86_64
systemd-libs-239-68.el8.i686
systemd-pam-debuginfo-239-68.el8.x86_64
systemd-libs-debuginfo-239-68.el8.i686
systemd-pam-239-68.el8.x86_64
systemd-239-68.el8.x86_64
systemd-devel-239-68.el8.x86_64
systemd-libs-239-68.el8.x86_64
systemd-debugsource-239-68.el8.i686
systemd-debugsource-239-68.el8.x86_64
python3-systemd-234-8.el8.x86_64
systemd-libs-debuginfo-239-68.el8.x86_64

kernel -
4.18.0-425.13.1.el8_7.x86_64

RH -
Red Hat Enterprise Linux release 8.7 (Ootpa)
~~~

How reproducible:
execute command as below
~~~
# stap -vge 'probe process("/usr/lib64/libsystemd.so.0.23.0").function("sd_bus_open_system") {print_ubacktrace() exit()}'
~~~

Steps to Reproduce:
1.
2.
3.

Actual results:
WARNING keep showing 

Expected results:
No warning 

Additional info:

Regards
Sam

Comment 1 Frank Ch. Eigler 2023-08-15 19:10:26 UTC
This sort of thing can happen if the installed and running copies of the binaries are different, such as during a dnf upgrade, or perhaps during mismatches between containerized and host software versions.  Perhaps the change occurred across a stap module compilation / run cycle.

In case it is the last problem, try:

# stap --poison-cache [...]

Comment 2 Frank Ch. Eigler 2023-08-15 19:17:04 UTC
By the way, can you share the contents of the /root/.systemtap/cache/7e/stap_7edbe3badfbf0f26279da1ce235*hash.log file?

Comment 3 XinhuaLi 2023-08-16 05:41:00 UTC
Hi Frank,

Thanks for your update and info as listed below.
~~~

# cat /root/.systemtap/cache/7e/stap_7edbe3badfbf0f26279da1ce235*hash.log
[Tue Aug 15 10:08:02 2023] script_hash:
Systemtap version: 4.7/0.187/0.186, rpm 4.7-1.el8
Kernel Release: 4.18.0-425.13.1.el8_7.x86_64
Kernel Build Tree Path: /lib/modules/4.18.0-425.13.1.el8_7.x86_64/build
Kernel Build Tree Size: 4096
Kernel Build Tree Timestamp: 1677588474
Architecture: x86_64
Kernel Build Tree .config Path: /lib/modules/4.18.0-425.13.1.el8_7.x86_64/build/.config
Kernel Build Tree .config Size: 198608
Kernel Build Tree .config Timestamp: 1675361720
Kernel Build Tree .version Path: /lib/modules/4.18.0-425.13.1.el8_7.x86_64/build/.version
Kernel Build Tree .version Size: -1
Kernel Build Tree .version Timestamp: -1
Kernel Build Tree compile.h Path: /lib/modules/4.18.0-425.13.1.el8_7.x86_64/build/include/linux/compile.h
Kernel Build Tree compile.h Size: -1
Kernel Build Tree compile.h Timestamp: -1
Kernel Build Tree version.h Path: /lib/modules/4.18.0-425.13.1.el8_7.x86_64/build/include/linux/version.h
Kernel Build Tree version.h Size: -1
Kernel Build Tree version.h Timestamp: -1
Kernel Build Tree utsrelease.h Path: /lib/modules/4.18.0-425.13.1.el8_7.x86_64/build/include/linux/utsrelease.h
Kernel Build Tree utsrelease.h Size: -1
Kernel Build Tree utsrelease.h Timestamp: -1
Guru mode 1
Runtime Path: /usr/share/systemtap/runtime
Runtime Size: 4096
Runtime Timestamp: 1668568962
Runtime transport Path: /usr/share/systemtap/runtime/transport
Runtime transport Size: 192
Runtime transport Timestamp: 1668568962
Runtime unwind Path: /usr/share/systemtap/runtime/unwind
Runtime unwind Size: 139
Runtime unwind Timestamp: 1668568962
Runtime sub Path: /usr/share/systemtap/runtime/linux
Runtime sub Size: 8192
Runtime sub Timestamp: 1668568962
Compiler Path: /usr/bin/gcc
Compiler Size: 1266664
Compiler Timestamp: 1658333104
Systemtap Path: /usr/bin/stap
Systemtap Size: 3860344
Systemtap Timestamp: 1651514709
UID: 0
Bulk Mode (-b): 0
Timing (-t): 0
Skip Badvars (--skip-badvars): 0
Privilege (--privilege): 8
Compatible (--compatible): 4.7
Error suppression (--suppress-handler-errors): 0
Suppress Time Limits (--suppress-time-limits): 0
Prologue Searching (--prologue-searching[=WHEN]): 1
Unwindsym Modules Path: /lib/modules/4.18.0-425.13.1.el8_7.x86_64/vdso/vdso32.so
Unwindsym Modules Size: 5492
Unwindsym Modules Timestamp: 1675362002
Unwindsym Modules Path: /lib/modules/4.18.0-425.13.1.el8_7.x86_64/vdso/vdso64.so
Unwindsym Modules Size: 5232
Unwindsym Modules Timestamp: 1675362003
Unwindsym Modules Path: /usr/lib64/libsystemd.so.0.23.0
Unwindsym Modules Size: 1383536
Unwindsym Modules Timestamp: 1664264120
Build ID: 2cc229fa40c8781cf10c5d5f692e5bfba709fa64
Script:
# functions
exit:unknown ()
{ %{ /* unprivileged */
   atomic_set (session_state(), STAP_SESSION_STOPPING);
   _stp_exit ();
%} }
print_ubacktrace:unknown ()
{ %{ /* pragma:unwind */ /* pragma:symbols */
/* myproc-unprivileged */ /* pragma:uprobes */ /* pragma:vma */
    _stp_stack_user_print(CONTEXT, _STP_SYM_FULL);
%} }
# probes
process("/usr/lib64/libsystemd.so.0.23.0").function("sd_bus_open_system@../src/libsystemd/sd-bus/sd-bus.c:1293") /* pc=.dynamic+0x52a50 */ /* <- process("/usr/lib64/libsystemd.so.0.23.0").function("sd_bus_open_system@../src/libsystemd/sd-bus/sd-bus.c:1293") */
{ {
print_ubacktrace();
exit();
} }


result:7edbe3badfbf0f26279da1ce235c1724_1672
~~~
Regards
Sam


Note You need to log in before you can comment on or make changes to this bug.