This bug has been migrated to another issue tracking site. It has been closed here and may no longer be being monitored.

If you would like to get updates for this issue, or to participate in it, you may do so at Red Hat Issue Tracker .
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2232047 - Build-id mismatch task_finder mmap inode-uprobes callback for task XXXX failed
Summary: Build-id mismatch task_finder mmap inode-uprobes callback for task XXXX failed
Keywords:
Status: CLOSED MIGRATED
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: systemtap
Version: 8.7
Hardware: x86_64
OS: Linux
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Frank Ch. Eigler
QA Contact: qe-baseos-tools-bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-08-15 02:27 UTC by XinhuaLi
Modified: 2023-09-22 16:01 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-09-22 16:01:42 UTC
Type: Bug
Target Upstream Version:
Embargoed:
pm-rhel: mirror+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker   RHEL-7333 0 None Migrated None 2023-09-22 16:01:33 UTC
Red Hat Issue Tracker RHELPLAN-165808 0 None None None 2023-08-15 02:29:14 UTC

Description XinhuaLi 2023-08-15 02:27:18 UTC
Description of problem:

When we try to execute stap command as below, we can see WARNING as listed below.
~~~
 # stap -vge 'probe process("/usr/lib64/libsystemd.so.0.23.0").function("sd_bus_open_system") {print_ubacktrace() exit()}'
Pass 1: parsed user script and 506 library scripts using 356700virt/154692res/17156shr/139544data kb, in 500usr/100sys/617real ms.
Pass 2: analyzed script: 1 probe, 2 functions, 0 embeds, 0 globals using 374716virt/173776res/18208shr/157560data kb, in 120usr/20sys/131real ms.
Pass 3: using cached /root/.systemtap/cache/7e/stap_7edbe3badfbf0f26279da1ce235c1724_1672.c
Pass 4: using cached /root/.systemtap/cache/7e/stap_7edbe3badfbf0f26279da1ce235c1724_1672.ko
Pass 5: starting run.
WARNING: Build-id mismatch [man warning::buildid]: "/usr/lib64/libsystemd.so.0.23.0" pid 4951 address 0x7fde382a3280, expected 2cc229fa40c8781cf10c5d5f692e5bfba709fa64 actual 01002501f8954c47fd3c0e45f3272230b17c64d9
WARNING: task_finder mmap inode-uprobes callback for task 4951 failed: 1
WARNING: Build-id mismatch [man warning::buildid]: "/usr/lib64/libsystemd.so.0.23.0" pid 4827 address 0x7f4b5019d280, expected 2cc229fa40c8781cf10c5d5f692e5bfba709fa64 actual 01002501f8954c47fd3c0e45f3272230b17c64d9
WARNING: task_finder mmap inode-uprobes callback for task 4827 failed: 1
WARNING: Build-id mismatch [man warning::buildid]: "/usr/lib64/libsystemd.so.0.23.0" pid 4949 address 0x7fde382a3280, expected 2cc229fa40c8781cf10c5d5f692e5bfba709fa64 actual 01002501f8954c47fd3c0e45f3272230b17c64d9
......
~~~

Version-Release number of selected component (if applicable):
~~~
# rpm -qa |grep systemtap
systemtap-client-4.7-1.el8.x86_64
systemtap-debugsource-4.7-1.el8.x86_64
systemtap-4.7-1.el8.x86_64
systemtap-runtime-4.7-1.el8.x86_64
systemtap-devel-4.7-1.el8.x86_64
systemtap-runtime-debuginfo-4.7-1.el8.x86_64
systemtap-sdt-devel-4.7-1.el8.x86_64
systemtap-runtime-virtguest-4.7-1.el8.x86_64
systemtap-debuginfo-4.7-1.el8.x86_64

# rpm -qa |grep systemd
systemd-udev-239-68.el8.x86_64
systemd-debuginfo-239-68.el8.x86_64
systemd-container-239-68.el8.x86_64
systemd-debuginfo-239-68.el8.i686
rpm-plugin-systemd-inhibit-4.14.3-24.el8_7.x86_64
systemd-libs-239-68.el8.i686
systemd-pam-debuginfo-239-68.el8.x86_64
systemd-libs-debuginfo-239-68.el8.i686
systemd-pam-239-68.el8.x86_64
systemd-239-68.el8.x86_64
systemd-devel-239-68.el8.x86_64
systemd-libs-239-68.el8.x86_64
systemd-debugsource-239-68.el8.i686
systemd-debugsource-239-68.el8.x86_64
python3-systemd-234-8.el8.x86_64
systemd-libs-debuginfo-239-68.el8.x86_64

kernel -
4.18.0-425.13.1.el8_7.x86_64

RH -
Red Hat Enterprise Linux release 8.7 (Ootpa)
~~~

How reproducible:
execute command as below
~~~
# stap -vge 'probe process("/usr/lib64/libsystemd.so.0.23.0").function("sd_bus_open_system") {print_ubacktrace() exit()}'
~~~

Steps to Reproduce:
1.
2.
3.

Actual results:
WARNING keep showing 

Expected results:
No warning 

Additional info:

Regards
Sam

Comment 1 Frank Ch. Eigler 2023-08-15 19:10:26 UTC
This sort of thing can happen if the installed and running copies of the binaries are different, such as during a dnf upgrade, or perhaps during mismatches between containerized and host software versions.  Perhaps the change occurred across a stap module compilation / run cycle.

In case it is the last problem, try:

# stap --poison-cache [...]

Comment 2 Frank Ch. Eigler 2023-08-15 19:17:04 UTC
By the way, can you share the contents of the /root/.systemtap/cache/7e/stap_7edbe3badfbf0f26279da1ce235*hash.log file?

Comment 3 XinhuaLi 2023-08-16 05:41:00 UTC
Hi Frank,

Thanks for your update and info as listed below.
~~~

# cat /root/.systemtap/cache/7e/stap_7edbe3badfbf0f26279da1ce235*hash.log
[Tue Aug 15 10:08:02 2023] script_hash:
Systemtap version: 4.7/0.187/0.186, rpm 4.7-1.el8
Kernel Release: 4.18.0-425.13.1.el8_7.x86_64
Kernel Build Tree Path: /lib/modules/4.18.0-425.13.1.el8_7.x86_64/build
Kernel Build Tree Size: 4096
Kernel Build Tree Timestamp: 1677588474
Architecture: x86_64
Kernel Build Tree .config Path: /lib/modules/4.18.0-425.13.1.el8_7.x86_64/build/.config
Kernel Build Tree .config Size: 198608
Kernel Build Tree .config Timestamp: 1675361720
Kernel Build Tree .version Path: /lib/modules/4.18.0-425.13.1.el8_7.x86_64/build/.version
Kernel Build Tree .version Size: -1
Kernel Build Tree .version Timestamp: -1
Kernel Build Tree compile.h Path: /lib/modules/4.18.0-425.13.1.el8_7.x86_64/build/include/linux/compile.h
Kernel Build Tree compile.h Size: -1
Kernel Build Tree compile.h Timestamp: -1
Kernel Build Tree version.h Path: /lib/modules/4.18.0-425.13.1.el8_7.x86_64/build/include/linux/version.h
Kernel Build Tree version.h Size: -1
Kernel Build Tree version.h Timestamp: -1
Kernel Build Tree utsrelease.h Path: /lib/modules/4.18.0-425.13.1.el8_7.x86_64/build/include/linux/utsrelease.h
Kernel Build Tree utsrelease.h Size: -1
Kernel Build Tree utsrelease.h Timestamp: -1
Guru mode 1
Runtime Path: /usr/share/systemtap/runtime
Runtime Size: 4096
Runtime Timestamp: 1668568962
Runtime transport Path: /usr/share/systemtap/runtime/transport
Runtime transport Size: 192
Runtime transport Timestamp: 1668568962
Runtime unwind Path: /usr/share/systemtap/runtime/unwind
Runtime unwind Size: 139
Runtime unwind Timestamp: 1668568962
Runtime sub Path: /usr/share/systemtap/runtime/linux
Runtime sub Size: 8192
Runtime sub Timestamp: 1668568962
Compiler Path: /usr/bin/gcc
Compiler Size: 1266664
Compiler Timestamp: 1658333104
Systemtap Path: /usr/bin/stap
Systemtap Size: 3860344
Systemtap Timestamp: 1651514709
UID: 0
Bulk Mode (-b): 0
Timing (-t): 0
Skip Badvars (--skip-badvars): 0
Privilege (--privilege): 8
Compatible (--compatible): 4.7
Error suppression (--suppress-handler-errors): 0
Suppress Time Limits (--suppress-time-limits): 0
Prologue Searching (--prologue-searching[=WHEN]): 1
Unwindsym Modules Path: /lib/modules/4.18.0-425.13.1.el8_7.x86_64/vdso/vdso32.so
Unwindsym Modules Size: 5492
Unwindsym Modules Timestamp: 1675362002
Unwindsym Modules Path: /lib/modules/4.18.0-425.13.1.el8_7.x86_64/vdso/vdso64.so
Unwindsym Modules Size: 5232
Unwindsym Modules Timestamp: 1675362003
Unwindsym Modules Path: /usr/lib64/libsystemd.so.0.23.0
Unwindsym Modules Size: 1383536
Unwindsym Modules Timestamp: 1664264120
Build ID: 2cc229fa40c8781cf10c5d5f692e5bfba709fa64
Script:
# functions
exit:unknown ()
{ %{ /* unprivileged */
   atomic_set (session_state(), STAP_SESSION_STOPPING);
   _stp_exit ();
%} }
print_ubacktrace:unknown ()
{ %{ /* pragma:unwind */ /* pragma:symbols */
/* myproc-unprivileged */ /* pragma:uprobes */ /* pragma:vma */
    _stp_stack_user_print(CONTEXT, _STP_SYM_FULL);
%} }
# probes
process("/usr/lib64/libsystemd.so.0.23.0").function("sd_bus_open_system@../src/libsystemd/sd-bus/sd-bus.c:1293") /* pc=.dynamic+0x52a50 */ /* <- process("/usr/lib64/libsystemd.so.0.23.0").function("sd_bus_open_system@../src/libsystemd/sd-bus/sd-bus.c:1293") */
{ {
print_ubacktrace();
exit();
} }


result:7edbe3badfbf0f26279da1ce235c1724_1672
~~~
Regards
Sam

Comment 4 William Cohen 2023-09-01 20:04:25 UTC
The build-id for that particular version of systemd-libs matches up with the 2cc229fa40c8781cf10c5d5f692e5bfba709fa64 reported as the hexstring_theory.  It appears that the checking might be getting some bogus build-id value for the hexstring_practice.  This results in the following printing the warning message:

https://sourceware.org/git/?p=systemtap.git;a=blob;f=runtime/sym.c;h=f5c52d6d4bb21b3d8ce9e8edcd348ac9596f643a;hb=HEAD#l729

However, I haven't been able to replicate the issue despite using identical kernel and systemd* rpms.  The only difference between the *_hash.log in the comment above and the one generated during the experiments are the lines with "*Timestap: *" and the "results:*" last line.

Comment 5 RHEL Program Management 2023-09-22 16:00:43 UTC
Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.

Comment 6 RHEL Program Management 2023-09-22 16:01:42 UTC
This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.


Note You need to log in before you can comment on or make changes to this bug.