Bug 1151196 - systemtap or glibc issue with context variables and a system process
Summary: systemtap or glibc issue with context variables and a system process
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Fedora
Classification: Fedora
Component: systemtap
Version: 20
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Frank Ch. Eigler
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: 1151226
TreeView+ depends on / blocked
 
Reported: 2014-10-09 18:34 UTC by Paulo Andrade
Modified: 2014-10-09 20:08 UTC (History)
9 users (show)

Fixed In Version:
Clone Of:
: 1151226 (view as bug list)
Environment:
Last Closed: 2014-10-09 19:44:01 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Paulo Andrade 2014-10-09 18:34:50 UTC
While experimenting with systemtap to write some experimental
leak check code, I could only get it to work to fetch some
variables when using this pattern:

probe process("/lib*/libc.so*").function("malloc") {
    if (pid() == target()) {
	printf("malloc(%s)", $$parms)
    }
}
probe process("/lib*/libc.so*").function("malloc").return {
    if (pid() == target()) {
	printf(" %s\n", $$return)
    }
}

that will output something like this:

malloc(bytes=0x6) return=0x7ff5380050d0

but frequently something like this:

malloc(bytes=0x7ff5787349c0) return=0x7ff5340bf480

that looks bogus, and probably had the argument,
in a register already overwritten.

If I try to use "context variables", e.g.
instead of
	printf("malloc(%s)", $$parms)
write
	printf("malloc(%d)", $bytes)
 it will just fail with a message like:

---%<---
Pass 1: parsed user script and 117 library script(s) using 225224virt/45440res/6136shr/40076data kb, in 190usr/20sys/205real ms.
semantic error: unresolved target-symbol expression: identifier '$bytes' at malloc.stp:3:23
        source:         printf("malloc(%d)", $bytes)
                                             ^

semantic error: unresolved target-symbol expression: identifier '$bytes' at :3:23
        source:         printf("malloc(%d)", $bytes)
---%<---

$ rpm -q glibc glibc-debuginfo glibc-debuginfo-common systemtap systemtap-devel 
glibc-2.18-16.fc20.x86_64
glibc-2.18-16.fc20.i686
glibc-debuginfo-2.18-16.fc20.x86_64
glibc-debuginfo-common-2.18-16.fc20.x86_64
systemtap-2.5-3.fc20.x86_64
systemtap-devel-2.5-3.fc20.x86_64

Comment 1 Lukas Berk 2014-10-09 18:50:55 UTC
Regarding the context variable issue.
Running 'stap -L 'process("/lib*/libc.so*").function("malloc")' returns

process("/usr/lib/libc-2.18.so").function("malloc")
process("/usr/lib64/libc-2.18.so").function("__libc_malloc@/usr/src/debug/glibc-2.18/malloc/malloc.c:2844") $bytes:size_t

on my system.  I believe this means the probe point you specified will try and probe both the 64 bit libc (for which you have debuginfo) and the 32 bit libc (which you don't have the debuginfo for, and subsequently can't probe for the context variables).

If you specify 'process("/usr/lib64/libc-2.18.so").function("malloc")' you should be able to print the $bytes context variable.

Comment 2 Josh Stone 2014-10-09 19:37:44 UTC
Lukas nailed the $bytes availability issue.  If you want to write a script which can work on either 64/32-bit (but not both since compat debuginfo doesn't install together), try alternates with '!' like:

probe process("/lib64/libc.so*").function("malloc")!,
      process("/lib/libc.so*").function("malloc")
{ /*...*/ }

You can also alias the first part:

probe libc = process("/lib64/libc.so*")!, process("/lib/libc.so*") {}
probe libc.function("malloc") { /*...*/ }
probe libc.function("malloc").return { /*...*/ }


Another tip is that stap can automatically save entry values, so you can write your script with one probe:

probe libc = process("/lib64/libc.so*")!, process("/lib/libc.so*") {}
probe libc.function("malloc").return {
    if (pid() == target()) {
	printf("malloc(%#x) %#x\n", $bytes, $return)
    }
}


As for the apparently-bad $bytes value, we're generally at the mercy of what gcc told us in debuginfo.  On my system it looks like:

 [189214]    subprogram
             external             (flag_present) Yes
             name                 (strp) "__libc_malloc"
             decl_file            (data1) 3
             decl_line            (data2) 2844
             linkage_name         (strp) "__GI___libc_malloc"
             prototyped           (flag_present) Yes
             type                 (ref4) [182c49]
             low_pc               (addr) +0x000000000007fca0
             high_pc              (data8) 244 (+0x000000000007fd94)
             frame_base           (exprloc) 
              [   0] call_frame_cfa
             sibling              (ref4) [18931e]
 [18923a]      formal_parameter
               name                 (strp) "bytes"
               decl_file            (data1) 3
               decl_line            (data2) 2844
               type                 (ref4) [182bce]
               location             (exprloc) 
                [   0] reg6

So the parameter is in DWARF reg6, which is x86_64 %rbp.  However, normal calling convention puts the first parameter in %rdi, and it's only moved to %rbp as part of the function prologue.

000000338927fca0 <__libc_malloc>:
  338927fca0:	55                   	push   %rbp
  338927fca1:	48 89 fd             	mov    %rdi,%rbp
  338927fca4:	53                   	push   %rbx
  338927fca5:	48 83 ec 08          	sub    $0x8,%rsp

Ideally, the debuginfo should be telling us to use %rdi from 7fca0..7fca4, and %rbp only from 7fca4 and on (or until that's invalidated).

I tried a similar "__libc_malloc" breakpoint in gdb, and it also prints the wrong value for "bytes" on function entry, but it's fine after stepping once.

Knowing all this, you can actually force stap into a prologue-searching mode with -P, and with that the data looks fine.

Comment 3 Paulo Andrade 2014-10-09 19:44:01 UTC
Many thanks for the explanations.

I am somewhat new to systemtap, but I am quite impressed with what
it can do :)

At first I changed to use /lib64/libc.so* instead of /lib*/libc.so*
and when using -P it appears to also always work as expected.


Note You need to log in before you can comment on or make changes to this bug.