Bug 643109

stap needs to know which process you want the ubacktrace for.
Try something like the following to tell stap that it needs the unwind info for bash and all shared libraries.

$ stap -d /bin/bash --ldd -e 'probe syscall.kill { print_ubacktrace () }'
$ kill -0 $$

 0x39b416 : __kernel_vsyscall+0x2/0x0 [vdso]
 0xbbbfe6 : kill+0x16/0x40 [libc-2.12.1.so]
 0x8080f7f : kill_pid+0x2f/0x260 [bash]
 0x80b3788 : kill_builtin+0x188/0x420 [bash]
 0x8070bc1 : execute_builtin+0x81/0x310 [bash]
 0x8072dfb : execute_simple_command+0xc7b/0xfc0 [bash]
 0x8073a22 : execute_command_internal+0x8e2/0x15f0 [bash]
 0x8074794 : execute_command+0x64/0xd0 [bash]
 0x80607d7 : reader_loop+0x97/0x2e0 [bash]
 0x805fefc : main+0xe1c/0x13e0 [bash]

Created attachment 453571 [details]
doc patch to suggest -d/--ldd at print_ubacktrace().

OK, thanks.  Still getting
# stap -d /bin/bash --ldd -e 'probe syscall.kill { print_ubacktrace () }'
WARNING: Couldn't register module '/lib64/ld-2.12.1.so' for pid 29040
WARNING: Couldn't register module '/lib64/libc-2.12.1.so' for pid 29040
WARNING: Couldn't register module '/lib64/libdl-2.12.1.so' for pid 29040
WARNING: Couldn't register module '/lib64/libtinfo.so.5.7' for pid 29040
WARNING: Couldn't register module '/lib64/ld-2.12.1.so' for pid 14435
WARNING: Couldn't register module '/lib64/libc-2.12.1.so' for pid 14435
WARNING: Couldn't register module '/lib64/libdl-2.12.1.so' for pid 14435
[...]
0x7fe8c7369c87
WARNING: Couldn't register module '/lib64/libdl-2.12.1.so' for pid 16967

I'll look into updating/clarifying the documentation. When hitting normal user space probe points (process("/bin/bash") for example) stap already knows it needs to collect the unwind data.

If one of those pids in those WARNINGS is equal to $$ then that would indeed be a problem. The WARNING doesn't say, but I assume it is because the vma table space is out of memory. Could you try with something like:

-DTASK_FINDER_VMA_ENTRY_ITEMS=4096

Jan, you may be hitting resource limits, as your last invocation of stap
says to probe the entire system (i.e., every process that links in any of
those basic libraries).

Either rerun stap with a -c/-x type process hierarchy filter, or consider
rerunning stap with a bigger -DMAXUPROBES=NNNN and/or -DTASK_FINDER_VMA_ENTRY_ITEMS=MMMM values.  Try MMMM=2000, and double it
for each retried failure.

(Mark, that latter parameter is not in the stap man page, but should be.)

-DTASK_FINDER_VMA_ENTRY_ITEMS=2000 works but it still gives WARNINGs, 4096 is OK.

My box is now running 449 tasks.  With 6GB RAM I do not mind if Systemtap takes for example 1GB.  Could be -DTASK_FINDER_VMA_ENTRY_ITEMS autodetected more appropriately for the system size?  Or at least that WARNING message should suggest -DTASK_FINDER_VMA_ENTRY_ITEMS.

In KVM (F14.x86_64) no -DTASK_FINDER_VMA_ENTRY_ITEMS is needed (114 tasks).

But it works great then, thanks.

I added the documentation suggestions and the warning message now mentions -DTASK_FINDER_VMA_ENTRY_ITEMS if the registration was because the limit was too low to upstream git.

Making the TASK_FINDER_VMA_ENTRY_ITEMS dynamically allocated is upstream bug http://sourceware.org/bugzilla/show_bug.cgi?id=11671