Description of problem:
SystemTap reports "semantic error: no match" on libjvm markers when the handler is using any $var access, even though the markers are found without $var use.

This has only been seen as an issue on i686; x86_64 seems fine.

Version-Release number of selected component (if applicable):
systemtap-1.8-1.fc17.i686
elfutils-0.154-1.fc17.i686
java-1.7.0-openjdk-1.7.0.3-2.2.1.fc17.8.i686

How reproducible:
100%

Steps to Reproduce:
1. stap -e 'probe process(@1).mark("*") { println($foo) }' -p2 $JVM
(The fact that $foo is bad here is irrelevant, just me being lazy.  The result should look more like the error x86_64 prints in "Expected results".)
  
Actual results:
> $ JVM=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.3/jre/lib/i386/server/libjvm.so
> $ stap -e 'probe process(@1).mark("*") { println($foo) }' -p2 $JVM
> semantic error: while resolving probe point: identifier 'process' at <input>:1:7
>         source: probe process(@1).mark("*") { println($foo) }
>                       ^
> 
> semantic error: no match
> Pass 2: analysis failed.  Try again with another '--vp 01' option.


Expected results:
> $ JVM=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.3.x86_64/jre/lib/amd64/server/libjvm.so
> $ stap -e 'probe process(@1).mark("*") { println($foo) }' -p2 $JVM
> semantic error: unable to find local 'foo' near pc 0x37342d in <unknown> /usr/src/debug/java-1.7.0-openjdk/openjdk/hotspot/src/share/vm/services/classLoadingService.cpp ( (alternatives: $data $len $name): identifier '$foo' at <input>:1:39
>         source: probe process(@1).mark("*") { println($foo) }
>                                                       ^
> 
> Pass 2: analysis failed.  Try again with another '--vp 01' option.

It's still an error, because I used invalid $foo instead of bothering to find a common $var across all markers, or to isolate a single marker.  But it's the *right* error.


Additional info:

I've started this bug against systemtap, but only because I don't know the root cause yet.  There seems to be a real problem in the debuginfo that leads us into this corner.

Certainly systemtap is wrong in the way it reacts to this failure, with a completely misleading "not found" error message.  I'll file a bug upstream to address that on its own.

When we locate an SDT marker, stap takes different paths depending on the presence of $vars in the handler.  Without any $var, we just build the probe directly, and life is good.  With a $var, we decide we need debuginfo, and so that goes down the query_addr path, until we reach a failure at calling elfutils dwfl_module_addrdie().  The given address is within the range of the queried module.

Then mjw asked me if elfutils is lying, i.e. is there a DIE/CU that ought to be returned here?  And now I find that something is really suspicious with the debuginfo.  The first clue is simply in size:

> $ ls -sh /usr/lib/debug/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.3/jre/lib/i386/server/libjvm.so.debug
> 32M /usr/lib/debug/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.3/jre/lib/i386/server/libjvm.so.debug
vs.
> $ ls -sh /usr/lib/debug/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.3.x86_64/jre/lib/amd64/server/libjvm.so.debug
> 200M /usr/lib/debug/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.3.x86_64/jre/lib/amd64/server/libjvm.so.debug

And looking at those with eu-readelf shows only *three* CUs in i386, vs. 555 CUs in amd64.  The only three survivors in i386 are:
> /usr/src/debug/java-1.7.0-openjdk/openjdk/hotspot/src/share/vm/memory/compactingPermGenGen.cpp
> /usr/src/debug/java-1.7.0-openjdk/openjdk/hotspot/src/share/vm/runtime/sharedRuntimeTrans.cpp
> /usr/src/debug/java-1.7.0-openjdk/openjdk/hotspot/src/share/vm/runtime/sharedRuntimeTrig.cpp

Surely there's more to libjvm.so than that! ;)  But whether this is an issue in compiler generation or debuginfo stripping or something else, I don't know...