Bug 463333

Putting in NEEDINFO for upstream code posting/acceptance.

(In reply to comment #4)
> ------- Comment From notting 2008-10-02 12:08:12 EDT-------
> Putting in NEEDINFO for upstream code posting/acceptance.
>
The upstream post to Performance Inspector was:
http://sourceforge.net/mailarchive/forum.php?thread_name=483E1962.8010002%40us.ibm.com&forum_name=perfinsp-list

The Fedora bugzilla  for the review request of the stapitrace package was:
https://bugzilla.redhat.com/show_bug.cgi?id=445224#add_comment

and a pointer to the Fedora cvs tree is:
http://cvs.fedora.redhat.com/viewvc/devel/stapitrace


The stapitrace RPM contains some SystemTap tapscripts/runtime support for instruction tracing.  All of these scripts are hosted by Performance Inspector on a temporary basis until their counterparts can be accepted into SystemTap.  So far, the basic instruction tracing pieces have been committed into the SystemTap git tree, but there remain a few other tapscripts that are specific to the capabilities specific to the goal of eventually replacing PI ITRACE functionality.

Eventually we would like to remove the tapscripts from stapitrace that are duplicated in SystemTap.

Can you tell me what version of CVS snapshot date will be included in RHEL6?

It's not a particular CVS revision, per se.

In the October 2008 comment by Dave Nomura, he mentioned that "Eventually we would like to remove the tapscripts from stapitrace that are duplicated in SystemTap".  I have updated the stapitrace package (hosted by the Performance Inspector project) to do just that.  So now stapitrace uses the instruction tracing support built into SystemTap's itrace probe point.  I had to make changes to both SystemTap and Performance Inspector code.  Below are the relevant URLs to show that the code is upstream.

Systemtap:
http://sources.redhat.com/git/gitweb.cgi?p=systemtap.git;a=commit;h=1324cc8267f44de856daa6aedd76d157e4259d28

Performance Inspector:
https://sourceforge.net/mailarchive/forum.php?thread_name=OF14824B21.0C88867C-ON8725756E.007DF053-8625756E.007E121F%40us.ibm.com&forum_name=perfinsp-list

As already mentioned by Dave, the original version of the stapitrace package is in Fedora 10.  If the SystemTap changes I just committed are pulled into Fedora 11, I intend to update the stapitrace package to the new code I committed to Performance Inspector.  That will be the basis of the RHEL 6 pacakge.

Systemtap's next release will include the itrace support
(= utrace-managed single-stepping of user threads).
However, this still needs a mild backport to the older RHEL5
utrace generation.

I've finished the backport to RHEL5, but all the changes didn't make the 0.9.7 release.  Commit 975a582 will need to be added to 0.9.7.  The RHEL5 problems are also being tracked in <http://sources.redhat.com/bugzilla/show_bug.cgi?id=10091>

Maynard, can you comment as to the current status of stapitrace
in fedora?  Reviews?  Portability?  Testing?  Kernel dependencies if any?

------- Comment From mpjohn.com 2009-06-29 19:01 EDT-------
(In reply to comment #11)
> Maynard, can you comment as to the current status of stapitrace
> in fedora?  Reviews?  Portability?  Testing?  Kernel dependencies if any?
The stapitrace package is current and up-to-date in Fedora 11, and I've tested it against systemtap 0.9.8.1 package.  The testing consists of using the 'itrace' program that's included with the package, passing it a program name and function name to trace, and specifying the option to output qtrace format data.  We then feed that qtrace data into a simulator and manually verify the output is as expected.

Other than the initial feature submission review, there have been no user reviews that I'm aware of (if that's what you mean).

As for portability, stapitrace currently only supports ppc64 and the package is only targeted for Fedora 10+ and RHEL 6.  Other platforms may be supported in the future.

No explicit kernel dependency is identified since our dependency is on systemtap with uprobes.

P.S. Frank, as you may recall, my recent stapitrace testing on Fedora 11 was resulting in some ugly error messages, even though the stapitrace program finished normally and gave expected results.  The error messages were:
================================
Unable to find certificate with nickname stap-server in /home/ppcteam/.systemtap
/ssl/server.
Unknown NSS error: -5950.
Copy failed ("/tmp/staprcUQdd/stap_9f414ca0e37a4c5ee8804a2b9ce19985_46713.ko.sgn
" to "/home/ppcteam/.systemtap/cache/9f/stap_9f414ca0e37a4c5ee8804a2b9ce19985_46
713.ko.sgn"): No such file or directory
Unable to obtain information on the signature file /tmp/staprcUQdd/stap_9f414ca0
e37a4c5ee8804a2b9ce19985_46713.ko.sgn.
Unknown NSS error: -5950.
================================

Discussion about this on the systemtap mailing list has stalled.  I've not been able to figure out how to eliminate the noise.  Any suggestions would be greatly appreciated.

Thanks.
>

------- Comment From mpjohn.com 2009-09-08 13:07 EDT-------
I don't yet see the stapitrace package in the RHEL 6 alpha ISOs.  Is this still a work in progress to pull that package into RHEL 6?  Thanks.

Changing summary line to clarify situation of what's being
contemplated: the inclusion of IBM's "stapitrace" package in
rhel6, which is in koji.fedora but not in base fedora-11 or -12
compositions.

http://koji.fedoraproject.org/koji/packageinfo?packageID=6494

I took a look at the srpm for itrace. I have some questions and comments about the current srpm.

Ran rpmlint on the srpm:

$ rpmlint /tmp/stapitrace-2.0.0-0.20090304cvs_alpha.fc12.1.src.rpm 
stapitrace.src: W: strange-permission stapitrace.spec 0755
stapitrace.src:64: E: hardcoded-library-path in $RPM_BUILD_ROOT/usr/lib
stapitrace.src: W: mixed-use-of-spaces-and-tabs (spaces: line 14, tab: line 6)
1 packages and 0 specfiles checked; 1 errors, 2 warnings.

The stapitrace.spec file permissions and the mixed-use-of-spaces-and-tabs (spaces used instead of tabs for Patch[01234]: lines") are trivial to fix.

What is the reason for forcing building 32-bit version of the software for ppc64? Having two different version of the 32-bit libraries in /usr/lib may cause problem (build-id will differ).

Is the  "-mminimal-toc" required to build the package? It would make things more portable not using ppc-only compiler options.

Software included in Red Hat Enterprise Linux (and Fedora) should be
portable between different architectures and minimize the attachment
to a particular hardware implementation. This source rpm for
stapitrace is very closely tied to ppc64; it only builds on
ppc64. Some quick hacks were made allow reviewing the code on x86_64
machine (allow generation of the stp tapset files). The patch and
changes to the spec file to get srpm to build on x86_64 architectures
so the generated .stp tapset files could be examined. The
modifications are attached. However, these are not expected to
function and additional work is required to erradicate the ppc64-ism
in the src/stap files.


The basic concept is that itrace generates a small systemtap script
that attaches to the appropriate process or executable and log a trace
through the code. The generated scripts uses process("*").insn or
pid("*").insn probe. It should be using the process("*").insn.block or
pid("*").insn probe.block instead. This works for x86 on Fedora and
should work on RHEL6. However, for insn.block to work on ppc64 need to
get the patches mentioned in the following URL in the utrace-devel
mailing list:

http://www.mail-archive.com/utrace-devel@redhat.com/msg00771.html

The pi_itrace.stp and pi_ppc_itrace.stp are generated during the
rpmbuild. The mechanism seems overly complicated to pull in bits and
pieces of other files for [DEFS] and [BODY]. Is a special special
program really needed for that? 

Having a hard coded limit of MAX_CPU of 32 might be low with upcoming
multicore systems (e.g.Nehalem-EX, 4 sockets with 8 core cores with
hyper threading (4*8*2=64 processors)). Is there some way to avoid
having this hard coded limit?

Rather than really using SystemTap mechanisms instrumentation to
collect data, SystemTap is being used more as patching system to pull
in the pieces from the existing performance inspector ppc64 driver as
guru mode code. There are large sections of code being copied over
from the ppc64 driver and the include files. This limits the ability
of systemtap to check and ensure that instrumentation is safe.  For
example task_lock() and task_unlock() (spinlocks) are used within the
SystemTap function pi_itrace_create_task().

Created attachment 360416 [details]
Patch to allow building code NON-working code on non-ppc64 machines

The patch just allows one to generate the *.stp files on non-ppc64 machine to allow analysis of the code. This is NOT intented to produce working code for x86 machines.

Created attachment 360417 [details]
Minimal spec file changes to allow building code on other non-ppc64 machines

Modified spec file to allow generation of .stp files on non-ppc64 machine for code review.

------- Comment From mpjohn.com 2009-09-10 15:19 EDT-------
(In reply to comment #16)
> Software included in Red Hat Enterprise Linux (and Fedora) should be
> portable between different architectures and minimize the attachment
> to a particular hardware implementation. This source rpm for
> stapitrace is very closely tied to ppc64; it only builds on
> ppc64. Some quick hacks were made allow reviewing the code on x86_64
> machine (allow generation of the stp tapset files). The patch and
> changes to the spec file to get srpm to build on x86_64 architectures
> so the generated .stp tapset files could be examined. The
> modifications are attached. However, these are not expected to
> function and additional work is required to erradicate the ppc64-ism
> in the src/stap files.
Will, I successfully tested the patch and the modified spec file on Fedora11, and I will test on RHEL6 alpha as soon as I can.  The reason this package is so ppc64-centric is because it is designed exclusively to collect instruction trace data that can be fed into a pipeline analyzer for the ppc64 processor line.  (NOTE:  The user must obtain this pipeline analyzer program from IBM's AlphaWorks website.)  The use of these tools helps performance analysts with deep-dive analysis to identify instruction sequences that cause pipeline stalls on the POWER processor line.
>
> The basic concept is that itrace generates a small systemtap script
> that attaches to the appropriate process or executable and log a trace
> through the code. The generated scripts uses process("*").insn or
> pid("*").insn probe. It should be using the process("*").insn.block or
> pid("*").insn probe.block instead.
The intent of this tool is to only do single-step instruction tracing, not block tracing.  Block trace data would not be of any benefit to feed into the pipeline analyzer.
> This works for x86 on Fedora and
> should work on RHEL6. However, for insn.block to work on ppc64 need to
> get the patches mentioned in the following URL in the utrace-devel
> mailing list:
>
> http://www.mail-archive.com/utrace-devel@redhat.com/msg00771.html
True, but this is a separate issue and not required for the functionality of this package.
>
> The pi_itrace.stp and pi_ppc_itrace.stp are generated during the
> rpmbuild. The mechanism seems overly complicated to pull in bits and
> pieces of other files for [DEFS] and [BODY]. Is a special special
> program really needed for that?
The Performance Inspector project pre-dates the stapitrace package by several years.  With that project, there's a tool called "ITrace" that includes a separate kernel module for controlling the instruction trace, managing certain quirks on the POWER processors, collecting the necessary instruction trace data, and writing the trace file.  In fact, we had tried to have that ITrace tool pulled into RHEL 4 (or was it 5 -- not sure), but we were told we should work with the SystemTap community to integrate instruction tracing.  We were successful in contributing base instruction tracing function into SystemTap, but this alone was not sufficient for our needs with the pipeline analysis tool.  Since we didn't think it would be appropriate to try to shove so much ppc64-specific code into SystemTap, so we took the route of leveraging the basic SystemTap instruction probe point to insert the ppc64-specific code that already exists in the Performance Inspector ITrace tool.  Merging code from the Performance Inspector source, using the [DEFS] and [BODY] tags is a technique to avoid duplication of code.  I'll grant you, it's not simple, but it beats dual maintenance.
>
> Having a hard coded limit of MAX_CPU of 32 might be low with upcoming
> multicore systems (e.g.Nehalem-EX, 4 sockets with 8 core cores with
> hyper threading (4*8*2=64 processors)). Is there some way to avoid
> having this hard coded limit?
I'll look into this.
>
> Rather than really using SystemTap mechanisms instrumentation to
> collect data, SystemTap is being used more as patching system to pull
> in the pieces from the existing performance inspector ppc64 driver as
> guru mode code. There are large sections of code being copied over
> from the ppc64 driver and the include files. This limits the ability
> of systemtap to check and ensure that instrumentation is safe.  For
> example task_lock() and task_unlock() (spinlocks) are used within the
> SystemTap function pi_itrace_create_task().
Yes, this is true -- but unavoidable, I think, given the detailed data collection that is needed to identify and eliminate pipeline hazards.
>

> The reason this package is so ppc64-centric is because it is designed
> exclusively to collect instruction trace data that can be fed into a pipeline 
> analyzer for the ppc64 processor line.  (NOTE:  The user must obtain this 
> pipeline analyzer program from IBM's AlphaWorks website.)

Do I understand correctly that this software is not useful
without a (closed-source?) package one must separately
download?

Can someone offer a rationale for having Red Hat maintain this
as a part of RHEL6?

------- Comment From mpjohn.com 2009-09-10 15:49 EDT-------
(In reply to comment #15)
> I took a look at the srpm for itrace. I have some questions and comments about
> the current srpm.
>
> Ran rpmlint on the srpm:
>
> $ rpmlint /tmp/stapitrace-2.0.0-0.20090304cvs_alpha.fc12.1.src.rpm
> stapitrace.src: W: strange-permission stapitrace.spec 0755
> stapitrace.src:64: E: hardcoded-library-path in $RPM_BUILD_ROOT/usr/lib
> stapitrace.src: W: mixed-use-of-spaces-and-tabs (spaces: line 14, tab: line 6)
> 1 packages and 0 specfiles checked; 1 errors, 2 warnings.
>
> The stapitrace.spec file permissions and the mixed-use-of-spaces-and-tabs
> (spaces used instead of tabs for Patch[01234]: lines") are trivial to fix.
Thanks.  I'm not very experienced with rpm development, so didn't even know about rpmlint.
>
> What is the reason for forcing building 32-bit version of the software for
> ppc64? Having two different version of the 32-bit libraries in /usr/lib may
> cause problem (build-id will differ).
In fact, there are no libraries that are included with this tool, so there's absolutely no reason to build anything other than the default 64-bit version.  Do you see something about building 32-bit that I'm missing?  Perhaps you're referring to the fact that there are both versions on Fedora.  This wasn't really by my choice.  I think the build infrastructure must automatically try to build every package both ways on bi-arch systems.

>
> Is the  "-mminimal-toc" required to build the package? It would make things
> more portable not using ppc-only compiler options.
It's not needed.  As I stated in my previous comment, I tested your updated spec file (which removes this compilation flag), and it worked fine.

Thanks for your support.
-Maynard
>

------- Comment From mpjohn.com 2009-09-10 16:57 EDT-------
(In reply to comment #20)
> > The reason this package is so ppc64-centric is because it is designed
> > exclusively to collect instruction trace data that can be fed into a pipeline
> > analyzer for the ppc64 processor line.  (NOTE:  The user must obtain this
> > pipeline analyzer program from IBM's AlphaWorks website.)
>
> Do I understand correctly that this software is not useful
> without a (closed-source?) package one must separately
> download?
At the present time, this is true, but we are working towards the goal of making an open source pipeline analyzer available, possibly as an Eclipse plug-in.  Having the stapitrace package available as a RHEL 6 RPM makes setup easier for users, whether they're downloading a closed source analysis tool or an OSS Eclipse plug-in.
>
> Can someone offer a rationale for having Red Hat maintain this
> as a part of RHEL6?
>

I believe it would be most appropriate to defer this until a
later point, when an open-source data consumer tool is sufficiently
mature and of general enough utility to be included in Fedora & RHEL.

Let's close for now, but we can reconsider in the future, as stapitrace develops.

Development Management has reviewed and declined this request.  You may appeal
this decision by reopening this request.

By the way, there still exists some problems in the
systemtap-side itrace prerequisite; see
<http://sourceware.org/bugzilla/show_bug.cgi?id=10216>.

------- Comment From mpjohn.com 2009-11-09 15:20 EDT-------
(In reply to comment #25)
> By the way, there still exists some problems in the
> systemtap-side itrace prerequisite; see
> <http://sourceware.org/bugzilla/show_bug.cgi?id=10216>.
Sorry for the delay in answering . . . I was out of the office for a week.  The bug you point to seems to be a problem with using the SystemTap instruction trace probe on an i586 platform.  I wasn't aware of this bug.  Nevertheless, since stapitrace currently only supports the ppc64 platform, the i586 problem is not an impediment to including this package in RHEL 6.
>

------- Comment From hohnbaum.com 2009-11-24 15:00 EDT-------
Moving bug state to rejected based on Red Hat's response.