Bug 463225

Summary: [LTC 6.0 FEAT] 200966:Per-thread debugging utility
Product: Red Hat Enterprise Linux 6 Reporter: IBM Bug Proxy <bugproxy>
Component: systemtapAssignee: Frank Ch. Eigler <fche>
Status: CLOSED DEFERRED QA Contact: Petr Muller <pmuller>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.1CC: dsmith, ejratl, jjarvis, mjw, ohudlick, woodard
Target Milestone: alphaKeywords: FutureFeature
Target Release: 6.1   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-02-11 22:32:04 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 538808, 580566    

Description IBM Bug Proxy 2008-09-22 16:50:54 UTC
=Comment: #0=================================================
Emily J. Ratliff <emilyr.com> - 2008-09-16 18:24 EDT
1. Feature Overview:
Feature Id:	[200966]
a. Name of Feature:	Per-thread debugging utility
b. Feature Description
The endeavor is to provide a tool that enables users to track the activities of a single user
thread. These activities include (but are not limited to) system call patterns, signals
sent/received, threads spawned, etc.

2. Feature Details:
Sponsor:	Linux Systems Tech
Architectures:
x86
x86_64
ppc64

Arch Specificity: Purely Common Code
Affects Core Kernel: Yes
Affects Kernel Modules: Yes
Delivery Mechanism: 
Category:	RAS
Request Type:	Package - Feature from IBM
d. Upstream Acceptance:	In Progress
Sponsor Priority	2
f. Severity: Medium
IBM Confidential:	no
Code Contribution:	IBM code
g. Component Version Target:	Upstream Kernel 2.6.27

3. Business Case
As DB/2 on Linux moves to adopting the threaded model for their next release, the DB/2 development
and service engineers want a tool that'd enable them to trace just at a thread level. They'd not
want other threads to pay a penalty when its certain to them what thread they'd  want to target
their debugging for. ptrace, strace, and other tools fall  short when it comes to debugging in
multithreaded scenarios. We need something new here, something we could easily build on the utrace
infrastructure.

Benefits
Value for SUSE customers:  SUSE customers who've been using tools such as ptrace, strace, etc.,
would find this tool increadibly useful for their debugging purposes. SUSE customers who run IBM SWG
products such as DB/2 will potentially have faster turnaround times for their support issues
involving complex  multithreaded applications.

4. Primary contact at Red Hat: 
John Jarvis
jjarvis

5. Primary contacts at Partner:
Project Management Contact:
Mounir Bsaibes, bsaibes.com, 512-838-1301

Technical contact(s):
Maneesh Soni, smaneesh.com
Ananth N Mavinakayanahall, mananth.com

IBM Manager:
Naren A Devaiah, naren.devaiah.com

Comment 1 Frank Ch. Eigler 2008-09-22 17:09:05 UTC
Can you specify how the target subset of threads may be identified by
the instrumentation?

Comment 2 IBM Bug Proxy 2008-09-25 12:30:36 UTC
(In reply to comment #4)
> ------- Comment From fche 2008-09-22 13:09:05 EDT-------
> Can you specify how the target subset of threads may be identified by
> the instrumentation?

The user specifies the thread-id of interest.
Since utrace provides a per thread engine with callbacks to syscalls, signals,
there would be minimal penalty on the other threads.
What is targeted is a strace like utility (or a systemtap script) that builds on
utrace implementation to list systemcalls, signals.

Comment 3 Frank Ch. Eigler 2008-09-25 13:28:38 UTC
OK, it sounds like all you want is that the syntax

  probe process(PID).* { }

allow PIDs to match kernel task->pid (thread-id) not just task->tgid (process-id).
It would not be hard to have some syntax for that:

  probe thread(TID).{begin,end,syscall,...} { }

though uprobes style probes like .function/.statement are per-process,
so would have to be emulated per-thread if they were at all desirable.

Comment 4 IBM Bug Proxy 2008-09-25 14:40:59 UTC
(In reply to comment #6)
> ------- Comment From fche 2008-09-25 09:28:38 EDT-------
> OK, it sounds like all you want is that the syntax
>
> probe process(PID).* { }
>
> allow PIDs to match kernel task->pid (thread-id) not just task->tgid (process-id).
> It would not be hard to have some syntax for that:
>
> probe thread(TID).{begin,end,syscall,...} { }
>
> though uprobes style probes like .function/.statement are per-process,
> so would have to be emulated per-thread if they were at all desirable.

yes the current requirement is for utrace style probes.
uprobes style probes are not a requirement but surely desirable to probe allow
function entry/exit
(more like per-thread-ltrace) but I feel statement probes would make it more heavy.

Comment 5 Frank Ch. Eigler 2009-09-21 18:19:24 UTC
No patch for this feature yet, design also incomplete.

Comment 6 IBM Bug Proxy 2009-10-01 08:20:53 UTC
------- Comment From srikar.ibm.com 2009-10-01 04:10 EDT-------
(In reply to comment #9)
> No patch for this feature yet, design also incomplete.
>

Some of the features requested by this bug are already implemented in utrace.
Here is the list of requests
1. Request for a thread to be quiesced without having to use SIGNALS.

(We already have support for this in utrace. However there is no easy way for current user space debuggers to use this feature. This could be addresssed if and when utrace has its own user space API)

2. Ability to trace just one thread of a multithreaded process for activity so that there is no penalty for other threads.
(We already have support for this in utrace/uprobes. Though SystemTap is able to exploit utrace/uprobes, It might not be complete with respect to no penalty on other threads. Also there could be user space debuggers who are  interested in exploiting this feature. This could also be addresssed if and when utrace has its own user space API).

As noted earlier, the request is for system call tracing per thread for process. However providing library function tracing per thread of a process would be great.

3. Reliably distinguish between different reasons for SIGTRAP.
(We already have support for this in utrace thro report_signal handler())

Reliable hardware-breakpoints was also part of the thread level debugging requests. However I think its been tracked by a different bug and also there is already considerable work done in this area, This is up for discussion in LKML by K Prasad thro this patches.

Comment 7 Frank Ch. Eigler 2010-01-22 21:22:00 UTC
We have been unable to build this in time for rhel6.0.

http://sourceware.org/bugzilla/show_bug.cgi?id=11212 now tracks this.

Comment 8 John Jarvis 2010-01-25 14:13:10 UTC
Removing from RHEL 6.0 tracker, adding to 6.1 tracker.

Comment 9 Frank Ch. Eigler 2011-01-28 22:27:00 UTC
Unfortunately, the systemtap PR11212 aspect has slipped through the cracks;
perhaps next version.

If there is still as sense that a non-systemtap solution to the above
issues is a priority, we should raise that in other longer-term fora.
How much interest is there at IBM to help define interfaces and build tools?

Comment 10 John Jarvis 2011-02-11 22:32:04 UTC
Closing as DEFERRED since this is not in plan for RHEL 6.1.  Please feel free to reopen when the 6.2 feature queue is opened.