RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 463225 - [LTC 6.0 FEAT] 200966:Per-thread debugging utility
Summary: [LTC 6.0 FEAT] 200966:Per-thread debugging utility
Keywords:
Status: CLOSED DEFERRED
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: systemtap
Version: 6.1
Hardware: All
OS: All
medium
medium
Target Milestone: alpha
: 6.1
Assignee: Frank Ch. Eigler
QA Contact: Petr Muller
URL:
Whiteboard:
Depends On:
Blocks: 538808 580566
TreeView+ depends on / blocked
 
Reported: 2008-09-22 16:50 UTC by IBM Bug Proxy
Modified: 2016-09-20 02:04 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Clone Of:
Environment:
Last Closed: 2011-02-11 22:32:04 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description IBM Bug Proxy 2008-09-22 16:50:54 UTC
=Comment: #0=================================================
Emily J. Ratliff <emilyr.com> - 2008-09-16 18:24 EDT
1. Feature Overview:
Feature Id:	[200966]
a. Name of Feature:	Per-thread debugging utility
b. Feature Description
The endeavor is to provide a tool that enables users to track the activities of a single user
thread. These activities include (but are not limited to) system call patterns, signals
sent/received, threads spawned, etc.

2. Feature Details:
Sponsor:	Linux Systems Tech
Architectures:
x86
x86_64
ppc64

Arch Specificity: Purely Common Code
Affects Core Kernel: Yes
Affects Kernel Modules: Yes
Delivery Mechanism: 
Category:	RAS
Request Type:	Package - Feature from IBM
d. Upstream Acceptance:	In Progress
Sponsor Priority	2
f. Severity: Medium
IBM Confidential:	no
Code Contribution:	IBM code
g. Component Version Target:	Upstream Kernel 2.6.27

3. Business Case
As DB/2 on Linux moves to adopting the threaded model for their next release, the DB/2 development
and service engineers want a tool that'd enable them to trace just at a thread level. They'd not
want other threads to pay a penalty when its certain to them what thread they'd  want to target
their debugging for. ptrace, strace, and other tools fall  short when it comes to debugging in
multithreaded scenarios. We need something new here, something we could easily build on the utrace
infrastructure.

Benefits
Value for SUSE customers:  SUSE customers who've been using tools such as ptrace, strace, etc.,
would find this tool increadibly useful for their debugging purposes. SUSE customers who run IBM SWG
products such as DB/2 will potentially have faster turnaround times for their support issues
involving complex  multithreaded applications.

4. Primary contact at Red Hat: 
John Jarvis
jjarvis

5. Primary contacts at Partner:
Project Management Contact:
Mounir Bsaibes, bsaibes.com, 512-838-1301

Technical contact(s):
Maneesh Soni, smaneesh.com
Ananth N Mavinakayanahall, mananth.com

IBM Manager:
Naren A Devaiah, naren.devaiah.com

Comment 1 Frank Ch. Eigler 2008-09-22 17:09:05 UTC
Can you specify how the target subset of threads may be identified by
the instrumentation?

Comment 2 IBM Bug Proxy 2008-09-25 12:30:36 UTC
(In reply to comment #4)
> ------- Comment From fche 2008-09-22 13:09:05 EDT-------
> Can you specify how the target subset of threads may be identified by
> the instrumentation?

The user specifies the thread-id of interest.
Since utrace provides a per thread engine with callbacks to syscalls, signals,
there would be minimal penalty on the other threads.
What is targeted is a strace like utility (or a systemtap script) that builds on
utrace implementation to list systemcalls, signals.

Comment 3 Frank Ch. Eigler 2008-09-25 13:28:38 UTC
OK, it sounds like all you want is that the syntax

  probe process(PID).* { }

allow PIDs to match kernel task->pid (thread-id) not just task->tgid (process-id).
It would not be hard to have some syntax for that:

  probe thread(TID).{begin,end,syscall,...} { }

though uprobes style probes like .function/.statement are per-process,
so would have to be emulated per-thread if they were at all desirable.

Comment 4 IBM Bug Proxy 2008-09-25 14:40:59 UTC
(In reply to comment #6)
> ------- Comment From fche 2008-09-25 09:28:38 EDT-------
> OK, it sounds like all you want is that the syntax
>
> probe process(PID).* { }
>
> allow PIDs to match kernel task->pid (thread-id) not just task->tgid (process-id).
> It would not be hard to have some syntax for that:
>
> probe thread(TID).{begin,end,syscall,...} { }
>
> though uprobes style probes like .function/.statement are per-process,
> so would have to be emulated per-thread if they were at all desirable.

yes the current requirement is for utrace style probes.
uprobes style probes are not a requirement but surely desirable to probe allow
function entry/exit
(more like per-thread-ltrace) but I feel statement probes would make it more heavy.

Comment 5 Frank Ch. Eigler 2009-09-21 18:19:24 UTC
No patch for this feature yet, design also incomplete.

Comment 6 IBM Bug Proxy 2009-10-01 08:20:53 UTC
------- Comment From srikar.ibm.com 2009-10-01 04:10 EDT-------
(In reply to comment #9)
> No patch for this feature yet, design also incomplete.
>

Some of the features requested by this bug are already implemented in utrace.
Here is the list of requests
1. Request for a thread to be quiesced without having to use SIGNALS.

(We already have support for this in utrace. However there is no easy way for current user space debuggers to use this feature. This could be addresssed if and when utrace has its own user space API)

2. Ability to trace just one thread of a multithreaded process for activity so that there is no penalty for other threads.
(We already have support for this in utrace/uprobes. Though SystemTap is able to exploit utrace/uprobes, It might not be complete with respect to no penalty on other threads. Also there could be user space debuggers who are  interested in exploiting this feature. This could also be addresssed if and when utrace has its own user space API).

As noted earlier, the request is for system call tracing per thread for process. However providing library function tracing per thread of a process would be great.

3. Reliably distinguish between different reasons for SIGTRAP.
(We already have support for this in utrace thro report_signal handler())

Reliable hardware-breakpoints was also part of the thread level debugging requests. However I think its been tracked by a different bug and also there is already considerable work done in this area, This is up for discussion in LKML by K Prasad thro this patches.

Comment 7 Frank Ch. Eigler 2010-01-22 21:22:00 UTC
We have been unable to build this in time for rhel6.0.

http://sourceware.org/bugzilla/show_bug.cgi?id=11212 now tracks this.

Comment 8 John Jarvis 2010-01-25 14:13:10 UTC
Removing from RHEL 6.0 tracker, adding to 6.1 tracker.

Comment 9 Frank Ch. Eigler 2011-01-28 22:27:00 UTC
Unfortunately, the systemtap PR11212 aspect has slipped through the cracks;
perhaps next version.

If there is still as sense that a non-systemtap solution to the above
issues is a priority, we should raise that in other longer-term fora.
How much interest is there at IBM to help define interfaces and build tools?

Comment 10 John Jarvis 2011-02-11 22:32:04 UTC
Closing as DEFERRED since this is not in plan for RHEL 6.1.  Please feel free to reopen when the 6.2 feature queue is opened.


Note You need to log in before you can comment on or make changes to this bug.