Bug 442990 - the guide is on the old latency_tracer, we need to update to ftrace
Summary: the guide is on the old latency_tracer, we need to update to ftrace
Alias: None
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: Realtime_Tuning_Guide   
(Show other bugs)
Version: beta
Hardware: All
OS: Linux
Target Milestone: ---
: ---
Assignee: Lana Brindley
QA Contact: Jeff Needle
Keywords: Documentation
: 436862 (view as bug list)
Depends On:
TreeView+ depends on / blocked
Reported: 2008-04-18 02:58 UTC by Steven Rostedt
Modified: 2013-10-23 23:06 UTC (History)
3 users (show)

Fixed In Version: 1.0
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2008-05-01 00:01:49 UTC
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

Description Steven Rostedt 2008-04-18 02:58:07 UTC
The tuning guide talks about latency tracer but the latest RT kernel implements
ftrace. The ftrace utility has a different interface altogether.

1) it now resides in the debugfs file system:

# mkdir /debug
# mount -t debugfs nodev /debug

2) we can compile in different tracers. /debug/tracing/current_tracer holds the
tracing that happens at that time. "none" is special and disables all tracing.
What we currently have in -rt is:

events wakeup preemptirqsoff preemptoff irqsoff ftrace sched_switch none

events - CONFIG_EVENT_TRACER - traces various events

wakeup - CONFIG_SCHED_TRACER - traces the wakeup latency of the highest RT
priority task.

preemptoff - CONFIG_PREEMPT_TRACER - traces the longest time preemption is disabled.

irqsoff - CONFIG_IRQSOFF_TRACER -traces the longest time interrupts are disabled.

preemptirqsoff - CONFIG_PREEMPT_TRACER && CONFIG_IRQSOFF_TRACER - traces the
time either preemption or interrupts are disabled.

ftrace - CONFIG_FTRACE - traces functions using the mcount mechanism

sched_switch - CONFIG_CONTEXT_SWITCH_TRACER - traces tasks as they are switched
by the scheduler.

simply echoing in one of the above will set the current tracer.

e.g. # echo ftrace > /debug/tracing/current_tracer

To see which tracers have been configured into the kernel do a

  # cat /debug/tracing/available_tracers

(I think) Tracing is default on and can be disabled via:

 # echo 0 > /debug/tracing/tracing_enabled

and reenabled

 # echo 1 > /debug/tracing/tracing_enabled

To see the traces you can look at


which is pretty much the same format as the latency_trace output of the previous

or you can see a simpler format in /debug/tracing/trace

/debug/tracing/iter_ctrl provides various changes to the output.

 # cat /debug/tracing/iter_ctrl
 noprint-parent nosym-offset nosym-addr noverbose 

Here we see everything is disabled. To enable one of the above, simply echo the
same name without the prefix "no".

 # echo sys-offset > /debug/tracing/iter_ctrl
 # cat /debug/tracing/iter_ctrl 
 noprint-parent sym-offset nosym-addr noverbose 

print-parent will show the parent of functions
sym-offset will add the offset into the function -0xa4/0x254
sym-addr will add the address of a symbol 0xc1042856
verbose changes latency_trace output to be much more verbose.

For those that trace max latencies (wakeup, irqsoff, preemptoff preemptirqsoff)
the recorded max latency will be stored in (as well as the trace)


If dynamic ftrace is configured in, ftrace mcount utility will have virtually
zero overhead.  But an extra feature of this is that the ftrace utility can be
configured to which functions you want to trace!

/debug/tracing/available_filter_functions lists the functions that have been
recorded by ftrace and can be traced.

/debug/tracing/dyn_ftrace_total_info is simply a information of the number of
functions that have been recorded.

/debug/tracing/set_ftrace_filter allows one to echo in the functions listed in
available_filter_functions to tune the trace. Simple glob works when echoing in
filter functions. If this has no functions in it, then all functions are traced.

 # echo schedule > /debug/tracing/set_ftrace_filter

When ftrace is enabled, then only schedule will be traced.

 # echo *lock > /debug/tracing/set_ftrace_filter

This will trace all functions that end with "lock" when ftrace is enabled.

 # echo spin_* > /debug/tracing/set_ftrace_filter

This will trace all functions that start with "spin_" when ftrace is enabled.

 # echo *cpu* > /debug/tracing/set_ftrace_filter

This will trace all functions with "cpu" in its name.

Note, that is the extent of globing. For example spin_*_lock will not work.

To keep with Unix conventions, using > will overwrite the previous filter, and
>> will append to it.

 # echo schedule > /debug/tracing/set_ftrace_filter

makes only schedule be traced.

  # echo 'spin_*' >> /debug/tracing/set_ftrace_filter

This adds all the functions that start with "spin_" to the current filter.

OK, that's a crash course in ftrace ;-)

Comment 1 Lana Brindley 2008-04-18 04:34:26 UTC
The Install Guide will also need to be updated to reflect this info. LKB

Comment 2 Steven Rostedt 2008-04-18 17:07:38 UTC
update on /debug/tracing/tracing_max_latency

It stores the time of the last max latency found (in microseconds). If a user
wants to set the time that a max latency should be recorded, they can simply do:

echo 100 > /debug/tracing/tracing_max_latency

to set the max to 100 microsecs and the next max latency that is greater than
100 microsecs will be recorded and tracing_max_latency will be updated.  Setting
it to 0 will record the next latency traced.

Comment 3 Lana Brindley 2008-04-22 00:52:24 UTC
*** Bug 436862 has been marked as a duplicate of this bug. ***

Comment 4 Steven Rostedt 2008-04-22 01:03:49 UTC
I'm replying to your question (a bit late ;-) about this comment:

"Another way of using the tracing infrastructure is by enabling
/proc/sys/kernel/trace_user_triggered. That will trace only processes that
explicitly used the prctl(0, 1) syscall just before the critical session where
tracing is desired and prctl(0, 0) after it."

The new ftrace code still has the prctl(0,1) and prctl(0,0). This is almost an
Easter Egg.  To facilitate tests like cyclictest we added an extension to the
prctl system call. The prctl system call is a Linux special system call that can
perform certain tasks using prctl. We took this hack and stole the command "0"
(the first argument) as a control to turn on and off tracing. By passing in
(0,1) we would turn on tracing, and (0,0) would turn it off. But this is really
a hack to facilitate some tracing. I would be hard pressed to actually include
it in real documentation since it is not something we plan on pushing upstream.
Since we are not pushing that upstream, the command "0" may actually be taken by
something else. Keeping this as a stable API is not a good idea.

Make a long story short, don't comment on prctl ;-)

Note You need to log in before you can comment on or make changes to this bug.