The tuning guide talks about latency tracer but the latest RT kernel implements ftrace. The ftrace utility has a different interface altogether. 1) it now resides in the debugfs file system: # mkdir /debug # mount -t debugfs nodev /debug 2) we can compile in different tracers. /debug/tracing/current_tracer holds the tracing that happens at that time. "none" is special and disables all tracing. What we currently have in -rt is: events wakeup preemptirqsoff preemptoff irqsoff ftrace sched_switch none events - CONFIG_EVENT_TRACER - traces various events wakeup - CONFIG_SCHED_TRACER - traces the wakeup latency of the highest RT priority task. preemptoff - CONFIG_PREEMPT_TRACER - traces the longest time preemption is disabled. irqsoff - CONFIG_IRQSOFF_TRACER -traces the longest time interrupts are disabled. preemptirqsoff - CONFIG_PREEMPT_TRACER && CONFIG_IRQSOFF_TRACER - traces the time either preemption or interrupts are disabled. ftrace - CONFIG_FTRACE - traces functions using the mcount mechanism sched_switch - CONFIG_CONTEXT_SWITCH_TRACER - traces tasks as they are switched by the scheduler. simply echoing in one of the above will set the current tracer. e.g. # echo ftrace > /debug/tracing/current_tracer To see which tracers have been configured into the kernel do a # cat /debug/tracing/available_tracers (I think) Tracing is default on and can be disabled via: # echo 0 > /debug/tracing/tracing_enabled and reenabled # echo 1 > /debug/tracing/tracing_enabled To see the traces you can look at /debug/tracing/latency_trace which is pretty much the same format as the latency_trace output of the previous tracer. or you can see a simpler format in /debug/tracing/trace /debug/tracing/iter_ctrl provides various changes to the output. # cat /debug/tracing/iter_ctrl noprint-parent nosym-offset nosym-addr noverbose Here we see everything is disabled. To enable one of the above, simply echo the same name without the prefix "no". # echo sys-offset > /debug/tracing/iter_ctrl # cat /debug/tracing/iter_ctrl noprint-parent sym-offset nosym-addr noverbose print-parent will show the parent of functions sym-offset will add the offset into the function -0xa4/0x254 sym-addr will add the address of a symbol 0xc1042856 verbose changes latency_trace output to be much more verbose. For those that trace max latencies (wakeup, irqsoff, preemptoff preemptirqsoff) the recorded max latency will be stored in (as well as the trace) /debug/tracing/tracing_max_latency If dynamic ftrace is configured in, ftrace mcount utility will have virtually zero overhead. But an extra feature of this is that the ftrace utility can be configured to which functions you want to trace! /debug/tracing/available_filter_functions lists the functions that have been recorded by ftrace and can be traced. /debug/tracing/dyn_ftrace_total_info is simply a information of the number of functions that have been recorded. /debug/tracing/set_ftrace_filter allows one to echo in the functions listed in available_filter_functions to tune the trace. Simple glob works when echoing in filter functions. If this has no functions in it, then all functions are traced. # echo schedule > /debug/tracing/set_ftrace_filter When ftrace is enabled, then only schedule will be traced. # echo *lock > /debug/tracing/set_ftrace_filter This will trace all functions that end with "lock" when ftrace is enabled. # echo spin_* > /debug/tracing/set_ftrace_filter This will trace all functions that start with "spin_" when ftrace is enabled. # echo *cpu* > /debug/tracing/set_ftrace_filter This will trace all functions with "cpu" in its name. Note, that is the extent of globing. For example spin_*_lock will not work. To keep with Unix conventions, using > will overwrite the previous filter, and >> will append to it. # echo schedule > /debug/tracing/set_ftrace_filter makes only schedule be traced. # echo 'spin_*' >> /debug/tracing/set_ftrace_filter This adds all the functions that start with "spin_" to the current filter. OK, that's a crash course in ftrace ;-)
The Install Guide will also need to be updated to reflect this info. LKB
update on /debug/tracing/tracing_max_latency It stores the time of the last max latency found (in microseconds). If a user wants to set the time that a max latency should be recorded, they can simply do: echo 100 > /debug/tracing/tracing_max_latency to set the max to 100 microsecs and the next max latency that is greater than 100 microsecs will be recorded and tracing_max_latency will be updated. Setting it to 0 will record the next latency traced.
*** Bug 436862 has been marked as a duplicate of this bug. ***
I'm replying to your question (a bit late ;-) about this comment: "Another way of using the tracing infrastructure is by enabling /proc/sys/kernel/trace_user_triggered. That will trace only processes that explicitly used the prctl(0, 1) syscall just before the critical session where tracing is desired and prctl(0, 0) after it." The new ftrace code still has the prctl(0,1) and prctl(0,0). This is almost an Easter Egg. To facilitate tests like cyclictest we added an extension to the prctl system call. The prctl system call is a Linux special system call that can perform certain tasks using prctl. We took this hack and stole the command "0" (the first argument) as a control to turn on and off tracing. By passing in (0,1) we would turn on tracing, and (0,0) would turn it off. But this is really a hack to facilitate some tracing. I would be hard pressed to actually include it in real documentation since it is not something we plan on pushing upstream. Since we are not pushing that upstream, the command "0" may actually be taken by something else. Keeping this as a stable API is not a good idea. Make a long story short, don't comment on prctl ;-)
Updated page available at: https://engineering.redhat.com/docbot/en-US/Red_Hat_Enterprise_MRG/1.0/html/Realtime_Tuning_Guide/sect-Realtime_Tuning_Guide-Realtime_Specific_Tuning-Using_the_ftrace_Utility_for_Tracing_Latencies.html