Red Hat Bugzilla – Bug 537472
ftrace function tracer triggers integrity check failure
Last modified: 2016-05-22 19:29:36 EDT
Description of problem: When enabling the function tracer "ftrace" on 2.6.24-rt, the integrity check for modifying the text triggers a failure. WARNING: at kernel/trace/ftrace.c:406 ftrace_bug() Pid: 2518, comm: kstopmachine Not tainted 2.6.24.7-137.el5rttrace #1 Call Trace: [<ffffffff882ac00d>] ? :acpi_cpufreq:extract_freq+0xd/0x92 [<ffffffff882ac00d>] ? :acpi_cpufreq:extract_freq+0xd/0x92 [<ffffffff81089a18>] ftrace_replace_code+0x25a/0x3fd [<ffffffff81089be6>] __ftrace_modify_code+0x2b/0x6a [<ffffffff8107622e>] do_stop+0xdb/0x13e [<ffffffff81076153>] ? do_stop+0x0/0x13e [<ffffffff810550ea>] kthread+0x4e/0x7c [<ffffffff8100d418>] child_rip+0xa/0x12 [<ffffffff8100cb00>] ? restore_args+0x0/0x30 [<ffffffff8105509c>] ? kthread+0x0/0x7c [<ffffffff8100d40e>] ? child_rip+0x0/0x12 --------------------------- | preempt count: 00000001 ] | 1-level deep critical section nesting: ---------------------------------------- .. [<ffffffff8107620b>] .... do_stop+0xb8/0x13e .....[<ffffffff810550ea>] .. ( <= kthread+0x4e/0x7c) ftrace failed to modify [<ffffffff882ac00d>] extract_freq+0xd/0x92 [acpi_cpufreq] actual: f8:01:74:07:83 ftrace failed to modify [<ffffffff882ac0c0>] acpi_cpufreq_resume+0x17/0x40 [acpi_cpufreq] actual: 88:48:c7:c6:78 ftrace failed to modify [<ffffffff882ac0e8>] acpi_cpufreq_resume+0x3f/0x40 [acpi_cpufreq] actual: c3:55:48:89:e5 ftrace failed to modify [<ffffffff882ac235>] get_cur_val+0xe1/0x108 [acpi_cpufreq] actual: c0:48:c7:c2:af ftrace failed to modify [<ffffffff882ac262>] acpi_cpufreq_target+0x6/0x32c [acpi_cpufreq] actual: 41:56:41:55:41 ftrace failed to modify [<ffffffff882ac294>] acpi_cpufreq_target+0x38/0x32c [acpi_cpufreq] actual: 00:00:00:c7:45 ftrace failed to modify [<ffffffff882ac2e6>] acpi_cpufreq_target+0x8a/0x32c [acpi_cpufreq] actual: c5:e8:50:20:f6 ftrace failed to modify [<ffffffff882ac323>] acpi_cpufreq_target+0xc7/0x32c [acpi_cpufreq] actual: 14:00:74:27:89 ftrace failed to modify [<ffffffff882ac43f>] acpi_cpufreq_target+0x1e3/0x32c [acpi_cpufreq] actual: 76:dc:65:48:8b ftrace failed to modify [<ffffffff882ac5ca>] get_cur_freq_on_cpu+0x42/0x8a [acpi_cpufreq] actual: 7b:08:00:75:04 ftrace failed to modify [<ffffffff882ac76f>] acpi_cpufreq_cpu_init+0x118/0x4a5 [acpi_cpufreq] actual: 7c:24:28:01:77 ftrace failed to modify [<ffffffff882ac78e>] acpi_cpufreq_cpu_init+0x137/0x4a5 [acpi_cpufreq] actual: f8:e9:21:03:00 ftrace failed to modify [<ffffffff882ac977>] acpi_cpufreq_cpu_init+0x320/0x4a5 [acpi_cpufreq] actual: 00:8d:04:09:45 ftrace failed to modify [<ffffffff882ac98e>] acpi_cpufreq_cpu_init+0x337/0x4a5 [acpi_cpufreq] actual: 03:42:30:48:69 ftrace failed to modify [<ffffffff882ac9b5>] acpi_cpufreq_cpu_init+0x35e/0x4a5 [acpi_cpufreq] actual: 44:89:02:eb:0d ftrace failed to modify [<ffffffff882acbf0>] get_measured_perf+0xf4/0x160 [acpi_cpufreq] actual: 65:48:8b:04:25 Looks like the acpi code may be read only, and it prevents ftrace from modifying that code. Will investigate. This may also be an issue with vanilla linux. I'll test the latest Linus tree first, to see if the issues is there as well.
Found the error. It was with the ftrace_release that is called by the module code to release the functions used by the module. It was missing some due to the way the hash check was implemented. Will fix.
Created attachment 369916 [details] ftrace: Only update $offset when we update $ref_func This is not the fix, but it is related. I backported this patch to 2.6.24-rt.
Created attachment 369917 [details] ftrace: Fix removing of function entries from modules This is the fix for the bug.
Technical note added. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. New Contents: The "function tracer" from ftrace could eventually present problems when a module was unloaded during a tracing session. Some of the related call site entries for that module were not removed from ftrace's internal lists and could lead to confusing error ("oops") messages.
Verified by reproducing the situation. reproducer: ------------------------------------------------------- - boot kernel-rt-trace kernel - Log in as root, and run the following commands: [root@host]# mount -t debugfs debugfs /sys/kernel/debug [root@host]# cd /sys/kernel/debug [root@host]# cat current_tracer none [root@host]# echo ftrace > current_tracer [root@host]# modprobe -r acpi ------------------------------------------------------- When checking dmesg, you'll find a similar backtrace to whats reported when running the kernel-rt-trace-2.6.24.7.139. When running the 2.6.24.7-146 version, no error messages are found.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2010-0041.html