Bug 655908
| Summary: | hang/crash on massive kprobing | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | Frank Ch. Eigler <fche> | |
| Component: | kernel | Assignee: | Petr Oros <poros> | |
| Status: | CLOSED WONTFIX | QA Contact: | Ziqian SUN (Zamir) <zsun> | |
| Severity: | medium | Docs Contact: | ||
| Priority: | low | |||
| Version: | 6.0 | CC: | chuhu, cye, dougsland, gansalmon, gbai, itamar, jonathan, kernel-maint, madhu.chinakonda, masami.hiramatsu.pt, mkolaja, panand, qzhao, ruyang, skozina, yshao, zsun | |
| Target Milestone: | rc | |||
| Target Release: | --- | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | 655904 | |||
| : | 831833 (view as bug list) | Environment: | ||
| Last Closed: | 2017-12-06 10:34:44 UTC | Type: | --- | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | 655904 | |||
| Bug Blocks: | 831833, 846704, 1270638, 1359574, 1496722 | |||
|
Description
Frank Ch. Eigler
2010-11-22 16:58:33 UTC
> probe kernel.function("*") {}
Does this systemtap script include all inlined functions too?
If no, we can test it on the kernel which supports kprobe-tracer of ftrace.
Following receipt should causes a kernel panic.
# sort /proc/kallsyms | egrep '[0-9a-f]+ [Tt] [^[]*$' | c -f 3 -d" " > syms.list
# for i in `cat syms.list`; do echo "p $i" >> /sys/kernel/debug/tracing/kprobe_events ;done
# echo 1 > /sys/kernel/debug/tracing/events/enable
This request was evaluated by Red Hat Product Management for inclusion in the current release of Red Hat Enterprise Linux. Because the affected component is not scheduled to be updated in the current release, Red Hat is unfortunately unable to address this request at this time. Red Hat invites you to ask your support representative to propose this request, if appropriate and relevant, in the next release of Red Hat Enterprise Linux. If you would like it considered as an exception in the current release, please ask your support representative. This request was erroneously denied for the current release of Red Hat Enterprise Linux. The error has been fixed and this request has been re-proposed for the current release. This request was evaluated by Red Hat Product Management for inclusion in the current release of Red Hat Enterprise Linux. Because the affected component is not scheduled to be updated in the current release, Red Hat is unfortunately unable to address this request at this time. Red Hat invites you to ask your support representative to propose this request, if appropriate and relevant, in the next release of Red Hat Enterprise Linux. If you would like it considered as an exception in the current release, please ask your support representative. This request was erroneously denied for the current release of Red Hat Enterprise Linux. The error has been fixed and this request has been re-proposed for the current release. Since RHEL 6.2 External Beta has begun, and this bug remains unresolved, it has been rejected as it is not proposed as exception or blocker. Red Hat invites you to ask your support representative to propose this request, if appropriate and relevant, in the next release of Red Hat Enterprise Linux. Please check my slide at the last year's LinuxCon Japan. http://events.linuxfoundation.jp/sites/events/files/slides/Handling%20the%20Massive%20Multiple%20Kprobes%20v2_1.pdf Unfortunately, that enhance is not accepted on upstream yet... (In reply to Masami Hiramatsu from comment #12) > Please check my slide at the last year's LinuxCon Japan. > > http://events.linuxfoundation.jp/sites/events/files/slides/ > Handling%20the%20Massive%20Multiple%20Kprobes%20v2_1.pdf > Thanks a lot for the pointer. So basically there are two aspects of this bug: a) When massive kprobe is enabled, system crashes. b) When massive kprobe is enabled, system becomes extremely slow. 'a' would mostly be arch specific and need to blacklist all those symbols which are not kprobable (mainly entry routines and subroutines which comes in path of exception used for breakpoint and single step handling). > Unfortunately, that enhance is not accepted on upstream yet... and 'b' should be resolved upto very much extent with your enhancement. Thanks for revisit to this patch series. I have rebased them to latest fedora-arm64 kernel [2] and tested with my ARM64 board. I was not subscribed to systemtap mailing list (I just subscribed it). However, I noticed your reply [1]. [1] https://www.sourceware.org/ml/systemtap/2015-q3/msg00039.html [2] https://github.com/pratyushanand/linux.git:fedora_arm64_uprobe_devel (880df93e2dac) Sorry for coming back late on it. Did work a bit for last few days, specifically on possibility of analyzing hot spots when massive kprobe is instrumented on ARM64 platform (As requested by Masami in [1]). We still had some issues of crash on ARM64(point 'a' in comment 13), which has been resolved upto some extent now. However, `perf report` did not show any load for arm64 kprobe_breakpoint_handler when massive kprobes were instrumented. It did not show any load, because current arm64 implementation does not support interrupt generation(and so PMU events) when we are handling debug exception. Having said that, there could be ways to do some patches and support profiling of kprobe handler on ARM64. I can work on that item, but it will take some time. However, that work is not directly related to resolution of this bug. Masami, since above work may take some time. Therefore, can I help in someway on top of the work [1, 2] which you have already done, so that up-streaming of patches for the resolution of this BZ becomes a bit faster. [1] https://www.sourceware.org/ml/systemtap/2015-q3/msg00039.html [2] https://lkml.org/lkml/2015/7/16/70 *** Bug 985734 has been marked as a duplicate of this bug. *** Red Hat Enterprise Linux 6 is in the Production 3 Phase. During the Production 3 Phase, Critical impact Security Advisories (RHSAs) and selected Urgent Priority Bug Fix Advisories (RHBAs) may be released as they become available. The official life cycle policy can be reviewed here: http://redhat.com/rhel/lifecycle This issue does not meet the inclusion criteria for the Production 3 Phase and will be marked as CLOSED/WONTFIX. If this remains a critical requirement, please contact Red Hat Customer Support to request a re-evaluation of the issue, citing a clear business justification. Note that a strong business justification will be required for re-evaluation. Red Hat Customer Support can be contacted via the Red Hat Customer Portal at the following URL: https://access.redhat.com/ |