Bug 655908 - hang/crash on massive kprobing [NEEDINFO]
hang/crash on massive kprobing
Status: NEW
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: kernel (Show other bugs)
6.0
Unspecified Unspecified
low Severity medium
: rc
: ---
Assigned To: Pratyush Anand
Ziqian SUN
:
: 985734 (view as bug list)
Depends On: 655904
Blocks: 831833 1359574 846704 1270638
  Show dependency treegraph
 
Reported: 2010-11-22 11:58 EST by Frank Ch. Eigler
Modified: 2016-08-14 23:42 EDT (History)
14 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 655904
: 831833 (view as bug list)
Environment:
Last Closed:
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
panand: needinfo? (masami.hiramatsu.pt)
zsun: needinfo? (panand)


Attachments (Terms of Use)

  None (edit)
Description Frank Ch. Eigler 2010-11-22 11:58:33 EST
+++ This bug was initially created as a clone of Bug #655904 +++
(This applies to all kernel versions > 2.6.9 we've ever seen.)

A longstanding problem in the linux kernel has been its failure to protect
itself against massive kprobe sessions, such as with systemtap scripts such as:

   probe kernel.function("*") {}

The important thing to note is that systemtap is not required to show
this problem.  "perf probe" can do it, as can the following recipe, which
builds an absolutely minimal kprobes-using kernel module, and applies it
to function entry points (as gleamed form /proc/kallsyms).  (With systemtap,
we can easily place probes into the bodies of functions too, and of course
that crashes even "harder", but let's leave that till later.)

  git clone git://sourceware.org/git/systemtap.git
  cd systemtap/scripts/kprobes_test
  sh gen_code_all.sh
  insmod kprobe_module.ko
  <bang>

There may be multiple causes, such as inadequate __kprobes markup, or
exception handling, or unknown factors.

See also

http://sourceware.org/PR275
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=604453
Comment 2 Masami Hiramatsu 2010-12-16 08:09:43 EST
>   probe kernel.function("*") {}

Does this systemtap script include all inlined functions too?
If no, we can test it on the kernel which supports kprobe-tracer of ftrace.
Following receipt should causes a kernel panic.

# sort /proc/kallsyms | egrep '[0-9a-f]+ [Tt] [^[]*$' | c -f 3 -d" " > syms.list
# for i in `cat syms.list`; do echo "p $i" >> /sys/kernel/debug/tracing/kprobe_events ;done
# echo 1 >  /sys/kernel/debug/tracing/events/enable
Comment 3 RHEL Product and Program Management 2011-01-06 23:22:07 EST
This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated
in the current release, Red Hat is unfortunately unable to
address this request at this time. Red Hat invites you to
ask your support representative to propose this request, if
appropriate and relevant, in the next release of Red Hat
Enterprise Linux. If you would like it considered as an
exception in the current release, please ask your support
representative.
Comment 4 Suzanne Yeghiayan 2011-01-07 11:19:21 EST
This request was erroneously denied for the current release of Red Hat
Enterprise Linux.  The error has been fixed and this request has been
re-proposed for the current release.
Comment 5 RHEL Product and Program Management 2011-02-01 00:53:07 EST
This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated
in the current release, Red Hat is unfortunately unable to
address this request at this time. Red Hat invites you to
ask your support representative to propose this request, if
appropriate and relevant, in the next release of Red Hat
Enterprise Linux. If you would like it considered as an
exception in the current release, please ask your support
representative.
Comment 6 RHEL Product and Program Management 2011-02-01 13:53:20 EST
This request was erroneously denied for the current release of
Red Hat Enterprise Linux.  The error has been fixed and this
request has been re-proposed for the current release.
Comment 10 RHEL Product and Program Management 2011-10-07 11:18:10 EDT
Since RHEL 6.2 External Beta has begun, and this bug remains
unresolved, it has been rejected as it is not proposed as
exception or blocker.

Red Hat invites you to ask your support representative to
propose this request, if appropriate and relevant, in the
next release of Red Hat Enterprise Linux.
Comment 12 Masami Hiramatsu 2015-06-16 06:40:29 EDT
Please check my slide at the last year's LinuxCon Japan.

http://events.linuxfoundation.jp/sites/events/files/slides/Handling%20the%20Massive%20Multiple%20Kprobes%20v2_1.pdf

Unfortunately, that enhance is not accepted on upstream yet...
Comment 13 Pratyush Anand 2015-07-16 02:35:26 EDT
(In reply to Masami Hiramatsu from comment #12)
> Please check my slide at the last year's LinuxCon Japan.
> 
> http://events.linuxfoundation.jp/sites/events/files/slides/
> Handling%20the%20Massive%20Multiple%20Kprobes%20v2_1.pdf
> 

Thanks a lot for the pointer. So basically there are two aspects of this bug:

a) When massive kprobe is enabled, system crashes.
b) When massive kprobe is enabled, system becomes extremely slow. 

'a' would mostly be arch specific and need to blacklist all those symbols which are not kprobable (mainly entry routines and subroutines which comes in path of exception used for breakpoint and single step handling).

> Unfortunately, that enhance is not accepted on upstream yet...

and 'b' should be resolved upto very much extent with your enhancement. Thanks for revisit to this patch series. I have rebased them to latest fedora-arm64 kernel [2] and tested with my ARM64 board.

I was not subscribed to systemtap mailing list (I just subscribed it). However, I noticed your reply [1]. 

[1] https://www.sourceware.org/ml/systemtap/2015-q3/msg00039.html
[2] https://github.com/pratyushanand/linux.git:fedora_arm64_uprobe_devel (880df93e2dac)
Comment 15 Ziqian SUN 2016-08-11 21:37:41 EDT
*** Bug 985734 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.