Bug 520618 - processing 'continue' command of gdb is slow, when many breakpoints are set
processing 'continue' command of gdb is slow, when many breakpoints are set
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: gdb (Show other bugs)
5.4
All Linux
high Severity high
: rc
: ---
Assigned To: Jan Kratochvil
BaseOS QE
:
Depends On: 526533
Blocks: 499522
  Show dependency treegraph
 
Reported: 2009-09-01 09:16 EDT by Martin Osvald
Modified: 2013-04-12 16:23 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-03-30 04:52:13 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
proposed patch, which fixes this issue (1.13 KB, patch)
2009-09-01 09:16 EDT, Martin Osvald
no flags Details | Diff
source file to compile and run under gdb to reveal this bug (58.67 KB, text/plain)
2009-09-01 09:19 EDT, Martin Osvald
no flags Details
commands for gdb with 2000 breakpoints (19.73 KB, text/plain)
2009-09-01 09:20 EDT, Martin Osvald
no flags Details
commands for gdb with 5000 breakpoints (49.02 KB, text/plain)
2009-09-01 09:21 EDT, Martin Osvald
no flags Details

  None (edit)
Description Martin Osvald 2009-09-01 09:16:29 EDT
Created attachment 359384 [details]
proposed patch, which fixes this issue

Description of problem:

Due to a patch, which introduced a mechanism, which hides breakpoint instructions and returns a 'shadow' content during every access to inferrior's memory by a function target_read_memory(), gdb, when thousands of breakpoints are inserted, is very slow when calling a continue command.


Version-Release number of selected component (if applicable):

gdb on RHEL5 (U3, U4), also on Fedora 10 and 11

How reproducible:

always


Steps to Reproduce:

1. Compile attached source file with debuging symbols turned on:

gcc -g -o testC5000 testC5000.c

2. Run gdb with attached command file:

$ gdb testC5000 -x cmd2000.txt

or

$ gdb testC5000 -x cmd5000.txt

3. Wait for gdb to insert all breakpoints and see that continuing is very slow
even at a first glance.


Actual results:

time gdb testC5000 -x cmd2000.txt

real    1m14.356s
user    0m59.857s
sys     0m13.444s


Expected results:

time gdb testC5000 -x cmd2000.txt

real    0m25.320s
user    0m11.769s
sys     0m13.033s


Additional info:

This issue in a case of RHEL5 is introduced by a patch:

gdb-6.8-bz467502-breakpoint_restore_shadows.patch

This patch joins the code responsible for restoring original content on place of breakpoint address from saved shadow_contents buffer:

(struct bp_location *)->(struct bp_target_info *)->shadow_contents)

from separate function read_memory_nobpt() into breakpoint_restore_shadows(), which is called by target_read_memory() - the function used by every single access (reading) to debugged application's memory.

According to posts on mailing list it is an intended behaviour:

http://sourceware.org/ml/gdb-patches/2008-02/msg00365.html
http://sourceware.org/ml/gdb-patches/2008-03/msg00106.html
http://sourceware.org/ml/gdb-patches/2008-02/msg00375.html

I can see from the discussion, that this is an intention to leave an user a chance to change this behaviour by calling "set breakpoint always-inserted on/off". But this option doesn't exist in RHEL5 even it is mentioned in testing scripts untroduced by a patch:

gdb-6.8-bz467502-ia64-breakpoint-restoration.patch

The original patch from Vladimir Prus contained an option "show-memory-breakpoints", but changes (which afterwards showed up in upstream sources) made by Daniel Jacobowitz removed it due to proposed always-inserted option. So no one of the mentioned options are presented in RHEL5 gdb sources (is this intended or is this ("always-inserted") option missing?).

So according to the fact that this is intended behaviour and by default turned on - global variable show_memory_breakpoints is zero, which enables automatic restoration of memory on address of breakpoint in every acces to debugged application's memory, I started to search for functions that doesn't need this restoration and are responsible for this slowdown. I have found a function default_memory_insert_breakpoint().

This function is called by:

insert_breakpoints()->insert_bp_location()->memory_insert_breakpoint()->gdbarch_memory_insert_breakpoint()->default_memory_insert_breakpoint()

A function memory_insert_breakpoint() does checking for existing and inserted breakpoints:

1032      if (!bpt->enabled || bpt->shlib_disabled || bpt->inserted || bpt->duplicate)
1033        return 0;

so trying to restore shadow content in default_memory_insert_breakpoint() is useless as there cannot be any inserted breakpoint by the time entering the function.

So I have created a patch which turns this restoration before reading memory content on address of future breakpoint off (see attached patch) and it solves the problem.

Someone, who is more familiar with the sources, should also check other target_read_memory() calls whether there is really needed to call breakpoint_restore_shadows() internally.

You can find an existing entry describing this bug in gdb bugzilla:

http://sourceware.org/bugzilla/show_bug.cgi?id=9335

Also one should look at sources in fedora 11, as the process of inserting breakpoints becomes slower and slower with increasing number of inserted breakpoints. This doesn't happen with gdb sources in RHEL5.
Comment 1 Martin Osvald 2009-09-01 09:19:00 EDT
Created attachment 359385 [details]
source file to compile and run under gdb to reveal this bug
Comment 2 Martin Osvald 2009-09-01 09:20:50 EDT
Created attachment 359387 [details]
commands for gdb with 2000 breakpoints
Comment 3 Martin Osvald 2009-09-01 09:21:42 EDT
Created attachment 359388 [details]
commands for gdb with 5000 breakpoints
Comment 22 Jan Kratochvil 2009-12-21 17:58:17 EST
Fixed in gdb-7.0-13.el5.
Comment 24 Chris Ward 2010-02-11 05:17:17 EST
~~ Attention Customers and Partners - RHEL 5.5 Beta is now available on RHN ~~

RHEL 5.5 Beta has been released! There should be a fix present in this 
release that addresses your request. Please test and report back results 
here, by March 3rd 2010 (2010-03-03) or sooner.

Upon successful verification of this request, post your results and update 
the Verified field in Bugzilla with the appropriate value.

If you encounter any issues while testing, please describe them and set 
this bug into NEED_INFO. If you encounter new defects or have additional 
patch(es) to request for inclusion, please clone this bug per each request
and escalate through your support representative.
Comment 25 Martin Osvald 2010-02-15 04:02:13 EST
Hello,

I have tested the gdb on both RHEL 5.4 and 5.5 with the following results:

RHEL5.5 gdb-7.0.1-20.el5:
=========================

$ time gdb testC5000 -x cmd2500.txt
...
--- snip ---
real    0m25.121s
user    0m8.855s
sys     0m9.097s
$

$ time gdb testC5000 -x cmd5000.txt
...
--- snip ---
real    1m28.025s
user    0m38.299s
sys     0m34.509s
$

RHEL5.4 gdb-6.8-37.el5:
=======================

$ time gdb testC5000 -x cmd2500.txt
...
--- snip ---
real    4m9.789s
user    3m40.934s
sys     0m11.037s
$

$ time gdb testC5000 -x cmd5000.txt
...
--- snip ---
real    36m58.809s
user    34m57.002s
sys     0m51.605s
$

As we can see there is a great improvement in gdb-7.0.1 in comparsion to gdb-6.8 in both phases - storing breakpoints into internal array and also processing breakpoints when a breakpoint is hit (restoring an original content on place of breakpoint address).

I will response soon with the customer's results.

Best regards,
-Martin
Comment 29 errata-xmlrpc 2010-03-30 04:52:13 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2010-0285.html

Note You need to log in before you can comment on or make changes to this bug.