Created attachment 359384 [details] proposed patch, which fixes this issue Description of problem: Due to a patch, which introduced a mechanism, which hides breakpoint instructions and returns a 'shadow' content during every access to inferrior's memory by a function target_read_memory(), gdb, when thousands of breakpoints are inserted, is very slow when calling a continue command. Version-Release number of selected component (if applicable): gdb on RHEL5 (U3, U4), also on Fedora 10 and 11 How reproducible: always Steps to Reproduce: 1. Compile attached source file with debuging symbols turned on: gcc -g -o testC5000 testC5000.c 2. Run gdb with attached command file: $ gdb testC5000 -x cmd2000.txt or $ gdb testC5000 -x cmd5000.txt 3. Wait for gdb to insert all breakpoints and see that continuing is very slow even at a first glance. Actual results: time gdb testC5000 -x cmd2000.txt real 1m14.356s user 0m59.857s sys 0m13.444s Expected results: time gdb testC5000 -x cmd2000.txt real 0m25.320s user 0m11.769s sys 0m13.033s Additional info: This issue in a case of RHEL5 is introduced by a patch: gdb-6.8-bz467502-breakpoint_restore_shadows.patch This patch joins the code responsible for restoring original content on place of breakpoint address from saved shadow_contents buffer: (struct bp_location *)->(struct bp_target_info *)->shadow_contents) from separate function read_memory_nobpt() into breakpoint_restore_shadows(), which is called by target_read_memory() - the function used by every single access (reading) to debugged application's memory. According to posts on mailing list it is an intended behaviour: http://sourceware.org/ml/gdb-patches/2008-02/msg00365.html http://sourceware.org/ml/gdb-patches/2008-03/msg00106.html http://sourceware.org/ml/gdb-patches/2008-02/msg00375.html I can see from the discussion, that this is an intention to leave an user a chance to change this behaviour by calling "set breakpoint always-inserted on/off". But this option doesn't exist in RHEL5 even it is mentioned in testing scripts untroduced by a patch: gdb-6.8-bz467502-ia64-breakpoint-restoration.patch The original patch from Vladimir Prus contained an option "show-memory-breakpoints", but changes (which afterwards showed up in upstream sources) made by Daniel Jacobowitz removed it due to proposed always-inserted option. So no one of the mentioned options are presented in RHEL5 gdb sources (is this intended or is this ("always-inserted") option missing?). So according to the fact that this is intended behaviour and by default turned on - global variable show_memory_breakpoints is zero, which enables automatic restoration of memory on address of breakpoint in every acces to debugged application's memory, I started to search for functions that doesn't need this restoration and are responsible for this slowdown. I have found a function default_memory_insert_breakpoint(). This function is called by: insert_breakpoints()->insert_bp_location()->memory_insert_breakpoint()->gdbarch_memory_insert_breakpoint()->default_memory_insert_breakpoint() A function memory_insert_breakpoint() does checking for existing and inserted breakpoints: 1032 if (!bpt->enabled || bpt->shlib_disabled || bpt->inserted || bpt->duplicate) 1033 return 0; so trying to restore shadow content in default_memory_insert_breakpoint() is useless as there cannot be any inserted breakpoint by the time entering the function. So I have created a patch which turns this restoration before reading memory content on address of future breakpoint off (see attached patch) and it solves the problem. Someone, who is more familiar with the sources, should also check other target_read_memory() calls whether there is really needed to call breakpoint_restore_shadows() internally. You can find an existing entry describing this bug in gdb bugzilla: http://sourceware.org/bugzilla/show_bug.cgi?id=9335 Also one should look at sources in fedora 11, as the process of inserting breakpoints becomes slower and slower with increasing number of inserted breakpoints. This doesn't happen with gdb sources in RHEL5.
Created attachment 359385 [details] source file to compile and run under gdb to reveal this bug
Created attachment 359387 [details] commands for gdb with 2000 breakpoints
Created attachment 359388 [details] commands for gdb with 5000 breakpoints
Fixed in gdb-7.0-13.el5.
~~ Attention Customers and Partners - RHEL 5.5 Beta is now available on RHN ~~ RHEL 5.5 Beta has been released! There should be a fix present in this release that addresses your request. Please test and report back results here, by March 3rd 2010 (2010-03-03) or sooner. Upon successful verification of this request, post your results and update the Verified field in Bugzilla with the appropriate value. If you encounter any issues while testing, please describe them and set this bug into NEED_INFO. If you encounter new defects or have additional patch(es) to request for inclusion, please clone this bug per each request and escalate through your support representative.
Hello, I have tested the gdb on both RHEL 5.4 and 5.5 with the following results: RHEL5.5 gdb-7.0.1-20.el5: ========================= $ time gdb testC5000 -x cmd2500.txt ... --- snip --- real 0m25.121s user 0m8.855s sys 0m9.097s $ $ time gdb testC5000 -x cmd5000.txt ... --- snip --- real 1m28.025s user 0m38.299s sys 0m34.509s $ RHEL5.4 gdb-6.8-37.el5: ======================= $ time gdb testC5000 -x cmd2500.txt ... --- snip --- real 4m9.789s user 3m40.934s sys 0m11.037s $ $ time gdb testC5000 -x cmd5000.txt ... --- snip --- real 36m58.809s user 34m57.002s sys 0m51.605s $ As we can see there is a great improvement in gdb-7.0.1 in comparsion to gdb-6.8 in both phases - storing breakpoints into internal array and also processing breakpoints when a breakpoint is hit (restoring an original content on place of breakpoint address). I will response soon with the customer's results. Best regards, -Martin
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2010-0285.html