Bug 1422836

Summary:	gdb gets stuck for unreasonable time with 100% CPU at program start
Product:	[Fedora] Fedora	Reporter:	Dov Grobgeld <dov.grobgeld>
Component:	gdb	Assignee:	Jan Kratochvil <jan.kratochvil>
Status:	CLOSED DUPLICATE	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	25	CC:	gbenson, jan.kratochvil, palves, pmuldoon, sergiodj, tom
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-02-16 12:30:33 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Dov Grobgeld 2017-02-16 11:23:33 UTC

Description of problem:

When starting debugging of a multithread program in gdb, gdb gets stuck with 100% CPU for about half a minute after gdb has output the line 'Using host libthread_db library "/lib64/libthread_db.so.1".' Before my latest dnf update the time spent on this line was negligable. 

Note that when being stuck Control-C does not stop the process immediately but is also delayed until the end of the "stuck" section. So perhaps the bug is in gdb and not in libthread_db.

The stack trace of gdb while being stuck is as follows. I stopped gdb several manually with Ctrl-C. And each time it was stuck somewhere in symfile.c and dwarf2read.c .

#0  0x00005555558bf9dc in read_attribute_value (reader=reader@entry=0x7fffffffc7d0, attr=attr@entry=0x7fffffffc660, form=14, info_ptr=info_ptr@entry=0x7fff7b486549 "/\354m") at ../../gdb/dwarf2read.c:16415
#1  0x00005555558c8086 in read_attribute (info_ptr=0x7fff7b486549 "/\354m", abbrev=<optimized out>, attr=0x7fffffffc660, reader=0x7fffffffc7d0) at ../../gdb/dwarf2read.c:16640
#2  read_partial_die (info_ptr=0x7fff7b486549 "/\354m", abbrev_len=<optimized out>, abbrev=0x55555fcd97f0, part_die=0x55555ffab7f0, reader=0x7fffffffc7d0) at ../../gdb/dwarf2read.c:15998
#3  load_partial_dies (reader=reader@entry=0x7fffffffc7d0, info_ptr=<optimized out>, info_ptr@entry=0x7fff7b42a663 "\002std", building_psymtab=building_psymtab@entry=1) at ../../gdb/dwarf2read.c:15799
#4  0x00005555558db4ab in process_psymtab_comp_unit_reader (reader=reader@entry=0x7fffffffc7d0, info_ptr=0x7fff7b42a663 "\002std", comp_unit_die=0x555562c356e0, has_children=<optimized out>, data=data@entry=0x7fffffffc850) at ../../gdb/dwarf2read.c:6039
#5  0x00005555558cb980 in init_cutu_and_read_dies (this_cu=this_cu@entry=0x555562f67860, abbrev_table=abbrev_table@entry=0x0, use_existing_cu=use_existing_cu@entry=0, keep=keep@entry=0, die_reader_func=die_reader_func@entry=0x5555558db140 <process_psymtab_comp_unit_reader(die_reader_specs const*, gdb_byte const*, die_info*, int, void*)>, data=data@entry=0x7fffffffc850) at ../../gdb/dwarf2read.c:5657
#6  0x00005555558da4a2 in process_psymtab_comp_unit (this_cu=0x555562f67860, want_partial_unit=0, pretend_language=language_minimal) at ../../gdb/dwarf2read.c:6124
#7  0x00005555558da61e in dwarf2_build_psymtabs_hard (objfile=0x55555ede1130) at ../../gdb/dwarf2read.c:6517
#8  dwarf2_build_psymtabs (objfile=0x55555ede1130) at ../../gdb/dwarf2read.c:4304
#9  0x000055555582755e in require_partial_symbols (objfile=objfile@entry=0x55555ede1130, verbose=verbose@entry=0) at ../../gdb/psymtab.c:91
#10 0x000055555582caa4 in read_symbols (objfile=objfile@entry=0x55555ede1130, add_flags=add_flags@entry=8) at ../../gdb/symfile.c:896
#11 0x000055555582c527 in syms_from_objfile_1 (add_flags=8, addrs=0x555559e79370, objfile=0x55555ede1130) at ../../gdb/symfile.c:1075
#12 syms_from_objfile (add_flags=8, addrs=0x8, objfile=0x55555ede1130) at ../../gdb/symfile.c:1091
#13 symbol_file_add_with_addrs (abfd=abfd@entry=0x55555eb8e5f0, name=name@entry=0x555557c9c2c0 "/usr/lib/debug/usr/lib64/libQt5WebEngineCore.so.5.7.1.debug", add_flags=add_flags@entry=8, addrs=addrs@entry=0x555559e79370, flags=3, parent=parent@entry=0x55555e725170) at ../../gdb/symfile.c:1188
#14 0x000055555582ca1d in symbol_file_add_separate (bfd=bfd@entry=0x55555eb8e5f0, name=name@entry=0x555557c9c2c0 "/usr/lib/debug/usr/lib64/libQt5WebEngineCore.so.5.7.1.debug", symfile_flags=symfile_flags@entry=8, objfile=objfile@entry=0x55555e725170) at ../../gdb/symfile.c:1262
#15 0x00005555557a4555 in elf_symfile_read (objfile=<optimized out>, symfile_flags=8) at ../../gdb/elfread.c:1273
#16 0x000055555582ca59 in read_symbols (objfile=objfile@entry=0x55555e725170, add_flags=add_flags@entry=8) at ../../gdb/symfile.c:871
#17 0x000055555582c527 in syms_from_objfile_1 (add_flags=8, addrs=0x55555ee6af30, objfile=0x55555e725170) at ../../gdb/symfile.c:1075
#18 syms_from_objfile (add_flags=8, addrs=0x55555ee6af30, objfile=0x55555e725170) at ../../gdb/symfile.c:1091
#19 symbol_file_add_with_addrs (abfd=<optimized out>, name=name@entry=0x5555569a6f60 "/lib64/libQt5WebEngineCore.so.5", add_flags=add_flags@entry=8, addrs=addrs@entry=0x55555ee6af30, flags=flags@entry=2, parent=parent@entry=0x0) at ../../gdb/symfile.c:1188
#20 0x000055555582caf5 in symbol_file_add_from_bfd (abfd=<optimized out>, name=name@entry=0x5555569a6f60 "/lib64/libQt5WebEngineCore.so.5", add_flags=add_flags@entry=8, addrs=addrs@entry=0x55555ee6af30, flags=flags@entry=2, parent=parent@entry=0x0) at ../../gdb/symfile.c:1277
#21 0x0000555555955e8b in solib_read_symbols (so=so@entry=0x5555569a6d50, flags=flags@entry=8) at ../../gdb/solib.c:703
#22 0x00005555559560ac in solib_add (pattern=pattern@entry=0x0, from_tty=from_tty@entry=0, target=<optimized out>, readsyms=1) at ../../gdb/solib.c:1030
#23 0x000055555595683f in handle_solib_event () at ../../gdb/solib.c:1346
#24 0x00005555557e2ed4 in bpstat_stop_status (aspace=0x555556356f00, bp_addr=bp_addr@entry=140737351889319, ptid=..., ws=ws@entry=0x7fffffffce60) at ../../gdb/breakpoint.c:5725
#25 0x000055555584cab3 in handle_signal_stop (ecs=0x7fffffffce40) at ../../gdb/infrun.c:5967
#26 handle_inferior_event_1 (ecs=0x7fffffffce40) at ../../gdb/infrun.c:5403
#27 handle_inferior_event (ecs=ecs@entry=0x7fffffffce40) at ../../gdb/infrun.c:5434
#28 0x000055555584e17b in fetch_inferior_event (client_data=<optimized out>) at ../../gdb/infrun.c:3982
#29 0x0000555555865735 in gdb_wait_for_event (block=block@entry=0) at ../../gdb/event-loop.c:859
#30 0x0000555555865847 in gdb_do_one_event () at ../../gdb/event-loop.c:322
#31 0x0000555555865996 in gdb_do_one_event () at ../../gdb/event-loop.c:353
#32 0x0000555555939c7c in wait_sync_command_done () at ../../gdb/top.c:568
#33 0x0000555555939ef6 in maybe_wait_sync_command_done (was_sync=0) at ../../gdb/top.c:587
#34 execute_command (p=<optimized out>, from_tty=0) at ../../gdb/top.c:676
#35 0x000055555576d415 in execute_gdb_command (self=<optimized out>, args=<optimized out>, kw=<optimized out>) at ../../gdb/python/python.c:693
#36 0x00007ffff5fa1809 in PyCFunction_Call () from /lib64/libpython3.5m.so.1.0
#37 0x00007ffff6018db0 in PyEval_EvalFrameEx () from /lib64/libpython3.5m.so.1.0
#38 0x00007ffff601a5e3 in _PyEval_EvalCodeWithName () from /lib64/libpython3.5m.so.1.0
#39 0x00007ffff601a6c3 in PyEval_EvalCodeEx () from /lib64/libpython3.5m.so.1.0
#40 0x00007ffff601a6eb in PyEval_EvalCode () from /lib64/libpython3.5m.so.1.0
#41 0x00007ffff60398e4 in run_mod () from /lib64/libpython3.5m.so.1.0
#42 0x00007ffff603be65 in PyRun_FileExFlags () from /lib64/libpython3.5m.so.1.0
#43 0x00007ffff603bfd5 in PyRun_SimpleFileExFlags () from /lib64/libpython3.5m.so.1.0
#44 0x000055555576efbf in run_python_script (argc=1, argv=0x7fffffffd698) at ../../gdb/python/python.c:1381
#45 0x000055555586054b in captured_main (data=<optimized out>) at ../../gdb/main.c:1234
#46 gdb_main (args=<optimized out>) at ../../gdb/main.c:1257
#47 0x000055555564ee48 in main (argc=<optimized out>, argv=<optimized out>) at ../../gdb/gdb.c:40



-

Version-Release number of selected component (if applicable):

gdb-7.12.1-41.fc25.x86_64
glibc-2.24-4.fc25.x86_64

How reproducible:

Every time.

Steps to Reproduce:

Sorry, meanwhile only happen when debugging a proprietary of my employer, that I cannot share. 

I can show its ldd dependencies though which are:
	linux-vdso.so.1 (0x00007ffc7bfd6000)
	libpng16.so.16 => /lib64/libpng16.so.16 (0x00007f41f0d98000)
	libz.so.1 => /lib64/libz.so.1 (0x00007f41f0b82000)
	libQHttpEngine.so.1 => /home/dov/git/SolarJet/XjetApps/MetalJet/Apps/Project/qt/BinLinux/libQHttpEngine.so.1 (0x00007f41f0956000)
	libhalcon.so => /home/dov/git/SolarJet/XjetApps/MetalJet/Apps/Project/qt/BinLinux/libhalcon.so (0x00007f41eefdf000)
	libhalconcpp.so => /home/dov/git/SolarJet/XjetApps/MetalJet/Apps/Project/qt/BinLinux/libhalconcpp.so (0x00007f41ee9da000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f41ee7d2000)
	libmariadb.so.2 => /usr/lib64/mariadb/libmariadb.so.2 (0x00007f41ee58c000)
	libfreetype.so.6 => /lib64/libfreetype.so.6 (0x00007f41ee2e1000)
	libpython2.7.so.1.0 => /lib64/libpython2.7.so.1.0 (0x00007f41edf01000)
	libexpat.so.1 => /lib64/libexpat.so.1 (0x00007f41edcd7000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f41edad1000)
	libutil.so.1 => /lib64/libutil.so.1 (0x00007f41ed8ce000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f41ed6b0000)
	libQt5Svg.so.5 => /lib64/libQt5Svg.so.5 (0x00007f41ed65a000)
	libQt5WebEngineWidgets.so.5 => /lib64/libQt5WebEngineWidgets.so.5 (0x00007f41ed61e000)
	libQt5PrintSupport.so.5 => /lib64/libQt5PrintSupport.so.5 (0x00007f41ed5ab000)
	libQt5Widgets.so.5 => /lib64/libQt5Widgets.so.5 (0x00007f41ecf44000)
	libQt5WebEngine.so.5 => /lib64/libQt5WebEngine.so.5 (0x00007f41eceeb000)
	libQt5WebEngineCore.so.5 => /lib64/libQt5WebEngineCore.so.5 (0x00007f41e81af000)
	libQt5Quick.so.5 => /lib64/libQt5Quick.so.5 (0x00007f41e7dc1000)
	libQt5Gui.so.5 => /lib64/libQt5Gui.so.5 (0x00007f41e791e000)
	libQt5WebChannel.so.5 => /lib64/libQt5WebChannel.so.5 (0x00007f41e78fe000)
	libQt5Qml.so.5 => /lib64/libQt5Qml.so.5 (0x00007f41e7501000)
	libQt5WebSockets.so.5 => /lib64/libQt5WebSockets.so.5 (0x00007f41e74d0000)
	libQt5Network.so.5 => /lib64/libQt5Network.so.5 (0x00007f41e7365000)
	libQt5Positioning.so.5 => /lib64/libQt5Positioning.so.5 (0x00007f41e7324000)
	libQt5Core.so.5 => /lib64/libQt5Core.so.5 (0x00007f41e6e9a000)
	libGL.so.1 => /lib64/libGL.so.1 (0x00007f41e6c0a000)
	libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f41e6880000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f41e6577000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f41e6360000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f41e5f9a000)
	libXext.so.6 => /lib64/libXext.so.6 (0x00007f41e5d88000)
	libX11.so.6 => /lib64/libX11.so.6 (0x00007f41e5a49000)
	libssl.so.10 => /lib64/libssl.so.10 (0x00007f41e57d5000)
	libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f41e5374000)
	libbz2.so.1 => /lib64/libbz2.so.1 (0x00007f41e5164000)
	/lib64/ld-linux-x86-64.so.2 (0x000055e1f23fe000)
	libglib-2.0.so.0 => /lib64/libglib-2.0.so.0 (0x00007f41e4e51000)
	libevent-2.0.so.5 => /lib64/libevent-2.0.so.5 (0x00007f41e4c08000)
	libicui18n.so.57 => /lib64/libicui18n.so.57 (0x00007f41e478e000)
	libicuuc.so.57 => /lib64/libicuuc.so.57 (0x00007f41e43e5000)
	libprotobuf.so.9 => /lib64/libprotobuf.so.9 (0x00007f41e40c6000)
	libsmime3.so => /lib64/libsmime3.so (0x00007f41e3e9f000)
	libnss3.so => /lib64/libnss3.so (0x00007f41e3b75000)
	libnssutil3.so => /lib64/libnssutil3.so (0x00007f41e3947000)
	libnspr4.so => /lib64/libnspr4.so (0x00007f41e3706000)
	libfontconfig.so.1 => /lib64/libfontconfig.so.1 (0x00007f41e34c2000)
	libharfbuzz.so.0 => /lib64/libharfbuzz.so.0 (0x00007f41e325e000)
	libjpeg.so.62 => /lib64/libjpeg.so.62 (0x00007f41e2ff4000)
	libXi.so.6 => /lib64/libXi.so.6 (0x00007f41e2de4000)
	libXcursor.so.1 => /lib64/libXcursor.so.1 (0x00007f41e2bd9000)
	libXfixes.so.3 => /lib64/libXfixes.so.3 (0x00007f41e29d1000)
	libXrender.so.1 => /lib64/libXrender.so.1 (0x00007f41e27c7000)
	libpci.so.3 => /lib64/libpci.so.3 (0x00007f41e25ba000)
	libre2.so.0 => /lib64/libre2.so.0 (0x00007f41e234a000)
	libXcomposite.so.1 => /lib64/libXcomposite.so.1 (0x00007f41e2147000)
	libasound.so.2 => /lib64/libasound.so.2 (0x00007f41e1e52000)
	libopus.so.0 => /lib64/libopus.so.0 (0x00007f41e1bff000)
	libvpx.so.4 => /lib64/libvpx.so.4 (0x00007f41e17f6000)
	libXdamage.so.1 => /lib64/libXdamage.so.1 (0x00007f41e15f3000)
	libXtst.so.6 => /lib64/libXtst.so.6 (0x00007f41e13ed000)
	libdbus-1.so.3 => /lib64/libdbus-1.so.3 (0x00007f41e119d000)
	libwebpdemux.so.2 => /lib64/libwebpdemux.so.2 (0x00007f41e0f98000)
	libwebp.so.6 => /lib64/libwebp.so.6 (0x00007f41e0d35000)
	libxml2.so.2 => /lib64/libxml2.so.2 (0x00007f41e09ce000)
	libxslt.so.1 => /lib64/libxslt.so.1 (0x00007f41e078f000)
	libsnappy.so.1 => /lib64/libsnappy.so.1 (0x00007f41e0587000)
	libminizip.so.1 => /lib64/libminizip.so.1 (0x00007f41e037b000)
	libsrtp.so.1 => /lib64/libsrtp.so.1 (0x00007f41e0162000)
	libproxy.so.1 => /lib64/libproxy.so.1 (0x00007f41dff41000)
	libicudata.so.57 => /lib64/libicudata.so.57 (0x00007f41de4c4000)
	libpcre16.so.0 => /lib64/libpcre16.so.0 (0x00007f41de25b000)
	libgthread-2.0.so.0 => /lib64/libgthread-2.0.so.0 (0x00007f41de059000)
	libsystemd.so.0 => /lib64/libsystemd.so.0 (0x00007f41ddfce000)
	libGLX.so.0 => /lib64/libGLX.so.0 (0x00007f41ddd9c000)
	libGLdispatch.so.0 => /lib64/libGLdispatch.so.0 (0x00007f41ddab3000)
	libxcb.so.1 => /lib64/libxcb.so.1 (0x00007f41dd88b000)
	libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f41dd63e000)
	libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f41dd358000)
	libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f41dd154000)
	libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f41dcf21000)
	libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f41dccae000)
	libplc4.so => /lib64/libplc4.so (0x00007f41dcaa9000)
	libplds4.so => /lib64/libplds4.so (0x00007f41dc8a5000)
	libgraphite2.so.3 => /lib64/libgraphite2.so.3 (0x00007f41dc679000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f41dc45c000)
	liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f41dc236000)
	libmodman.so.1 => /lib64/libmodman.so.1 (0x00007f41dc02e000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f41dbe07000)
	libcap.so.2 => /lib64/libcap.so.2 (0x00007f41dbc02000)
	liblz4.so.1 => /lib64/liblz4.so.1 (0x00007f41db9ec000)
	libgcrypt.so.20 => /lib64/libgcrypt.so.20 (0x00007f41db706000)
	libgpg-error.so.0 => /lib64/libgpg-error.so.0 (0x00007f41db4f2000)
	libXau.so.6 => /lib64/libXau.so.6 (0x00007f41db2ee000)
	libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f41db0df000)
	libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f41daedb000)


Actual results:


Expected results:


Additional info:

I would very much appreciate to be guided what additional information to provide  to solve this bug.

Comment 1 Dov Grobgeld 2017-02-16 12:02:31 UTC

FYI, I just tested compling the latest binutils-gdb from source (commit 90571206059), and it does *not* suffer from the delay reported in this bug.

Comment 2 Jan Kratochvil 2017-02-16 12:30:33 UTC

(In reply to Dov Grobgeld from comment #0)
> #7  0x00005555558da61e in dwarf2_build_psymtabs_hard
...
> #13 symbol_file_add_with_addrs (abfd=abfd@entry=0x55555eb8e5f0, name=name@entry=0x555557c9c2c0 "/usr/lib/debug/usr/lib64/libQt5WebEngineCore.so.5.7.1.debug", add_flags=add_flags@entry=8,

F-25 is unfortunately accidentally missing .gdb_index to accelerate opening of the debuginfo files - Bug 1410907.  It is fixed now but the debuginfo rpms which have not been rebuilt yet will still lack this way.  F-24 and F-26 should be OK.

I have tested:

FAIL:
/tmp/qt5-qtwebengine-debuginfo-5.7.1-4.fc25.x86_64$ readelf -WS ./usr/lib/debug/usr/lib64/libQt5WebEngineCore.so.5.7.1.debug|grep gdb_index
/tmp/qt5-qtwebengine-debuginfo-5.7.1-4.fc25.x86_64$ _

PASS:
/tmp/qt5-qtwebengine-debuginfo-5.7.1-6.fc26.x86_64$ readelf -WS ./usr/lib/debug/usr/lib64/libQt5WebEngineCore.so.5.7.1.debug|grep gdb_index
  [37] .gdb_index        PROGBITS        0000000000000000 8c2023b4 38037c3 00      0   0  1
/tmp/qt5-qtwebengine-debuginfo-5.7.1-6.fc26.x86_64$ _


(In reply to Dov Grobgeld from comment #1)
> FYI, I just tested compling the latest binutils-gdb from source (commit
> 90571206059), and it does *not* suffer from the delay reported in this bug.

I guess this is because you haven't used:
  configure --with-separate-debug-dir=/usr/lib/debug
And therefore your GDB does not load the debuginfo rpm for real debugging of the libraries.
You can simulate the same with Fedora GDB using
  --readnever
option.  Although that will not read even the main executable debug info so maybe you may prefer to use the option:
  -iex 'set debug-file-directory /mordor'

*** This bug has been marked as a duplicate of bug 1410907 ***