Bug 861844 - Backtrace parsing failed for . ... 11:0: "Thread" header expected [NEEDINFO]
Backtrace parsing failed for . ... 11:0: "Thread" header expected
Status: CLOSED WONTFIX
Product: Fedora
Classification: Fedora
Component: gdb (Show other bugs)
18
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Jan Kratochvil
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-10-01 01:40 EDT by Steve Tyler
Modified: 2014-01-06 09:32 EST (History)
15 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-01-06 09:32:41 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
jan.kratochvil: needinfo? (jmoskovc)


Attachments (Terms of Use)
backtrace (213.16 KB, text/plain)
2012-10-01 01:40 EDT, Steve Tyler
no flags Details
event_log (14.08 KB, text/plain)
2012-10-01 01:45 EDT, Steve Tyler
no flags Details
gdb log showing output from "t a a bt full" (307.19 KB, text/plain)
2012-10-01 02:44 EDT, Steve Tyler
no flags Details
backtrace from reproducer (212.47 KB, text/plain)
2012-10-22 12:39 EDT, Steve Tyler
no flags Details
event_log from reproducer (16.02 KB, text/plain)
2012-10-22 12:41 EDT, Steve Tyler
no flags Details

  None (edit)
Description Steve Tyler 2012-10-01 01:40:48 EDT
Created attachment 619619 [details]
backtrace

Description of problem:
The attached backtrace cannot be parsed:

[snippet from event_log]
2012-09-30-21:50:18  All debuginfo files are available
2012-09-30-21:50:18  Generating backtrace
2012-09-30-21:50:27  Backtrace is generated and saved, 218280 bytes
2012-09-30-21:50:27  Backtrace parsing failed for .
2012-09-30-21:50:27  11:0: "Thread" header expected

Version-Release number of selected component (if applicable):
abrt-2.0.13-1.fc17.x86_64
btparser-0.19-1.fc17.x86_64
gdb-7.4.50.20120120-50.fc17.x86_64

How reproducible:
Only tried this once.

Steps to Reproduce:
1. Crash qemu-kvm.
2. Retrace service failed because the coredump is too large.
3. Quit abrt.
3. Download some symbols manually.
4. Restart abrt.
5. Choose local debugging.
  
Actual results:
backtrace cannot be parsed.

Expected results:
backtrace can be parsed.

Additional info:
The coredump is huge:
$ ls -l coredump 
-rw-r--r--. 1 stephent stephent 2376052736 Sep 30 21:09 coredump
Comment 1 Steve Tyler 2012-10-01 01:45:18 EDT
Created attachment 619620 [details]
event_log
Comment 2 Steve Tyler 2012-10-01 01:56:02 EDT
$ cat cmdline 
qemu-kvm -m 2048 -hda f18-test-2.img -vga qxl
Comment 3 Steve Tyler 2012-10-01 02:44:10 EDT
Created attachment 619637 [details]
gdb log showing output from "t a a bt full"

$ gdb qemu-kvm ../ccpp-2012-09-30-21:09:00-2240/coredump
GNU gdb (GDB) Fedora (7.4.50.20120120-50.fc17)
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/qemu-kvm...Reading symbols from /usr/lib/debug/usr/bin/qemu-kvm.debug...done.
done.
[New LWP 2240]
[New LWP 2430]
[New LWP 2244]
[New LWP 2243]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `qemu-kvm -m 2048 -hda f18-test-2.img -vga qxl'.
Program terminated with signal 11, Segmentation fault.
#0  __memcpy_sse2 () at ../sysdeps/x86_64/memcpy.S:202
202		movq	%rax,   (%rdi)
Missing separate debuginfos, use: debuginfo-install celt051-0.5.1.3-4.fc17.x86_64 cryptopp-5.6.1-7.fc17.x86_64 dbus-libs-1.4.10-5.fc17.x86_64 flac-1.2.1-9.fc17.x86_64 gsm-1.0.13-6.fc17.x86_64 json-c-0.9-4.fc17.x86_64 keyutils-libs-1.5.5-2.fc17.x86_64 krb5-libs-1.10.2-6.fc17.x86_64 libICE-1.0.8-1.fc17.x86_64 libSM-1.2.1-1.fc17.x86_64 libXau-1.0.6-3.fc17.x86_64 libXcursor-1.1.13-1.fc17.x86_64 libXext-1.3.1-1.fc17.x86_64 libXfixes-5.0-2.fc17.x86_64 libXi-1.6.1-1.fc17.x86_64 libXrandr-1.3.1-3.fc17.x86_64 libXrender-0.9.7-1.fc17.x86_64 libXtst-1.2.0-3.fc17.x86_64 libasyncns-0.8-3.fc17.x86_64 libcom_err-1.42.3-2.fc17.x86_64 libdb-5.2.36-5.fc17.x86_64 libgcc-4.7.2-2.fc17.x86_64 libgcrypt-1.5.0-3.fc17.x86_64 libgpg-error-1.10-2.fc17.x86_64 libidn-1.24-1.fc17.x86_64 libogg-1.3.0-1.fc17.x86_64 libselinux-2.1.10-3.fc17.x86_64 libsndfile-1.0.25-2.fc17.x86_64 libssh2-1.4.1-2.fc17.x86_64 libstdc++-4.7.2-2.fc17.x86_64 libtasn1-2.12-1.fc17.x86_64 libvorbis-1.3.3-1.fc17.x86_64 libxcb-1.8.1-1.fc17.x86_64 nss-softokn-freebl-3.13.5-1.fc17.x86_64 openldap-2.4.32-2.fc17.x86_64 openssl-1.0.0j-2.fc17.x86_64 p11-kit-0.12-1.fc17.x86_64 pixman-0.24.4-2.fc17.x86_64 tcp_wrappers-libs-7.6-69.fc17.x86_64
(gdb) set logging on
Copying output to gdb.txt.
(gdb) t a a bt full
...
Comment 4 Steve Tyler 2012-10-01 03:14:42 EDT
See also:
Bug 852897 - qemu-kvm crash in validate_surface 

Created attachment 608006 [details]
Backtrace generated by abrt, which it fails to parse
Comment 5 Karel Klíč 2012-10-01 09:29:47 EDT
This seems to be a bug in backtrace generation and not in backtrace parsing.  Btparser assumes that GDB uses the Thread line to introduce a new thread in a backtrace, and comment #3 indicates this is true also for this bug when the backtrace is generated manually.   Why the backtrace is different when generated by ABRT?
Comment 6 Jiri Moskovcak 2012-10-01 09:46:23 EDT
(In reply to comment #5)
> This seems to be a bug in backtrace generation and not in backtrace parsing.
> Btparser assumes that GDB uses the Thread line to introduce a new thread in
> a backtrace, and comment #3 indicates this is true also for this bug when
> the backtrace is generated manually.   Why the backtrace is different when
> generated by ABRT?

The only difference between those backtraces is that GDB can't find debuginfo if you run the gdb manually, the rest seems the same (at least to me...)
Comment 7 Jiri Moskovcak 2012-10-22 06:40:51 EDT
Please disregard comment#6 I didn't notice the missing [Thread ..] line. ABRT uses gdb to generate the backtrace (it runs gdb as a child) so I suspect there is a bug in GDB - is there some option which could make GDB to not output these lines?
Comment 8 Jan Kratochvil 2012-10-22 11:12:01 EDT
As was already told parsing GDB output will never work as GDB output changes.

What is the expecte and actual output?

When loading a core file GDB prints:
[New LWP 30612]
[New LWP 30611]
This seems to always have been in recent history, the threads are then shown in 'info threads':
(gdb) info threads 
  Id   Target Id         Frame 
  2    Thread 0x7f02d9ec6700 (LWP 30611) 0x37e4609080 in pthread_join [...]
* 1    Thread 0x7f02d9ec4700 (LWP 30612) 0x0000000000 in ?? ()

With live processes GDB prints:
[New Thread 0x7ffff7fd3700 (LWP 21540)]

GDB probably could print [New Thread...] even for core files but I do not think it ever was that way, was it?
Comment 9 Steve Tyler 2012-10-22 11:58:22 EDT
This recent bug was reported with abrt 2.0.16, and the backtrace has:
Thread 6 (Thread 0x7f4729e22700 (LWP 10964)):

Bug 868974 - [abrt] nautilus-3.6.1-1.fc18: g_type_check_instance_cast: Process /usr/libexec/nautilus-shell-search-provider was killed by signal 11 (SIGSEGV)

What is different about the qemu-kvm examples?

The coredump was huge (~2GB):
$ ls -l coredump 
-rw-r--r--. 1 stephent stephent 2376052736 Sep 30 21:09 coredump

Thread 3, Frame 1 is huge:
gdb log showing output from "t a a bt full"
Attachment 619637 [details]
Comment 10 Steve Tyler 2012-10-22 12:35:22 EDT
Here's a reproducer: Crash qemu-kvm with 2048MB memory running a Live CD.

1. $ qemu-kvm -m 2048 -cdrom ~/xfr/fedora/F18/F18-Beta/TC6/Fedora-18-Beta-TC6-x86_64-Live-Desktop.iso -usb -vga qxl -usbdevice mouse
2. Boot to the Live CD desktop.
3. From another terminal:
   $ kill -ABRT <pid-of-qemu-kvm-process>

NB: The retrace server will reject the coredump because it is too large. I had to download one debuginfo package with yum, because the connection was so slow.

===== snippet from event_log =====
...
2012-10-22-09:05:13  Retrace server can not be used, because the crash is too large. Try local retracing.
2012-10-22-09:05:13  The size of your crash is 2391605326 bytes, but the retrace server only accepts crashes smaller or equal to 1342177280 bytes.
...
2012-10-22-09:23:34  All debuginfo files are available
2012-10-22-09:23:34  Generating backtrace
2012-10-22-09:23:45  Backtrace is generated and saved, 217572 bytes
2012-10-22-09:23:45  Backtrace parsing failed for .
2012-10-22-09:23:45  10:0: "Thread" header expected

===== head of backtrace =====
[New LWP 4537]
[New LWP 4540]
[New LWP 4541]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `qemu-kvm -m 2048 -cdrom /home/stephent/xfr/fedora/F18/F18-Beta/TC6/Fedora-18-Be'.
Program terminated with signal 6, Aborted.
#0  0x00007f6aed7ac9e3 in select () at ../sysdeps/unix/syscall-template.S:82
82      ../sysdeps/unix/syscall-template.S: No such file or directory.
#0  0x00007f6aed7ac9e3 in select () at ../sysdeps/unix/syscall-template.S:82
No locals.
#1  0x00007f6af30516f8 in main_loop_wait (nonblocking=<optimized out>) at main-loop.c:456
        rfds = {fds_bits = {32, 0 <repeats 15 times>}}
        wfds = {fds_bits = {0 <repeats 16 times>}}
        xfds = {fds_bits = {0 <repeats 16 times>}}
        ret = <optimized out>
        nfds = 28
        tv = {tv_sec = 0, tv_usec = 998107}
        timeout = 1000
#2  0x00007f6af2f92729 in main_loop () at /usr/src/debug/qemu-kvm-1.0.1/vl.c:1482
        nonblocking = <optimized out>
        last_io = 1
Comment 11 Steve Tyler 2012-10-22 12:39:47 EDT
Created attachment 631634 [details]
backtrace from reproducer

$ rpm -q abrt libreport btparser gdb | sort
abrt-2.0.13-1.fc17.x86_64
btparser-0.19-1.fc17.x86_64
gdb-7.4.50.20120120-50.fc17.x86_64
libreport-2.0.14-1.fc17.x86_64
Comment 12 Steve Tyler 2012-10-22 12:41:01 EDT
Created attachment 631635 [details]
event_log from reproducer
Comment 13 Steve Tyler 2012-10-22 18:38:15 EDT
Here's a reproducer that can use either the retrace server or the local debugger:

1. Specify 64 MB of memory and boot to the syslinux menu that is first displayed when booting from the installer DVD. For consistency, use the up arrow to select the first menu item. This also stops the 60 second countdown.

$ qemu-kvm -m 64 -cdrom ~/xfr/fedora/F18/F18-Beta/TC6/Fedora-18-Beta-TC6-x86_64-DVD.iso -usb -vga qxl -usbdevice mouse

2. $ kill -ABRT <pid-of-qemu-kvm-process>

Summary:

With retrace server: Thread header present
With local debugger: Thread header absent
Comment 14 Fedora End Of Life 2013-07-03 18:52:10 EDT
This message is a reminder that Fedora 17 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 17. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '17'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 17's end of life.

Bug Reporter:  Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 17 is end of life. If you 
would still like  to see this bug fixed and are able to reproduce it 
against a later version  of Fedora, you are encouraged  change the 
'version' to a later Fedora version prior to Fedora 17's end of life.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.
Comment 15 Fedora End Of Life 2013-12-21 03:59:57 EST
This message is a reminder that Fedora 18 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 18. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '18'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 18's end of life.

Thank you for reporting this issue and we are sorry that we may not be 
able to fix it before Fedora 18 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior to Fedora 18's end of life.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.
Comment 16 Jan Kratochvil 2014-01-06 09:32:41 EST
ABRT should not depend on GDB output, it should use the GDB Python scripting.

Note You need to log in before you can comment on or make changes to this bug.