Bug 966726

Summary: abrt-action-generate-core-backtrace in abrt-addon-vmcore fails Segmentation fault (SIGSEGV or killed by signal 11)
Product: [Fedora] Fedora Reporter: James Hartsock <hartsjc>
Component: abrtAssignee: abrt <abrt-devel-list>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: medium    
Version: 18CC: abrt-devel-list, dvlasenk, iprikryl, jfilak, jmoskovc, mmilata, mtoman
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: abrt-2.1.5-1.fc18 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-07-02 00:26:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description James Hartsock 2013-05-23 19:22:07 UTC
Description of problem:
  When running report on vmcore get (killed by signal 11)
  When running abrt-action-generate-core-backtrace manually get Segmenation fault

Version-Release number of selected component (if applicable):
  Fedora release 18 (Spherical Cow)
  abrt-addon-vmcore-2.1.2-2.fc18.x86_64

How reproducible:
Always


Steps to Reproduce:
1.  Setup system to generate vmcore
2.  install & enable abrt-addon-vmcore
3.  trigger a core (echo c > /proc/sysrq-trigger)
4.  I manually get debuginfo downloaded & extraced for abrt
    # yumdownloader --destdir=/var/cache/abrt-di/ --enablerepo=\* kernel-debuginfo-$(uname -r)
    # cd /var/cache/abrt-di && rpm2cpio kernel-debuginfo-$(uname -r).rpm | cpio -idv ./usr/lib/debug/lib/modules/*/vmlinux


5.  Validate have abrt for vmcore
    # abrt-cli list
    @0
    Directory:      /var/tmp/abrt/vmcore-127.0.0.1-2013.05.23-14:35:46
    count:          1
    time:           Thu 23 May 2013 09:36:00 AM CDT
    uid:            0

6.  Run the abrt report on event
    # abrt-cli report /var/tmp/abrt/vmcore-127.0.0.1-2013.05.23-14:35:46
    Required vmlinux debuginfo is installed in /var/cache/abrt-di
   Generating backtrace
    abrt-dump-oops: Found oopses: 1
   (killed by signal 11)

7.  Manually reproduce issue
    # cd /var/tmp/abrt/vmcore-127.0.0.1-2013.05.23-14:35:46
    # abrt-action-generate-core-backtrace
    Segmentation fault

8.  Debug manual issue, see expecting kernel file
    # strace -e open abrt-action-generate-core-backtrace 2>&1| tail -n 4
    open("./backtrace", O_RDONLY|O_NOFOLLOW) = 3
    open("./kernel", O_RDONLY|O_NOFOLLOW)   = -1 ENOENT (No such file or directory)
    --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0} ---
    +++ killed by SIGSEGV +++





Actual results:
    # abrt-cli report /var/tmp/abrt/vmcore-127.0.0.1-2013.05.23-14:35:46
    Required vmlinux debuginfo is installed in /var/cache/abrt-di
    Generating backtrace
    abrt-dump-oops: Found oopses: 1
    (killed by signal 11)


    # cd /var/tmp/abrt/vmcore-127.0.0.1-2013.05.23-14:35:46
    # abrt-action-generate-core-backtrace
    Segmentation fault

    # strace -e open abrt-action-generate-core-backtrace 2>&1| tail -n 4
    open("./backtrace", O_RDONLY|O_NOFOLLOW) = 3
    open("./kernel", O_RDONLY|O_NOFOLLOW)   = -1 ENOENT (No such file or directory)
    --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0} ---
    +++ killed by SIGSEGV +++


Expected results for abrt report:
  Everything works if manually create kernel file in abrt config first
    # head  -n 6 /etc/libreport/events.d/vmcore_event.conf
    # analyze
    EVENT=analyze_VMcore analyzer=vmcore
            uname -r > kernel
            abrt-action-analyze-vmcore
            abrt-action-generate-core-backtrace

    # abrt-cli report /var/tmp/abrt/vmcore-127.0.0.1-2013.05.23-14:35:46
    Required vmlinux debuginfo is installed in /var/cache/abrt-di
    Generating backtrace
    abrt-dump-oops: Found oopses: 1
    Core backtrace is generated and saved, 368 bytes

Expected results for manual run:
  After creating 'kernel' file everything works
    # uname -r > /var/tmp/abrt/vmcore-127.0.0.1-2013.05.23-14:35:46/kernel
    # abrt-action-generate-core-backtrace
    Core backtrace is generated and saved, 368 bytes



Additional info:

Comment 1 James Hartsock 2013-05-23 19:24:36 UTC
FYI: 'kernel' file can be empty from what I can tell too, I just put uname -r output in it for something of value.

Comment 2 Fedora Update System 2013-06-14 22:37:25 UTC
abrt-2.1.5-1.fc18,libreport-2.1.5-1.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/abrt-2.1.5-1.fc18,libreport-2.1.5-1.fc18

Comment 3 Fedora Update System 2013-06-16 05:28:26 UTC
Package abrt-2.1.5-1.fc18, libreport-2.1.5-1.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing abrt-2.1.5-1.fc18 libreport-2.1.5-1.fc18'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-10938/abrt-2.1.5-1.fc18,libreport-2.1.5-1.fc18
then log in and leave karma (feedback).

Comment 4 James Hartsock 2013-06-17 17:24:44 UTC
Making sure issue still existed on my test system:
  # abrt-cli report /var/tmp/abrt/vmcore-127.0.0.1-2013.06.17-16:19:49
  Required vmlinux debuginfo is installed in /var/cache/abrt-di
  Generating backtrace
  abrt-dump-oops: Found oopses: 1
  (killed by signal 11)


  # cd /var/tmp/abrt/vmcore-127.0.0.1-2013.06.17-16:19:49
  # abrt-action-generate-core-backtrace
  Segmentation fault (core dumped)

Now update the RPMs
  # yum update --enablerepo=updates-testing abrt-2.1.5-1.fc18
  # rpm -q abrt abrt-addon-vmcore abrt-addon-ccpp
  abrt-2.1.5-1.fc18.x86_64
  abrt-addon-vmcore-2.1.5-1.fc18.x86_64
  abrt-addon-ccpp-2.1.5-1.fc18.x86_64


And then test make sure issue is fixed.  Did get message about report_uReport, but backtrace is working.
  # abrt-cli report /var/tmp/abrt/vmcore-127.0.0.1-2013.06.17-16:19:49
  Required vmlinux debuginfo is installed in /var/cache/abrt-di
  Generating backtrace
  abrt-dump-oops: Found oopses: 1
  abrt-dump-oops: Updating problem directory
  Generating core_backtrace
  Core backtrace is generated and saved, 368 bytes
  Error: no processing is specified for event 'report_uReport'

  # cd /var/tmp/abrt/vmcore-127.0.0.1-2013.06.17-16:19:49
  # abrt-action-generate-core-backtrace
  Generating core_backtrace
  Core backtrace is generated and saved, 368 bytes



Thank you,
/James

Comment 5 Jakub Filak 2013-06-18 07:04:07 UTC
(In reply to James Hartsock from comment #4)
Thank you for your excellent collaboration.

The message about report_uReport is strange but it is not a disaster. Does your /etc/libreport/events.d/vmcore_event.conf file look like the default one at https://github.com/abrt/abrt/blob/master/src/plugins/vmcore_event.conf ? If so then it could be cause by damaged analyzer file (/var/tmp/abrt/vmcore-127.0.0.1-2013.06.17-16:19:49/analyzer). The file should contain single line "vmcore" without trailing new line character.

Comment 6 James Hartsock 2013-06-18 14:24:29 UTC
(In reply to Jakub Filak from comment #5)

Good catch, I should have looked a little closer after the RPM update.
  # ls -1 /etc/libreport/events.d/vmcore_event.conf*
  /etc/libreport/events.d/vmcore_event.conf
  /etc/libreport/events.d/vmcore_event.conf.rpmnew

  # mv -f /etc/libreport/events.d/vmcore_event.conf.rpmnew   /etc/libreport/events.d/vmcore_event.conf


I don't want to report to bugzilla:
  # sed -i -e 's/-e report_Bugzilla//' /etc/libreport/events.d/vmcore_event.conf


Now run, and report_uReport is a success.  But post_report isn't defined in my vmcore_event.conf (or the one you link to on github).
  # abrt-cli report /var/tmp/abrt/vmcore-127.0.0.1-2013.06.18-13:11:33
  Required vmlinux debuginfo is installed in /var/cache/abrt-di
  Generating backtrace
  abrt-dump-oops: Found oopses: 1
  abrt-dump-oops: Updating problem directory
  Generating core_backtrace
  Core backtrace is generated and saved, 368 bytes
  Looking for kernel package
  Kernel package kernel-3.9.2-200.fc18.x86_64 found
  ('report_uReport' completed successfully)
  Error: no processing is specified for event 'post_report'

  # echo $?
  1



If I disable the non defined post_report, all is a success
  # sed -i -e 's/-e post_report//' /etc/libreport/events.d/vmcore_event.conf

  # abrt-cli report /var/tmp/abrt/vmcore-127.0.0.1-2013.06.18-13:11:33
  Required vmlinux debuginfo is installed in /var/cache/abrt-di
  Generating backtrace
  abrt-dump-oops: Found oopses: 1
  abrt-dump-oops: Updating problem directory
  Generating core_backtrace
  Core backtrace is generated and saved, 368 bytes
  Looking for kernel package
  Kernel package kernel-3.9.2-200.fc18.x86_64 found
  ('report_uReport' completed successfully)

  # echo $?
  0

Comment 7 Jakub Filak 2013-06-18 15:29:27 UTC
(In reply to James Hartsock from comment #6)
If you don't want to report to bugzilla, you even don't want to run post_report event. The event connects an ureport together with a bugzilla bug id on the ureport server. And it doesn't make sense when you don't have the bug id. So, you can save safely disable the post_report event.

But anyway, you've found an another bug. I've forgot to submit a patch for the missing post_report event problem. I didn't want to add an another simply stupid type specific post_create definition, I wanted to fix this kind of problems once and forever. Thank you!

Comment 8 Fedora Update System 2013-07-02 00:26:20 UTC
abrt-2.1.5-1.fc18, libreport-2.1.5-1.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.