Bug 1306903 - abrt-action-generate-core-backtrace consumed all memory resources.
abrt-action-generate-core-backtrace consumed all memory resources.
Status: CLOSED WONTFIX
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: abrt (Show other bugs)
6.7
x86_64 Linux
unspecified Severity high
: rc
: ---
Assigned To: abrt
Jakub Filak
:
Depends On:
Blocks: 1442328
  Show dependency treegraph
 
Reported: 2016-02-12 01:21 EST by Jay Shin
Modified: 2017-04-14 02:25 EDT (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1442328 (view as bug list)
Environment:
Last Closed: 2017-04-14 02:23:21 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
coredump generated by abrtd (2.56 MB, application/x-gzip)
2016-02-12 01:21 EST, Jay Shin
no flags Details

  None (edit)
Description Jay Shin 2016-02-12 01:21:09 EST
Created attachment 1123349 [details]
coredump generated by abrtd

Description of problem:
abrt-action-generate-core-backtrace consumed all memory resources.
This issue is relating to the vim bug : https://bugzilla.redhat.com/show_bug.cgi?id=1306099

Version-Release number of selected component (if applicable):

ENVIRONMENT:
  PowerEdge R920     Vers: 1.0.7    Date: 02/24/2014    BIOS Rev: 1.0
  Intel(R) Xeon(R) CPU E7-4880 v2 @ 2.50GHz / 256 GB Memory
  Vxfs Oracle server
  2.6.32-573.3.1.el6.x86_64
  vim-enhanced-7.4.629-5.el6
  abrt-2.0.8-34.el6.x86_64

How reproducible:
  

Steps to Reproduce:
1.
2.
3.

Actual results:
  abrt-action-generate-core-backtrace consumed all memory resources
  
Expected results:
  abrt process should not consume all memory resources.

Additional info:
Comment 1 Brad Hubbard 2016-02-12 01:30:11 EST
SFDC : 01580348 https://c.na7.visual.force.com/apex/Case_View?id=500A000000TTK7xIAH&sfdc.override=1

https://optimus.gsslab.rdu2.redhat.com/manager/778320950
retrace-server-interact 778320950 crash
 
ENVIRONMENT:
  PowerEdge R920     Vers: 1.0.7    Date: 02/24/2014    BIOS Rev: 1.0
  Intel(R) Xeon(R) CPU E7-4880 v2 @ 2.50GHz / 256 GB Memory
  Vxfs Oracle server
  2.6.32-573.3.1.el6.x86_64
  vim-enhanced-7.4.629-5.el6 abrt-2.0.8-34.el6.x86_64

SPECIFICS:
  abrt-action-generate-core-backtrace consumed all memory resources after vim killed
  call trace of coredump is repeating infinitely.

  from customer :
    Update I found one of our ADBA team members uses a login script that he’s used for years that issues a sig kill to clean up when he exits.  
    The sig kill seems to be triggering the abrtd process (performs crash analysis) to start, which eventually consumes all memory causing the crash.  
    Our ADBA has modified his script to not issue the sig kill. I tght this info may help help RH narrow down

RESEARCH:
  
STEPS TAKEN: 

SUPPORTING DATA:
  Attached to Case: 
   coredump : 01580348_data_bundle.tgz 
   sosreport

  Elsewhere: vmcore : retrace-server-interact 778320950 crash
Comment 2 Brad Hubbard 2016-02-12 01:34:52 EST
The vim crash leading to this issue generated a coredump with over nine million frames and the vim process was repeastedly generating a segfault which it was catching with its signal handler which in turn segfaulted.

It could be argued that abrtd was just doing its job but it should not be bringing a system to it's knees in the process. The following is similar to what the vim process was doing but, in the case of my binary the stacks frames do not increase as was seen in the vim coredump. Not sure why that is but I'm out of time on this one I'm afraid.

# cat sigtest.c 
#include <unistd.h>
#include <signal.h>
#include <stdlib.h>
#include <stdio.h>

void wayout() {
    raise(SIGSEGV);
}

void deathwish(int sig) {

    printf("%s","*");
    wayout();
}

int main() {

    signal(SIGINT, deathwish);
    signal(SIGSEGV, deathwish);

    while(1)
        sleep(1);

    return 0;
}
Comment 3 Jakub Filak 2016-02-16 14:24:24 EST
I thought abrt-action-generate-core-backtrace is stackoverflow resistant. We have an automatic test that verifies that abrt-generate-core-backtrace saves only the botton 256 frames. We forgot to backport the test to rhel6 too but if I run the test utility 'will_stackoverflow' (will-crash package from EPEL), abrt-generate-core-backtrace finishes almost immediately.

We can run all abrtd post-processing with increased nice:
https://github.com/abrt/abrt/commit/7451d0fa8f2d551a73ceb12a47ff03b083eb6d66

I would really love to have reproducer here to be 100% sure that there is not a hiudden bug.

Can you reproduce the issue with will_stackoverflow?
Comment 6 Brad Hubbard 2016-02-16 16:19:45 EST
(In reply to Jakub Filak from comment #3)
> I thought abrt-action-generate-core-backtrace is stackoverflow resistant. We
> have an automatic test that verifies that abrt-generate-core-backtrace saves
> only the botton 256 frames. We forgot to backport the test to rhel6 too but
> if I run the test utility 'will_stackoverflow' (will-crash package from
> EPEL), abrt-generate-core-backtrace finishes almost immediately.
> 
> We can run all abrtd post-processing with increased nice:
> https://github.com/abrt/abrt/commit/7451d0fa8f2d551a73ceb12a47ff03b083eb6d66
> 
> I would really love to have reproducer here to be 100% sure that there is
> not a hiudden bug.
> 
> Can you reproduce the issue with will_stackoverflow?

I don't think that is a good simulation of what is going on in the vim core. That core shows unbounded stack growth *and* repeated SIGSEGV being generated every 20 frames.
Comment 9 Matej Habrnal 2016-02-26 08:37:51 EST
I tried to reproduce the issue with the code which is mentioned in Comment 2 and even with will_stackoverflow and everything seems to work right. 

In my case abrt-action-generate-core-backtrace do not consumed all memory resources.
Comment 10 Matej Habrnal 2016-02-26 08:51:05 EST
I also tried to reproduce the issue with given coredump and it worked right as well.

Note You need to log in before you can comment on or make changes to this bug.