Bug 1306903

SFDC : 01580348 https://c.na7.visual.force.com/apex/Case_View?id=500A000000TTK7xIAH&sfdc.override=1

https://optimus.gsslab.rdu2.redhat.com/manager/778320950
retrace-server-interact 778320950 crash
 
ENVIRONMENT:
  PowerEdge R920     Vers: 1.0.7    Date: 02/24/2014    BIOS Rev: 1.0
  Intel(R) Xeon(R) CPU E7-4880 v2 @ 2.50GHz / 256 GB Memory
  Vxfs Oracle server
  2.6.32-573.3.1.el6.x86_64
  vim-enhanced-7.4.629-5.el6 abrt-2.0.8-34.el6.x86_64

SPECIFICS:
  abrt-action-generate-core-backtrace consumed all memory resources after vim killed
  call trace of coredump is repeating infinitely.

  from customer :
    Update I found one of our ADBA team members uses a login script that he’s used for years that issues a sig kill to clean up when he exits.  
    The sig kill seems to be triggering the abrtd process (performs crash analysis) to start, which eventually consumes all memory causing the crash.  
    Our ADBA has modified his script to not issue the sig kill. I tght this info may help help RH narrow down

RESEARCH:
  
STEPS TAKEN: 

SUPPORTING DATA:
  Attached to Case: 
   coredump : 01580348_data_bundle.tgz 
   sosreport

  Elsewhere: vmcore : retrace-server-interact 778320950 crash

The vim crash leading to this issue generated a coredump with over nine million frames and the vim process was repeastedly generating a segfault which it was catching with its signal handler which in turn segfaulted.

It could be argued that abrtd was just doing its job but it should not be bringing a system to it's knees in the process. The following is similar to what the vim process was doing but, in the case of my binary the stacks frames do not increase as was seen in the vim coredump. Not sure why that is but I'm out of time on this one I'm afraid.

# cat sigtest.c 
#include <unistd.h>
#include <signal.h>
#include <stdlib.h>
#include <stdio.h>

void wayout() {
    raise(SIGSEGV);
}

void deathwish(int sig) {

    printf("%s","*");
    wayout();
}

int main() {

    signal(SIGINT, deathwish);
    signal(SIGSEGV, deathwish);

    while(1)
        sleep(1);

    return 0;
}

I thought abrt-action-generate-core-backtrace is stackoverflow resistant. We have an automatic test that verifies that abrt-generate-core-backtrace saves only the botton 256 frames. We forgot to backport the test to rhel6 too but if I run the test utility 'will_stackoverflow' (will-crash package from EPEL), abrt-generate-core-backtrace finishes almost immediately.

We can run all abrtd post-processing with increased nice:
https://github.com/abrt/abrt/commit/7451d0fa8f2d551a73ceb12a47ff03b083eb6d66

I would really love to have reproducer here to be 100% sure that there is not a hiudden bug.

Can you reproduce the issue with will_stackoverflow?

(In reply to Jakub Filak from comment #3)
> I thought abrt-action-generate-core-backtrace is stackoverflow resistant. We
> have an automatic test that verifies that abrt-generate-core-backtrace saves
> only the botton 256 frames. We forgot to backport the test to rhel6 too but
> if I run the test utility 'will_stackoverflow' (will-crash package from
> EPEL), abrt-generate-core-backtrace finishes almost immediately.
> 
> We can run all abrtd post-processing with increased nice:
> https://github.com/abrt/abrt/commit/7451d0fa8f2d551a73ceb12a47ff03b083eb6d66
> 
> I would really love to have reproducer here to be 100% sure that there is
> not a hiudden bug.
> 
> Can you reproduce the issue with will_stackoverflow?

I don't think that is a good simulation of what is going on in the vim core. That core shows unbounded stack growth *and* repeated SIGSEGV being generated every 20 frames.

I tried to reproduce the issue with the code which is mentioned in Comment 2 and even with will_stackoverflow and everything seems to work right. 

In my case abrt-action-generate-core-backtrace do not consumed all memory resources.

I also tried to reproduce the issue with given coredump and it worked right as well.