Bug 591923 - RFE: Add abrt data (crashes) to beaker's test report
RFE: Add abrt data (crashes) to beaker's test report
Status: CLOSED WONTFIX
Product: Beaker
Classification: Community
Component: tests (Show other bugs)
0.5
All Linux
low Severity medium (vote)
: ---
: ---
Assigned To: Nick Coghlan
MC
: FutureFeature
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2010-05-13 09:50 EDT by Roman Rakus
Modified: 2014-01-12 19:11 EST (History)
12 users (show)

See Also:
Fixed In Version:
Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
: 679315 (view as bug list)
Environment:
Last Closed: 2012-11-07 02:23:30 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Roman Rakus 2010-05-13 09:50:43 EDT
Description of problem:
During the test it's possible that some process crashes, abrt will catch this crash; creating core dumo and so on. But these crashes are not recognized in tests if crash occurred for non tested package. For example you are testing procps, running many processes in background with bash, but when bash crashes the beaker will not inform about it.
Comment 1 Ales Zelinka 2010-05-19 08:30:31 EDT
This would be a great time-saver. I.e. when there's little time to re-run failed tests to get coredumps or the issue isn't 100% reproducible. 
Consider making it optional though - sometime we make stuff segfault/traceback on purpose and we don't want that to cause the tests to fail. Thanks
Comment 2 Michal Nowak 2011-02-18 08:14:22 EST
We always want to know that a crash in packaged binary was detected.

Set: OpenGPGCheck = no && restart abrtd service

On crash some sort of [ FAIL ] warning with custom created log from /var/spool/abrt/$reporter-$date-$pid/ directory should be produced. But do not generate, by default, backtrace because it's time consuming. Also do not rlBundleLog corefile -- it's quite huge.

One can place such a log generating command or link to the script to RunApp() function, e.g.

[ Common ]

ActionsAndReporters = RunApp("echo abrt: Crash in $(cat package) detected:
$(cat reason) | wall", "/tmp/__beakerlib_abrt_crash")

Perhaps `/tmp/__beakerlib_abrt_crash` could be socket not just regular file, so abrt can "append" to it.

By using wall(1) we can see the msg on stdout (and TESTOUT.log).

Ideas? Comments? Code?
Comment 3 Ondrej Hudlicky 2011-02-18 16:13:13 EST
Should there be separate bug for beakerlib then? 

This bug is a request for Beaker enhancement, I suppose original idea was to have beakerlib independent solution - using benefits of abrt as another monitoring tool for test jobs, which could be light integrated with Beaker harness.

I think we need more configuration levels, one is just not enough:

1. [OFF] abrt not installed or disabled (default to RHEL5 and older)
 -> preferred if running abrt tests for example
2. [ON] Monitoring with backtrace disabled (as described in Comment 2)
 -> candidate for default option for RHEL6+, Fedora 
3. [Tracing] Monitoring with tracing and full abrt reports in test logs
 -> it crash occurred, might want to rerun test job with full traceback

Interesting solution might be to configure these levels in bkr workflow-* options. Simple workaround is to implement configuration tests/tasks for this purpose.

Ideally if there is an option to change monitoring level during the test or on reserved system for better notifications in console (similar as bug 678564). 

What about simple command line tool which would be able to change levels
$ abrtconf [off|on|tracing]    (for example)
Comment 4 Bill Peck 2011-02-18 16:23:40 EST
(In reply to comment #3)

> Interesting solution might be to configure these levels in bkr workflow-*
> options. Simple workaround is to implement configuration tests/tasks for this
> purpose.
> 
> Ideally if there is an option to change monitoring level during the test or on
> reserved system for better notifications in console (similar as bug 678564). 
> 


Could this whole feature be enabled via a task?  I worry that we are adding way too many layers and addons to the test harness.  With each thing added we change the underlying system under test.  If this was implemented purely as a task then it could be easily turned on/off.

Just like we can enable kdump/netdump now.
Comment 5 Ondrej Hudlicky 2011-02-21 05:44:10 EST
Good point - then task based solution would be probably best choice, I agree. 

Can you pls link the mentioned kdump/netdump examples for inspiration?
Comment 6 Michal Nowak 2011-02-22 04:07:57 EST
Proposition from comment #2 moved to separate bug 679315.
Comment 7 David Kutálek 2011-04-04 05:14:53 EDT
Regarding tracing - generating backtraces.

As written in comment #1, sometimes crash is not 100% reproducible, sometimes you don't have time for second run to create backtraces. Therefore it would be very helpful if we could use external retrace server [1] to create backtraces directly from the first run/crash.

It could work like this:

1) When crash during Beaker test is catched by Abrt, it should automatically send crash to RH internal retrace server, getting task id and password. 

2) Task id and password should be logged to TESTOUT.log to give user possibility of manually grabbing backtrace from retrace server. This can be done e.g. by using wall as described in bug 679315.

3) Cron job should be setup in Beaker to check whether backtrace is ready on retrace server. When backtrace is ready, it should be grabbed and added to Beaker job outputs as separate file, and cron job removed.

Step 1) should be quite quick when network throughput between beaker machine and retrace server is sufficient. I was told that core dumps are being compressed before sending, so it shouldn't be major problem.

This way, backtrace should be available for each crash, without significant delay for beaker job processing.

Any ideas and comments are welcome.


[1] http://fedoraproject.org/wiki/Features/RetraceServer
Comment 8 Nick Coghlan 2012-10-17 00:39:27 EDT
Bulk reassignment of issues as Bill has moved to another team.
Comment 9 Min Shin 2012-11-07 02:23:30 EST
This bugs is closed as it is either not in the current Beaker scope or we could not find sufficient data in the bug report for consideration.
Please feel free to reopen the bug with additional information and/or business cases behind it.
Comment 10 Ales Zelinka 2012-11-07 12:57:46 EST
FYI this functionality is provided by Richard Marko's 
/distribution/crashes/enable-abrt & /distribution/crashes/check-abrt tasks.
(--abrt for workflow-tomorrow users)
Comment 11 Roman Rakus 2012-11-29 06:04:04 EST
(In reply to comment #9)
> This bugs is closed as it is either not in the current Beaker scope or we
> could not find sufficient data in the bug report for consideration.
> Please feel free to reopen the bug with additional information and/or
> business cases behind it.

I don't understand this; which case is it? Not in the scope or insufficient data?

If this bug rather belongs to beakerlib - can we change component?
If it is insufficient data - which data you need?
Comment 12 Dan Callaghan 2012-11-29 17:35:51 EST
(In reply to comment #11)

The actual resolution should be WONTFIX. As Bill said in comment 4, this functionality should be in a Beaker task, not the harness itself. In comment 10 Ales points out that such a task is already available.

Note You need to log in before you can comment on or make changes to this bug.