Red Hat Bugzilla – Bug 591923
RFE: Add abrt data (crashes) to beaker's test report
Last modified: 2014-01-12 19:11:42 EST
Description of problem:
During the test it's possible that some process crashes, abrt will catch this crash; creating core dumo and so on. But these crashes are not recognized in tests if crash occurred for non tested package. For example you are testing procps, running many processes in background with bash, but when bash crashes the beaker will not inform about it.
This would be a great time-saver. I.e. when there's little time to re-run failed tests to get coredumps or the issue isn't 100% reproducible.
Consider making it optional though - sometime we make stuff segfault/traceback on purpose and we don't want that to cause the tests to fail. Thanks
We always want to know that a crash in packaged binary was detected.
Set: OpenGPGCheck = no && restart abrtd service
On crash some sort of [ FAIL ] warning with custom created log from /var/spool/abrt/$reporter-$date-$pid/ directory should be produced. But do not generate, by default, backtrace because it's time consuming. Also do not rlBundleLog corefile -- it's quite huge.
One can place such a log generating command or link to the script to RunApp() function, e.g.
[ Common ]
ActionsAndReporters = RunApp("echo abrt: Crash in $(cat package) detected:
$(cat reason) | wall", "/tmp/__beakerlib_abrt_crash")
Perhaps `/tmp/__beakerlib_abrt_crash` could be socket not just regular file, so abrt can "append" to it.
By using wall(1) we can see the msg on stdout (and TESTOUT.log).
Ideas? Comments? Code?
Should there be separate bug for beakerlib then?
This bug is a request for Beaker enhancement, I suppose original idea was to have beakerlib independent solution - using benefits of abrt as another monitoring tool for test jobs, which could be light integrated with Beaker harness.
I think we need more configuration levels, one is just not enough:
1. [OFF] abrt not installed or disabled (default to RHEL5 and older)
-> preferred if running abrt tests for example
2. [ON] Monitoring with backtrace disabled (as described in Comment 2)
-> candidate for default option for RHEL6+, Fedora
3. [Tracing] Monitoring with tracing and full abrt reports in test logs
-> it crash occurred, might want to rerun test job with full traceback
Interesting solution might be to configure these levels in bkr workflow-* options. Simple workaround is to implement configuration tests/tasks for this purpose.
Ideally if there is an option to change monitoring level during the test or on reserved system for better notifications in console (similar as bug 678564).
What about simple command line tool which would be able to change levels
$ abrtconf [off|on|tracing] (for example)
(In reply to comment #3)
> Interesting solution might be to configure these levels in bkr workflow-*
> options. Simple workaround is to implement configuration tests/tasks for this
> Ideally if there is an option to change monitoring level during the test or on
> reserved system for better notifications in console (similar as bug 678564).
Could this whole feature be enabled via a task? I worry that we are adding way too many layers and addons to the test harness. With each thing added we change the underlying system under test. If this was implemented purely as a task then it could be easily turned on/off.
Just like we can enable kdump/netdump now.
Good point - then task based solution would be probably best choice, I agree.
Can you pls link the mentioned kdump/netdump examples for inspiration?
Proposition from comment #2 moved to separate bug 679315.
Regarding tracing - generating backtraces.
As written in comment #1, sometimes crash is not 100% reproducible, sometimes you don't have time for second run to create backtraces. Therefore it would be very helpful if we could use external retrace server  to create backtraces directly from the first run/crash.
It could work like this:
1) When crash during Beaker test is catched by Abrt, it should automatically send crash to RH internal retrace server, getting task id and password.
2) Task id and password should be logged to TESTOUT.log to give user possibility of manually grabbing backtrace from retrace server. This can be done e.g. by using wall as described in bug 679315.
3) Cron job should be setup in Beaker to check whether backtrace is ready on retrace server. When backtrace is ready, it should be grabbed and added to Beaker job outputs as separate file, and cron job removed.
Step 1) should be quite quick when network throughput between beaker machine and retrace server is sufficient. I was told that core dumps are being compressed before sending, so it shouldn't be major problem.
This way, backtrace should be available for each crash, without significant delay for beaker job processing.
Any ideas and comments are welcome.
Bulk reassignment of issues as Bill has moved to another team.
This bugs is closed as it is either not in the current Beaker scope or we could not find sufficient data in the bug report for consideration.
Please feel free to reopen the bug with additional information and/or business cases behind it.
FYI this functionality is provided by Richard Marko's
/distribution/crashes/enable-abrt & /distribution/crashes/check-abrt tasks.
(--abrt for workflow-tomorrow users)
(In reply to comment #9)
> This bugs is closed as it is either not in the current Beaker scope or we
> could not find sufficient data in the bug report for consideration.
> Please feel free to reopen the bug with additional information and/or
> business cases behind it.
I don't understand this; which case is it? Not in the scope or insufficient data?
If this bug rather belongs to beakerlib - can we change component?
If it is insufficient data - which data you need?
(In reply to comment #11)
The actual resolution should be WONTFIX. As Bill said in comment 4, this functionality should be in a Beaker task, not the harness itself. In comment 10 Ales points out that such a task is already available.