Bug 464155 - [RFE] summary of phase results in logfile
Summary: [RFE] summary of phase results in logfile
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: beakerlib
Version: rawhide
Hardware: All
OS: Linux
medium
medium
Target Milestone: ---
Assignee: Petr Šplíchal
QA Contact:
URL:
Whiteboard:
Depends On: 479435
Blocks: 893059
TreeView+ depends on / blocked
 
Reported: 2008-09-26 14:39 UTC by Petr Muller
Modified: 2016-06-01 01:36 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 893059 (view as bug list)
Environment:
Last Closed: 2013-06-27 16:32:08 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
patch + tests (3.47 KB, patch)
2009-09-04 13:12 UTC, Ales Zelinka
no flags Details | Diff
fixed patch (3.14 KB, patch)
2009-09-04 13:26 UTC, Ales Zelinka
no flags Details | Diff
Implement summary of phase results in logfile (3.67 KB, patch)
2011-07-18 12:29 UTC, Jan Hutař
no flags Details | Diff
testsuite improvement (1.80 KB, patch)
2011-07-22 18:25 UTC, Petr Muller
no flags Details | Diff

Description Petr Muller 2008-09-26 14:39:08 UTC
Description of problem:

just like every phase at its end summarizes its assertions, there should be summary of all the phases at the end of the test telling how many phases were run and how many failed/passed. This would be equivalent to reporting results for test "/" (no subtest) in non-rhtslib rhts
<jhutar>: +1, please do this, would be nice

Comment 1 Petr Šplíchal 2008-10-07 09:52:45 UTC
I'll take this one...

Comment 2 Jan Hutař 2009-03-18 10:31:37 UTC
Recently, we had a discussion with Petr about phases and sub-phases. What about changing journal to be completely recursive - so test is actually 1 phase containing other and each phase compiles results of it's childs to itself, so the helpers propposed ^^^ could be completely general:

<?xml version="1.0" ?>
<RHTS_TEST>
  <test_info>
    <test_id>
      [...]
    </purpose>
  </test_info>
  <phase name='ROOT' result='FAIL' score='5'>
    <phase name="Preparing something" result="PASS" score="0" type="ABORT">
      <test message="Checking something" result='PASS'>
        ...output of the assert...
      </test>
    </phase>
    <phase name="Testing separate files"  result="FAIL" score="5" type="FAIL">
      <phase name="File1" result="FAIL" score="2" type="FAIL">
        <test message="Checking something" result='FAIL'>
          ...output of the assert...
        </test>
        <test message="Checking something else" result='FAIL'>
          ...output of the assert...
        </test>
        <test message="Doing something" result='FAIL'>
          ...output of the assert...
        </test>
        <test message="Doing something else" result='PASS'>
          ...output of the assert...
        </test>
      </phase>
      <phase name="File2" result="FAIL" score="3" type="FAIL">
        <test message="Checking something" result='FAIL'>
          ...output of the assert...
        </test>
        <test message="Checking something else" result='FAIL'>
          ...output of the assert...
        </test>
        <test message="Doing something" result='FAIL'>
          ...output of the assert...
        </test>
        <test message="Doing something else" result='PASS'>
          ...output of the assert...
        </test>
      </phase>
    </phase>
    <phase name="Cleaning up" result="PASS" score="0" type="WARN">
      <test message="Removing xml files" result='PASS'>
        ...output...
      </test>
    </phase>
  </phase>
</RHTS_TEST>

Comment 3 Jan Hutař 2009-03-18 10:33:55 UTC
Err, just a typo in assert "ROOT/Testing separate files/File1/Doing something" - it's result should be "PASS", so "ROOT/Testing separate files/File1" really have "score='2'".

Comment 4 Petr Muller 2009-03-18 13:57:03 UTC
Agreed on this. I *think* that journalling could use a small rewrite (objectify), so it could be incorporated...

Comment 5 Petr Šplíchal 2009-03-18 13:59:16 UTC
The discussion about phase-nesting should be moved to bug 485457.

Comment 6 Petr Šplíchal 2009-07-13 13:22:30 UTC
Adding just one more small RFE related to the phase summary:

Total number of bad asserts (in case of FAIL) or good asserts (in
case of PASS) could be reported as SCORE to RHTS. In this way it'd
be clear at the first sight in the web UI how many asserts really
caused the PASS/FAIL.

Comment 7 Petr Muller 2009-07-23 13:10:00 UTC
I'm 99% sure this worked in the past. I'll check why it stopped to do that.

Comment 8 Ales Zelinka 2009-09-04 13:12:42 UTC
Created attachment 359803 [details]
patch + tests

Here's the patch and unit tests - In case you don't mind including this feature before phase nesting is fully finished.

Comment 9 Ales Zelinka 2009-09-04 13:26:46 UTC
Created attachment 359808 [details]
fixed patch

New version fixing stupid mistake (counting asserts instead of phases). Tests included.

Comment 10 Petr Muller 2009-09-04 14:45:34 UTC
seems good

Comment 11 Ondrej Moriš 2011-04-21 09:48:26 UTC
Well, guys... it is quite a long time and we are still looking forward to the solution. What about to apply that patch? :)

Comment 14 Jan Hutař 2011-07-18 12:29:16 UTC
Created attachment 513607 [details]
Implement summary of phase results in logfile

Comment 15 Jan Hutař 2011-07-18 12:30:26 UTC
Re-formated patch. Slightly modified tests & ran them.

Comment 16 Petr Muller 2011-07-22 18:25:39 UTC
Created attachment 514763 [details]
testsuite improvement

This broke the 'rlAssertRpm increases SCORE when package is not found' test. During investigation, I've found 2 things:

1) That test used a custom output | head | tail | grep construction instead of assertGoodBad function (it was probably old): I've converted it to assertGoodBad

2) I've found out that the additional output produced by this RFE could interfere with how assertGoodBad works: until now, it checked for presence of 'X good' or 'Y bad' only, which could be satisfied even with new 'Phases: 6 good, 6 bad' line when sufficiently unlucky. So I've improved assertGoodBad function to search for stronger regexp.

Comment 20 Petr Muller 2013-01-08 14:23:07 UTC
beakerlib-1.6-1 is already in fedora


Note You need to log in before you can comment on or make changes to this bug.