Bug 464155 - [RFE] summary of phase results in logfile
[RFE] summary of phase results in logfile
Status: CLOSED RAWHIDE
Product: Fedora
Classification: Fedora
Component: beakerlib (Show other bugs)
rawhide
All Linux
medium Severity medium
: ---
: ---
Assigned To: Petr Šplíchal
: Patch
Depends On: 479435
Blocks: 893059
  Show dependency treegraph
 
Reported: 2008-09-26 10:39 EDT by Petr Muller
Modified: 2016-05-31 21:36 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 893059 (view as bug list)
Environment:
Last Closed: 2013-06-27 12:32:08 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
patch + tests (3.47 KB, patch)
2009-09-04 09:12 EDT, Ales Zelinka
no flags Details | Diff
fixed patch (3.14 KB, patch)
2009-09-04 09:26 EDT, Ales Zelinka
no flags Details | Diff
Implement summary of phase results in logfile (3.67 KB, patch)
2011-07-18 08:29 EDT, Jan Hutař
no flags Details | Diff
testsuite improvement (1.80 KB, patch)
2011-07-22 14:25 EDT, Petr Muller
no flags Details | Diff

  None (edit)
Description Petr Muller 2008-09-26 10:39:08 EDT
Description of problem:

just like every phase at its end summarizes its assertions, there should be summary of all the phases at the end of the test telling how many phases were run and how many failed/passed. This would be equivalent to reporting results for test "/" (no subtest) in non-rhtslib rhts
<jhutar>: +1, please do this, would be nice
Comment 1 Petr Šplíchal 2008-10-07 05:52:45 EDT
I'll take this one...
Comment 2 Jan Hutař 2009-03-18 06:31:37 EDT
Recently, we had a discussion with Petr about phases and sub-phases. What about changing journal to be completely recursive - so test is actually 1 phase containing other and each phase compiles results of it's childs to itself, so the helpers propposed ^^^ could be completely general:

<?xml version="1.0" ?>
<RHTS_TEST>
  <test_info>
    <test_id>
      [...]
    </purpose>
  </test_info>
  <phase name='ROOT' result='FAIL' score='5'>
    <phase name="Preparing something" result="PASS" score="0" type="ABORT">
      <test message="Checking something" result='PASS'>
        ...output of the assert...
      </test>
    </phase>
    <phase name="Testing separate files"  result="FAIL" score="5" type="FAIL">
      <phase name="File1" result="FAIL" score="2" type="FAIL">
        <test message="Checking something" result='FAIL'>
          ...output of the assert...
        </test>
        <test message="Checking something else" result='FAIL'>
          ...output of the assert...
        </test>
        <test message="Doing something" result='FAIL'>
          ...output of the assert...
        </test>
        <test message="Doing something else" result='PASS'>
          ...output of the assert...
        </test>
      </phase>
      <phase name="File2" result="FAIL" score="3" type="FAIL">
        <test message="Checking something" result='FAIL'>
          ...output of the assert...
        </test>
        <test message="Checking something else" result='FAIL'>
          ...output of the assert...
        </test>
        <test message="Doing something" result='FAIL'>
          ...output of the assert...
        </test>
        <test message="Doing something else" result='PASS'>
          ...output of the assert...
        </test>
      </phase>
    </phase>
    <phase name="Cleaning up" result="PASS" score="0" type="WARN">
      <test message="Removing xml files" result='PASS'>
        ...output...
      </test>
    </phase>
  </phase>
</RHTS_TEST>
Comment 3 Jan Hutař 2009-03-18 06:33:55 EDT
Err, just a typo in assert "ROOT/Testing separate files/File1/Doing something" - it's result should be "PASS", so "ROOT/Testing separate files/File1" really have "score='2'".
Comment 4 Petr Muller 2009-03-18 09:57:03 EDT
Agreed on this. I *think* that journalling could use a small rewrite (objectify), so it could be incorporated...
Comment 5 Petr Šplíchal 2009-03-18 09:59:16 EDT
The discussion about phase-nesting should be moved to bug 485457.
Comment 6 Petr Šplíchal 2009-07-13 09:22:30 EDT
Adding just one more small RFE related to the phase summary:

Total number of bad asserts (in case of FAIL) or good asserts (in
case of PASS) could be reported as SCORE to RHTS. In this way it'd
be clear at the first sight in the web UI how many asserts really
caused the PASS/FAIL.
Comment 7 Petr Muller 2009-07-23 09:10:00 EDT
I'm 99% sure this worked in the past. I'll check why it stopped to do that.
Comment 8 Ales Zelinka 2009-09-04 09:12:42 EDT
Created attachment 359803 [details]
patch + tests

Here's the patch and unit tests - In case you don't mind including this feature before phase nesting is fully finished.
Comment 9 Ales Zelinka 2009-09-04 09:26:46 EDT
Created attachment 359808 [details]
fixed patch

New version fixing stupid mistake (counting asserts instead of phases). Tests included.
Comment 10 Petr Muller 2009-09-04 10:45:34 EDT
seems good
Comment 11 Ondrej Moriš 2011-04-21 05:48:26 EDT
Well, guys... it is quite a long time and we are still looking forward to the solution. What about to apply that patch? :)
Comment 14 Jan Hutař 2011-07-18 08:29:16 EDT
Created attachment 513607 [details]
Implement summary of phase results in logfile
Comment 15 Jan Hutař 2011-07-18 08:30:26 EDT
Re-formated patch. Slightly modified tests & ran them.
Comment 16 Petr Muller 2011-07-22 14:25:39 EDT
Created attachment 514763 [details]
testsuite improvement

This broke the 'rlAssertRpm increases SCORE when package is not found' test. During investigation, I've found 2 things:

1) That test used a custom output | head | tail | grep construction instead of assertGoodBad function (it was probably old): I've converted it to assertGoodBad

2) I've found out that the additional output produced by this RFE could interfere with how assertGoodBad works: until now, it checked for presence of 'X good' or 'Y bad' only, which could be satisfied even with new 'Phases: 6 good, 6 bad' line when sufficiently unlucky. So I've improved assertGoodBad function to search for stronger regexp.
Comment 20 Petr Muller 2013-01-08 09:23:07 EST
beakerlib-1.6-1 is already in fedora

Note You need to log in before you can comment on or make changes to this bug.