Description of problem: just like every phase at its end summarizes its assertions, there should be summary of all the phases at the end of the test telling how many phases were run and how many failed/passed. This would be equivalent to reporting results for test "/" (no subtest) in non-rhtslib rhts <jhutar>: +1, please do this, would be nice
I'll take this one...
Recently, we had a discussion with Petr about phases and sub-phases. What about changing journal to be completely recursive - so test is actually 1 phase containing other and each phase compiles results of it's childs to itself, so the helpers propposed ^^^ could be completely general: <?xml version="1.0" ?> <RHTS_TEST> <test_info> <test_id> [...] </purpose> </test_info> <phase name='ROOT' result='FAIL' score='5'> <phase name="Preparing something" result="PASS" score="0" type="ABORT"> <test message="Checking something" result='PASS'> ...output of the assert... </test> </phase> <phase name="Testing separate files" result="FAIL" score="5" type="FAIL"> <phase name="File1" result="FAIL" score="2" type="FAIL"> <test message="Checking something" result='FAIL'> ...output of the assert... </test> <test message="Checking something else" result='FAIL'> ...output of the assert... </test> <test message="Doing something" result='FAIL'> ...output of the assert... </test> <test message="Doing something else" result='PASS'> ...output of the assert... </test> </phase> <phase name="File2" result="FAIL" score="3" type="FAIL"> <test message="Checking something" result='FAIL'> ...output of the assert... </test> <test message="Checking something else" result='FAIL'> ...output of the assert... </test> <test message="Doing something" result='FAIL'> ...output of the assert... </test> <test message="Doing something else" result='PASS'> ...output of the assert... </test> </phase> </phase> <phase name="Cleaning up" result="PASS" score="0" type="WARN"> <test message="Removing xml files" result='PASS'> ...output... </test> </phase> </phase> </RHTS_TEST>
Err, just a typo in assert "ROOT/Testing separate files/File1/Doing something" - it's result should be "PASS", so "ROOT/Testing separate files/File1" really have "score='2'".
Agreed on this. I *think* that journalling could use a small rewrite (objectify), so it could be incorporated...
The discussion about phase-nesting should be moved to bug 485457.
Adding just one more small RFE related to the phase summary: Total number of bad asserts (in case of FAIL) or good asserts (in case of PASS) could be reported as SCORE to RHTS. In this way it'd be clear at the first sight in the web UI how many asserts really caused the PASS/FAIL.
I'm 99% sure this worked in the past. I'll check why it stopped to do that.
Created attachment 359803 [details] patch + tests Here's the patch and unit tests - In case you don't mind including this feature before phase nesting is fully finished.
Created attachment 359808 [details] fixed patch New version fixing stupid mistake (counting asserts instead of phases). Tests included.
seems good
Well, guys... it is quite a long time and we are still looking forward to the solution. What about to apply that patch? :)
Created attachment 513607 [details] Implement summary of phase results in logfile
Re-formated patch. Slightly modified tests & ran them.
Created attachment 514763 [details] testsuite improvement This broke the 'rlAssertRpm increases SCORE when package is not found' test. During investigation, I've found 2 things: 1) That test used a custom output | head | tail | grep construction instead of assertGoodBad function (it was probably old): I've converted it to assertGoodBad 2) I've found out that the additional output produced by this RFE could interfere with how assertGoodBad works: until now, it checked for presence of 'X good' or 'Y bad' only, which could be satisfied even with new 'Phases: 6 good, 6 bad' line when sufficiently unlucky. So I've improved assertGoodBad function to search for stronger regexp.
Both pushed: http://git.fedorahosted.org/git?p=beakerlib.git;a=commit;h=25d4133f9fc3a5c47d274220bd9a8b01c826c61e http://git.fedorahosted.org/git?p=beakerlib.git;a=commit;h=9324f75ce76bb6454b1d45c46abcc168f70bcc28
beakerlib-1.6-1 is already in fedora