Bug 1415309 - jUnit xml results not jenkins compatible
Summary: jUnit xml results not jenkins compatible
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Beaker
Classification: Community
Component: tests
Version: 23
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified vote
Target Milestone: 24.2
Assignee: Dan Callaghan
QA Contact: tools-bugs
URL:
Whiteboard:
Keywords: Patch
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-20 20:31 UTC by John Bieren
Modified: 2017-03-30 03:23 UTC (History)
5 users (show)

(edit)
Clone Of:
(edit)
Last Closed: 2017-03-30 03:23:08 UTC


Attachments (Terms of Use)

Description John Bieren 2017-01-20 20:31:07 UTC
Description of problem:
bkr job-results --format=junit-xml J:XXXXXX does not produce output that is compatible with jenkins xunit publisher plugin.  The xunit plugin complains because there is one testcase per beaker task that does not have a name (it has a classname, but not just 'name').  There needs to be a name="<some string>" for each testcase.  
Furthermore, the current xunit results report more successes than actually occurred, as there will always be a testcase with no error message per task, even if the entire task failed.


Version-Release number of selected component (if applicable):
bkr --version
23.3

How reproducible:
Always

Steps to Reproduce:
1. bkr job-results --format=junit-xml J:XXXXXX
2.
3.

Actual results:
xUnit file with a testcase without a name= per beaker task.
<testcase ... </testcase> without any error message per beaker task in the result of a beaker task that actually failed.


Expected results:
xUnit file that has a name= field per testcase so that it is jenkins compatible.
Some error message present in the testcases that represent the actual task name (these are currently the ones with no name=) in the case that the task fails, to prevent success from being reported by xunit parsers in these cases.


Additional info:
Example of false success in next comment.

Comment 2 Bill Peck 2017-02-13 20:37:08 UTC
WIP https://gerrit.beaker-project.org/#/c/5607/

May need to update test cases.

Comment 3 Dan Callaghan 2017-02-22 13:42:09 UTC
(In reply to John Bieren from comment #1)
> As you can see, lines such as
> <testcase classname="/distribution/install">
> have no name= field, only a classname=
> This causes the jenkins xunit publisher plugin to fail.  Also,
> because that same testcase has no error message, despite the beaker job in
> the UI clearly showing that that task failed, that testcase will report as a
> success by xUnit parsers, which is incorrect.

Just for future reference... That first <testcase/> element without any name="" attribute represents the overall task (and then the subsequent <testcase/> elements are for each result in the task).

The only reason we have it is so that we have a place to put the task logs. Otherwise we could just skip it and only report back the individual results, which would actually be closer to the JUnit model.

Comment 4 Dan Callaghan 2017-02-22 13:47:40 UTC
(In reply to Bill Peck from comment #2)

Thanks for the patch, Bill! It looks fine to me.

To summarize, the changes are:

* it now always has skipped="", failures="", and errors="" attributes
* skipped tests are counted in the overall total
* the <testcase/> element representing the overall task result now has name="(main)" and contains <failure/> or <error/> if it was Fail or Aborted

Comment 5 Bill Peck 2017-02-22 13:54:14 UTC
Hi Dan,

Your summary is correct.  Should I rebase the patch against release-0.24?

Comment 6 Dan Callaghan 2017-02-22 13:57:47 UTC
I used some Gerrit magic to take care of that.

Bill, have you tried using the new JUnit XML output in Jenkins to confirm if it fixes the issues Johnny was hitting?

Comment 7 Bill Peck 2017-02-22 13:59:11 UTC
I confirmed the output looked correctly by using Beaker in a Box but haven't imported it into jenkins yet.  I'll do that Today.

Comment 8 John Bieren 2017-02-22 14:15:53 UTC
Bill, if you send me an example new JUnit XML output I can plug it into my jenkins test job and confirm it fixes it pretty easily.  I already have a jenkins job set up for testing JUnit files.

Comment 9 Roman Joost 2017-03-06 23:56:57 UTC
We'll have to do QE on the items before we can tag a release.

Comment 10 Roman Joost 2017-03-22 06:00:30 UTC
Verified by looking at JUnit.xml view on my local Beaker instance 24.2.git.9.d4c983d from a job which ran. Used comment 4 as a comparison.

Comment 11 Dan Callaghan 2017-03-30 03:23:08 UTC
Beaker 24.2 has been released.


Note You need to log in before you can comment on or make changes to this bug.