Bug 1415309

Summary:	jUnit xml results not jenkins compatible
Product:	[Retired] Beaker	Reporter:	John Bieren <jbieren>
Component:	tests	Assignee:	Dan Callaghan <dcallagh>
Status:	CLOSED CURRENTRELEASE	QA Contact:	tools-bugs <tools-bugs>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	23	CC:	bpeck, dcallagh, jburke, mjia, rjoost
Target Milestone:	24.2	Keywords:	Patch
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-03-30 03:23:08 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description John Bieren 2017-01-20 20:31:07 UTC

Description of problem:
bkr job-results --format=junit-xml J:XXXXXX does not produce output that is compatible with jenkins xunit publisher plugin.  The xunit plugin complains because there is one testcase per beaker task that does not have a name (it has a classname, but not just 'name').  There needs to be a name="<some string>" for each testcase.  
Furthermore, the current xunit results report more successes than actually occurred, as there will always be a testcase with no error message per task, even if the entire task failed.


Version-Release number of selected component (if applicable):
bkr --version
23.3

How reproducible:
Always

Steps to Reproduce:
1. bkr job-results --format=junit-xml J:XXXXXX
2.
3.

Actual results:
xUnit file with a testcase without a name= per beaker task.
<testcase ... </testcase> without any error message per beaker task in the result of a beaker task that actually failed.


Expected results:
xUnit file that has a name= field per testcase so that it is jenkins compatible.
Some error message present in the testcases that represent the actual task name (these are currently the ones with no name=) in the case that the task fails, to prevent success from being reported by xunit parsers in these cases.


Additional info:
Example of false success in next comment.

Comment 2 Bill Peck 2017-02-13 20:37:08 UTC

WIP https://gerrit.beaker-project.org/#/c/5607/

May need to update test cases.

Comment 3 Dan Callaghan 2017-02-22 13:42:09 UTC

(In reply to John Bieren from comment #1)
> As you can see, lines such as
> <testcase classname="/distribution/install">
> have no name= field, only a classname=
> This causes the jenkins xunit publisher plugin to fail.  Also,
> because that same testcase has no error message, despite the beaker job in
> the UI clearly showing that that task failed, that testcase will report as a
> success by xUnit parsers, which is incorrect.

Just for future reference... That first <testcase/> element without any name="" attribute represents the overall task (and then the subsequent <testcase/> elements are for each result in the task).

The only reason we have it is so that we have a place to put the task logs. Otherwise we could just skip it and only report back the individual results, which would actually be closer to the JUnit model.

Comment 4 Dan Callaghan 2017-02-22 13:47:40 UTC

(In reply to Bill Peck from comment #2)

Thanks for the patch, Bill! It looks fine to me.

To summarize, the changes are:

* it now always has skipped="", failures="", and errors="" attributes
* skipped tests are counted in the overall total
* the <testcase/> element representing the overall task result now has name="(main)" and contains <failure/> or <error/> if it was Fail or Aborted

Comment 5 Bill Peck 2017-02-22 13:54:14 UTC

Hi Dan,

Your summary is correct.  Should I rebase the patch against release-0.24?

Comment 6 Dan Callaghan 2017-02-22 13:57:47 UTC

I used some Gerrit magic to take care of that.

Bill, have you tried using the new JUnit XML output in Jenkins to confirm if it fixes the issues Johnny was hitting?

Comment 7 Bill Peck 2017-02-22 13:59:11 UTC

I confirmed the output looked correctly by using Beaker in a Box but haven't imported it into jenkins yet.  I'll do that Today.

Comment 8 John Bieren 2017-02-22 14:15:53 UTC

Bill, if you send me an example new JUnit XML output I can plug it into my jenkins test job and confirm it fixes it pretty easily.  I already have a jenkins job set up for testing JUnit files.

Comment 9 Roman Joost 2017-03-06 23:56:57 UTC

We'll have to do QE on the items before we can tag a release.

Comment 10 Roman Joost 2017-03-22 06:00:30 UTC

Verified by looking at JUnit.xml view on my local Beaker instance 24.2.git.9.d4c983d from a job which ran. Used comment 4 as a comparison.

Comment 11 Dan Callaghan 2017-03-30 03:23:08 UTC

Beaker 24.2 has been released.