Description of problem: bkr job-results --format=junit-xml J:XXXXXX does not produce output that is compatible with jenkins xunit publisher plugin. The xunit plugin complains because there is one testcase per beaker task that does not have a name (it has a classname, but not just 'name'). There needs to be a name="<some string>" for each testcase. Furthermore, the current xunit results report more successes than actually occurred, as there will always be a testcase with no error message per task, even if the entire task failed. Version-Release number of selected component (if applicable): bkr --version 23.3 How reproducible: Always Steps to Reproduce: 1. bkr job-results --format=junit-xml J:XXXXXX 2. 3. Actual results: xUnit file with a testcase without a name= per beaker task. <testcase ... </testcase> without any error message per beaker task in the result of a beaker task that actually failed. Expected results: xUnit file that has a name= field per testcase so that it is jenkins compatible. Some error message present in the testcases that represent the actual task name (these are currently the ones with no name=) in the case that the task fails, to prevent success from being reported by xunit parsers in these cases. Additional info: Example of false success in next comment.
WIP https://gerrit.beaker-project.org/#/c/5607/ May need to update test cases.
(In reply to John Bieren from comment #1) > As you can see, lines such as > <testcase classname="/distribution/install"> > have no name= field, only a classname= > This causes the jenkins xunit publisher plugin to fail. Also, > because that same testcase has no error message, despite the beaker job in > the UI clearly showing that that task failed, that testcase will report as a > success by xUnit parsers, which is incorrect. Just for future reference... That first <testcase/> element without any name="" attribute represents the overall task (and then the subsequent <testcase/> elements are for each result in the task). The only reason we have it is so that we have a place to put the task logs. Otherwise we could just skip it and only report back the individual results, which would actually be closer to the JUnit model.
(In reply to Bill Peck from comment #2) Thanks for the patch, Bill! It looks fine to me. To summarize, the changes are: * it now always has skipped="", failures="", and errors="" attributes * skipped tests are counted in the overall total * the <testcase/> element representing the overall task result now has name="(main)" and contains <failure/> or <error/> if it was Fail or Aborted
Hi Dan, Your summary is correct. Should I rebase the patch against release-0.24?
I used some Gerrit magic to take care of that. Bill, have you tried using the new JUnit XML output in Jenkins to confirm if it fixes the issues Johnny was hitting?
I confirmed the output looked correctly by using Beaker in a Box but haven't imported it into jenkins yet. I'll do that Today.
Bill, if you send me an example new JUnit XML output I can plug it into my jenkins test job and confirm it fixes it pretty easily. I already have a jenkins job set up for testing JUnit files.
We'll have to do QE on the items before we can tag a release.
Verified by looking at JUnit.xml view on my local Beaker instance 24.2.git.9.d4c983d from a job which ran. Used comment 4 as a comparison.
Beaker 24.2 has been released.