Bug 1128114 - Support xUnit result reporting
Summary: Support xUnit result reporting
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: beakerlib
Version: 27
Hardware: Unspecified
OS: Unspecified
low
unspecified
Target Milestone: ---
Assignee: Ondrej Hudlicky
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: 1173652 Fedora-beakerlib
TreeView+ depends on / blocked
 
Reported: 2014-08-08 10:23 UTC by Namita Soman
Modified: 2018-11-30 19:29 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-11-30 19:29:09 UTC


Attachments (Terms of Use)
junit style XML from nose that plays nice with Polarion (1.32 KB, text/plain)
2014-09-04 17:55 UTC, Ben Turner
no flags Details
random journal.xml downloaded from beaker (5.05 KB, application/xml)
2015-03-25 15:59 UTC, Ales Zelinka
no flags Details
journal in the xunit format (2.08 KB, application/xml)
2015-03-25 16:01 UTC, Ales Zelinka
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Bugzilla 1123244 None None None Never

Internal Links: 1123244

Description Namita Soman 2014-08-08 10:23:40 UTC
Description of problem:
For log results to be consumed by CI, we would like to be able to export beaker job results in xUnit format using beakerlib.

There is a bz requesting the same with Beaker - bz 1123244, but need this support in beakerlib as well.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Jenny Severance 2014-08-08 19:31:15 UTC
This is actually in support of using the Polarion APIs to get the results of the tests back into Polarion.  The APIs want the results in xUnit format.  Everyone would need to do this so let's make it part of beakerlib.

Comment 2 Jenny Severance 2014-08-08 19:38:09 UTC
Just a side comment, beakerlib is different than beaker the provisioning system.  We need this in beakerlib, so we can take advantage of other provisioning options when appropriate like openStack and docker containers.  If we keep confusing the two and making beaker provisioning system tied to beakerlib, we will not be flexible.  We need to be able to target specific hardware etc so beaker needs to be an option.  Let's just de-couple them and make sure we are not making ourselves dependent on one way to provision.

Comment 3 Petr Muller 2014-08-08 22:32:08 UTC
1) There's nothing in BeakerLib specific to Beaker (the name is a historic artifact), just the RPMs shipped in Beaker are configured to use Beaker API to report resuls, submit logs etc.

2) Emitting xUnit should not be that hard. There is no concept of "Phase" in xUnit, so my first take on the mapping:

<testsuites> = single BL test run
<property> = header records
<testsuite> = phase
<testcase> = assert

Comment 4 Gowrishankar Rajaiyan 2014-09-02 05:38:04 UTC
Also, in a multi-host environment the xUnit results should be unique instead of separate ones per host.

Comment 5 Fedora Admin XMLRPC Client 2014-09-02 12:13:44 UTC
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 6 Ales Zelinka 2014-09-04 09:45:05 UTC
Is there a schema for the xUnit accepted by Polarion?

(In reply to Gowrishankar Rajaiyan from comment #4)
> Also, in a multi-host environment the xUnit results should be unique instead
> of separate ones per host.

beakerlib runs on each worker machine and only logs what happens on that machine. Creating an aggregated log over all machines in a multihost test (be it classic journal or the xUnit log) would require some form of log synchronization over network. Or gathering and aggregating the logs outside of beakerlib. 

I guess the usual applies here: Patches are welcome!

Comment 7 Ben Turner 2014-09-04 17:54:03 UTC
@Ales, Polarion should accept any junit style XML.  I attached an example XML that was generated from nose, I was able to import it to Polarion successfully.

Comment 8 Ben Turner 2014-09-04 17:55:01 UTC
Created attachment 934564 [details]
junit style XML from nose that plays nice with Polarion

Comment 9 Petr Muller 2014-09-04 18:19:12 UTC
I plan to do this in September (not the multihost part, though)

Comment 10 Petr Muller 2014-11-11 15:39:36 UTC
Seems like this mapping will work better:

<testsuites> : A set of BL tests
<testsuite>  : Single BL test
<property>   : Single header record, not having a good <testsuite> attribute
<testcase>   : Phase
<failure>/<error>  : Failed assert

I'm dropping this scheme here for reference:
https://svn.jenkins-ci.org/trunk/hudson/dtkit/dtkit-format/dtkit-junit-model/src/main/resources/com/thalesgroup/dtkit/junit/model/xsd/junit-4.xsd

Comment 11 Petr Muller 2014-11-20 15:12:24 UTC
I have it working, see my GH branch:

https://github.com/petr-muller/beakerlib/compare/xunit-xslt

The general approach is as follows. I'm using XSLT template to translate BeakerLib journal XML to xUnit XML. The binary required is called 'xsltproc', coming from package 'libxslt', which seems to be available & installed by default (at least in Beaker) on RHEL5, RHEL6, RHEL7. This means we can easily do the transformation *in* BeakerLib functions rlPhaseEnd and rlJournalEnd and then either submit the result using harness or report its location locally. The other approach would be to translate the journal by client tool or harness, but there is no reason doing that, as xsltproc is easily available everywhere.

I have included tests for both the XSLT transformation itself (=sanitized real result journals and their "golden" xUnit counterparts) and for testing that rlJournalEnd and rlPhaseEnd actually produce the xUnit output too.

On RHEL5, RHEL6, RHEL7, F20: No testsuite regressions. XSLT tests all PASS. xUnit tests both PASS. 

This is needed for CI progress (Jenkins eats xUnit): I would like this to be reviewed soon. When it is OK for upstream, I will push and build an experimental package with this feature for use in CI.

Comment 12 Jaroslav Reznik 2015-03-03 17:03:21 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 22 development cycle.
Changing version to '22'.

More information and reason for this action is here:
https://fedoraproject.org/wiki/Fedora_Program_Management/HouseKeeping/Fedora22

Comment 13 Petr Muller 2015-03-24 17:10:52 UTC
Is there anything I can do to make this patch go in?

Comment 15 Ales Zelinka 2015-03-25 15:57:19 UTC
I took xunit.xsl from your github beakerlib/xunit-xslt (comment #11), used it to convert journal.xml (attached) to journal.xslt (attached) and then checked its validity on http://www.freeformatter.com/xml-validator-xsd.html using http://windyroad.org/dl/Open%20Source/JUnit.xsd. 

These are the errors I got:

    	
    Cvc-complex-type.3.2.2: Attribute 'id' Is Not Allowed To Appear In Element 'testsuite'.. Line '1', Column '201'.
    Cvc-complex-type.3.2.2: Attribute 'package' Is Not Allowed To Appear In Element 'testsuite'.. Line '1', Column '201'.
    Cvc-complex-type.4: Attribute 'timestamp' Must Appear On Element 'testsuite'.. Line '1', Column '201'.
    Cvc-complex-type.4: Attribute 'time' Must Appear On Element 'testsuite'.. Line '1', Column '201'.
    Cvc-complex-type.3.2.2: Attribute 'assertions' Is Not Allowed To Appear In Element 'testcase'.. Line '15', Column '41'.
    Cvc-complex-type.4: Attribute 'classname' Must Appear On Element 'testcase'.. Line '15', Column '41'.
    Cvc-complex-type.4: Attribute 'time' Must Appear On Element 'testcase'.. Line '15', Column '41'.
    Cvc-complex-type.4: Attribute 'type' Must Appear On Element 'failure'.. Line '16', Column '66'.
    Cvc-complex-type.3.2.2: Attribute 'assertions' Is Not Allowed To Appear In Element 'testcase'.. Line '18', Column '41'.
    Cvc-complex-type.4: Attribute 'classname' Must Appear On Element 'testcase'.. Line '18', Column '41'.
    Cvc-complex-type.4: Attribute 'time' Must Appear On Element 'testcase'.. Line '18', Column '41'.
    Cvc-complex-type.4: Attribute 'type' Must Appear On Element 'error'.. Line '19', Column '64'.
    Cvc-complex-type.2.4.d: Invalid Content Was Found Starting With Element 'error'. No Child Element Is Expected At This Point.. Line '20', Column '61'.
    Cvc-complex-type.4: Attribute 'type' Must Appear On Element 'error'.. Line '20', Column '61'.
    Cvc-complex-type.4: Attribute 'type' Must Appear On Element 'error'.. Line '21', Column '60'.
    Cvc-complex-type.4: Attribute 'type' Must Appear On Element 'error'.. Line '22', Column '78'.
    Cvc-complex-type.4: Attribute 'type' Must Appear On Element 'error'.. Line '23', Column '78'.
    Cvc-complex-type.4: Attribute 'type' Must Appear On Element 'error'.. Line '24', Column '77'.
    Cvc-complex-type.4: Attribute 'type' Must Appear On Element 'error'.. Line '25', Column '79'.
    Cvc-complex-type.4: Attribute 'type' Must Appear On Element 'error'.. Line '26', Column '107'.
    Cvc-complex-type.4: Attribute 'type' Must Appear On Element 'error'.. Line '27', Column '106'.
    Cvc-complex-type.4: Attribute 'type' Must Appear On Element 'error'.. Line '28', Column '99'.
    Cvc-complex-type.3.2.2: Attribute 'assertions' Is Not Allowed To Appear In Element 'testcase'.. Line '30', Column '44'.
    Cvc-complex-type.4: Attribute 'classname' Must Appear On Element 'testcase'.. Line '30', Column '44'.
    Cvc-complex-type.4: Attribute 'time' Must Appear On Element 'testcase'.. Line '30', Column '44'.
    Cvc-complex-type.2.4.b: The Content Of Element 'testsuite' Is Not Complete. One Of '{testcase, System-out}' Is Expected.. Line '31', Column '13'.

Comment 17 Ales Zelinka 2015-03-25 15:59:16 UTC
Created attachment 1006368 [details]
random journal.xml downloaded from beaker

Comment 18 Ales Zelinka 2015-03-25 16:01:35 UTC
Created attachment 1006369 [details]
journal in the xunit format

ignore the xslt extension, it's xml/xunit, I just got confused by all the x-words...

Comment 20 Petr Muller 2015-03-25 16:59:56 UTC
I would suggest to respect the missing attributes/elements warnings (to fix consumers which expect these to exist) and ignore the warnings about forbidden elements/attributes. If some consumer is particularly picky in not accepting XMLs with additional attributes/elements, I would handle that when needed.

Comment 21 Ales Zelinka 2015-03-26 09:04:03 UTC
Can we satisfy both CI/jenkins and the "Picky customer"? C#1 suggests this bug is mostly about the later.

Comment 23 Petr Muller 2015-03-26 09:36:03 UTC
OK, we agreed to go with the following approach:

1) I'll add the missing fields and investigate what is the problem with the two messages unrelated to attributes:

Cvc-complex-type.2.4.d: Invalid Content Was Found Starting With Element 'error'. No Child Element Is Expected At This Point.. Line '20', Column '61'.
Cvc-complex-type.2.4.b: The Content Of Element 'testsuite' Is Not Complete. One Of '{testcase, System-out}' Is Expected.. Line '31', Column '13'

2) We'll leave the additional attributes (those flagged with 'Is Not Allowed to Appear In Element') in for now

3) If it turns out that Polarion really cannot import xUnit XMLs with additional attributes, we'll solve it separately as needed.

Comment 26 Fedora End Of Life 2016-07-19 20:47:04 UTC
Fedora 22 changed to end-of-life (EOL) status on 2016-07-19. Fedora 22 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 29 Fedora End Of Life 2017-07-25 18:41:36 UTC
This message is a reminder that Fedora 24 is nearing its end of life.
Approximately 2 (two) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 24. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '24'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 24 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Comment 30 Jan Kurik 2017-08-15 09:04:19 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 27 development cycle.
Changing version to '27'.

Comment 31 Ben Cotton 2018-11-27 17:18:30 UTC
This message is a reminder that Fedora 27 is nearing its end of life.
On 2018-Nov-30  Fedora will stop maintaining and issuing updates for
Fedora 27. It is Fedora's policy to close all bug reports from releases
that are no longer maintained. At that time this bug will be closed as
EOL if it remains open with a Fedora  'version' of '27'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 27 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 32 Ben Cotton 2018-11-30 19:29:09 UTC
Fedora 27 changed to end-of-life (EOL) status on 2018-11-30. Fedora 27 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.