Bug 1128114

This is actually in support of using the Polarion APIs to get the results of the tests back into Polarion.  The APIs want the results in xUnit format.  Everyone would need to do this so let's make it part of beakerlib.

Just a side comment, beakerlib is different than beaker the provisioning system.  We need this in beakerlib, so we can take advantage of other provisioning options when appropriate like openStack and docker containers.  If we keep confusing the two and making beaker provisioning system tied to beakerlib, we will not be flexible.  We need to be able to target specific hardware etc so beaker needs to be an option.  Let's just de-couple them and make sure we are not making ourselves dependent on one way to provision.

1) There's nothing in BeakerLib specific to Beaker (the name is a historic artifact), just the RPMs shipped in Beaker are configured to use Beaker API to report resuls, submit logs etc.

2) Emitting xUnit should not be that hard. There is no concept of "Phase" in xUnit, so my first take on the mapping:

<testsuites> = single BL test run
<property> = header records
<testsuite> = phase
<testcase> = assert

Also, in a multi-host environment the xUnit results should be unique instead of separate ones per host.

This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Is there a schema for the xUnit accepted by Polarion?

(In reply to Gowrishankar Rajaiyan from comment #4)
> Also, in a multi-host environment the xUnit results should be unique instead
> of separate ones per host.

beakerlib runs on each worker machine and only logs what happens on that machine. Creating an aggregated log over all machines in a multihost test (be it classic journal or the xUnit log) would require some form of log synchronization over network. Or gathering and aggregating the logs outside of beakerlib. 

I guess the usual applies here: Patches are welcome!

@Ales, Polarion should accept any junit style XML.  I attached an example XML that was generated from nose, I was able to import it to Polarion successfully.

Created attachment 934564 [details]
junit style XML from nose that plays nice with Polarion

I plan to do this in September (not the multihost part, though)

Seems like this mapping will work better:

<testsuites> : A set of BL tests
<testsuite>  : Single BL test
<property>   : Single header record, not having a good <testsuite> attribute
<testcase>   : Phase
<failure>/<error>  : Failed assert

I'm dropping this scheme here for reference:
https://svn.jenkins-ci.org/trunk/hudson/dtkit/dtkit-format/dtkit-junit-model/src/main/resources/com/thalesgroup/dtkit/junit/model/xsd/junit-4.xsd

I have it working, see my GH branch:

https://github.com/petr-muller/beakerlib/compare/xunit-xslt

The general approach is as follows. I'm using XSLT template to translate BeakerLib journal XML to xUnit XML. The binary required is called 'xsltproc', coming from package 'libxslt', which seems to be available & installed by default (at least in Beaker) on RHEL5, RHEL6, RHEL7. This means we can easily do the transformation *in* BeakerLib functions rlPhaseEnd and rlJournalEnd and then either submit the result using harness or report its location locally. The other approach would be to translate the journal by client tool or harness, but there is no reason doing that, as xsltproc is easily available everywhere.

I have included tests for both the XSLT transformation itself (=sanitized real result journals and their "golden" xUnit counterparts) and for testing that rlJournalEnd and rlPhaseEnd actually produce the xUnit output too.

On RHEL5, RHEL6, RHEL7, F20: No testsuite regressions. XSLT tests all PASS. xUnit tests both PASS. 

This is needed for CI progress (Jenkins eats xUnit): I would like this to be reviewed soon. When it is OK for upstream, I will push and build an experimental package with this feature for use in CI.

This bug appears to have been reported against 'rawhide' during the Fedora 22 development cycle.
Changing version to '22'.

More information and reason for this action is here:
https://fedoraproject.org/wiki/Fedora_Program_Management/HouseKeeping/Fedora22

Is there anything I can do to make this patch go in?

I took xunit.xsl from your github beakerlib/xunit-xslt (comment #11), used it to convert journal.xml (attached) to journal.xslt (attached) and then checked its validity on http://www.freeformatter.com/xml-validator-xsd.html using http://windyroad.org/dl/Open%20Source/JUnit.xsd. 

These are the errors I got:

    	
    Cvc-complex-type.3.2.2: Attribute 'id' Is Not Allowed To Appear In Element 'testsuite'.. Line '1', Column '201'.
    Cvc-complex-type.3.2.2: Attribute 'package' Is Not Allowed To Appear In Element 'testsuite'.. Line '1', Column '201'.
    Cvc-complex-type.4: Attribute 'timestamp' Must Appear On Element 'testsuite'.. Line '1', Column '201'.
    Cvc-complex-type.4: Attribute 'time' Must Appear On Element 'testsuite'.. Line '1', Column '201'.
    Cvc-complex-type.3.2.2: Attribute 'assertions' Is Not Allowed To Appear In Element 'testcase'.. Line '15', Column '41'.
    Cvc-complex-type.4: Attribute 'classname' Must Appear On Element 'testcase'.. Line '15', Column '41'.
    Cvc-complex-type.4: Attribute 'time' Must Appear On Element 'testcase'.. Line '15', Column '41'.
    Cvc-complex-type.4: Attribute 'type' Must Appear On Element 'failure'.. Line '16', Column '66'.
    Cvc-complex-type.3.2.2: Attribute 'assertions' Is Not Allowed To Appear In Element 'testcase'.. Line '18', Column '41'.
    Cvc-complex-type.4: Attribute 'classname' Must Appear On Element 'testcase'.. Line '18', Column '41'.
    Cvc-complex-type.4: Attribute 'time' Must Appear On Element 'testcase'.. Line '18', Column '41'.
    Cvc-complex-type.4: Attribute 'type' Must Appear On Element 'error'.. Line '19', Column '64'.
    Cvc-complex-type.2.4.d: Invalid Content Was Found Starting With Element 'error'. No Child Element Is Expected At This Point.. Line '20', Column '61'.
    Cvc-complex-type.4: Attribute 'type' Must Appear On Element 'error'.. Line '20', Column '61'.
    Cvc-complex-type.4: Attribute 'type' Must Appear On Element 'error'.. Line '21', Column '60'.
    Cvc-complex-type.4: Attribute 'type' Must Appear On Element 'error'.. Line '22', Column '78'.
    Cvc-complex-type.4: Attribute 'type' Must Appear On Element 'error'.. Line '23', Column '78'.
    Cvc-complex-type.4: Attribute 'type' Must Appear On Element 'error'.. Line '24', Column '77'.
    Cvc-complex-type.4: Attribute 'type' Must Appear On Element 'error'.. Line '25', Column '79'.
    Cvc-complex-type.4: Attribute 'type' Must Appear On Element 'error'.. Line '26', Column '107'.
    Cvc-complex-type.4: Attribute 'type' Must Appear On Element 'error'.. Line '27', Column '106'.
    Cvc-complex-type.4: Attribute 'type' Must Appear On Element 'error'.. Line '28', Column '99'.
    Cvc-complex-type.3.2.2: Attribute 'assertions' Is Not Allowed To Appear In Element 'testcase'.. Line '30', Column '44'.
    Cvc-complex-type.4: Attribute 'classname' Must Appear On Element 'testcase'.. Line '30', Column '44'.
    Cvc-complex-type.4: Attribute 'time' Must Appear On Element 'testcase'.. Line '30', Column '44'.
    Cvc-complex-type.2.4.b: The Content Of Element 'testsuite' Is Not Complete. One Of '{testcase, System-out}' Is Expected.. Line '31', Column '13'.

Created attachment 1006368 [details]
random journal.xml downloaded from beaker

Created attachment 1006369 [details]
journal in the xunit format

ignore the xslt extension, it's xml/xunit, I just got confused by all the x-words...

I would suggest to respect the missing attributes/elements warnings (to fix consumers which expect these to exist) and ignore the warnings about forbidden elements/attributes. If some consumer is particularly picky in not accepting XMLs with additional attributes/elements, I would handle that when needed.

Can we satisfy both CI/jenkins and the "Picky customer"? C#1 suggests this bug is mostly about the later.

OK, we agreed to go with the following approach:

1) I'll add the missing fields and investigate what is the problem with the two messages unrelated to attributes:

Cvc-complex-type.2.4.d: Invalid Content Was Found Starting With Element 'error'. No Child Element Is Expected At This Point.. Line '20', Column '61'.
Cvc-complex-type.2.4.b: The Content Of Element 'testsuite' Is Not Complete. One Of '{testcase, System-out}' Is Expected.. Line '31', Column '13'

2) We'll leave the additional attributes (those flagged with 'Is Not Allowed to Appear In Element') in for now

3) If it turns out that Polarion really cannot import xUnit XMLs with additional attributes, we'll solve it separately as needed.

Fedora 22 changed to end-of-life (EOL) status on 2016-07-19. Fedora 22 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

This message is a reminder that Fedora 24 is nearing its end of life.
Approximately 2 (two) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 24. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '24'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 24 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

This bug appears to have been reported against 'rawhide' during the Fedora 27 development cycle.
Changing version to '27'.

This message is a reminder that Fedora 27 is nearing its end of life.
On 2018-Nov-30  Fedora will stop maintaining and issuing updates for
Fedora 27. It is Fedora's policy to close all bug reports from releases
that are no longer maintained. At that time this bug will be closed as
EOL if it remains open with a Fedora  'version' of '27'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 27 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Fedora 27 changed to end-of-life (EOL) status on 2018-11-30. Fedora 27 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.