Bug 1473130 - Adjust test execution settings in Python %check section
Summary: Adjust test execution settings in Python %check section
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: python3
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Miro Hrončok
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-07-20 05:37 UTC by Nick Coghlan
Modified: 2019-01-07 12:20 UTC (History)
7 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2019-01-07 12:16:46 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Nick Coghlan 2017-07-20 05:37:12 UTC
The Python spec file currently runs the test suite using the "--verbose" flag, which makes it hard to find the details of which tests actually failed when something goes wrong.

Two other flags are likely to produce easier to read results (these can both be set at the same time):

-w: Immediately re-runs failed tests in verbose mode
-W: Prints captured output when a test fails

Another potentially useful option for the %check section may be "--failfast", which will abort the test run as soon as any test fails, rather than always continuing to run the full test suite.

Some other ideas potentially worth exploring:

* using the `--findleaks` option forces serialisation of test execution on a single CPU. If it is omitted (effectively reclassifying leak hunting in the Python runtime as an integration test rather than a unit test), then the `-j` option could instead be passed to run tests in parallel across multiple CPUs, which may significantly reduce the overall RPM build time (since the test suite gets run twice, once against the debug binaries, once against the release binaries). If we did this, then it would make sense to instead set up a Taskotron task to replace it in the Fedora release pipeline.

* the `--slowest` option lists the 10 slowest tests at the end of the run

Comment 1 Charalampos Stratakis 2017-07-20 16:02:38 UTC
First step done for F27 and F26 [0]

[0] http://pkgs.fedoraproject.org/cgit/rpms/python3.git/commit/?id=5b6a3e0e34f46702773ad1082df3b782c1a8b7ac

Comment 2 Jan Kurik 2017-08-15 06:25:57 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 27 development cycle.
Changing version to '27'.

Comment 3 Petr Viktorin (pviktori) 2018-01-18 14:08:54 UTC
[x] -wW was added
[x] --failfast is not worth it
[ ] --findleaks removal is waiting for Taskotron or Standard test interface
[x] --slowest is there already

Comment 4 Fedora End Of Life 2018-02-20 15:33:43 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 28 development cycle.
Changing version to '28'.

Comment 5 Petr Viktorin (pviktori) 2018-04-19 13:59:21 UTC
Miro is looking into running integration tests first.

Comment 6 Miro Hrončok 2018-09-24 12:37:15 UTC
I'll try to add --findleaks to:

  https://src.fedoraproject.org/tests/python

Comment 8 Miro Hrončok 2019-01-07 12:16:46 UTC
This has been done. --findleaks is now only used on the Fedora CI.

Remarks: The CI only runs on x86_64 (we agreed that this is OK for now).

RFE @ Fedora CI: https://pagure.io/fedora-ci/general/issue/16

Comment 9 Petr Viktorin (pviktori) 2019-01-07 12:20:57 UTC
Miro, thanks for your CI work. The progress on Fedora Python's testing in the last months is pretty amazing.


Note You need to log in before you can comment on or make changes to this bug.