Bug 1473130

Summary:	Adjust test execution settings in Python %check section
Product:	[Fedora] Fedora	Reporter:	Nick Coghlan <ncoghlan>
Component:	python3	Assignee:	Miro Hrončok <mhroncok>
Status:	CLOSED RAWHIDE	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	rawhide	CC:	bkabrda, cstratak, mhroncok, pviktori, rkuska, tomspur, torsava
Target Milestone:	---	Keywords:	FutureFeature
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-01-07 12:16:46 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Nick Coghlan 2017-07-20 05:37:12 UTC

The Python spec file currently runs the test suite using the "--verbose" flag, which makes it hard to find the details of which tests actually failed when something goes wrong.

Two other flags are likely to produce easier to read results (these can both be set at the same time):

-w: Immediately re-runs failed tests in verbose mode
-W: Prints captured output when a test fails

Another potentially useful option for the %check section may be "--failfast", which will abort the test run as soon as any test fails, rather than always continuing to run the full test suite.

Some other ideas potentially worth exploring:

* using the `--findleaks` option forces serialisation of test execution on a single CPU. If it is omitted (effectively reclassifying leak hunting in the Python runtime as an integration test rather than a unit test), then the `-j` option could instead be passed to run tests in parallel across multiple CPUs, which may significantly reduce the overall RPM build time (since the test suite gets run twice, once against the debug binaries, once against the release binaries). If we did this, then it would make sense to instead set up a Taskotron task to replace it in the Fedora release pipeline.

* the `--slowest` option lists the 10 slowest tests at the end of the run

Comment 1 Charalampos Stratakis 2017-07-20 16:02:38 UTC

First step done for F27 and F26 [0]

[0] http://pkgs.fedoraproject.org/cgit/rpms/python3.git/commit/?id=5b6a3e0e34f46702773ad1082df3b782c1a8b7ac

Comment 2 Jan Kurik 2017-08-15 06:25:57 UTC

This bug appears to have been reported against 'rawhide' during the Fedora 27 development cycle.
Changing version to '27'.

Comment 3 Petr Viktorin (pviktori) 2018-01-18 14:08:54 UTC

[x] -wW was added
[x] --failfast is not worth it
[ ] --findleaks removal is waiting for Taskotron or Standard test interface
[x] --slowest is there already

Comment 4 Fedora End Of Life 2018-02-20 15:33:43 UTC

This bug appears to have been reported against 'rawhide' during the Fedora 28 development cycle.
Changing version to '28'.

Comment 5 Petr Viktorin (pviktori) 2018-04-19 13:59:21 UTC

Miro is looking into running integration tests first.

Comment 6 Miro Hrončok 2018-09-24 12:37:15 UTC

I'll try to add --findleaks to:

  https://src.fedoraproject.org/tests/python

Comment 7 Miro Hrončok 2018-11-12 13:11:20 UTC

https://src.fedoraproject.org/tests/python/pull-request/4

Comment 8 Miro Hrončok 2019-01-07 12:16:46 UTC

This has been done. --findleaks is now only used on the Fedora CI.

Remarks: The CI only runs on x86_64 (we agreed that this is OK for now).

RFE @ Fedora CI: https://pagure.io/fedora-ci/general/issue/16

Comment 9 Petr Viktorin (pviktori) 2019-01-07 12:20:57 UTC

Miro, thanks for your CI work. The progress on Fedora Python's testing in the last months is pretty amazing.