The Python spec file currently runs the test suite using the "--verbose" flag, which makes it hard to find the details of which tests actually failed when something goes wrong. Two other flags are likely to produce easier to read results (these can both be set at the same time): -w: Immediately re-runs failed tests in verbose mode -W: Prints captured output when a test fails Another potentially useful option for the %check section may be "--failfast", which will abort the test run as soon as any test fails, rather than always continuing to run the full test suite. Some other ideas potentially worth exploring: * using the `--findleaks` option forces serialisation of test execution on a single CPU. If it is omitted (effectively reclassifying leak hunting in the Python runtime as an integration test rather than a unit test), then the `-j` option could instead be passed to run tests in parallel across multiple CPUs, which may significantly reduce the overall RPM build time (since the test suite gets run twice, once against the debug binaries, once against the release binaries). If we did this, then it would make sense to instead set up a Taskotron task to replace it in the Fedora release pipeline. * the `--slowest` option lists the 10 slowest tests at the end of the run
First step done for F27 and F26 [0] [0] http://pkgs.fedoraproject.org/cgit/rpms/python3.git/commit/?id=5b6a3e0e34f46702773ad1082df3b782c1a8b7ac
This bug appears to have been reported against 'rawhide' during the Fedora 27 development cycle. Changing version to '27'.
[x] -wW was added [x] --failfast is not worth it [ ] --findleaks removal is waiting for Taskotron or Standard test interface [x] --slowest is there already
This bug appears to have been reported against 'rawhide' during the Fedora 28 development cycle. Changing version to '28'.
Miro is looking into running integration tests first.
I'll try to add --findleaks to: https://src.fedoraproject.org/tests/python
https://src.fedoraproject.org/tests/python/pull-request/4
This has been done. --findleaks is now only used on the Fedora CI. Remarks: The CI only runs on x86_64 (we agreed that this is OK for now). RFE @ Fedora CI: https://pagure.io/fedora-ci/general/issue/16
Miro, thanks for your CI work. The progress on Fedora Python's testing in the last months is pretty amazing.