Red Hat Bugzilla – Bug 885397
'make check' shouldn't halt on first test failure
Last modified: 2012-12-09 14:32:52 EST
Similar to libvirt'd test suite, I think it would be useful if the libguestfs suite didn't halt on the first failure.
For example, I'm running it for the first time on F18 and getting failures from the btrfs tests. I see those tests have an option to skip them with an environment variable, which I do, but then the suite is failing at test-parallel-mount.
But as a random developer, all I really care about is 'does my change make the test suite regress' which can't be answered definitively if the test suite bails out at first failure.
By coincidence I recently changed the RHEL 7 libguestfs.spec
so it uses 'make check -k'.
The thing is, I have no idea how to implement this generally,
since the test suite is spread over many different directories.
However suggestions are welcome ...
(In reply to comment #0)
> For example, I'm running it for the first time on F18 and getting failures
> from the btrfs tests.
This is a known (and massive) bug in btrfs with no upstream
resolution. I raised this months ago and no one's bothered to
do anything about it, which makes me worried for the quality
of btrfs ...
> I see those tests have an option to skip them with an
> environment variable, which I do,
By the way, you can create a file called 'localenv' containing
> but then the suite is failing at
This really shouldn't fail, except perhaps if you have a really
small amount of memory. What's the failure?
FYI here is my Fedora 18 'localenv' file:
$ cat localenv
# NB: -*- Makefile -*- AND a shell script.
# Test python3, since defaults test ordinary python.
# Skip all tests that involve btrfs because of RHBZ#863978.
Doh, I completely forgot about make --keep-going... which is sufficient to meet my needs. Closing this bug
And now I can't reproduce the test-parallel-mount failure, but I'll let you know if I see it again.