Bug 885397 - 'make check' shouldn't halt on first test failure
'make check' shouldn't halt on first test failure
Status: CLOSED NOTABUG
Product: Virtualization Tools
Classification: Community
Component: libguestfs (Show other bugs)
unspecified
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Richard W.M. Jones
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-12-08 21:33 EST by Cole Robinson
Modified: 2012-12-09 14:32 EST (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-12-09 14:32:52 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Cole Robinson 2012-12-08 21:33:51 EST
Similar to libvirt'd test suite, I think it would be useful if the libguestfs suite didn't halt on the first failure.

For example, I'm running it for the first time on F18 and getting failures from the btrfs tests. I see those tests have an option to skip them with an environment variable, which I do, but then the suite is failing at test-parallel-mount.

But as a random developer, all I really care about is 'does my change make the test suite regress' which can't be answered definitively if the test suite bails out at first failure.
Comment 1 Richard W.M. Jones 2012-12-09 07:21:38 EST
By coincidence I recently changed the RHEL 7 libguestfs.spec
so it uses 'make check -k'.

The thing is, I have no idea how to implement this generally,
since the test suite is spread over many different directories.
However suggestions are welcome ...

(In reply to comment #0)
> For example, I'm running it for the first time on F18 and getting failures
> from the btrfs tests.

This is a known (and massive) bug in btrfs with no upstream
resolution.  I raised this months ago and no one's bothered to
do anything about it, which makes me worried for the quality
of btrfs ...

http://article.gmane.org/gmane.comp.file-systems.btrfs/20257

> I see those tests have an option to skip them with an
> environment variable, which I do,

By the way, you can create a file called 'localenv' containing
skipped tests:

http://libguestfs.org/guestfs.3.html#local-files

> but then the suite is failing at
> test-parallel-mount.

This really shouldn't fail, except perhaps if you have a really
small amount of memory.  What's the failure?
Comment 2 Richard W.M. Jones 2012-12-09 07:22:33 EST
FYI here is my Fedora 18 'localenv' file:

$ cat localenv 
# NB: -*- Makefile -*- AND a shell script.

# Test python3, since defaults test ordinary python.
#export PYTHON=python3

# Skip all tests that involve btrfs because of RHBZ#863978.
export SKIP_TEST_BTRFS_FSCK=1
export SKIP_TEST_BTRFS_SET_SEEDING=1
export SKIP_TEST_BTRFS_FILESYSTEM_SYNC=1
export SKIP_TEST_BTRFS_SUBVOLUME_DELETE=1
export SKIP_TEST_BTRFS_SUBVOLUME_SNAPSHOT=1
export SKIP_TEST_MKFS_BTRFS=1
export SKIP_TEST_BTRFS_DEVICES_SH=1
export SKIP_TEST_BTRFS_SUBVOLUME_DEFAULT_PL=1
export SKIP_TEST_CHARSET_FIDELITY=1
Comment 3 Cole Robinson 2012-12-09 14:32:52 EST
Doh, I completely forgot about make --keep-going... which is sufficient to meet my needs. Closing this bug

And now I can't reproduce the test-parallel-mount failure, but I'll let you know if I see it again.

Note You need to log in before you can comment on or make changes to this bug.