Red Hat Bugzilla – Bug 784007
Server jar unit tests failing due to poorly handled test interactions
Last modified: 2012-02-10 10:32:54 EST
The addition of a new server jar test class for bug 782612 started causing
test failures generated solely due to unhandled test interactions. Meaning,
any test class run by itself would pass but running the tests together
The server jar tests are fragile with respect to test invocation ordering.
testng does not guarantee test execution ordering within a test class,
nor does it guarantee that a test class Begin/testExecution/After methods
will run "atomically".
Work here aims to improve the situation, by getting the server jar tests
to again pass and hopefully be more stable going forward It is not to re-architect the unit testing completely.
master commit 24d89dc93ac47178812ce63c213d176fa265e915
It seems we didn't really understand how testng was executing tests
when we submitted many test classes in a run. For example, when we
submit the server jar tests we assumed that the test classes would
be run sequentially, in perhaps some random order, and that each would
have its Before/AfterClass methods executed at the start and end of each
test class execution.
That is not true.
Although testng seems to run most tests from a test class together, it
does not seem to make that guarantee. From latest traces it seems that
the test classes are, toa degree, pooled and:
1) you can not assume that all test methods for a test class will execute
before execution of test methods from another test class.
2) you can not assume that multiple BeforeClass methods will not execute
prior to an AfterClass. For example, this can happen:
Put another way: You can only assume that BeforeClass and AfterClass
execute in order, for a single test class. You can not predict the
execution order of all Before|AfterClass methods for all test classes.
3) The same rules as described in 2 apply to Before|AfterGroups
What this means is that you must be very careful with what is done in
Before|AfterClass and BeforeAfter|Groups. The work must not conflict
with work done in other test classes that will execute in the same test
This is very important when multiple test classes are sharing some sort
of test resource.
In general, it is much safer, although perhaps a bit slower, to use
@BeforeMethod() and @AfterMethod(alwaysRun=true).
This misunderstanding was particularly bad for us in the server jar tests,
where many test classes extend AbstractEJB3Test and utilize the jboss
mbean server under the covers, in the prepareXXX methods. This caused
clashes described above.
So, for server jar testing this reworks things to better share the mbean
server, even among tests in the same class. And migrates all
AbstractEJB3Test.prepare/unprepare calls from Before|AfterClass|Groups to
also - remove/suppress IDE warnings from touched test classes
master commit 68a688bc7bf11d2d6cfe22b08a1ffb1d282eb804
More work to prevent server jar tests from stepping on each other, or
- tests now pass with testng 6.1.1 (current version) and also 6.3.1,
latest version. Each version brought out different issues due to
different execution ordering and constraint checking.
- Avoid the use of DatabaseOperation.CLEAN_INSERT. This deletes all rows
in tables used by the data set. This does not play well with other modules.
Instead use DatabaseOperation.REFRESH and also clean up using
DatabaseOperation.DELETE. These options affect only the rows in the
data set. (see DeleteAlertsTest)
- Unless absolutely necessary, use Before|AfterMethod for database setup
and teardown. The perf hit is typically minimal but the risk of
test class conflict is high. (see DatabaseAndFilePluginDeloymentTest)
- Be more diligent to keep the database clean. For example, in addition to
DatabaseOperation.DELETE, there may be additional data created by the
tests. (see DiscoveryBossBeanTest)
- When possible, assertions should be narrowed to deal with only the data
relevant to the test or test class. Don't assume the database is
necessarily clean. This helps ensure that other test data does not cause
failures, or is not manipulated accidentally. Queries should try to be
qualified with the ids known to be relevant. (see InventoryManagerBeanTest)
- Prefer to use slsb calls to do cleanup, these calls are often better at
ensuring related data is also cleaned up. Hibernate cascading is often
not enough. And straight SQl requires even more diligence. Cleaning up
plugins is very important (see MetadataBeanTest).
- If a test does specific dbsetup make sure it also cleans it up, or move
it to Before|AfterMethod. (see PluginManagerBeanTest)
- Make sure your tests can executein any order, or add specific dependencies
to force ordering. (see ResourceMetadataManagerBean)
- Use unusual/varying ids for canned data. Always starting ids at id 1 is more likely to cause conflicts.
- Always run your test class twice in a row. It should work more than once. If
it doesn't it probably changed state of the db or other shared resource.
- Start your tests with a clean db (dbsetup) and manually inspect the
db to see if any data was left behind (or better, write a tool to do it).
Needs to be triaged to determine if needs backporting to the jon3.0.1 branch
fyi: this will replicate the problem and will run fast (you don't have to run the entire server/jar test suite):
mvn -Dtest=EventMetadataManagerBeanTest,CoreServerServiceImplTest install
closing, the relevant commits were backported in bug 782612.