Bug 1118593 - Karaf cause OOME when running OSGI integration tests on all environments
Summary: Karaf cause OOME when running OSGI integration tests on all environments
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: JBoss Data Grid 6
Classification: JBoss
Component: Infinispan
Version: 6.3.1
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ER8
: 6.4.0
Assignee: Martin Gencur
QA Contact: Martin Gencur
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-07-11 06:57 UTC by Vitalii Chepeliuk
Modified: 2015-01-28 13:26 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-01-28 13:26:30 UTC
Type: Bug


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
JBoss Issue Tracker ISPN-4500 Major Closed Karaf cause OOME when running iOSGI integration tests on all environments 2017-05-05 08:00:17 UTC

Description Vitalii Chepeliuk 2014-07-11 06:57:29 UTC
See description of linked ISPN-4500

Comment 2 Ion Savin 2014-07-11 18:01:13 UTC
The tests fail in the 'infinispan-cachestore-leveldb' module and because of this the install step doesn't complete and the features.xml file is not pushed to the local repo.

The <hr> comes from a "not found" reply from one of the remote maven repos configured to resolve artifacts inside Karaf.

Upgrading the leveldbjni version to 1.13-redhat.002 + clean install in infinispan-cachestore-leveldb solved the problem.

The OOME and other errors are probably side-effects of the feature not being installed. Please open a new issue if they still occur with CR3 (which contains the version fix).

See the leveldbjni issue for details.

*** This bug has been marked as a duplicate of bug 1118785 ***

Comment 3 JBoss JIRA Server 2014-07-11 18:02:27 UTC
Ion Savin <isavin@redhat.com> updated the status of jira ISPN-4500 to Closed

Comment 5 Alan Field 2014-09-03 14:20:23 UTC
Moving target version to JDG 6.4.0

Comment 6 Ion Savin 2014-09-12 06:32:10 UTC
Just like before the OSGi test failure is a side-effect of the tests in LevelDB failing -> the bundle/feature is not installed and the OSGi tests can't load them.

The failure in the LevelDB module is:

Test suite progress: tests succeeded: 28, failed: 0, skipped: 0.
org.infinispan.commons.CacheConfigurationException: Unable to instantiate class org.fusesource.leveldbjni.JniDBFactory
	at org.infinispan.commons.util.Util.loadClass(Util.java:85)
	at org.infinispan.commons.util.Util.getInstance(Util.java:223)
	at org.infinispan.persistence.leveldb.LevelDBStore.newDbFactory(LevelDBStore.java:78)
	at org.infinispan.persistence.leveldb.LevelDBStore.init(LevelDBStore.java:60)
	at org.infinispan.persistence.leveldb.LevelDBStoreTest.createStore(LevelDBStoreTest.java:83)
	at org.infinispan.persistence.BaseStoreTest.setUp(BaseStoreTest.java:78)
	at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
	at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:564)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:213)
	at org.testng.internal.Invoker.invokeMethod(Invoker.java:653)
	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
	at org.testng.TestRunner.privateRun(TestRunner.java:767)
	at org.testng.TestRunner.run(TestRunner.java:617)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
	at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
	at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
	at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.ClassNotFoundException: Could not load requested class : org.fusesource.leveldbjni.JniDBFactory
	at org.infinispan.commons.util.OsgiClassLoader.findClass(OsgiClassLoader.java:89)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:270)
	at org.infinispan.commons.util.Util.loadClassStrict(Util.java:139)
	at org.infinispan.commons.util.Util.loadClass(Util.java:83)
	... 26 more

The OsgiClassLoader appears in the exception because it appears to be the last classloader from the list which failed to load that class.

Locally the test for the leveldb module run OK on the branch mentioned.

Comment 7 Ion Savin 2014-09-15 10:56:48 UTC
Please check the environment setup. On a new run there's this failure which could and it could be the cause for the tests failing in cachestore-leveldb and then integrationtests/osgi:

Caused by: java.lang.UnsatisfiedLinkError: Could not load library. Reasons: [no leveldbjni32-1.13-redhat.002 in java.library.path, no leveldbjni-1.13-redhat.002 in java.library.path, no leveldbjni in java.library.path]

The build is:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/JDG/view/FUNC/job/edg-60-ispn-testsuite-solaris/211/USE_JDK=JDK7,jdk=java17_default,label=sol11_sparc64/

Comment 8 Jakub Markos 2014-09-15 12:40:36 UTC
Leveldb not running on solaris is an old bug https://bugzilla.redhat.com/show_bug.cgi?id=1020411. So maybe we could just disable them and see if the osgi tests pass.

Comment 9 Martin Gencur 2014-11-25 08:28:54 UTC
Jakub, what is the current status of this issue?

Comment 11 Martin Gencur 2014-12-09 16:03:16 UTC
PR: https://github.com/infinispan/jdg/pull/391

See the PR for details on how to run tests.

Comment 12 Ion Savin 2014-12-10 09:35:40 UTC
Looking at the CI test failures for the last mentioned failure I can see failures in other modules besides the OSGi one. For example:

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/JDG/view/FUNC/job/edg-60-ispn-testsuite-windows/334/USE_JDK=JDK7,jdk=java17_default,label_exp=w2k8r2%20&&%20x86_64/testReport/junit/org.infinispan.persistence.remote/RemoteStoreRawValuesTest/testLoadAndStoreMarshalledValues/

If the tests are running using the "ignore test failure" flag the build won't fail when there are test failures in other modules but the install phase for that module won't execute either.

Because the install phase doesn't execute the bundles and feature files for those modules won't be present in the local maven repository and as a result the OSGi tests will fail but they are not at fault.

The solution is to fix the failures in the other modules.

Comment 13 Vojtech Juranek 2014-12-10 09:49:26 UTC
Sorry for ignorance, I'm not very familiar with OSGi ... but shouldn't maven download missing artifacts from remote repository in this case?

Comment 14 Martin Gencur 2014-12-10 09:51:26 UTC
I think my PR will ensure that we will get a better error message when the artifacts are not in the local repo nor in any of the remote ones.

Comment 15 Vojtech Juranek 2014-12-10 10:00:37 UTC
so if I get it right, as appropriate maven repo is part of the release, IMHO we can fix it in our test suite scripts by skipping maven install and instead of it downloading the maven repo from release, right?

Comment 16 Dave Stahl 2014-12-18 14:46:13 UTC
PR integrated into jdg-6.4.x


Note You need to log in before you can comment on or make changes to this bug.