Bug 818863

Summary: Tests for UNORDERED and LRU eviction strategies fail on IBM JDK
Product: [JBoss] JBoss Data Grid 6 Reporter: Martin Gencur <mgencur>
Component: InfinispanAssignee: Tristan Tarrant <ttarrant>
Status: ASSIGNED --- QA Contact:
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.0.0CC: jdg-bugs, tsykora
Target Milestone: ---Keywords: Reopened
Target Release: 6.2.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Known Issue
Doc Text:
The LinkedHashMap implementation in IBM's JDK behaves erratically when extended (as is done by the eviction strategy code). This incorrect behavior is exposed by the Red Hat JBoss Data Grid Test Suite. It is recommended, if using eviction, to use LIRS eviction strategy, which is not affected by this issue. Only LRU eviction strategy is affected.
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-05-16 20:09:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Martin Gencur 2012-05-04 08:26:14 UTC
Description of problem:

Running Infinispan testsuite for JDG reveals problems with IBM JDK:
java version "1.6.0"
Java(TM) SE Runtime Environment (build pxi3260sr9fp1-20110208_03(SR9 FP1))
IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux x86-32 jvmxi3260sr9-20110203_74623 (JIT enabled, AOT enabled))

Stacktraces can be found here:

https://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-FUNC/job/edg-60-infinispan-testsuite-rhel/37/jdk=ibm16,label=RHEL5_x86/

The stacktrace basically says the following:

java.lang.AssertionError: cache size too big: 182
	at org.infinispan.eviction.BaseEvictionFunctionalTest.testSimpleEvictionMaxEntries(BaseEvictionFunctionalTest.java:67)

I encountered this problem on RHEL5_x86 and RHEL6_x86_64 platforms, ran the testsuite twice (each time on different machine of the same type) and got the same results.

This is probably related to different implementation of ConcurrentHashMap in IBM JDK.

Comment 1 Tristan Tarrant 2012-05-16 20:09:20 UTC

*** This bug has been marked as a duplicate of bug 822135 ***

Comment 2 Tristan Tarrant 2012-05-16 20:11:36 UTC
*** Bug 822135 has been marked as a duplicate of this bug. ***

Comment 3 Tristan Tarrant 2012-05-16 20:16:43 UTC
This is due to a bug in the IBM JDK (possibly http://www-01.ibm.com/support/docview.wss?uid=swg1IZ63859)

Comment 4 Martin Gencur 2012-05-22 11:41:12 UTC
I'm seeing this in ER10 too. Shouldn't we close this as WONTFIX when it's a bug in IBM JDK?

Comment 5 mark yarborough 2012-06-06 13:31:47 UTC
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Tristan will provide CCFR or will route to appropriate developer.

Comment 6 Misha H. Ali 2012-06-07 03:19:26 UTC
Flagging Tristan for information we want to convey to customers about this bug.

Comment 7 Tristan Tarrant 2012-06-12 07:34:24 UTC
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1 +1 @@
-Tristan will provide CCFR or will route to appropriate developer.+The LinkedHashMap implementation in IBM's JDK sometimes behaves erratically when extended (as is done by the eviction strategy code). This wrong behaviour is exposed by the JDG Test Suite. The recommendation is, if using eviction, to use other JDKs (Oracle or OpenJDK) which are not affected by this issue.

Comment 10 mark yarborough 2012-11-14 14:42:21 UTC
ttarrant will add jira links as appropriate.

Comment 13 JBoss JIRA Server 2013-02-12 16:01:25 UTC
Martin Gencur <mgencur> made a comment on jira ISPN-1822

I've been trying to fix this issues but it seems I'm running in circles. My previous fix was running in single-threaded environment and therefore EvictionWithPassivationTest is passing. However, if MapStressTest is executed, there are always more entries left in the cache than expected. I simply got stuck with this issue.

Comment 14 JBoss JIRA Server 2013-02-14 16:52:28 UTC
Martin Gencur <mgencur> made a comment on jira ISPN-1822

So I gave it another try and got much better results. The commit that fixes this issue is located at https://github.com/mgencur/infinispan/commit/575dbf1837803d4422725119082157e968f647d2

Some results from MapStressTest: https://gist.github.com/mgencur/4954139

The first two scenarios maintain the capacity <= 100000. The WriteOnMiss scenario leaves just a few redundant entries in the map after the test ends. I executed the same test also with 10-minute duration and it seems the number of redundant entries does not increase.

Other functional tests like LRUEvictionFunctionalTest or EvictionWithPassivationTest are passing now (fixes ISPN-2623 too).

I tested my solution with IBM JDK 7 64bit:

java version "1.7.0"
Java(TM) SE Runtime Environment (build pxa6470sr3-20121025_01(SR3))
IBM J9 VM (build 2.6, JRE 1.7.0 Linux amd64-64 20121024_126071 (JIT enabled, AOT enabled)


I will leave it to your judgment whether it's a good fix or not. IMO that's definitely better than the current behavior of ISPN.

Comment 15 Misha H. Ali 2013-05-07 03:23:52 UTC
Nominated for 6.2 release notes.

Comment 17 Martin Gencur 2013-12-18 12:06:55 UTC
This is no longer a problem on IBM JDK 1.7
Tested version:
java version "1.7.0"
Java(TM) SE Runtime Environment (build pxa6470sr4ifix-20130305_01(SR4+IV37419))
IBM J9 VM (build 2.6, JRE 1.7.0 Linux amd64-64 Compressed References 20130205_137358 (JIT enabled, AOT enabled))

I was not able to verify IBM JDK 1.6 due to https://bugzilla.redhat.com/show_bug.cgi?id=1044480

Comment 18 Martin Gencur 2013-12-18 16:18:18 UTC
It turns out this is still a problem for 
java version "1.6.0"
Java(TM) SE Runtime Environment (build pxa6460sr13ifix-20130303_02(SR13+IV37419))
IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux amd64-64 jvmxa6460sr13-20130114_134867 (JIT enabled, AOT enabled)

Maybe a newer version is available with the fix?

Comment 19 gsheldon 2014-01-09 06:44:19 UTC
Removing requires_doc_text flag as this bug is not required for release notes.

Comment 20 Martin Gencur 2014-01-10 10:50:18 UTC
Gemma, this is required for release notes in the end. Can you please add it? Thanks