Bug 1040346

Summary: ISPN test suite fails on Unix due to memory issues
Product: [JBoss] JBoss Data Grid 6 Reporter: Vojtech Juranek <vjuranek>
Component: TestSuiteAssignee: Tristan Tarrant <ttarrant>
Status: NEW --- QA Contact: Martin Gencur <mgencur>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: unspecifiedCC: jdg-bugs, ttarrant
Target Milestone: ---   
Target Release: ---   
Hardware: sparc64   
OS: Solaris   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Vojtech Juranek 2013-12-11 09:06:57 UTC 
This issue is just for tracking problems related to running ISPN test suite on Solaris Sparc. It's very likely not a bug in ISPN test suite itself, but probably some misconfiguration of Solaris Sparc machines.

Description of problem:
ISPN testsuite fails randomly on Solaris Sparc64 with some memory issue. Typical error looks like this one:

ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.11-redhat-1:test (default-test) on project infinispan-core: Error occured in starting fork, check output in log -> [Help 1]
 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.11-redhat-1:test (default-test) on project infinispan-query-dsl: Error while executing forked tests.; nested exception is java.io.IOException: Cannot run program "/bin/sh" (in directory "/mnt/hudson_workspace/workspace/edg-60-ispn-testsuite-solaris/61ab75ab/infinispan/query-dsl"): error=12, Not enough space -> [Help 1]
Comment 3 Vojtech Juranek 2013-12-17 19:54:42 UTC 
PR #2290 [1] seems to help, not sure it if fixes the issue completely, though

[1] https://github.com/infinispan/infinispan/pull/2290
Comment 4 Vojtech Juranek 2014-03-17 13:38:08 UTC 
Possible root cause is that Solaris use fork() which creates deep copy of process:
http://docs.oracle.com/cd/E19455-01/806-5257/6je9h0341/index.html
Comment 5 Vojtech Juranek 2015-03-17 14:39:48 UTC 
Similar memory issue happens on HP-UX: it fails after some time with 

Caused by: java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:714)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
	at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1590)
	at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:333)
	at java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:594)
	at java.util.concurrent.Executors$DelegatedScheduledExecutorService.scheduleWithFixedDelay(Executors.java:698)
	at org.infinispan.executors.LazyInitializingScheduledExecutorService.scheduleWithFixedDelay(LazyInitializingScheduledExecutorService.java:145)
	at org.infinispan.eviction.EvictionManagerImpl.start(EvictionManagerImpl.java:78)
	at sun.reflect.GeneratedMethodAccessor74.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)
	... 57 more
Comment 6 Vojtech Juranek 2015-03-17 23:38:08 UTC 
Obviously not insufficient memory issue - happens also on machine wih ~ 180GB RAM. I.e. some setup (env. or test) issue.
Comment 7 Vojtech Juranek 2015-03-19 13:22:50 UTC 
Lowering stack size to 264k (-Xss264k) seems to fix the problem on HP. It seems to work also with 512k.