Created attachment 909947 [details] reproducer to show number of leaked threads Description of problem: After each reload of EAP server there is one new thread "ServerDeploymentRepository-temp-threads - 1" created and it seems they are waiting forever. This eventually leads to OutOfMemory error after many reloads. Although such many reloads is not a common use case, in long term it can be reached. Error in server log: 11:57:42,456 ERROR [stderr] (Remoting "erebos-rh:MANAGEMENT" task-4) Exception in thread "Remoting "erebos-rh:MANAGEMENT" task-4" java.lang.OutOfMemoryError: unable to create new native thread 11:57:42,457 ERROR [stderr] (Remoting "erebos-rh:MANAGEMENT" task-4) at java.lang.Thread.start0(Native Method) 11:57:42,457 ERROR [stderr] (Remoting "erebos-rh:MANAGEMENT" task-4) at java.lang.Thread.start(Thread.java:693) 11:57:42,457 ERROR [stderr] (Remoting "erebos-rh:MANAGEMENT" task-4) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949) 11:57:42,457 ERROR [stderr] (Remoting "erebos-rh:MANAGEMENT" task-4) at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017) 11:57:42,458 ERROR [stderr] (Remoting "erebos-rh:MANAGEMENT" task-4) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163) 11:57:42,458 ERROR [stderr] (Remoting "erebos-rh:MANAGEMENT" task-4) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 11:57:42,458 ERROR [stderr] (Remoting "erebos-rh:MANAGEMENT" task-4) at java.lang.Thread.run(Thread.java:724) thread info from jstack: "ServerDeploymentRepository-temp-threads - 1" prio=10 tid=0x00007f715c275800 nid=0x2793 waiting on condition [0x00007f71333f2000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000afd2cf20> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) at org.jboss.threads.JBossThread.run(JBossThread.java:122) Version-Release number of selected component (if applicable): EAP 6.3.0.ER7 How reproducible: always Steps to Reproduce: 1. Start server 2. run jconsole, connect to server and filter threads on "ServerDeploymentRepository-temp-threads" 3. reload server with CLI command 4. check filtered threads in jconsole 5. repeat step 3 and 4 as many times you wanted 6. wait some time (even few minutes) till threads count get stabilized -> many threads "ServerDeploymentRepository-temp-threads" still there Or you can also use attached simple reproducer, just edit it with right paths. Actual results: one new "ServerDeploymentRepository-temp-threads" thread is created after each reload Expected results: no new "ServerDeploymentRepository-temp-threads" thread is created after reload
You can also use this simple steps to reproduce: 1) start EAP 2) run: RELOAD_BATCH="batch, reload"; for i in $(seq 1000); do RELOAD_BATCH="$RELOAD_BATCH, reload"; done; RELOAD_BATCH="$RELOAD_BATCH, run-batch"; $JBOSS_HOME/bin/jboss-cli.sh -c "$RELOAD_BATCH" 3) OutOfMemory occurs after couple of minutes.
Brian Stansberry <brian.stansberry> updated the status of jira WFLY-3517 to Coding In Progress
Trying this with EAP 6.3.0.ER9 - Start the standalone: 'ServerDeploymentRepository-temp-threads - 1' and 'ServerDeploymentRepository-temp-threads - 2' are created. - A new 'ServerDeploymentRepository-temp-threads - 1' is created with every reload operation - Running Radim's reproducer (comment 1) I am getting 'java.lang.OutOfMemoryError: unable to create new native thread' in less than 5 minutes.
Verified on EAP 6.4.0.DR1.1