Bug 1110757
| Summary: | Thread leak of ServerDeploymentRepository-temp-threads after reload | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [JBoss] JBoss Enterprise Application Platform 6 | Reporter: | Jan Blizňák <jbliznak> | ||||
| Component: | Domain Management | Assignee: | Brian Stansberry <brian.stansberry> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Petr Kremensky <pkremens> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 6.3.0 | CC: | dandread, kkhan | ||||
| Target Milestone: | DR1 | ||||||
| Target Release: | EAP 6.4.0 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2019-08-19 12:38:53 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
You can also use this simple steps to reproduce: 1) start EAP 2) run: RELOAD_BATCH="batch, reload"; for i in $(seq 1000); do RELOAD_BATCH="$RELOAD_BATCH, reload"; done; RELOAD_BATCH="$RELOAD_BATCH, run-batch"; $JBOSS_HOME/bin/jboss-cli.sh -c "$RELOAD_BATCH" 3) OutOfMemory occurs after couple of minutes. Brian Stansberry <brian.stansberry> updated the status of jira WFLY-3517 to Coding In Progress Trying this with EAP 6.3.0.ER9 - Start the standalone: 'ServerDeploymentRepository-temp-threads - 1' and 'ServerDeploymentRepository-temp-threads - 2' are created. - A new 'ServerDeploymentRepository-temp-threads - 1' is created with every reload operation - Running Radim's reproducer (comment 1) I am getting 'java.lang.OutOfMemoryError: unable to create new native thread' in less than 5 minutes. Verified on EAP 6.4.0.DR1.1 |
Created attachment 909947 [details] reproducer to show number of leaked threads Description of problem: After each reload of EAP server there is one new thread "ServerDeploymentRepository-temp-threads - 1" created and it seems they are waiting forever. This eventually leads to OutOfMemory error after many reloads. Although such many reloads is not a common use case, in long term it can be reached. Error in server log: 11:57:42,456 ERROR [stderr] (Remoting "erebos-rh:MANAGEMENT" task-4) Exception in thread "Remoting "erebos-rh:MANAGEMENT" task-4" java.lang.OutOfMemoryError: unable to create new native thread 11:57:42,457 ERROR [stderr] (Remoting "erebos-rh:MANAGEMENT" task-4) at java.lang.Thread.start0(Native Method) 11:57:42,457 ERROR [stderr] (Remoting "erebos-rh:MANAGEMENT" task-4) at java.lang.Thread.start(Thread.java:693) 11:57:42,457 ERROR [stderr] (Remoting "erebos-rh:MANAGEMENT" task-4) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949) 11:57:42,457 ERROR [stderr] (Remoting "erebos-rh:MANAGEMENT" task-4) at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017) 11:57:42,458 ERROR [stderr] (Remoting "erebos-rh:MANAGEMENT" task-4) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163) 11:57:42,458 ERROR [stderr] (Remoting "erebos-rh:MANAGEMENT" task-4) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 11:57:42,458 ERROR [stderr] (Remoting "erebos-rh:MANAGEMENT" task-4) at java.lang.Thread.run(Thread.java:724) thread info from jstack: "ServerDeploymentRepository-temp-threads - 1" prio=10 tid=0x00007f715c275800 nid=0x2793 waiting on condition [0x00007f71333f2000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000afd2cf20> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) at org.jboss.threads.JBossThread.run(JBossThread.java:122) Version-Release number of selected component (if applicable): EAP 6.3.0.ER7 How reproducible: always Steps to Reproduce: 1. Start server 2. run jconsole, connect to server and filter threads on "ServerDeploymentRepository-temp-threads" 3. reload server with CLI command 4. check filtered threads in jconsole 5. repeat step 3 and 4 as many times you wanted 6. wait some time (even few minutes) till threads count get stabilized -> many threads "ServerDeploymentRepository-temp-threads" still there Or you can also use attached simple reproducer, just edit it with right paths. Actual results: one new "ServerDeploymentRepository-temp-threads" thread is created after each reload Expected results: no new "ServerDeploymentRepository-temp-threads" thread is created after reload