Bug 1193926

Summary: [GSS] (6.4.z) RejectedExecutionException when closing ModelControllerClient client
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Matthew Robson <mrobson>
Component: CLI, Remoting, Domain ManagementAssignee: Brad Maxwell <bmaxwell>
Status: CLOSED CURRENTRELEASE QA Contact: Petr Kremensky <pkremens>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.3.3CC: bbaranow, bmaxwell, brian.stansberry, cdewolf, chaowan, david.lloyd, dereed, jawilson, jmartisk, joallen, rhatlapa, rsvoboda, sjadhav
Target Milestone: CR1   
Target Release: EAP 6.4.2   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1199231, 1215733, 1219165, 1235389    

Description Matthew Robson 2015-02-18 15:03:40 UTC
Description of problem:

Feb 17 07:08:04 com..Utility : DEBUG: groups command reports: qwert users shadow qwert-users qwert
Feb 17 07:08:04 com.runclicommand : DEBUG: Calling run_cli_command for: connect,/subsystem=logging:read-children-names(child-type=logger),quit
Feb 17 07:08:04 com.application_server_jbcp6_impl : DEBUG: Cli java options set: -Xms64m -Xmx512m -XX:MaxPermSize=256m
Feb 17 07:08:04 com.application_server_jbcp6_impl : DEBUG: Attempting jboss CLI command: /usr/bin/sh /opt/qwert/appserver/bin/jboss-cli.sh -c --controller=192.168.0.3:30001 "connect,/subsystem=logging:read-children-names(child-type=logger),quit"
Feb 17 07:08:07 com.application_server_jbcp6_impl : DEBUG: Successfully ran jboss CLI command: connect,/subsystem=logging:read-children-names(child-type=logger),quit
Feb 17 07:08:07 com.application_server_jbcp6_impl : DEBUG: CLI returned: {
    "outcome" => "success",
    "result" => [
        "com.arjuna",
        "com.qwert",
        "com.qwert.logging.LevelManager",
        "com.qwert.logging.LoggerLevels",
        "com.qwert.logging.LoggingConfig",
        "com.qwert.logging.RollingOutputStream",
        "com.qwert.logging.SafLoggingManager",
        "com.qwert.logging.SafStreamConfig",
        "jacorb",
        "jacorb.config",
        "org.apache.tomcat.util.modeler",
        "org.jboss",
        "org.jboss.as.config",
        "sun.rmi"
    ]
}
Exception in thread "Remoting "cli-client" task-14" java.util.concurrent.RejectedExecutionException: Task org.jboss.remoting3.remote.RemoteConnectionChannel$3@47478104 rejected from org.xnio.XnioWorker$TaskPool@2767522b[Shutting down, pool size = 14, active threads = 2, queued tasks = 0, completed tasks = 13]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
	at org.xnio.XnioWorker.execute(XnioWorker.java:572)
	at org.jboss.remoting3.remote.RemoteConnectionChannel.receiveMessage(RemoteConnectionChannel.java:363)
	at org.jboss.as.protocol.mgmt.ManagementChannelReceiver.handleMessage(ManagementChannelReceiver.java:99)
	at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:451)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:
Exception running CLI commands

Expected results:
No exception running CLI commands

Additional info:

Comment 3 Brian Stansberry 2015-02-18 17:01:23 UTC
David,

Do you think this is the same issue as WFCORE-227, and if so do you think the xnio version used in EAP 6.x branch already has the fix?

Comment 20 Rostislav Svoboda 2015-04-23 13:27:57 UTC
Please provide test (as discussed with Carlo) to include this in CP01 payload.
I will qa_ack afterwards.

Comment 23 Rostislav Svoboda 2015-04-27 10:05:03 UTC
qa_acking, thank you for the test

Comment 37 Petr Kremensky 2015-06-25 15:23:49 UTC
Verified on EAP 6.4.2.CP.CR1.

RejectedExecutionException still can be seen while looping through these operations (approximately once in 100000 tries):

client = ModelControllerClient.Factory.create(host,port);
client.execute(operation);
client.close();

However this seems to be the limitation of the implementation as above scenario creates/destroys 30000 clients every minute.

Not seen RejectedExecutionException using jboss-cli.

Comment 38 Petr Penicka 2017-01-17 10:16:15 UTC
Retroactively bulk-closing issues from released EAP 6.4 cumulative patches.

Comment 39 Petr Penicka 2017-01-17 10:16:23 UTC
Retroactively bulk-closing issues from released EAP 6.4 cumulative patches.