Bug 1401399

Summary: [GSS](6.4.z) Race condition on creating Xnio instance
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Osamu Nagano <onagano>
Component: RemotingAssignee: Petr Jurak <pjurak>
Status: CLOSED CURRENTRELEASE QA Contact: Jiří Bílek <jbilek>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.2.4CC: bbaranow, bmaxwell, david.lloyd, jbilek, pjurak, rnetuka
Target Milestone: CR1   
Target Release: EAP 6.4.15   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-05-19 08:04:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1403684, 1419353    

Description Osamu Nagano 2016-12-05 07:01:53 UTC
EAP failed to start up because 2 threads were instantiating Xnio instance at exactly the same time.

~~~
09:23:08,823 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-10) MSC000001: Failed to start service jboss.remoting.endpoint.subsystem: org.jboss.msc.service.StartException in service jboss.remoting.endpoint.subsystem: Failed to start service
	at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1767) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_75]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_75]
	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_75]
Caused by: java.util.ServiceConfigurationError: org.xnio.XnioProvider: Provider org.xnio.nio.NioXnioProvider could not be instantiated
	at java.util.ServiceLoader.fail(ServiceLoader.java:224) [rt.jar:1.7.0_75]
	at java.util.ServiceLoader.access$100(ServiceLoader.java:181) [rt.jar:1.7.0_75]
	at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:377) [rt.jar:1.7.0_75]
	at java.util.ServiceLoader$1.next(ServiceLoader.java:445) [rt.jar:1.7.0_75]
	at org.xnio.Xnio.doGetInstance(Xnio.java:187) [xnio-api-3.0.10.GA-redhat-1.jar:3.0.10.GA-redhat-1]
	at org.xnio.Xnio.getInstance(Xnio.java:146) [xnio-api-3.0.10.GA-redhat-1.jar:3.0.10.GA-redhat-1]
	at org.jboss.remoting3.Remoting.createEndpoint(Remoting.java:73) [jboss-remoting-3.2.19.GA-redhat-1.jar:3.2.19.GA-redhat-1]
	at org.jboss.as.remoting.EndpointService.start(EndpointService.java:71) [jboss-as-remoting-7.3.4.Final-redhat-1.jar:7.3.4.Final-redhat-1]
	at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1]
	at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1]
	... 3 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.xnio.nio.NioXnioProvider
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) [rt.jar:1.7.0_75]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) [rt.jar:1.7.0_75]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [rt.jar:1.7.0_75]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526) [rt.jar:1.7.0_75]
	at java.lang.Class.newInstance(Class.java:379) [rt.jar:1.7.0_75]
	at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:373) [rt.jar:1.7.0_75]
	... 10 more

09:23:08,823 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-3) MSC000001: Failed to start service jboss.remoting.endpoint.management: org.jboss.msc.service.StartException in service jboss.remoting.endpoint.management: Failed to start service
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1767) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_75]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_75]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_75]
Caused by: java.util.ServiceConfigurationError: org.xnio.XnioProvider: Provider org.xnio.nio.NioXnioProvider could not be instantiated
        at java.util.ServiceLoader.fail(ServiceLoader.java:224) [rt.jar:1.7.0_75]
        at java.util.ServiceLoader.access$100(ServiceLoader.java:181) [rt.jar:1.7.0_75]
        at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:377) [rt.jar:1.7.0_75]
        at java.util.ServiceLoader$1.next(ServiceLoader.java:445) [rt.jar:1.7.0_75]
        at org.xnio.Xnio.doGetInstance(Xnio.java:187)
        at org.xnio.Xnio.getInstance(Xnio.java:146)
        at org.jboss.remoting3.Remoting.createEndpoint(Remoting.java:73)
        at org.jboss.as.remoting.EndpointService.start(EndpointService.java:71)
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1]
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1]
        ... 3 more
Caused by: java.lang.ExceptionInInitializerError
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) [rt.jar:1.7.0_75]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) [rt.jar:1.7.0_75]
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [rt.jar:1.7.0_75]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526) [rt.jar:1.7.0_75]
        at java.lang.Class.newInstance(Class.java:379) [rt.jar:1.7.0_75]
        at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:373) [rt.jar:1.7.0_75]
        ... 10 more
Caused by: java.lang.IllegalStateException: No functional selector provider is available
        at org.xnio.nio.NioXnio$2.run(NioXnio.java:158)
        at org.xnio.nio.NioXnio$2.run(NioXnio.java:81)
        at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.7.0_75]
        at org.xnio.nio.NioXnio.<init>(NioXnio.java:80)
        at org.xnio.nio.NioXnioProvider.<clinit>(NioXnioProvider.java:34)
        ... 16 more
~~~

This happened only once in a customer's environment. EAP version is 6.2.4 (Xnio 3.0.10.GA-redhat-1).

It seems synchronizing "org.xnio.Xnio.doGetInstance()" can be a workaround. The current Xnio 3.x branch is still not synchronized. Or a proper service dependency can be defined in the container side.

Comment 1 David M. Lloyd 2016-12-05 17:44:08 UTC
I agree this is a correct workaround that should be applied to 3.x and 3.4 branches, and as many early branches as have this same behavior.

Please file an upstream JIRA in the XNIO project.

Comment 2 Osamu Nagano 2016-12-07 06:47:35 UTC
PRs have been sent in XNIO-284. I'll arrange more JIRA tickets for EAP 7.1.z and 7.0.z.

Comment 3 Petr Jurak 2016-12-12 14:36:57 UTC
PR: https://github.com/xnio/xnio/pull/107 (merged already into 3.0)

Comment 5 Jiří Bílek 2017-04-19 12:37:00 UTC
Verified with EAP 6.4.15.CP.CR2

Comment 6 Petr Penicka 2017-05-19 08:04:57 UTC
Released on May 18 as part of EAP 6.4.15.