Bug 1027738 - Re-deployment of a clustered application fail due to race-condition with infinispan
Re-deployment of a clustered application fail due to race-condition with infi...
Status: CLOSED CURRENTRELEASE
Product: JBoss Enterprise Application Platform 6
Classification: JBoss
Component: Clustering (Show other bugs)
6.1.1,6.2.0
Unspecified Unspecified
unspecified Severity unspecified
: CR2
: EAP 6.2.0
Assigned To: Paul Ferraro
Jitka Kozana
Russell Dickenson
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-11-07 06:03 EST by wfink
Modified: 2013-12-15 11:12 EST (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-12-15 11:12:59 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
JBoss Issue Tracker ISPN-3698 Blocker Resolved Failure during concurrent restart of cache/manager 2017-07-28 10:45 EDT
JBoss Issue Tracker WFLY-2458 Major Resolved Re-deployment of a clustered application fail due to race-condition with infinispan 2017-07-28 10:45 EDT

  None (edit)
Description wfink 2013-11-07 06:03:13 EST
If a clustered application should be redeployed it looks like that infinispan is not able to stop complete until the deployer finish undeployment and start to deploy the new application. See the Exception below.
The behaviour is the same for managed or unmanged deployments.
If it is unmanaged the .failed marker file exists.



11:58:17,988 INFO  [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-9) ISPN000029: Passivating all entries to disk
11:58:17,989 INFO  [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-9) ISPN000030: Passivated 0 entries in 0 milliseconds
11:58:17,992 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-1) JBAS015877: Stopped deployment null (runtime-name: ejb.jar) in 16ms
11:58:17,992 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-14) JBAS010282: Stopped remote-connector-client-mappings cache from ejb container
11:58:17,993 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-9) JBAS010282: Stopped repl cache from ejb container
11:58:17,993 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-14) JBAS015877: Stopped deployment appone.ear (runtime-name: appone.ear) in 18ms
11:58:17,994 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000080: Disconnecting and closing JGroups Channel
11:58:17,995 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-7) JBAS015876: Starting deployment of "appone.ear" (runtime-name: "appone.ear")
11:58:17,999 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-13) JBAS015876: Starting deployment of "null" (runtime-name: "ejb.jar")
11:58:18,017 INFO  [org.jboss.as.ejb3.deployment.processors.EjbJndiBindingsDeploymentUnitProcessor] (MSC service thread 1-8) JNDI bindings for session bean named AppOneBean in deployment unit subdeployment "ejb.jar" of deployment "appone.ear" are as follows:

	java:global/appone/ejb/AppOneBean!org.jboss.as.quickstarts.ejb.multi.server.app.AppOne
	java:app/ejb/AppOneBean!org.jboss.as.quickstarts.ejb.multi.server.app.AppOne
	java:module/AppOneBean!org.jboss.as.quickstarts.ejb.multi.server.app.AppOne
	java:jboss/exported/appone/ejb/AppOneBean!org.jboss.as.quickstarts.ejb.multi.server.app.AppOne
	java:global/appone/ejb/AppOneBean
	java:app/ejb/AppOneBean
	java:module/AppOneBean

11:58:18,350 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000082: Stopping the RpcDispatcher
11:58:18,377 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 22) ISPN000078: Starting JGroups Channel
11:58:18,380 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 22) MSC000001: Failed to start service jboss.infinispan.ejb.remote-connector-client-mappings: org.jboss.msc.service.StartException in service jboss.infinispan.ejb.remote-connector-client-mappings: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type JGroupsTransport
	at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:87)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25]
	at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]
	at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.0.Final-redhat-1.jar:2.1.0.Final-redhat-1]
Caused by: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type JGroupsTransport
	at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:247)
	at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:678)
	at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:649)
	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:545)
	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:559)
	at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:109)
	at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:100)
	at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78)
	at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:82)
	... 4 more
Caused by: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type JGroupsTransport
	at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:205)
	at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:886)
	at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:657)
	at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:646)
	at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:549)
	at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:225)
	... 12 more
Caused by: org.infinispan.CacheException: Unable to start JGroups Channel
	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.startJGroupsChannelIfNeeded(JGroupsTransport.java:209)
	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.start(JGroupsTransport.java:198)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_25]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_25]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25]
	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25]
	at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203)
	... 17 more
Caused by: java.lang.IllegalStateException: channel is closed
	at org.jgroups.JChannel.checkClosed(JChannel.java:865)
	at org.jgroups.JChannel._preConnect(JChannel.java:520)
	at org.jgroups.JChannel.connect(JChannel.java:277)
	at org.jgroups.JChannel.connect(JChannel.java:268)
	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.startJGroupsChannelIfNeeded(JGroupsTransport.java:207)
	... 23 more

11:58:18,385 INFO  [org.infinispan.factories.TransactionManagerFactory] (ServerService Thread Pool -- 4) ISPN000161: Using a batchMode transaction manager
11:58:18,388 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 4) ISPN000078: Starting JGroups Channel
11:58:18,389 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 4) MSC000001: Failed to start service jboss.infinispan.ejb.repl: org.jboss.msc.service.StartException in service jboss.infinispan.ejb.repl: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type JGroupsTransport
	at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:87)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25]
	at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]
	at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.0.Final-redhat-1.jar:2.1.0.Final-redhat-1]
Caused by: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type JGroupsTransport
	at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:205)
	at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:886)
	at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:657)
	at org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:226)
	at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175)
	at org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:296)
	at org.infinispan.factories.ComponentRegistry.getOrCreateComponent(ComponentRegistry.java:158)
	at org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246)
	at org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:86)
	at org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:814)
	at org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220)
	at org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:188)
	at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175)
	at org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:296)
	at org.infinispan.factories.ComponentRegistry.getOrCreateComponent(ComponentRegistry.java:161)
	at org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246)
	at org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:86)
	at org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:814)
	at org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220)
	at org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:188)
	at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175)
	at org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:296)
	at org.infinispan.factories.ComponentRegistry.getOrCreateComponent(ComponentRegistry.java:161)
	at org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246)
	at org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:86)
	at org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:814)
	at org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220)
	at org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:188)
	at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175)
	at org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:296)
	at org.infinispan.factories.ComponentRegistry.getOrCreateComponent(ComponentRegistry.java:161)
	at org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246)
	at org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:86)
	at org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:814)
	at org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220)
	at org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:188)
	at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175)
	at org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:296)
	at org.infinispan.factories.ComponentRegistry.getOrCreateComponent(ComponentRegistry.java:161)
	at org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246)
	at org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:86)
	at org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:814)
	at org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220)
	at org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:188)
	at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175)
	at org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:296)
	at org.infinispan.factories.ComponentRegistry.getOrCreateComponent(ComponentRegistry.java:161)
	at org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246)
	at org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:86)
	at org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:814)
	at org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220)
	at org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:188)
	at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175)
	at org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:296)
	at org.infinispan.factories.ComponentRegistry.getOrCreateComponent(ComponentRegistry.java:161)
	at org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246)
	at org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:86)
	at org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:814)
	at org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220)
	at org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:188)
	at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175)
	at org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:296)
	at org.infinispan.factories.ComponentRegistry.getOrCreateComponent(ComponentRegistry.java:161)
	at org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246)
	at org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:86)
	at org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:814)
	at org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220)
	at org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:188)
	at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175)
	at org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:296)
	at org.infinispan.factories.ComponentRegistry.getOrCreateComponent(ComponentRegistry.java:161)
	at org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246)
	at org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:86)
	at org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:814)
	at org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220)
	at org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:188)
	at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175)
	at org.infinispan.factories.InternalCacheFactory.bootstrap(InternalCacheFactory.java:101)
	at org.infinispan.factories.InternalCacheFactory.createAndWire(InternalCacheFactory.java:80)
	at org.infinispan.factories.InternalCacheFactory.createCache(InternalCacheFactory.java:64)
	at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:682)
	at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:649)
	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:545)
	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:559)
	at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:109)
	at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:100)
	at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78)
	at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:82)
	... 4 more
Caused by: org.infinispan.CacheException: Unable to start JGroups Channel
	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.startJGroupsChannelIfNeeded(JGroupsTransport.java:209)
	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.start(JGroupsTransport.java:198)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_25]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_25]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25]
	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25]
	at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203)
	... 91 more
Caused by: java.lang.IllegalStateException: channel is closed
	at org.jgroups.JChannel.checkClosed(JChannel.java:865)
	at org.jgroups.JChannel._preConnect(JChannel.java:520)
	at org.jgroups.JChannel.connect(JChannel.java:277)
	at org.jgroups.JChannel.connect(JChannel.java:268)
	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.startJGroupsChannelIfNeeded(JGroupsTransport.java:207)
	... 97 more

11:58:18,630 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-4) JBAS015877: Stopped deployment null (runtime-name: ejb.jar) in 30ms
11:58:18,632 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-9) JBAS015877: Stopped deployment appone.ear (runtime-name: appone.ear) in 33ms
11:58:18,634 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-13) JBAS015876: Starting deployment of "appone.ear" (runtime-name: "appone.ear")
11:58:18,634 ERROR [org.jboss.as.server] (DeploymentScanner-threads - 1) JBAS015863: Replacement of deployment "appone.ear" by deployment "appone.ear" was rolled back with the following failure message: "JBAS014750: Operation handler failed to complete"
11:58:18,643 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-14) JBAS015876: Starting deployment of "null" (runtime-name: "ejb.jar")
11:58:18,674 INFO  [org.jboss.as.ejb3.deployment.processors.EjbJndiBindingsDeploymentUnitProcessor] (MSC service thread 1-16) JNDI bindings for session bean named AppOneBean in deployment unit subdeployment "ejb.jar" of deployment "appone.ear" are as follows:

	java:global/appone/ejb/AppOneBean!org.jboss.as.quickstarts.ejb.multi.server.app.AppOne
	java:app/ejb/AppOneBean!org.jboss.as.quickstarts.ejb.multi.server.app.AppOne
	java:module/AppOneBean!org.jboss.as.quickstarts.ejb.multi.server.app.AppOne
	java:jboss/exported/appone/ejb/AppOneBean!org.jboss.as.quickstarts.ejb.multi.server.app.AppOne
	java:global/appone/ejb/AppOneBean
	java:app/ejb/AppOneBean
	java:module/AppOneBean

11:58:18,790 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 4) ISPN000078: Starting JGroups Channel
11:58:18,792 WARN  [org.jgroups.protocols.UDP] (ServerService Thread Pool -- 4) [JGRP00014] the send buffer of socket DatagramSocket was set to 640KB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)
11:58:18,792 WARN  [org.jgroups.protocols.UDP] (ServerService Thread Pool -- 4) [JGRP00014] the receive buffer of socket DatagramSocket was set to 20MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)
11:58:18,792 WARN  [org.jgroups.protocols.UDP] (ServerService Thread Pool -- 4) [JGRP00014] the send buffer of socket MulticastSocket was set to 640KB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)
11:58:18,792 WARN  [org.jgroups.protocols.UDP] (ServerService Thread Pool -- 4) [JGRP00014] the receive buffer of socket MulticastSocket was set to 25MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)
11:58:18,792 INFO  [stdout] (ServerService Thread Pool -- 4) 
11:58:18,793 INFO  [stdout] (ServerService Thread Pool -- 4) -------------------------------------------------------------------
11:58:18,793 INFO  [stdout] (ServerService Thread Pool -- 4) GMS: address=home/ejb, cluster=ejb, physical address=127.0.0.1:55200
11:58:18,793 INFO  [stdout] (ServerService Thread Pool -- 4) -------------------------------------------------------------------
11:58:18,799 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 4) ISPN000094: Received new cluster view: [node2/ejb|7] [node2/ejb, home/ejb]
11:58:18,821 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 4) ISPN000079: Cache local address is home/ejb, physical addresses are [127.0.0.1:55200]
11:58:18,823 INFO  [org.infinispan.factories.TransactionManagerFactory] (ServerService Thread Pool -- 4) ISPN000161: Using a batchMode transaction manager
11:58:18,828 INFO  [org.infinispan.factories.TransactionManagerFactory] (ServerService Thread Pool -- 22) ISPN000161: Using a batchMode transaction manager
11:58:18,831 INFO  [org.infinispan.jmx.CacheJmxRegistration] (ServerService Thread Pool -- 4) ISPN000031: MBeans were successfully registered to the platform MBean server.
11:58:18,837 INFO  [org.infinispan.jmx.CacheJmxRegistration] (ServerService Thread Pool -- 22) ISPN000031: MBeans were successfully registered to the platform MBean server.
11:58:18,841 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 4) JBAS010281: Started remote-connector-client-mappings cache from ejb container
11:58:18,845 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 22) JBAS010281: Started repl cache from ejb container
11:58:18,846 INFO  [org.jboss.as.clustering] (MSC service thread 1-10) JBAS010226: New cluster view for partition ejb: 7 (org.jboss.as.clustering.impl.CoreGroupCommunicationService$GroupView@215dd258 delta: 0, merge: false)
Comment 2 Paul Ferraro 2013-11-08 10:17:38 EST
https://github.com/jbossas/jboss-eap/pull/684
Comment 5 Ladislav Thon 2013-11-11 11:55:06 EST
When using 6.2.0.CR1 preview bits, the issue is still present.

I'm using the reproducer from WFLY-2458. That is:

1. unzip jboss-eap-6.2.0.CR1.zip twice to create 2 separate EAP dirs, say node1 and node2
2. cp appone.ear node1/standalone/deployments/
3. cp appone.ear node2/standalone/deployments/
4. node1/bin/standalone.sh -c standalone-ha.xml -Djboss.node.name=node1
5. node2/bin/standalone.sh -c standalone-ha.xml -Djboss.node.name=node2 -Djboss.socket.binding.port-offset=100
6. touch node1/standalone/deployments/appone.ear
7. watch the console log of node1

I still see the error:

17:47:43,543 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 58) MSC000001: Failed to start service jboss.infinispan.ejb.global-component-registry: org.jboss.msc.service.StartException in service jboss.infinispan.ejb.global-component-registry: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type JGroupsTransport
	at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:91) [jboss-as-clustering-common-7.3.0.Final-redhat-12.jar:7.3.0.Final-redhat-12]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_45]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_45]
	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_45]
	at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.1.Final-redhat-1.jar:2.1.1.Final-redhat-1]
Caused by: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type JGroupsTransport
	at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:247)
	at org.jboss.as.clustering.infinispan.subsystem.GlobalComponentRegistryService.start(GlobalComponentRegistryService.java:57)
	at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:86) [jboss-as-clustering-common-7.3.0.Final-redhat-12.jar:7.3.0.Final-redhat-12]
	... 4 more
Caused by: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type JGroupsTransport
	at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:205)
	at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:886)
	at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:657)
	at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:646)
	at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:549)
	at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:225)
	... 6 more
Caused by: org.infinispan.CacheException: Unable to start JGroups Channel
	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.startJGroupsChannelIfNeeded(JGroupsTransport.java:209)
	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.start(JGroupsTransport.java:198)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_45]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_45]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_45]
	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_45]
	at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203)
	... 11 more
Caused by: java.lang.IllegalStateException: channel is closed
	at org.jgroups.JChannel.checkClosed(JChannel.java:865)
	at org.jgroups.JChannel._preConnect(JChannel.java:520)
	at org.jgroups.JChannel.connect(JChannel.java:277)
	at org.jgroups.JChannel.connect(JChannel.java:268)
	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.startJGroupsChannelIfNeeded(JGroupsTransport.java:207)
	... 17 more

When I built WildFly master, which already has the fix, it failed too, so this is not EAP-specific. Moving back to assigned.
Comment 6 Paul Ferraro 2013-11-11 13:06:02 EST
OK - so the previous fix ensured that the channel exists before Infinispan's global component registry starts - however, this appears to be insufficient.
If undeploy doesn't trigger a full restart of the JGroups channel service, Infinispan's restarted cache manager attempts to reuse the channel that was previously closed, hence the exception.
The underlying problem is that Infinispan now controls the lifecycle of the channel, while the channel is provided by EAP.  However, Infinispan should only be connecting and disconnecting the channel.  The channel should not close until EAP says so.
I've opened an upstream issue for this:
https://issues.jboss.org/browse/ISPN-3697

Until then, we can trick Infinispan into not closing the channel - and instead make that the responsibility of the channel service.

I'll have a PR ready momentarily.

I should note that this issue has a workaround.
Instead of redeploying, the user can undeploy and deploy again.  The introduction of a brief period of time allows the channel service to stop - thus causing the redeploy to use a fresh channel, instead of reusing the old, closed one.
Comment 7 Paul Ferraro 2013-11-11 13:30:08 EST
https://github.com/jbossas/jboss-eap/pull/693
Comment 8 wfink 2013-11-12 02:13:55 EST
(In reply to Paul Ferraro from comment #6)
> I should note that this issue has a workaround.
> Instead of redeploying, the user can undeploy and deploy again.

Often this will not accepted by the customer as the wanted behaviour is that the action is automatically rolled-back if the new deployment failed. Also the time where the applicaiton is not reachable is longer.
Comment 9 Ladislav Thon 2013-11-12 02:40:10 EST
I just built the GitHub EAP 6.x branch, which contains Paul's fix from the pull request, and it's no good. When replaying the reproducer I described in comment 5, I no longer get the "java.lang.IllegalStateException: channel is closed" exception. Instead, I see a series of lovely NPEs, looking like this:

08:28:58,082 WARN  [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (OOB-19,shared=udp) Problems unmarshalling remote command from byte buffer: java.lang.NullPointerException
	at org.infinispan.marshall.jboss.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:164)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:355)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:213)
	at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:45)
	at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:163)
	...

08:28:58,083 ERROR [org.jgroups.blocks.RequestCorrelator] (OOB-20,shared=udp) failed unmarshalling buffer into return value: java.lang.NullPointerException
	at org.infinispan.marshall.jboss.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:164)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:355)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:213)
	at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:45)
	at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:163)
	...

And there's even this:

08:28:57,806 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 57) MSC000001: Failed to start service jboss.infinispan.ejb.repl: org.jboss.msc.service.StartException in service jboss.infinispan.ejb.repl: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
	at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:91) [jboss-as-clustering-common-7.3.0.Final-redhat-SNAPSHOT.jar:7.3.0.Final-redhat-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_45]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_45]
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45]
	at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.1.Final-redhat-1.jar:2.1.1.Final-redhat-1]
Caused by: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
	at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:205)
	at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:886)
	at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:657)
	at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:646)
	at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:549)
	...
Caused by: org.infinispan.CacheException: java.lang.NullPointerException
	at org.infinispan.util.Util.rewrapAsCacheException(Util.java:542)
	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:186)
	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:515)
	at org.infinispan.topology.LocalTopologyManagerImpl.executeOnCoordinator(LocalTopologyManagerImpl.java:293)
	at org.infinispan.topology.LocalTopologyManagerImpl.join(LocalTopologyManagerImpl.java:107)
	...
Caused by: java.lang.NullPointerException
	at org.infinispan.marshall.jboss.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:164)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:355)
	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:213)
	at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:45)
	at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:163)
	...

But it's all essentially the same.

Note that there is a LOT of these messages, they are being reported for 1 minute, after that, the log shows:

08:28:58,090 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000080: Disconnecting and closing JGroups Channel
08:28:58,455 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000082: Stopping the RpcDispatcher

Please, take this as a preliminary feedback. Once we get an official build, we'll provide official QA statement.
Comment 10 Paul Ferraro 2013-11-12 08:30:26 EST
This is a similar issue, but a separate cause - and needs to be addressed by upstream.  See: https://issues.jboss.org/browse/ISPN-3698
Comment 11 JBoss JIRA Server 2013-11-12 10:29:30 EST
Galder Zamarreño <galder.zamarreno@redhat.com> updated the status of jira ISPN-3698 to Coding In Progress
Comment 12 Paul Ferraro 2013-11-12 11:25:11 EST
This issue only appears to happen when the redeploy is triggered on multiple servers simultaneously.  Unfortunately, triggering redeploy via the CLI does exactly this.  As workaround a rollout plan can be used to perform redeploy.  In the HA case, this is probably preferable anyway, to avoid loss of service.
Comment 13 Paul Ferraro 2013-11-12 14:28:41 EST
This PR adds a workaround for ISPN-3698, and should completely fix this issue:
https://github.com/jbossas/jboss-eap/pull/697
Comment 14 JBoss JIRA Server 2013-11-12 15:34:43 EST
Paul Ferraro <paul.ferraro@redhat.com> made a comment on jira ISPN-3698

Bumping priority and setting target release flags where the fix is needed.
Comment 15 Ladislav Thon 2013-11-13 02:49:10 EST
I built the current EAP "upstream" branch which has the fix, and I can confirm that the issue is indeed fixed. Please take this as a preliminary feedback; once we get official build, we will provide official QA statement.
Comment 16 Ladislav Thon 2013-11-18 05:32:53 EST
Verified with EAP 6.2.0.CR2 preview.

Note You need to log in before you can comment on or make changes to this bug.