777669 – (SOA-181) Cannot start multiple nodes at the same time

Bug 777669 (SOA-181) - Cannot start multiple nodes at the same time

Summary: Cannot start multiple nodes at the same time

Keywords:
Status:	CLOSED NEXTRELEASE
Alias:	SOA-181
Product:	JBoss Enterprise SOA Platform 4
Classification:	JBoss
Component:	JBoss Messaging, Deployment
Sub Component:
Version:	4.2 IR6
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	---
Target Release:	4.2 IR9
Assignee:	Martin Vecera
QA Contact:
Docs Contact:
URL:	http://jira.jboss.org/jira/browse/SOA...
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2007-11-27 13:13 UTC by Martin Vecera
Modified:	2007-12-21 12:26 UTC (History)
CC List:	0 users
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:	[Server] Release ID: JBoss [EAP] 4.3.0.GA (build: SVNTag=JBPAPP_4_3_0_GA date=200710252032) all nodes are using the same Oracle database
Last Closed:	2007-12-21 12:26:27 UTC
Type:	Bug

Attachments	(Terms of Use)
logs.tar.bz2 (638.41 KB, application/x-bzip) 2007-11-27 13:16 UTC, Martin Vecera	no flags	Details
logs.tar.bz2 (169.35 KB, application/x-bzip) 2007-12-03 11:42 UTC, Martin Vecera	no flags	Details
View All Add an attachment (proposed patch, testcase, etc.)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	SOA-181	0	Major	Closed	Cannot start multiple nodes at the same time	2014-01-17 18:36:17 UTC

Description Martin Vecera 2007-11-27 13:13:13 UTC

Date of First Response: 2007-11-27 08:44:20
Workaround: Workaround Exists
Workaround Description: Start the nodes one by one.
project_key: SOA

There's no problem to start the nodes in a cluster one by one. Everything works perfectly here.
But when I fired the execution at almost the same time I've got the following exceptions:

Node 1:
Stopped after a few seconds
...
14:00:39,016 INFO  [SettingsFactory] Structured second-level cache entries: disabled
14:00:39,016 INFO  [SettingsFactory] Statistics: disabled
14:00:39,017 INFO  [SettingsFactory] Deleted entity synthetic identifier rollback: disabled
14:00:39,017 INFO  [SettingsFactory] Default entity-mode: pojo
14:00:39,017 INFO  [SettingsFactory] Named query checking : enabled
14:00:39,023 INFO  [SessionFactoryImpl] building session factory
14:00:39,069 INFO  [SessionFactoryObjectFactory] Factory name: java:/hibernate/MonitoringSessionFactory
14:00:39,069 INFO  [NamingHelper] JNDI InitialContext properties:{}
14:00:39,070 INFO  [NamingHelper] Creating subcontext: hibernate
14:00:39,071 INFO  [SessionFactoryObjectFactory] Bound factory to JNDI name: java:/hibernate/MonitoringSessionFactory
14:00:39,072 WARN  [SessionFactoryObjectFactory] InitialContext did not implement EventContext

Node 2:
2007-11-27 14:00:22,598 DEBUG [org.jboss.remoting.MicroRemoteClientInvoker] SocketClientInvoker[426a086a, bisocket://10.34.32.154:4457] connected
2007-11-27 14:00:22,600 DEBUG [org.jboss.remoting.ConnectionValidator] ConnectionValidator[SocketClientInvoker[426a086a, bisocket://10.34.32.154:4457], pingPeriod=2000 ms] started
2007-11-27 14:00:22,645 DEBUG [org.jboss.jms.server.connectionfactory.ConnectionFactoryJNDIMapper] Server[3].ConnFactoryJNDIMapper received notification from node 3
2007-11-27 14:00:22,670 WARN  [org.jboss.messaging.core.impl.JDBCSupport] SQLException caught, SQLState 23000 code:1- assuming deadlock detected, try:1
java.sql.SQLException: ORA-00001: unique constraint (SOAESB1.SYS_C0013883) violated

        at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
        at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:331)
        at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:288)
        at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:743)
        at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:216)
        at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:955)
        at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1168)
        at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3316)
        at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:3400)
        at org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeUpdate(WrappedPreparedStatement.java:251)
        at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice$1InsertBindings.doTransaction(MessagingPostOffice.java:2524)
        at org.jboss.messaging.core.impl.JDBCSupport$JDBCTxRunner.execute(JDBCSupport.java:385)
        at org.jboss.messaging.core.impl.JDBCSupport$JDBCTxRunner.executeWithRetry(JDBCSupport.java:407)
        at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.insertBindingInStorage(MessagingPostOffice.java:2535)
        at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.internalAddBinding(MessagingPostOffice.java:1797)
        at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.addBinding(MessagingPostOffice.java:431)
        at org.jboss.jms.server.destination.QueueService.startService(QueueService.java:110)
        at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
        at org.jboss.system.ServiceMBeanSupport.start(ServiceMBeanSupport.java:196)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
        at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
        at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
        at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
        at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
        at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
        at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
        at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
        at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:995)
        at $Proxy0.start(Unknown Source)
        at org.jboss.system.ServiceController.start(ServiceController.java:417)
        at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
        at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
        at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
        at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
        at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
        at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
        at $Proxy4.start(Unknown Source)
        at org.jboss.deployment.SARDeployer.start(SARDeployer.java:302)
        at org.jboss.deployment.MainDeployer.start(MainDeployer.java:1025)
        at org.jboss.deployment.MainDeployer.start(MainDeployer.java:1015)
        at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:819)
        at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:782)
        at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
        at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
        at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
        at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
        at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
        at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
        at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
        at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
        at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
        at $Proxy9.deploy(Unknown Source)
        at org.jboss.deployment.scanner.URLDeploymentScanner.deploy(URLDeploymentScanner.java:421)
        at org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:634)
        at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.doScan(AbstractDeploymentScanner.java:263)
        at org.jboss.deployment.scanner.AbstractDeploymentScanner.startService(AbstractDeploymentScanner.java:336)
        at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
        at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:245)
        at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
        at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
        at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
        at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
        at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
        at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:978)
        at $Proxy0.start(Unknown Source)
        at org.jboss.system.ServiceController.start(ServiceController.java:417)
        at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
        at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
        at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
        at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
        at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
        at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
        at $Proxy4.start(Unknown Source)
        at org.jboss.deployment.SARDeployer.start(SARDeployer.java:302)
        at org.jboss.deployment.MainDeployer.start(MainDeployer.java:1025)
        at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:819)
        at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:782)
        at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:766)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
        at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
        at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
        at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
        at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
        at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
        at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
        at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
        at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
        at $Proxy5.deploy(Unknown Source)
        at org.jboss.system.server.ServerImpl.doStart(ServerImpl.java:482)
        at org.jboss.system.server.ServerImpl.start(ServerImpl.java:362)
        at org.jboss.Main.boot(Main.java:200)
        at org.jboss.Main$1.run(Main.java:508)
        at java.lang.Thread.run(Thread.java:595)
...
2007-11-27 14:00:30,679 ERROR [org.jboss.messaging.core.impl.JDBCSupport] Retried 25 times, now giving up
And was unable to start any queue.

Node 3:
14:00:45,952 INFO  [EJBDeployer] Deployed: file:/opt/jboss-soa-p.4.2.0.IR6/server/oracle/farm/farmtestfile.jar
14:00:46,651 INFO  [JBoss4ESBDeployer] create esb service, jbossesb.esb
14:00:46,993 INFO  [ProxyFactory] Bound EJB Home 'GangDispute' to jndi 'GangDispute'
14:00:47,598 WARN  [JDBCSupport] SQLException caught, SQLState 23000 code:1- assuming deadlock detected, try:1
java.sql.SQLException: ORA-00001: unique constraint (SOAESB1.SYS_C0013883) violated

        at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
        at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:331)
        at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:288)
        at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:743)
        at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:216)
        at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:955)
        at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1168)
        at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3316)
        at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:3400)
        at org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeUpdate(WrappedPreparedStatement.java:251)
        at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice$1InsertBindings.doTransaction(MessagingPostOffice.java:2524)
        at org.jboss.messaging.core.impl.JDBCSupport$JDBCTxRunner.execute(JDBCSupport.java:385)
        at org.jboss.messaging.core.impl.JDBCSupport$JDBCTxRunner.executeWithRetry(JDBCSupport.java:407)
        at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.insertBindingInStorage(MessagingPostOffice.java:2535)
        at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.internalAddBinding(MessagingPostOffice.java:1797)
        at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.addBinding(MessagingPostOffice.java:431)
        at org.jboss.jms.server.destination.QueueService.startService(QueueService.java:110)
        at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
        at org.jboss.system.ServiceMBeanSupport.start(ServiceMBeanSupport.java:196)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
        at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
        at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
        at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
...
Very similar like node 2.

See atached server logs.

It is 100% reproducible. I would expect the nodes to start simultaneously without problems.

Comment 1 Martin Vecera 2007-11-27 13:16:37 UTC

server.log files from individual nodes.

Comment 2 Martin Vecera 2007-11-27 13:16:37 UTC

Attachment: Added: logs.tar.bz2

Comment 3 Mark Little 2007-11-27 13:44:20 UTC

Have you tried this with EAP, i.e., without ESB, but with JBoss Messaging?

Comment 6 Mark Little 2007-11-28 11:40:49 UTC

Please try with vanilla EAP so we can isolate the cause.

Comment 7 Martin Vecera 2007-12-03 11:42:58 UTC

I tried it with EAP 4.3.0 Beta4 (because 4.2.0 doesn't have JBM and next SOA build will be based on 4.3.0). The second node experienced some problems (see server.2.log)

Comment 8 Martin Vecera 2007-12-03 11:42:58 UTC

Attachment: Added: logs.tar.bz2

Comment 9 Mark Little 2007-12-13 17:29:45 UTC

So this is an EAP (maybe JBM) issue. Can you raise it there and link this one to the new JIRA?

Comment 10 Martin Vecera 2007-12-13 17:36:37 UTC

Link: Added: This issue incorporates JBPAPP-505

Comment 11 Clebert Suconic 2007-12-18 00:44:34 UTC

Do you still see this issue?


It should been fixed a while ago:

JBMESSAGING-1142

Comment 12 Clebert Suconic 2007-12-18 03:32:37 UTC

Those stack traces on the log are from an old version... Please validate the issue with the latest version of JBM.

Comment 13 Martin Vecera 2007-12-18 09:38:12 UTC

I hope there will be new version (1.4.0.SP2) of JBM in IR9 so we can verify the fix.

Comment 14 Martin Vecera 2007-12-18 09:38:38 UTC

Link: Added: This issue is related to JBMESSAGING-1142

Comment 15 Alex Pinkin 2007-12-18 14:32:42 UTC

closing since JBPAPP-505 can not be reproduced with the EAP 4.3.0.GA

Comment 16 Martin Vecera 2007-12-18 15:33:06 UTC

This will be closed when verified with IR9.

Comment 17 Martin Vecera 2007-12-21 12:26:27 UTC

Verified with SOA-P beta1

Note You need to log in before you can comment on or make changes to this bug.