Bug 1315393 - [GSS](JDG 6.6) Hot Rod client stuck sporadical if a server node is shutting down
Summary: [GSS](JDG 6.6) Hot Rod client stuck sporadical if a server node is shutting down
Keywords:
Status: VERIFIED
Alias: None
Product: JBoss Data Grid 6
Classification: JBoss
Component: Server
Version: 6.6.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ER1
: 6.6.1
Assignee: Tristan Tarrant
QA Contact: Martin Gencur
URL:
Whiteboard:
Depends On:
Blocks: 1309749 1316132
TreeView+ depends on / blocked
 
Reported: 2016-03-07 15:54 UTC by wfink
Modified: 2022-03-03 09:11 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1316132 (view as bug list)
Environment:
Last Closed:
Type: Bug
Embargoed:

Attachments (Terms of Use)
client/server log and jstack files (2sec delay) - several times stop/start server2 (deleted)
2016-03-07 15:54 UTC, wfink 		no flags Details
client/server log and jstack files (2sec delay) - several times stop/start server2 (1.65 MB, application/zip)
2016-03-07 16:03 UTC, wfink 		no flags Details
second try server with ispn=TRACE (5.97 MB, application/zip)
2016-03-07 18:35 UTC, wfink 		no flags Details
Show Obsolete (1) View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker ISPN-5507 0 Blocker Resolved Transactions committed immediately before cache stop can block shutdown 2018-11-16 11:45:52 UTC
Red Hat Issue Tracker ISPN-6341 0 Major Resolved StateTransferManager should be the first component to stop 2018-11-16 11:45:52 UTC
Red Hat Issue Tracker JDG-83 0 Major Verified StateTransferManager should be the first component to stop 2018-11-16 11:45:52 UTC
Red Hat Knowledge Base (Solution) 2195381 0 None None None 2016-03-10 12:52:18 UTC

Description wfink 2016-03-07 15:54:26 UTC 
Description of problem:
If a multithreaded client fire put operations in parallel and one server of the cluster (here one of two) will be stopped the client will see sporadical an Exception but might continue after the server is down completely.

org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for message id[19416] returned server error (status=0x85): org.infinispan.commons.CacheException: Could not commit implicit transaction
	at org.infinispan.client.hotrod.impl.protocol.Codec20.checkForErrorsInResponseStatus(Codec20.java:343)
	at org.infinispan.client.hotrod.impl.protocol.Codec20.readPartialHeader(Codec20.java:133)
	at org.infinispan.client.hotrod.impl.protocol.Codec20.readHeader(Codec20.java:119)
	at org.infinispan.client.hotrod.impl.operations.HotRodOperation.readHeaderAndValidate(HotRodOperation.java:56)
	at org.infinispan.client.hotrod.impl.operations.AbstractKeyValueOperation.sendPutOperation(AbstractKeyValueOperation.java:56)
	at org.infinispan.client.hotrod.impl.operations.PutIfAbsentOperation.executeOperation(PutIfAbsentOperation.java:37)
	at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:56)
	at org.infinispan.client.hotrod.impl.RemoteCacheImpl.putIfAbsent(RemoteCacheImpl.java:263)
	at org.infinispan.client.hotrod.impl.RemoteCacheSupport.putIfAbsent(RemoteCacheSupport.java:54)
	at HotRodTestClient.updateCache(HotRodTestClient.java:69)
	at HotRodTestClient.lambda$queuePut$0(HotRodTestClient.java:89)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Comment 2 wfink 2016-03-07 16:03:02 UTC 
Created attachment 1133828 [details]
client/server log and jstack files (2sec delay) - several times stop/start server2
Comment 3 wfink 2016-03-07 18:33:08 UTC 
The client stuck for about 30sec until server2 is shutting down and show the following exception after continue:

2016-03-07 16:38:05,070 WARN  [Codec21] (pool-1-thread-3:) ISPN004005: Error received from the server: org.infinispan.commons.CacheException: Could not commit implicit transaction
Exception in thread 14, thread will die.
org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for message id[19416] returned server error (status=0x85): org.infinispan.commons.CacheException: Could not commit implicit transaction
	at org.infinispan.client.hotrod.impl.protocol.Codec20.checkForErrorsInResponseStatus(Codec20.java:343)
	at org.infinispan.client.hotrod.impl.protocol.Codec20.readPartialHeader(Codec20.java:133)
	at org.infinispan.client.hotrod.impl.protocol.Codec20.readHeader(Codec20.java:119)
	at org.infinispan.client.hotrod.impl.operations.HotRodOperation.readHeaderAndValidate(HotRodOperation.java:56)
	at org.infinispan.client.hotrod.impl.operations.AbstractKeyValueOperation.sendPutOperation(AbstractKeyValueOperation.java:56)
	at org.infinispan.client.hotrod.impl.operations.PutIfAbsentOperation.executeOperation(PutIfAbsentOperation.java:37)
	at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:56)
	at org.infinispan.client.hotrod.impl.RemoteCacheImpl.putIfAbsent(RemoteCacheImpl.java:263)
	at org.infinispan.client.hotrod.impl.RemoteCacheSupport.putIfAbsent(RemoteCacheSupport.java:54)
	at HotRodTestClient.updateCache(HotRodTestClient.java:69)
	at HotRodTestClient.lambda$queuePut$0(HotRodTestClient.java:89)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Comment 4 wfink 2016-03-07 18:35:10 UTC 
Created attachment 1133890 [details]
second try server with ispn=TRACE

This time the client and server1 did not recover after server2 finished shutdown
Comment 8 Mike McCune 2016-03-28 23:42:22 UTC 
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions
Comment 9 JBoss JIRA Server 2016-04-04 16:34:01 UTC 
Sebastian Łaskawiec <slaskawi> updated the status of jira ISPN-5507 to Closed
Comment 10 JBoss JIRA Server 2016-04-04 16:34:05 UTC 
Sebastian Łaskawiec <slaskawi> updated the status of jira ISPN-6341 to Closed
Comment 11 JBoss JIRA Server 2016-04-30 05:24:51 UTC 
Tristan Tarrant <ttarrant> updated the status of jira ISPN-5507 to Reopened
Comment 12 JBoss JIRA Server 2016-04-30 05:25:06 UTC 
Tristan Tarrant <ttarrant> updated the status of jira ISPN-5507 to Resolved
Comment 13 JBoss JIRA Server 2016-04-30 05:25:57 UTC 
Tristan Tarrant <ttarrant> updated the status of jira ISPN-6341 to Reopened
Comment 14 JBoss JIRA Server 2016-04-30 05:26:17 UTC 
Tristan Tarrant <ttarrant> updated the status of jira ISPN-6341 to Resolved
Comment 15 JBoss JIRA Server 2016-05-03 11:10:06 UTC 
Dan Berindei <dberinde> updated the status of jira JDG-83 to Resolved
Note You need to log in before you can comment on or make changes to this bug.