Bug 1315393 - [GSS](JDG 6.6) Hot Rod client stuck sporadical if a server node is shutting down
Summary: [GSS](JDG 6.6) Hot Rod client stuck sporadical if a server node is shutting down
Keywords:
Status: VERIFIED
Alias: None
Product: JBoss Data Grid 6
Classification: JBoss
Component: Server
Version: 6.6.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ER1
: 6.6.1
Assignee: Tristan Tarrant
QA Contact: Martin Gencur
URL:
Whiteboard:
Depends On:
Blocks: 1309749 1316132
TreeView+ depends on / blocked
 
Reported: 2016-03-07 15:54 UTC by wfink
Modified: 2022-03-03 09:11 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1316132 (view as bug list)
Environment:
Last Closed:
Type: Bug


Attachments (Terms of Use)
client/server log and jstack files (2sec delay) - several times stop/start server2 (deleted)
2016-03-07 15:54 UTC, wfink
no flags Details
client/server log and jstack files (2sec delay) - several times stop/start server2 (1.65 MB, application/zip)
2016-03-07 16:03 UTC, wfink
no flags Details
second try server with ispn=TRACE (5.97 MB, application/zip)
2016-03-07 18:35 UTC, wfink
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker ISPN-5507 0 Blocker Resolved Transactions committed immediately before cache stop can block shutdown 2018-11-16 11:45:52 UTC
Red Hat Issue Tracker ISPN-6341 0 Major Resolved StateTransferManager should be the first component to stop 2018-11-16 11:45:52 UTC
Red Hat Issue Tracker JDG-83 0 Major Verified StateTransferManager should be the first component to stop 2018-11-16 11:45:52 UTC
Red Hat Knowledge Base (Solution) 2195381 0 None None None 2016-03-10 12:52:18 UTC

Description wfink 2016-03-07 15:54:26 UTC
Description of problem:
If a multithreaded client fire put operations in parallel and one server of the cluster (here one of two) will be stopped the client will see sporadical an Exception but might continue after the server is down completely.

org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for message id[19416] returned server error (status=0x85): org.infinispan.commons.CacheException: Could not commit implicit transaction
	at org.infinispan.client.hotrod.impl.protocol.Codec20.checkForErrorsInResponseStatus(Codec20.java:343)
	at org.infinispan.client.hotrod.impl.protocol.Codec20.readPartialHeader(Codec20.java:133)
	at org.infinispan.client.hotrod.impl.protocol.Codec20.readHeader(Codec20.java:119)
	at org.infinispan.client.hotrod.impl.operations.HotRodOperation.readHeaderAndValidate(HotRodOperation.java:56)
	at org.infinispan.client.hotrod.impl.operations.AbstractKeyValueOperation.sendPutOperation(AbstractKeyValueOperation.java:56)
	at org.infinispan.client.hotrod.impl.operations.PutIfAbsentOperation.executeOperation(PutIfAbsentOperation.java:37)
	at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:56)
	at org.infinispan.client.hotrod.impl.RemoteCacheImpl.putIfAbsent(RemoteCacheImpl.java:263)
	at org.infinispan.client.hotrod.impl.RemoteCacheSupport.putIfAbsent(RemoteCacheSupport.java:54)
	at HotRodTestClient.updateCache(HotRodTestClient.java:69)
	at HotRodTestClient.lambda$queuePut$0(HotRodTestClient.java:89)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Comment 2 wfink 2016-03-07 16:03:02 UTC
Created attachment 1133828 [details]
client/server log and jstack files (2sec delay) - several times stop/start server2

Comment 3 wfink 2016-03-07 18:33:08 UTC
The client stuck for about 30sec until server2 is shutting down and show the following exception after continue:

2016-03-07 16:38:05,070 WARN  [Codec21] (pool-1-thread-3:) ISPN004005: Error received from the server: org.infinispan.commons.CacheException: Could not commit implicit transaction
Exception in thread 14, thread will die.
org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for message id[19416] returned server error (status=0x85): org.infinispan.commons.CacheException: Could not commit implicit transaction
	at org.infinispan.client.hotrod.impl.protocol.Codec20.checkForErrorsInResponseStatus(Codec20.java:343)
	at org.infinispan.client.hotrod.impl.protocol.Codec20.readPartialHeader(Codec20.java:133)
	at org.infinispan.client.hotrod.impl.protocol.Codec20.readHeader(Codec20.java:119)
	at org.infinispan.client.hotrod.impl.operations.HotRodOperation.readHeaderAndValidate(HotRodOperation.java:56)
	at org.infinispan.client.hotrod.impl.operations.AbstractKeyValueOperation.sendPutOperation(AbstractKeyValueOperation.java:56)
	at org.infinispan.client.hotrod.impl.operations.PutIfAbsentOperation.executeOperation(PutIfAbsentOperation.java:37)
	at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:56)
	at org.infinispan.client.hotrod.impl.RemoteCacheImpl.putIfAbsent(RemoteCacheImpl.java:263)
	at org.infinispan.client.hotrod.impl.RemoteCacheSupport.putIfAbsent(RemoteCacheSupport.java:54)
	at HotRodTestClient.updateCache(HotRodTestClient.java:69)
	at HotRodTestClient.lambda$queuePut$0(HotRodTestClient.java:89)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Comment 4 wfink 2016-03-07 18:35:10 UTC
Created attachment 1133890 [details]
second try server with ispn=TRACE

This time the client and server1 did not recover after server2 finished shutdown

Comment 8 Mike McCune 2016-03-28 23:42:22 UTC
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune@redhat.com with any questions

Comment 9 JBoss JIRA Server 2016-04-04 16:34:01 UTC
Sebastian Łaskawiec <slaskawi@redhat.com> updated the status of jira ISPN-5507 to Closed

Comment 10 JBoss JIRA Server 2016-04-04 16:34:05 UTC
Sebastian Łaskawiec <slaskawi@redhat.com> updated the status of jira ISPN-6341 to Closed

Comment 11 JBoss JIRA Server 2016-04-30 05:24:51 UTC
Tristan Tarrant <ttarrant@redhat.com> updated the status of jira ISPN-5507 to Reopened

Comment 12 JBoss JIRA Server 2016-04-30 05:25:06 UTC
Tristan Tarrant <ttarrant@redhat.com> updated the status of jira ISPN-5507 to Resolved

Comment 13 JBoss JIRA Server 2016-04-30 05:25:57 UTC
Tristan Tarrant <ttarrant@redhat.com> updated the status of jira ISPN-6341 to Reopened

Comment 14 JBoss JIRA Server 2016-04-30 05:26:17 UTC
Tristan Tarrant <ttarrant@redhat.com> updated the status of jira ISPN-6341 to Resolved

Comment 15 JBoss JIRA Server 2016-05-03 11:10:06 UTC
Dan Berindei <dberinde@redhat.com> updated the status of jira JDG-83 to Resolved


Note You need to log in before you can comment on or make changes to this bug.