Bug 1315393

Summary: [GSS](JDG 6.6) Hot Rod client stuck sporadical if a server node is shutting down
Product: [JBoss] JBoss Data Grid 6 Reporter: wfink
Component: ServerAssignee: Tristan Tarrant <ttarrant>
Status: CLOSED UPSTREAM QA Contact: Martin Gencur <mgencur>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.6.0CC: jdg-bugs, vjuranek
Target Milestone: ER1   
Target Release: 6.6.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1316132 (view as bug list) Environment:
Last Closed: 2025-02-10 03:48:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1309749, 1316132    
Attachments:
Description Flags
client/server log and jstack files (2sec delay) - several times stop/start server2
none
client/server log and jstack files (2sec delay) - several times stop/start server2
none
second try server with ispn=TRACE none

Description wfink 2016-03-07 15:54:26 UTC
Description of problem:
If a multithreaded client fire put operations in parallel and one server of the cluster (here one of two) will be stopped the client will see sporadical an Exception but might continue after the server is down completely.

org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for message id[19416] returned server error (status=0x85): org.infinispan.commons.CacheException: Could not commit implicit transaction
	at org.infinispan.client.hotrod.impl.protocol.Codec20.checkForErrorsInResponseStatus(Codec20.java:343)
	at org.infinispan.client.hotrod.impl.protocol.Codec20.readPartialHeader(Codec20.java:133)
	at org.infinispan.client.hotrod.impl.protocol.Codec20.readHeader(Codec20.java:119)
	at org.infinispan.client.hotrod.impl.operations.HotRodOperation.readHeaderAndValidate(HotRodOperation.java:56)
	at org.infinispan.client.hotrod.impl.operations.AbstractKeyValueOperation.sendPutOperation(AbstractKeyValueOperation.java:56)
	at org.infinispan.client.hotrod.impl.operations.PutIfAbsentOperation.executeOperation(PutIfAbsentOperation.java:37)
	at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:56)
	at org.infinispan.client.hotrod.impl.RemoteCacheImpl.putIfAbsent(RemoteCacheImpl.java:263)
	at org.infinispan.client.hotrod.impl.RemoteCacheSupport.putIfAbsent(RemoteCacheSupport.java:54)
	at HotRodTestClient.updateCache(HotRodTestClient.java:69)
	at HotRodTestClient.lambda$queuePut$0(HotRodTestClient.java:89)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Comment 2 wfink 2016-03-07 16:03:02 UTC
Created attachment 1133828 [details]
client/server log and jstack files (2sec delay) - several times stop/start server2

Comment 3 wfink 2016-03-07 18:33:08 UTC
The client stuck for about 30sec until server2 is shutting down and show the following exception after continue:

2016-03-07 16:38:05,070 WARN  [Codec21] (pool-1-thread-3:) ISPN004005: Error received from the server: org.infinispan.commons.CacheException: Could not commit implicit transaction
Exception in thread 14, thread will die.
org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for message id[19416] returned server error (status=0x85): org.infinispan.commons.CacheException: Could not commit implicit transaction
	at org.infinispan.client.hotrod.impl.protocol.Codec20.checkForErrorsInResponseStatus(Codec20.java:343)
	at org.infinispan.client.hotrod.impl.protocol.Codec20.readPartialHeader(Codec20.java:133)
	at org.infinispan.client.hotrod.impl.protocol.Codec20.readHeader(Codec20.java:119)
	at org.infinispan.client.hotrod.impl.operations.HotRodOperation.readHeaderAndValidate(HotRodOperation.java:56)
	at org.infinispan.client.hotrod.impl.operations.AbstractKeyValueOperation.sendPutOperation(AbstractKeyValueOperation.java:56)
	at org.infinispan.client.hotrod.impl.operations.PutIfAbsentOperation.executeOperation(PutIfAbsentOperation.java:37)
	at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:56)
	at org.infinispan.client.hotrod.impl.RemoteCacheImpl.putIfAbsent(RemoteCacheImpl.java:263)
	at org.infinispan.client.hotrod.impl.RemoteCacheSupport.putIfAbsent(RemoteCacheSupport.java:54)
	at HotRodTestClient.updateCache(HotRodTestClient.java:69)
	at HotRodTestClient.lambda$queuePut$0(HotRodTestClient.java:89)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Comment 4 wfink 2016-03-07 18:35:10 UTC
Created attachment 1133890 [details]
second try server with ispn=TRACE

This time the client and server1 did not recover after server2 finished shutdown

Comment 8 Mike McCune 2016-03-28 23:42:22 UTC
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions

Comment 9 JBoss JIRA Server 2016-04-04 16:34:01 UTC
Sebastian Łaskawiec <slaskawi> updated the status of jira ISPN-5507 to Closed

Comment 10 JBoss JIRA Server 2016-04-04 16:34:05 UTC
Sebastian Łaskawiec <slaskawi> updated the status of jira ISPN-6341 to Closed

Comment 11 JBoss JIRA Server 2016-04-30 05:24:51 UTC
Tristan Tarrant <ttarrant> updated the status of jira ISPN-5507 to Reopened

Comment 12 JBoss JIRA Server 2016-04-30 05:25:06 UTC
Tristan Tarrant <ttarrant> updated the status of jira ISPN-5507 to Resolved

Comment 13 JBoss JIRA Server 2016-04-30 05:25:57 UTC
Tristan Tarrant <ttarrant> updated the status of jira ISPN-6341 to Reopened

Comment 14 JBoss JIRA Server 2016-04-30 05:26:17 UTC
Tristan Tarrant <ttarrant> updated the status of jira ISPN-6341 to Resolved

Comment 15 JBoss JIRA Server 2016-05-03 11:10:06 UTC
Dan Berindei <dberinde> updated the status of jira JDG-83 to Resolved

Comment 17 Red Hat Bugzilla 2025-02-10 03:48:51 UTC
This product has been discontinued or is no longer tracked in Red Hat Bugzilla.