Bug 1312185

Summary: [GSS](JDG 6.x) Double invalidate of invalid Hot Rod connections
Product: [JBoss] JBoss Data Grid 6 Reporter: dereed
Component: InfinispanAssignee: Tristan Tarrant <ttarrant>
Status: VERIFIED --- QA Contact: Martin Gencur <mgencur>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.4.1, 6.5.0, 6.6.0, 6.5.1CC: jdg-bugs, vjuranek, wfink
Target Milestone: ER1   
Target Release: 6.6.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1309749, 1324651    

Description dereed 2016-02-26 02:54:06 UTC
When there's a problem with a Hot Rod operation, RetryOnFailureOperation invalidates the connection twice (once in a catch block, and once in a finally block).

This causes the GenericKeyedObjectPool counts to get off, and anything relying on that count (such as the maxTotal configuration for the pool) to break.

Comment 1 dereed 2016-02-26 02:54:28 UTC
org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation

    public T execute() {
        ...
        try {
            transport = getTransport(retryCount, failedServers);
            return executeOperation(transport);
        } catch (TransportException te) {
            ...
            if (transport != null)
                transportFactory.invalidateTransport(te.getServerAddress(), transport);
        ...
        } finally {
            releaseTransport(transport);
        } 
        ...

   protected void releaseTransport(Transport transport) {
      if (transport != null)
         transportFactory.releaseTransport(transport);
   }


org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory:

   public void invalidateTransport(SocketAddress serverAddress, Transport transport) {  
       ...
       pool.invalidateObject(serverAddress, (TcpTransport) transport);

   public void releaseTransport(Transport transport) {
      TcpTransport tcpTransport = (TcpTransport) transport;
      if (!tcpTransport.isValid()) {
          ...
          pool.invalidateObject(tcpTransport.getServerAddress(), tcpTransport);

Comment 4 Shay Matasaro 2016-02-29 20:03:00 UTC
the impact of this issue for this customer is that more sockets then expected are getting created and using more memory , which results in server stability issues

Comment 6 JBoss JIRA Server 2016-03-16 15:16:01 UTC
Galder Zamarreño <galder.zamarreno> updated the status of jira ISPN-6275 to Coding In Progress

Comment 7 JBoss JIRA Server 2016-04-04 16:34:04 UTC
Sebastian Łaskawiec <slaskawi> updated the status of jira ISPN-6275 to Closed

Comment 8 JBoss JIRA Server 2016-04-14 13:40:20 UTC
Galder Zamarreño <galder.zamarreno> updated the status of jira JDG-82 to Resolved