Bug 1396541

Summary: [GSS](6.4.z) HornetQ Enable BACKLOG_PROP_NAME in hornetq-client
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Shaun Appleton <sappleto>
Component: HornetQAssignee: Petr Jurak <pjurak>
Status: CLOSED CURRENTRELEASE QA Contact: Peter Mackay <pmackay>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 6.4.9CC: bmaxwell, csuconic, msochure, msvehla, pjurak, pmackay, rstancel
Target Milestone: CR1   
Target Release: EAP 6.4.13   
Hardware: Unspecified   
OS: Unspecified   
URL: https://c.na7.visual.force.com/apex/Case_View?id=500A000000Us4vm
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1396599 (view as bug list) Environment:
Last Closed: 2017-02-03 16:41:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1386335, 1390788, 1396599    
Attachments:
Description Flags
patch the shows the change none

Description Shaun Appleton 2016-11-18 15:27:08 UTC
Description of problem:
BACKLOG_PROP_NAME is not enabled n hornetq-client

Version-Release number of selected component (if applicable):
2.3.25.SP11-redhat-1

How reproducible:
very


Actual results:
can lead to repeated
ERROR [org.hornetq.core.client] (Thread-60 (HornetQ-client-factory-threads-1408487265-733221974)) HQ214003: Failed to handle failover: HornetQException[errorType=CONNECTION_TIMEDOUT message=HQ119014: Timed out waiting for response when sending packet 32]
.
.
.

Expected results:
with this fix these results no longer occur

Comment 1 Clebert Suconic 2016-11-18 16:54:41 UTC
What is the BACKLOG_PROP_NAME ? I'm confused.

Comment 2 Shaun Appleton 2016-11-18 17:12:02 UTC
BACKLOG_PROP_NAME is backlog

  public static final String BACKLOG_PROP_NAME = "backlog";

and backlog is parameter that can be set when opening a socket
The backlog is the number of requests allowable in the socket listen backlog[1].

By default java uses 50 [2].

This can be seen as follows when using ss as follows - 

[sappleton@dhcppc10 ifdownTest]$ ss -l | grep 5545
tcp    LISTEN     0      50     127.0.0.1:5545                  *:* 

If the backog limit gets hit a you get 'kernel: Possible SYN flooding on port X. Sending cookies' as detailed in [1].

Which on the HQ client results in repeated errors like:

ERROR [org.hornetq.core.client] (Thread-60 (HornetQ-client-factory-threads-1408487265-733221974)) HQ214003: Failed to handle failover: HornetQException[errorType=CONNECTION_TIMEDOUT message=HQ119014: Timed out waiting for response when sending packet 32]

The default for linux is SOMAXCONN = 128 , windows is 200
Enabling this will allow HornetQ to use a user defined value.

The customer has applied the fix and recompiled HornetQ + added the parameter with the same value as their SOMMAXCANN value as follows:

                <acceptors>
                    <netty-acceptor name="netty" socket-binding="messaging">
                        <param key="use-nio" value="true"/>
                        <param key="backlog" value="1028"/>
                    </netty-acceptor>
                    <netty-acceptor name="netty-throughput" socket-binding="messaging-throughput">
                        <param key="batch-delay" value="50"/>
                        <param key="direct-deliver" value="false"/>
                        <param key="use-nio" value="true"/>
                        <param key="backlog" value="1028"/>
                    </netty-acceptor>
                </acceptors>

they thensee it as then being used for port 5445 :

[root@ieatrcxb3804-1 ~]# sysctl -a | grep -i maxc
net.core.somaxconn = 1028
[root@ieatrcxb3804-1 ~]# ss -l | grep -i -E "5445|5455"
LISTEN     0      1028        10.247.244.26:5445                     *:*       
LISTEN     0      1028        10.247.244.26:apc-5455                  *:* 

and the repeated ERROR no longer occurs

[1] https://access.redhat.com/solutions/30453
[2] http://docs.oracle.com/javase/7/docs/api/java/net/ServerSocket.html#ServerSocket(int)

Comment 3 Shaun Appleton 2016-11-18 18:37:20 UTC
Created attachment 1221936 [details]
patch the shows the change

Comment 9 Peter Mackay 2017-01-03 15:54:33 UTC
Verified with EAP 6.4.13.CP.CR1

Comment 10 Petr Penicka 2017-02-03 16:41:58 UTC
Released with EAP 6.4.13 on Feb 02 2017.