Description of problem: When bind-insecure is turned on (which is the default now), it may happen that brick is not able to bind to port assigned by Glusterd for example 49192-49195... It seems to be because the rpc_clnt connections are binding to ports in the same range. One small fix for now could be to make rpc_clnt to get port numbers from 65535 in a descending order.
For more details refer to http://build.gluster.org/job/rackspace-regression-2GB-triggered/11740/consoleFull and lines like [2015-07-01 07:33:25.069736] E [socket.c:818:__socket_server_bind] 0-tcp.patchy-server: binding to failed: Address already in use
REVIEW: http://review.gluster.org/11512 (rpc: fix binding brick issue while bind-insecure is enabled) posted (#1) for review on master by Prasanna Kumar Kalever
REVIEW: http://review.gluster.org/11512 (rpc: fix binding brick issue while bind-insecure is enabled) posted (#2) for review on master by Prasanna Kumar Kalever
REVIEW: http://review.gluster.org/11512 (rpc: fix binding brick issue while bind-insecure is enabled) posted (#3) for review on master by Prasanna Kumar Kalever
COMMIT: http://review.gluster.org/11512 committed in master by Raghavendra G (rgowdapp) ------ commit 9442e7bf80f5c2189140918bb5588868204521ed Author: Prasanna Kumar Kalever <prasanna.kalever> Date: Wed Jun 24 12:21:02 2015 +0530 rpc: fix binding brick issue while bind-insecure is enabled problem: When bind-insecure is turned on (which is the default now), it may happen that brick is not able to bind to port assigned by Glusterd for example 49192-49195... It seems to occur because the rpc_clnt connections are binding to ports in the same range. so brick fails to bind to a port which is already used by someone else solution: fix for now is to make rpc_clnt to get port numbers from 65535 in a descending order, as a result port clash is minimized other fixes: previously rdma binds to port >= 1024 if it cannot find a free port < 1024, even when bind insecure was turned off(ref to commit '0e3fd04e'), this patch add's a check for bind-insecure in gf_rdma_client_bind function This patch also re-enable bind-insecure and allow insecure by default which was reverted (ref: commit cef1720) previously Change-Id: Ia1cfa93c5454e2ae0ff57813689b75de282ebd07 BUG: 1238661 Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever> Reviewed-on: http://review.gluster.org/11512 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Raghavendra G <rgowdapp>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report. glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/ [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user