Bug 1165010

Summary: Regression TestFrameWork : Starting a process fails with "Port already in use" error in our regression test framework
Product: [Community] GlusterFS Reporter: Sachin Pandit <spandit>
Component: glusterdAssignee: bugs <bugs>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: mainlineCC: amukherj, bugs, ndevos, smohan
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-15 14:53:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Sachin Pandit 2014-11-18 06:04:11 UTC
Description of problem:

The root cause is the way in which a new node is added to the cluster.

Now we have N1(127.1.1.1) and N2(127.1.1.2) as two nodes in the cluster, each having a brick N1:B1 (127.1.1.1 : 49146) and N2:B2 (127.1.1.2 : 49147)

Now lets peer probe N3(127.1.1.3) from N1

1) Friend request is sent from N1 to N3. N3 added N1 in the peerinfo list i.e N1 and its uuid say [UUID1]
2) N3 get the brick infos from N1
3) N3 tries to start the bricks
       1) N3 tries to start the brick B1 and find its not a local brick, using the logic MY_UUID == brickinfuuid, which is false in this case,
          as the UUID of brickinfhostname (N1) is [UUID1] (as suggested by the peerinfo list) and MY_UUID is [UUID3]. Hence doesn't start it.
       2) N3 tries to start the brick B2. Now the problem lies here. N3 uses glusterd_resolve_brick() to resolve the UUID of B2->hostname(N2).
          In glusterd_resolve_brick(), it cannot find  N2 in the peerinfo list. Then it checks if N2 is a local loop back address. Since N2(127.1.1.2) starts with
          "127" it decides that its a local loop back address. Thus glusterd_resolve_brick() fills brickinfuuid with [UUID3]. Now as brickinfuuid == MY_UUID is
          true, N3 initiates the brick process B2 with -s 127.1.1.2 and *-posix.glusterd-uuid=[UUID3]. This process dies off immediately, But for a short amount of
          time it holds on to the  --brick-port, say for example 49155



Actual results:
Starting a process fails some of the times in our regression test framework


Expected results:


Additional info:

Comment 1 Niels de Vos 2014-11-18 12:28:51 UTC
You can probably use the "transport.socket.bind-address" option in the glusterd.vol file for this? See http://review.gluster.org/8910 for some more details.

Comment 2 Sachin Pandit 2014-11-19 05:41:24 UTC
I will give it a try. Thanks for the information.