Bug 810089

Summary: glusterd remove brick does not close the unix domain sockets.
Product: [Community] GlusterFS Reporter: Raghavendra G <rgowdapp>
Component: glusterdAssignee: Kaushal <kaushal>
Status: CLOSED CURRENTRELEASE QA Contact: shylesh <shmohan>
Severity: high Docs Contact:
Priority: high    
Version: mainlineCC: amarts, gluster-bugs, sgowda
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.4.0 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-07-24 13:13:06 EDT Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Bug Depends On:    
Bug Blocks: 817967    
Attachments:
Description Flags
glusterd log file none

Description Raghavendra G 2012-04-05 02:26:29 EDT
Created attachment 575293 [details]
glusterd log file

Description of problem:

After remove-brick glusterd doesn't close the unix-domain sockets. It appears (from the series of netstat outputs given below) as though it closes the sockets but reacquires them.

root@booradley:~# netstat -nap | grep -i glusterd | grep unix | wc -l
1

root@booradley:~# while true; do gluster volume add-brick replica booradley:/home/export/repl-$i;  export i=$(($i + 1)); sleep 30; done
Add Brick successful
^C
root@booradley:~# netstat -nap | grep -i glusterd | grep unix | wc -l
4
root@booradley:~#  gluster volume remove-brick replica booradley:/home/export/repl-65
Removing brick(s) can result in data loss. Do you want to Continue? (y/n) y
Remove Brick successful
root@booradley:~# netstat -nap | grep -i glusterd | grep unix | wc -l
3
root@booradley:~# netstat -nap | grep -i glusterd | grep unix | wc -l
4
root@booradley:~#  gluster volume add-brick replica booradley:/home/export/repl-3
Add Brick successful
root@booradley:~# netstat -nap | grep -i glusterd | grep unix | wc -l
5
root@booradley:~# netstat -nap | grep -i glusterd | grep unix | wc -l
6
root@booradley:~# gluster volume remove-brick replica booradley:/home/export/repl-3
Removing brick(s) can result in data loss. Do you want to Continue? (y/n) y
Remove Brick successful

root@booradley:~# netstat -nap | grep -i glusterd | grep unix | wc -l
2
root@booradley:~# netstat -nap | grep -i glusterd | grep unix | wc -l
3
root@booradley:~# netstat -nap | grep -i glusterd | grep unix | wc -l
5
root@booradley:~# netstat -nap | grep -i glusterd | grep unix | wc -l
6

root@booradley:~# gluster volume add-brick replica booradley:/home/export/repl-4
Add Brick successful
root@booradley:~# netstat -nap | grep -i glusterd | grep unix | wc -l
8
root@booradley:~# gluster volume remove-brick replica booradley:/home/export/repl-4
Removing brick(s) can result in data loss. Do you want to Continue? (y/n) y
Remove Brick successful
root@booradley:~# netstat -nap | grep -i glusterd | grep unix | wc -l
5
root@booradley:~# netstat -nap | grep -i glusterd | grep unix | wc -l
7
root@booradley:~# netstat -nap | grep -i glusterd | grep unix | wc -l
8
root@booradley:~# gluster volume add-brick replica booradley:/home/export/repl-3
Add Brick successful
root@booradley:~# netstat -nap | grep -i glusterd | grep unix | wc -l
6
root@booradley:~# netstat -nap | grep -i glusterd | grep unix | wc -l
10

root@booradley:~# gluster volume remove-brick replica booradley:/home/export/repl-3
Removing brick(s) can result in data loss. Do you want to Continue? (y/n) y
Remove Brick successful
root@booradley:~# netstat -nap | grep -i glusterd | grep unix | wc -l
9
root@booradley:~# netstat -nap | grep -i glusterd | grep unix | wc -l
10


And in glusterd log files, I am seeing following errors:

[2012-04-05 11:43:29.115905] E [glusterd-utils.c:1297:glusterd_brick_unlink_socket_file] 0-glusterd: Failed to remove /tmp/ca3e5f544d226f222f5e841579aa616c.socket error:
 Resource temporarily unavailable

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:
Comment 1 Anand Avati 2012-04-23 21:55:30 EDT
CHANGE: http://review.gluster.com/3168 (glusterd,glusterfsd : Perform proper cleanup of connections and socket files) merged in master by Anand Avati (avati@redhat.com)
Comment 2 shishir gowda 2012-05-29 06:39:43 EDT
root@shishirng:~# gluster volume info
 
Volume Name: new
Type: Distribute
Volume ID: 72ea227e-91cd-4af9-b5b6-306989684562
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: sng:/export/dir1
Brick2: sng:/export/dir2

root@shishirng:~# netstat -nap | grep -i glusterd | grep unix | wc -l
4

root@shishirng:~# gluster volume remove-brick new sng:/export/dir2
Removing brick(s) can result in data loss. Do you want to Continue? (y/n) y
Remove Brick commit force successful

root@shishirng:~# netstat -nap | grep -i glusterd | grep unix | wc -l
3