Created attachment 575293 [details] glusterd log file Description of problem: After remove-brick glusterd doesn't close the unix-domain sockets. It appears (from the series of netstat outputs given below) as though it closes the sockets but reacquires them. root@booradley:~# netstat -nap | grep -i glusterd | grep unix | wc -l 1 root@booradley:~# while true; do gluster volume add-brick replica booradley:/home/export/repl-$i; export i=$(($i + 1)); sleep 30; done Add Brick successful ^C root@booradley:~# netstat -nap | grep -i glusterd | grep unix | wc -l 4 root@booradley:~# gluster volume remove-brick replica booradley:/home/export/repl-65 Removing brick(s) can result in data loss. Do you want to Continue? (y/n) y Remove Brick successful root@booradley:~# netstat -nap | grep -i glusterd | grep unix | wc -l 3 root@booradley:~# netstat -nap | grep -i glusterd | grep unix | wc -l 4 root@booradley:~# gluster volume add-brick replica booradley:/home/export/repl-3 Add Brick successful root@booradley:~# netstat -nap | grep -i glusterd | grep unix | wc -l 5 root@booradley:~# netstat -nap | grep -i glusterd | grep unix | wc -l 6 root@booradley:~# gluster volume remove-brick replica booradley:/home/export/repl-3 Removing brick(s) can result in data loss. Do you want to Continue? (y/n) y Remove Brick successful root@booradley:~# netstat -nap | grep -i glusterd | grep unix | wc -l 2 root@booradley:~# netstat -nap | grep -i glusterd | grep unix | wc -l 3 root@booradley:~# netstat -nap | grep -i glusterd | grep unix | wc -l 5 root@booradley:~# netstat -nap | grep -i glusterd | grep unix | wc -l 6 root@booradley:~# gluster volume add-brick replica booradley:/home/export/repl-4 Add Brick successful root@booradley:~# netstat -nap | grep -i glusterd | grep unix | wc -l 8 root@booradley:~# gluster volume remove-brick replica booradley:/home/export/repl-4 Removing brick(s) can result in data loss. Do you want to Continue? (y/n) y Remove Brick successful root@booradley:~# netstat -nap | grep -i glusterd | grep unix | wc -l 5 root@booradley:~# netstat -nap | grep -i glusterd | grep unix | wc -l 7 root@booradley:~# netstat -nap | grep -i glusterd | grep unix | wc -l 8 root@booradley:~# gluster volume add-brick replica booradley:/home/export/repl-3 Add Brick successful root@booradley:~# netstat -nap | grep -i glusterd | grep unix | wc -l 6 root@booradley:~# netstat -nap | grep -i glusterd | grep unix | wc -l 10 root@booradley:~# gluster volume remove-brick replica booradley:/home/export/repl-3 Removing brick(s) can result in data loss. Do you want to Continue? (y/n) y Remove Brick successful root@booradley:~# netstat -nap | grep -i glusterd | grep unix | wc -l 9 root@booradley:~# netstat -nap | grep -i glusterd | grep unix | wc -l 10 And in glusterd log files, I am seeing following errors: [2012-04-05 11:43:29.115905] E [glusterd-utils.c:1297:glusterd_brick_unlink_socket_file] 0-glusterd: Failed to remove /tmp/ca3e5f544d226f222f5e841579aa616c.socket error: Resource temporarily unavailable Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
CHANGE: http://review.gluster.com/3168 (glusterd,glusterfsd : Perform proper cleanup of connections and socket files) merged in master by Anand Avati (avati)
root@shishirng:~# gluster volume info Volume Name: new Type: Distribute Volume ID: 72ea227e-91cd-4af9-b5b6-306989684562 Status: Started Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: sng:/export/dir1 Brick2: sng:/export/dir2 root@shishirng:~# netstat -nap | grep -i glusterd | grep unix | wc -l 4 root@shishirng:~# gluster volume remove-brick new sng:/export/dir2 Removing brick(s) can result in data loss. Do you want to Continue? (y/n) y Remove Brick commit force successful root@shishirng:~# netstat -nap | grep -i glusterd | grep unix | wc -l 3