Description of problem:

device remove fails when it has over 200 volumes carved from it although there is a free device available for replacement.

Error: Failed to remove device, error: Unable to replace brick 10.70.46.165:/var/lib/heketi/mounts/vg_2da562dbb930fc8d90f78f70075f8c2a/brick_015f4d4b0a1fe112807c36424993e06d/brick with 10.70.46.165:/var/lib/heketi/mounts/vg_03279ea02e78439382a985db0c4d92ed/brick_d22e5522be2657b399324e5f1cd4f81a/brick for volume vol_3c0d06f8338e11ff98c1de5bf1315acc

sh-4.2# gluster vol status vol_3c0d06f8338e11ff98c1de5bf1315acc
Status of volume: vol_3c0d06f8338e11ff98c1de5bf1315acc
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.46.165:/var/lib/heketi/mounts/v
g_03279ea02e78439382a985db0c4d92ed/brick_d2
2e5522be2657b399324e5f1cd4f81a/brick        N/A       N/A        N       N/A  
Brick 10.70.47.21:/var/lib/heketi/mounts/vg
_775747b65771ac369114ce29bf62b12c/brick_ded
fff49941c5960ccc8321426e65ccf/brick         49350     0          Y       97136
Brick 10.70.47.51:/var/lib/heketi/mounts/vg
_71049fe1ea71ce342fa4110c04cc7f98/brick_130
706b8fe642159156bdf428c55f3df/brick         49350     0          Y       28808
Self-heal Daemon on localhost               N/A       N/A        Y       100861
Self-heal Daemon on 10.70.47.21             N/A       N/A        Y       98156
Self-heal Daemon on 10.70.47.51             N/A       N/A        Y       29818
 
Task Status of Volume vol_3c0d06f8338e11ff98c1de5bf1315acc
------------------------------------------------------------------------------
There are no active volume tasks

The failure seems to be from the gluster (brick crash?). however, we should understand if having 200 volumes under a device is causing this failure. If yes, we must define the number of volumes a device can hold.

snippet of brick log from 10.70.46.165
=======================================
[2017-03-30 15:32:18.997153] I [rpc-clnt.c:1999:rpc_clnt_reconfig] 0-vol_3c0d06f8338e11ff98c1de5bf1315acc-client-2: changing port to 49350 (from 0)
[2017-03-30 15:32:18.999869] E [socket.c:2314:socket_connect_finish] 0-vol_3c0d06f8338e11ff98c1de5bf1315acc-client-1: connection to 10.70.47.21:49350 failed (No route to host)
[2017-03-30 15:32:18.999910] W [MSGID: 108001] [afr-common.c:4773:afr_notify] 0-vol_3c0d06f8338e11ff98c1de5bf1315acc-replicate-0: Client-quorum is not met
[2017-03-30 15:32:19.002853] E [socket.c:2314:socket_connect_finish] 0-vol_3c0d06f8338e11ff98c1de5bf1315acc-client-2: connection to 10.70.47.51:49350 failed (No route to host)
[2017-03-30 15:32:19.002882] E [MSGID: 108006] [afr-common.c:4684:afr_notify] 0-vol_3c0d06f8338e11ff98c1de5bf1315acc-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
[2017-03-30 15:32:19.007244] I [fuse-bridge.c:5241:fuse_graph_setup] 0-fuse: switched to graph 0
[2017-03-30 15:32:19.007517] I [fuse-bridge.c:4153:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22
[2017-03-30 15:32:19.007666] I [MSGID: 108006] [afr-common.c:4827:afr_local_init] 0-vol_3c0d06f8338e11ff98c1de5bf1315acc-replicate-0: no subvolumes up
[2017-03-30 15:32:19.008165] W [fuse-resolve.c:132:fuse_resolve_gfid_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001: failed to resolve (Transport endpoint is not connected)
[2017-03-30 15:32:19.008183] W [fuse-bridge.c:3174:fuse_setxattr_resume] 0-glusterfs-fuse: 2: SETXATTR 00000000-0000-0000-0000-000000000001/1 (trusted.replace-brick) resolution failed
[2017-03-30 15:32:19.150804] I [fuse-bridge.c:5082:fuse_thread_proc] 0-fuse: unmounting /tmp/mntc7tn20
[2017-03-30 15:32:19.008158] I [MSGID: 108006] [afr-common.c:4827:afr_local_init] 0-vol_3c0d06f8338e11ff98c1de5bf1315acc-replicate-0: no subvolumes up
[2017-03-30 15:32:19.151333] W [glusterfsd.c:1288:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) [0x7fd237548dc5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7fd238bdcc45] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7fd238bdcabb] ) 0-: received signum (15), shutting down
[2017-03-30 15:32:19.151379] I [fuse-bridge.c:5793:fini] 0-fuse: Unmounting '/tmp/mntc7tn20'.


Version-Release number of selected component (if applicable):
heketi-client-4.0.0-4.el7rhgs.x86_64

How reproducible:
1/1

Steps to Reproduce:
1. On a CNS system, have 200 volumes from a single device
2. Run remove device on this device

Actual results:
remove device failed

Expected results:
remove device should go through

Additional info:
logs shall be attached shortly