Description of problem: **************************** After updating to latest gluster-block build, created a new block on the existing volume and trying to list and find info of the block created, It is seen that the list or info fails to show any output and gluster-blockd service stopped running. These results are observed when created a block and tried to delete the block and when tried to create a new block again with the same name the cli hangs and there is no output shown.After that none of the cli commands succeeds. [1494854519] ERROR: block_info_cli_1: RPC: Timed out block testBlock info on volume testvol failed [at gluster-block.c+140 :<glusterBlockCliRPC_1>] [1494854519] ERROR: failed getting info of block testBlock on volume testvol [at gluster-block.c+514 :<glusterBlockInfo>] [1494854519] ERROR: failed in info [at gluster-block.c+572 :<glusterBlockParseArgs>] [1494855032] ERROR: failed getting info of block testBlock on volume testvol [at gluster-block.c+514 :<glusterBlockInfo>] [1494855032] ERROR: failed in info [at gluster-block.c+572 :<glusterBlockParseArgs>] [1494855122] ERROR: failed getting info of block testBlock on volume testvol [at gluster-block.c+514 :<glusterBlockInfo>] [1494855122] ERROR: failed in info [at gluster-block.c+572 :<glusterBlockParseArgs>] [1494854577] WARNING: glusterBlockCreateRemoteAsync: return -1 failed in remote async create for block testNewblk on volume testvol with hosts 10.70.46.65 [at block_svc_routines.c+1156 :<glusterBlockAuditRequest>] [1494854577] INFO: Trying to serve request for (testNewblk) on volume testvol from spare machines [at block_svc_routines.c+1149 :<glusterBlockAuditRequest>] [1494854577] ERROR: connect on 10.70.46.65 failed (Connection refused) [at block_svc_routines.c+320 :<glusterBlockCallRPC_1>] [1494854577] ERROR: Connection refused hence failed in remote create for block testNewblk on host 10.70.46.65 volume (null) [at block_svc_routines.c+483 :<glusterBlockCreateRemote>] ************************************************** pstack output: 7d80c86e0, reply=reply@entry=0x7f17de3f4038) at block_svc_routines.c:1163 #345 0x0000000000405671 in glusterBlockAuditRequest (glfs=glfs@entry=0x7f17d80c3700, blk=blk@entry=0x7f17de3f4540, cobj=cobj@entry=0x7f17de3f4070, list=list@entry=0x7f17d80c86e0, reply=reply@entry=0x7f17de3f4038) at block_svc_routines.c:1163 #346 0x0000000000405671 in glusterBlockAuditRequest (glfs=glfs@entry=0x7f17d80c3700, blk=blk@entry=0x7f17de3f4540, cobj=cobj@entry=0x7f17de3f4070, list=list@entry=0x7f17d80c86e0, reply=reply@entry=0x7f17de3f4038) at block_svc_routines.c:1163 #347 0x0000000000405671 in glusterBlockAuditRequest (glfs=glfs@entry=0x7f17d80c3700, blk=blk@entry=0x7f17de3f4540, cobj=cobj@entry=0x7f17de3f4070, list=list@entry=0x7f17d80c86e0, reply=reply@entry=0x7f17de3f4038) at block_svc_routines.c:1163 #348 0x0000000000405671 in glusterBlockAuditRequest (glfs=glfs@entry=0x7f17d80c3700, blk=blk@entry=0x7f17de3f4540, cobj=cobj@entry=0x7f17de3f4070, list=list@entry=0x7f17d80c86e0, reply=reply@entry=0x7f17de3f4038) at block_svc_routines.c:1163 #349 0x0000000000405671 in glusterBlockAuditRequest (glfs=glfs@entry=0x7f17d80c3700, blk=blk@entry=0x7f17de3f4540, cobj=cobj@entry=0x7f17de3f4070, list=list@entry=0x7f17d80c86e0, reply=reply@entry=0x7f17de3f4038) at block_svc_routines.c:1163 #350 0x0000000000405671 in glusterBlockAuditRequest (glfs=glfs@entry=0x7f17d80c3700, blk=blk@entry=0x7f17de3f4540, cobj=cobj@entry=0x7f17de3f4070, list=list@entry=0x7f17d80c86e0, reply=reply@entry=0x7f17de3f4038) at block_svc_routines.c:1163 #351 0x0000000000405671 in glusterBlockAuditRequest (glfs=glfs@entry=0x7f17d80c3700, blk=blk@entry=0x7f17de3f4540, cobj=cobj@entry=0x7f17de3f4070, list=list@entry=0x7f17d80c86e0, reply=reply@entry=0x7f17de3f4038) at block_svc_routines.c:1163 #352 0x0000000000405671 in glusterBlockAuditRequest (glfs=glfs@entry=0x7f17d80c3700, blk=blk@entry=0x7f17de3f4540, cobj=cobj@entry=0x7f17de3f4070, list=list@entry=0x7f17d80c86e0, reply=reply@entry=0x7f17de3f4038) at block_svc_routines.c:1163 #353 0x0000000000405671 in glusterBlockAuditRequest (glfs=glfs@entry=0x7f17d80c3700, blk=blk@entry=0x7f17de3f4540, cobj=cobj@entry=0x7f17de3f4070, list=list@entry=0x7f17d80c86e0, reply=reply@entry=0x7f17de3f4038) at block_svc_routines.c:1163 #354 0x0000000000405671 in glusterBlockAuditRequest (glfs=glfs@entry=0x7f17d80c3700, blk=blk@entry=0x7f17de3f4540, cobj=cobj@entry=0x7f17de3f4070, list=list@entry=0x7f17d80c86e0, reply=reply@entry=0x7f17de3f4038) at block_svc_routines.c:1163 #355 0x0000000000405671 in glusterBlockAuditRequest (glfs=glfs@entry=0x7f17d80c3700, blk=blk@entry=0x7f17de3f4540, cobj=cobj@entry=0x7f17de3f4070, list=list@entry=0x7f17d80c86e0, reply=reply@entry=0x7f17de3f4038) at block_svc_routines.c:1163 #356 0x0000000000405671 in glusterBlockAuditRequest (glfs=glfs@entry=0x7f17d80c3700, blk=blk@entry=0x7f17de3f4540, cobj=cobj@entry=0x7f17de3f4070, list=list@entry=0x7f17d80c86e0, reply=reply@entry=0x7f17de3f4038) at block_svc_routines.c:1163 #357 0x0000000000405671 in glusterBlockAuditRequest (glfs=glfs@entry=0x7f17d80c3700, blk=blk@entry=0x7f17de3f4540, cobj=cobj@entry=0x7f17de3f4070, list=list@entry=0x7f17d80c86e0, reply=reply@entry=0x7f17de3f4038) at block_svc_routines.c:1163 #358 0x0000000000405671 in glusterBlockAuditRequest (glfs=glfs@entry=0x7f17d80c3700, blk=blk@entry=0x7f17de3f4540, cobj=cobj@entry=0x7f17de3f4070, list=list@entry=0x7f17d80c86e0, reply=reply@entry=0x7f17de3f4038) at block_svc_routines.c:1163 #359 0x0000000000405671 in glusterBlockAuditRequest (glfs=glfs@entry=0x7f17d80c3700, blk=blk@entry=0x7f17de3f4540, cobj=cobj@entry=0x7f17de3f4070, list=list@entry=0x7f17d80c86e0, reply=reply@entry=0x7f17de3f4038) at block_svc_routines.c:1163 #360 0x0000000000405671 in glusterBlockAuditRequest (glfs=glfs@entry=0x7f17d80c3700, blk=blk@entry=0x7f17de3f4540, cobj=cobj@entry=0x7f17de3f4070, list=list@entry=0x7f17d80c86e0, reply=reply@entry=0x7f17de3f4038) at block_svc_routines.c:1163 #361 0x0000000000405671 in glusterBlockAuditRequest (glfs=glfs@entry=0x7f17d80c3700, blk=blk@entry=0x7f17de3f4540, cobj=cobj@entry=0x7f17de3f4070, list=list@entry=0x7f17d80c86e0, reply=reply@entry=0x7f17de3f4038) at block_svc_routines.c:1163 #362 0x0000000000405671 in glusterBlockAuditRequest (glfs=glfs@entry=0x7f17d80c3700, blk=blk@entry=0x7f17de3f4540, cobj=cobj@entry=0x7f17de3f4070, list=list@entry=0x7f17d80c86e0, reply=reply@entry=0x7f17de3f4038) at block_svc_routines.c:1163 #363 0x0000000000405671 in glusterBlockAuditRequest (glfs=glfs@entry=0x7f17d80c3700, blk=blk@entry=0x7f17de3f4540, cobj=cobj@entry=0x7f17de3f4070, list=list@entry=0x7f17d80c86e0, reply=reply@entry=0x7f17de3f4038) at block_svc_routines.c:1163 #364 0x000000000040bffe in block_create_cli_1_svc (blk=blk@entry=0x7f17de3f4540, rqstp=rqstp@entry=0x7f17de3f47a0) at block_svc_routines.c:1772 #365 0x000000000040332f in gluster_block_cli_1 (rqstp=0x7f17de3f47a0, transp=0x7f17d80c23d0) at block_svc.c:132 #366 0x00007f17df430fc1 in svc_getreq_common () from /lib64/libc.so.6 #367 0x00007f17df43110e in svc_getreq_poll () from /lib64/libc.so.6 #368 0x00007f17df4348ff in svc_run () from /lib64/libc.so.6 #369 0x0000000000403025 in glusterBlockCliThreadProc (vargp=<optimized out>) at gluster-blockd.c:101 #370 0x00007f17e063edc5 in start_thread () from /lib64/libpthread.so.0 #371 0x00007f17df40073d in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f17ddbf4700 (LWP 21910)): #0 0x00007f17df3c72c9 in waitpid () from /lib64/libc.so.6 #1 0x00007f17df34ac72 in do_system () from /lib64/libc.so.6 #2 0x00007f17df34b021 in system () from /lib64/libc.so.6 #3 0x000000000040c6df in block_create_1_svc (blk=blk@entry=0x7f17ddbf33b0, rqstp=rqstp@entry=0x7f17ddbf3800) at block_svc_routines.c:1832 #4 0x000000000040318f in gluster_block_1 (rqstp=0x7f17ddbf3800, transp=0x7f17d014be70) at block_svc.c:63 #5 0x00007f17df430fc1 in svc_getreq_common () from /lib64/libc.so.6 #6 0x00007f17df43110e in svc_getreq_poll () from /lib64/libc.so.6 #7 0x00007f17df4348ff in svc_run () from /lib64/libc.so.6 #8 0x0000000000402c65 in glusterBlockServerThreadProc (vargp=<optimized out>) at gluster-blockd.c:156 #9 0x00007f17e063edc5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f17df40073d in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f17dc8f3700 (LWP 21919)): #0 0x00007f17e0642a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f17dff755b8 in syncenv_task () from /lib64/libglusterfs.so.0 #2 0x00007f17dff76400 in syncenv_processor () from /lib64/libglusterfs.so.0 #3 0x00007f17e063edc5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f17df40073d in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f17d7fff700 (LWP 21920)): #0 0x00007f17e0642a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f17dff755b8 in syncenv_task () from /lib64/libglusterfs.so.0 #2 0x00007f17dff76400 in syncenv_processor () from /lib64/libglusterfs.so.0 #3 0x00007f17e063edc5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f17df40073d in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f17ad186700 (LWP 22004)): #0 0x00007f17e0642a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f17dff755b8 in syncenv_task () from /lib64/libglusterfs.so.0 #2 0x00007f17dff76400 in syncenv_processor () from /lib64/libglusterfs.so.0 #3 0x00007f17e063edc5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f17df40073d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f17a96e5700 (LWP 22005)): #0 0x00007f17e0642a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f17dff755b8 in syncenv_task () from /lib64/libglusterfs.so.0 #2 0x00007f17dff76400 in syncenv_processor () from /lib64/libglusterfs.so.0 #3 0x00007f17e063edc5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f17df40073d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f17ab035700 (LWP 22006)): #0 0x00007f17e0645bdd in nanosleep () from /lib64/libpthread.so.0 #1 0x00007f17dff4a24e in gf_timer_proc () from /lib64/libglusterfs.so.0 #2 0x00007f17e063edc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f17df40073d in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f17aa834700 (LWP 22007)): #0 0x00007f17e063fef7 in pthread_join () from /lib64/libpthread.so.0 #1 0x00007f17dff97290 in event_dispatch_epoll () from /lib64/libglusterfs.so.0 #2 0x00007f17e020e254 in glfs_poller () from /lib64/libgfapi.so.0 #3 0x00007f17e063edc5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f17df40073d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f17a8ee4700 (LWP 22008)): #0 0x00007f17df400d13 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f17dff96ce0 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #2 0x00007f17e063edc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f17df40073d in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f17c2b9c700 (LWP 22017)): #0 0x00007f17e0642a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f17a82d390c in iot_worker () from /usr/lib64/glusterfs/3.8.4/xlator/performance/io-threads.so #2 0x00007f17e063edc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f17df40073d in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f17a7594700 (LWP 22018)): #0 0x00007f17df400d13 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f17dff96ce0 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #2 0x00007f17e063edc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f17df40073d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f17a6d93700 (LWP 30867)): #0 0x00007f17df3f5dfd in poll () from /lib64/libc.so.6 #1 0x00007f17df42d7e9 in readtcp () from /lib64/libc.so.6 #2 0x00007f17df427ad1 in xdrrec_getbytes () from /lib64/libc.so.6 #3 0x00007f17df427ce1 in xdrrec_getlong () from /lib64/libc.so.6 #4 0x00007f17df432ea9 in xdr_u_long () from /lib64/libc.so.6 #5 0x00007f17df426871 in xdr_replymsg () from /lib64/libc.so.6 #6 0x00007f17df42d5af in clnttcp_call () from /lib64/libc.so.6 #7 0x0000000000411199 in block_create_1 (argp=argp@entry=0x7f17a6d92a80, clnt=clnt@entry=0x7f17a06386e0) at block_clnt.c:18 #8 0x000000000040601b in glusterBlockCallRPC_1 (host=0x7f17d80dbc50 "10.70.46.65", cobj=cobj@entry=0x7f17a6d92a80, opt=opt@entry=CREATE_SRV, out=out@entry=0x7f17d80c0560) at block_svc_routines.c:337 #9 0x00000000004065e6 in glusterBlockCreateRemote (data=0x7f17d80c0540) at block_svc_routines.c:476 #10 0x00007f17e063edc5 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f17df40073d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f17e0a5e780 (LWP 21908)): #0 0x00007f17e063fef7 in pthread_join () from /lib64/libpthread.so.0 #1 0x0000000000402636 in main (argc=<optimized out>, argv=<optimized out>) at gluster-blockd.c:208 Version-Release number of selected component (if applicable): ********************** gluster-block-0.2-1.x86_64 How reproducible: ******************** 2/2 Steps to Reproduce: 1. Create a new block on existing volume 2. Try to list the block and get info on the block ------- hangs, after restarting gluster-blockd listed 3. Delete the block and do the list again. 4. Now again create the block with the same name ---create also hangs Actual results: ******************* The cli hangs to show the list or info on the block. The cli hangs to create the block Also seen gluster-blockd service crash once. Expected results: ****************** The block should get listed successfully and after create the block should get deleted and recreated again successfully without any hangs or crashes. Additional info: Sosreports to follow soon.
Related Patch: https://review.gluster.org/#/c/17340/
After creating , deleting and creating block for multiple times , there are no issues seen. gluster-block create blockstor/block6 ha 2 10.70.46.152,10.70.46.144 1GiB IQN: iqn.2016-12.org.gluster-block:2b0b6eec-cae2-44b6-914e-289a5ce83227 PORTAL(S): 10.70.46.152:3260 10.70.46.144:3260 RESULT: SUCCESS [root@dhcp46-152 ~]# gluster-block list blockstor block1 bluff-block block3 block6 [root@dhcp46-152 ~]# gluster-block delete blockstor/block6 SUCCESSFUL ON: 10.70.46.152 10.70.46.144 RESULT: SUCCESS [root@dhcp46-152 ~]# gluster-block create blockstor/block6 ha 2 10.70.46.152,10.70.46.144 1GiB IQN: iqn.2016-12.org.gluster-block:b24aa6f5-ae06-4f04-8336-b922d1e8e217 PORTAL(S): 10.70.46.152:3260 10.70.46.144:3260 RESULT: SUCCESS Marking the BZ verified. gluster-block-0.2.1-1.el7rhgs.x86_64 tcmu-runner-1.2.0-4.el7rhgs.x86_64
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:2773