Description of problem: glusterd : all gluster commands are hung or not giving output on one server and after that commands are not giving output on other servers also Version-Release number of selected component (if applicable): 3.4.0.8rhs-1.el6rhs.x86_64 Steps to Reproduce: 1. had a cluster of four RHS server - mia, fred, fan, cutlass 2. power off one server - fred 3. try to detach that server from cluster forcefully from cutlass and it was taking too long, so killed that using Ctrl + c (more than 3 min) [root@cutlass ~]# gluster peer detach fred.lab.eng.blr.redhat.com force ^C 4. detach server form another peer [root@mia ~]# gluster peer detach fred.lab.eng.blr.redhat.com force peer detach: success [root@mia ~]# gluster peer status Number of Peers: 2 Hostname: 10.70.34.116 Uuid: c5154da1-be15-40e2-b5f3-9be6dadafd43 State: Peer in Cluster (Connected) Hostname: fan.lab.eng.blr.redhat.com Uuid: a37ff566-da82-4ae4-90c6-17763 5. try to execute volume info and other gluster command but no output root@cutlass ~]# gluster peer status ^C [root@cutlass ~]# gluster peer status ^C [root@cutlass ~]# service glusterd status \glusterd (pid 1542) is running... [root@cutlass ~]# gluster v info No volumes present [root@cutlass ~]# gluster v info No volumes present [root@mia ~]# gluster v status Tfile [root@mia ~]# Actual results: all commands ara hung or not giving output as expected Expected results: Additional info: [root@cutlass ~]# pgrep glusterd 1542 [root@cutlass ~]# gdb -p 1542 GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Attaching to process 1542 Reading symbols from /usr/sbin/glusterfsd...(no debugging symbols found)...done. Reading symbols from /usr/lib64/libglusterfs.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libglusterfs.so.0 Reading symbols from /usr/lib64/libgfrpc.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libgfrpc.so.0 Reading symbols from /usr/lib64/libgfxdr.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libgfxdr.so.0 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. [New LWP 1564] [New LWP 1563] [New LWP 1545] [New LWP 1544] [New LWP 1543] [Thread debugging using libthread_db enabled] Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libcrypto.so.10 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libz.so.1 Reading symbols from /usr/lib64/glusterfs/3.4.0.8rhs/xlator/mgmt/glusterd.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/glusterfs/3.4.0.8rhs/xlator/mgmt/glusterd.so Reading symbols from /usr/lib64/libxml2.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libxml2.so.2 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /usr/lib64/glusterfs/3.4.0.8rhs/rpc-transport/socket.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/glusterfs/3.4.0.8rhs/rpc-transport/socket.so Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libssl.so.10 Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libgssapi_krb5.so.2 Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done. Loaded symbols for /lib64/libkrb5.so.3 Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libcom_err.so.2 Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done. Loaded symbols for /lib64/libk5crypto.so.3 Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done. Loaded symbols for /lib64/libkrb5support.so.0 Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libkeyutils.so.1 Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libresolv.so.2 Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libselinux.so.1 Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libnss_files.so.2 Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libnss_dns.so.2 Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libgcc_s.so.1 0x0000003676a0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 Missing separate debuginfos, use: debuginfo-install glusterfs-3.4.0.8rhs-1.el6rhs.x86_64 (gdb) bt #0 0x0000003676a0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000003677244b33 in ?? () from /usr/lib64/libglusterfs.so.0 #2 0x0000003677244bce in synclock_lock () from /usr/lib64/libglusterfs.so.0 #3 0x00007faacd81baa1 in glusterd_big_locked_cbk () from /usr/lib64/glusterfs/3.4.0.8rhs/xlator/mgmt/glusterd.so #4 0x0000003677a0d153 in saved_frames_unwind () from /usr/lib64/libgfrpc.so.0 #5 0x0000003677a0d23e in saved_frames_destroy () from /usr/lib64/libgfrpc.so.0 #6 0x0000003677a0db23 in rpc_clnt_connection_cleanup () from /usr/lib64/libgfrpc.so.0 #7 0x00007faacd812f9b in glusterd_friend_cleanup () from /usr/lib64/glusterfs/3.4.0.8rhs/xlator/mgmt/glusterd.so #8 0x00007faacd7eba38 in glusterd_friend_remove () from /usr/lib64/glusterfs/3.4.0.8rhs/xlator/mgmt/glusterd.so #9 0x00007faacd7ebabc in glusterd_handle_friend_update_delete () from /usr/lib64/glusterfs/3.4.0.8rhs/xlator/mgmt/glusterd.so #10 0x00007faacd7f882c in __glusterd_handle_friend_update () from /usr/lib64/glusterfs/3.4.0.8rhs/xlator/mgmt/glusterd.so #11 0x00007faacd7ea26f in glusterd_big_locked_handler () from /usr/lib64/glusterfs/3.4.0.8rhs/xlator/mgmt/glusterd.so #12 0x0000003677a08655 in rpcsvc_handle_rpc_call () from /usr/lib64/libgfrpc.so.0 #13 0x0000003677a087a3 in rpcsvc_notify () from /usr/lib64/libgfrpc.so.0 #14 0x0000003677a098d8 in rpc_transport_notify () from /usr/lib64/libgfrpc.so.0 #15 0x00007faacd55a3a4 in socket_event_poll_in () from /usr/lib64/glusterfs/3.4.0.8rhs/rpc-transport/socket.so #16 0x00007faacd55a4fd in socket_event_handler () from /usr/lib64/glusterfs/3.4.0.8rhs/rpc-transport/socket.so #17 0x000000367725a9b7 in ?? () from /usr/lib64/libglusterfs.so.0 #18 0x0000000000406776 in main () (gdb) info th 6 Thread 0x7faacf883700 (LWP 1543) 0x0000003676a0f2a5 in sigwait () from /lib64/libpthread.so.0 5 Thread 0x7faacee82700 (LWP 1544) 0x0000003676a0b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 4 Thread 0x7faace481700 (LWP 1545) 0x0000003676a0b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 3 Thread 0x7faaccb76700 (LWP 1563) 0x0000003676a0ed2d in nanosleep () from /lib64/libpthread.so.0 2 Thread 0x7faac7fff700 (LWP 1564) 0x0000003676a0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 * 1 Thread 0x7faad0e1e700 (LWP 1542) 0x0000003676a0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 (gdb) t 2 [Switching to thread 2 (Thread 0x7faac7fff700 (LWP 1564))]#0 0x0000003676a0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 (gdb) bt #0 0x0000003676a0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007faacd84e253 in ?? () from /usr/lib64/glusterfs/3.4.0.8rhs/xlator/mgmt/glusterd.so #2 0x0000003676a07851 in start_thread () from /lib64/libpthread.so.0 #3 0x00000036762e890d in clone () from /lib64/libc.so.6 (gdb) l No symbol table is loaded. Use the "file" command. (gdb) info th 6 Thread 0x7faacf883700 (LWP 1543) 0x0000003676a0f2a5 in sigwait () from /lib64/libpthread.so.0 5 Thread 0x7faacee82700 (LWP 1544) 0x0000003676a0b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 4 Thread 0x7faace481700 (LWP 1545) 0x0000003676a0b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 3 Thread 0x7faaccb76700 (LWP 1563) 0x0000003676a0ed2d in nanosleep () from /lib64/libpthread.so.0 * 2 Thread 0x7faac7fff700 (LWP 1564) 0x0000003676a0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 1 Thread 0x7faad0e1e700 (LWP 1542) 0x0000003676a0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 (gdb) t 3 [Switching to thread 3 (Thread 0x7faaccb76700 (LWP 1563))]#0 0x0000003676a0ed2d in nanosleep () from /lib64/libpthread.so.0 (gdb) bt #0 0x0000003676a0ed2d in nanosleep () from /lib64/libpthread.so.0 #1 0x0000003677229dac in gf_timer_proc () from /usr/lib64/libglusterfs.so.0 #2 0x0000003676a07851 in start_thread () from /lib64/libpthread.so.0 #3 0x00000036762e890d in clone () from /lib64/libc.so.6 (gdb) t 4 [Switching to thread 4 (Thread 0x7faace481700 (LWP 1545))]#0 0x0000003676a0b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 (gdb) bt #0 0x0000003676a0b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000003677244caf in syncenv_task () from /usr/lib64/libglusterfs.so.0 #2 0x0000003677248aa0 in syncenv_processor () from /usr/lib64/libglusterfs.so.0 #3 0x0000003676a07851 in start_thread () from /lib64/libpthread.so.0 #4 0x00000036762e890d in clone () from /lib64/libc.so.6 (gdb) t 5 [Switching to thread 5 (Thread 0x7faacee82700 (LWP 1544))]#0 0x0000003676a0b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 (gdb) bt #0 0x0000003676a0b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000003677244caf in syncenv_task () from /usr/lib64/libglusterfs.so.0 #2 0x0000003677248aa0 in syncenv_processor () from /usr/lib64/libglusterfs.so.0 #3 0x0000003676a07851 in start_thread () from /lib64/libpthread.so.0 #4 0x00000036762e890d in clone () from /lib64/libc.so.6 (gdb) t 6 [Switching to thread 6 (Thread 0x7faacf883700 (LWP 1543))]#0 0x0000003676a0f2a5 in sigwait () from /lib64/libpthread.so.0 (gdb) bt #0 0x0000003676a0f2a5 in sigwait () from /lib64/libpthread.so.0 #1 0x000000000040523b in glusterfs_sigwaiter () #2 0x0000003676a07851 in start_thread () from /lib64/libpthread.so.0 #3 0x00000036762e890d in clone () from /lib64/libc.so.6 (gdb) t 7 Thread ID 7 not known. (gdb) t [Current thread is 6 (Thread 0x7faacf883700 (LWP 1543))] (gdb) t 1 [Switching to thread 1 (Thread 0x7faad0e1e700 (LWP 1542))]#0 0x0000003676a0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 (gdb) bt #0 0x0000003676a0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000003677244b33 in ?? () from /usr/lib64/libglusterfs.so.0 #2 0x0000003677244bce in synclock_lock () from /usr/lib64/libglusterfs.so.0 #3 0x00007faacd81baa1 in glusterd_big_locked_cbk () from /usr/lib64/glusterfs/3.4.0.8rhs/xlator/mgmt/glusterd.so #4 0x0000003677a0d153 in saved_frames_unwind () from /usr/lib64/libgfrpc.so.0 #5 0x0000003677a0d23e in saved_frames_destroy () from /usr/lib64/libgfrpc.so.0 #6 0x0000003677a0db23 in rpc_clnt_connection_cleanup () from /usr/lib64/libgfrpc.so.0 #7 0x00007faacd812f9b in glusterd_friend_cleanup () from /usr/lib64/glusterfs/3.4.0.8rhs/xlator/mgmt/glusterd.so #8 0x00007faacd7eba38 in glusterd_friend_remove () from /usr/lib64/glusterfs/3.4.0.8rhs/xlator/mgmt/glusterd.so #9 0x00007faacd7ebabc in glusterd_handle_friend_update_delete () from /usr/lib64/glusterfs/3.4.0.8rhs/xlator/mgmt/glusterd.so #10 0x00007faacd7f882c in __glusterd_handle_friend_update () from /usr/lib64/glusterfs/3.4.0.8rhs/xlator/mgmt/glusterd.so #11 0x00007faacd7ea26f in glusterd_big_locked_handler () from /usr/lib64/glusterfs/3.4.0.8rhs/xlator/mgmt/glusterd.so #12 0x0000003677a08655 in rpcsvc_handle_rpc_call () from /usr/lib64/libgfrpc.so.0 #13 0x0000003677a087a3 in rpcsvc_notify () from /usr/lib64/libgfrpc.so.0 #14 0x0000003677a098d8 in rpc_transport_notify () from /usr/lib64/libgfrpc.so.0 #15 0x00007faacd55a3a4 in socket_event_poll_in () from /usr/lib64/glusterfs/3.4.0.8rhs/rpc-transport/socket.so #16 0x00007faacd55a4fd in socket_event_handler () from /usr/lib64/glusterfs/3.4.0.8rhs/rpc-transport/socket.so #17 0x000000367725a9b7 in ?? () from /usr/lib64/libglusterfs.so.0 #18 0x0000000000406776 in main () (gdb) thread apply all bt full Thread 6 (Thread 0x7faacf883700 (LWP 1543)): #0 0x0000003676a0f2a5 in sigwait () from /lib64/libpthread.so.0 No symbol table info available. #1 0x000000000040523b in glusterfs_sigwaiter () No symbol table info available. #2 0x0000003676a07851 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #3 0x00000036762e890d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 5 (Thread 0x7faacee82700 (LWP 1544)): #0 0x0000003676a0b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x0000003677244caf in syncenv_task () from /usr/lib64/libglusterfs.so.0 No symbol table info available. #2 0x0000003677248aa0 in syncenv_processor () from /usr/lib64/libglusterfs.so.0 No symbol table info available. #3 0x0000003676a07851 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #4 0x00000036762e890d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 4 (Thread 0x7faace481700 (LWP 1545)): #0 0x0000003676a0b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x0000003677244caf in syncenv_task () from /usr/lib64/libglusterfs.so.0 No symbol table info available. #2 0x0000003677248aa0 in syncenv_processor () from /usr/lib64/libglusterfs.so.0 No symbol table info available. #3 0x0000003676a07851 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. ---Type <return> to continue, or q <return> to quit--- #4 0x00000036762e890d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 3 (Thread 0x7faaccb76700 (LWP 1563)): #0 0x0000003676a0ed2d in nanosleep () from /lib64/libpthread.so.0 No symbol table info available. #1 0x0000003677229dac in gf_timer_proc () from /usr/lib64/libglusterfs.so.0 No symbol table info available. #2 0x0000003676a07851 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #3 0x00000036762e890d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 2 (Thread 0x7faac7fff700 (LWP 1564)): #0 0x0000003676a0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007faacd84e253 in ?? () from /usr/lib64/glusterfs/3.4.0.8rhs/xlator/mgmt/glusterd.so No symbol table info available. #2 0x0000003676a07851 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #3 0x00000036762e890d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 1 (Thread 0x7faad0e1e700 (LWP 1542)): #0 0x0000003676a0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x0000003677244b33 in ?? () from /usr/lib64/libglusterfs.so.0 No symbol table info available. #2 0x0000003677244bce in synclock_lock () from /usr/lib64/libglusterfs.so.0 No symbol table info available. #3 0x00007faacd81baa1 in glusterd_big_locked_cbk () from /usr/lib64/glusterfs/3.4.0.8rhs/xlator/mgmt/glusterd.so No symbol table info available. ---Type <return> to continue, or q <return> to quit--- #4 0x0000003677a0d153 in saved_frames_unwind () from /usr/lib64/libgfrpc.so.0 No symbol table info available. #5 0x0000003677a0d23e in saved_frames_destroy () from /usr/lib64/libgfrpc.so.0 No symbol table info available. #6 0x0000003677a0db23 in rpc_clnt_connection_cleanup () from /usr/lib64/libgfrpc.so.0 No symbol table info available. #7 0x00007faacd812f9b in glusterd_friend_cleanup () from /usr/lib64/glusterfs/3.4.0.8rhs/xlator/mgmt/glusterd.so No symbol table info available. #8 0x00007faacd7eba38 in glusterd_friend_remove () from /usr/lib64/glusterfs/3.4.0.8rhs/xlator/mgmt/glusterd.so No symbol table info available. #9 0x00007faacd7ebabc in glusterd_handle_friend_update_delete () from /usr/lib64/glusterfs/3.4.0.8rhs/xlator/mgmt/glusterd.so No symbol table info available. #10 0x00007faacd7f882c in __glusterd_handle_friend_update () from /usr/lib64/glusterfs/3.4.0.8rhs/xlator/mgmt/glusterd.so No symbol table info available. #11 0x00007faacd7ea26f in glusterd_big_locked_handler () from /usr/lib64/glusterfs/3.4.0.8rhs/xlator/mgmt/glusterd.so No symbol table info available. #12 0x0000003677a08655 in rpcsvc_handle_rpc_call () from /usr/lib64/libgfrpc.so.0 No symbol table info available. #13 0x0000003677a087a3 in rpcsvc_notify () from /usr/lib64/libgfrpc.so.0 No symbol table info available. #14 0x0000003677a098d8 in rpc_transport_notify () from /usr/lib64/libgfrpc.so.0 No symbol table info available. #15 0x00007faacd55a3a4 in socket_event_poll_in () from /usr/lib64/glusterfs/3.4.0.8rhs/rpc-transport/socket.so No symbol table info available. #16 0x00007faacd55a4fd in socket_event_handler () from /usr/lib64/glusterfs/3.4.0.8rhs/rpc-transport/socket.so No symbol table info available. #17 0x000000367725a9b7 in ?? () from /usr/lib64/libglusterfs.so.0 No symbol table info available. #18 0x0000000000406776 in main () No symbol table info available.
Root cause: ----------- From the backtrace we can see that glusterd has deadlocked trying to lock that it has already acquired - ie, trying to acquire a lock in a nested manner. This happens because of the following sequence of events, 1) glusterd_friend_cleanup function 'flushes' out messages in the rpc layer's queue, that haven't received a response. At this point, glusterd has already acquired the big lock[1]. 2) The side-effect of flushing out the messages is that the corresponding call backs are called. Call backs themselves are executed after acquiring the big lock. This results in the big lock being acquired in a nested manner [1] For more info on big-lock - http://review.gluster.org/#/c/4784/9//COMMIT_MSG
verified with 3.4.0.9rhs-1.el6.x86_64 Not able to reproduce, hence moving to verified state
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-1262.html