Description of problem: ======================= was running a script to create and delete volumes over a period of 2 days. noticed that glusterd crashed after about 2 days Looks like glusterd crashed as there were more thant 1900 volumes I have been creating new volumes(ie about 50) in every iteration, but not deleting them. After about 34 iterations and about 1500+ volumes present, below issues were seen 1) volume create failed on 35th loop for one volume "volume create: 35-indiaArb-10: failed: Locking failed on rhs-client38.lab.eng.blr.redhat.com. Please check log file for details." 2)one of the nodes ie rhs-client38.lab.eng.blr.redhat.com seems to have got disconnected and the node seemed to have got rebooted, and post that volume create started to fail on volumes which are being created on one of the nodes due to "failed: Host rhs-client38.lab.eng.blr.redhat.com not connected" 3) glusterd crashed, due to the possible load on the node where the script was running 4) few of the nodes got rebooted by themselves(possibly due to load) Note: IOs were planned for a few volumes, but they didn't succeed due to too much load on clients,and failed initially itself, so IOs don't come into picture for this issue Core was generated by `/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO'. Program terminated with signal 11, Segmentation fault. #0 0x00007f2cc839a44e in gd_mgmt_v3_unlock_timer_cbk () from /usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so Missing separate debuginfos, use: debuginfo-install glusterfs-fuse-3.12.2-13.el7rhgs.x86_64 (gdb) bt #0 0x00007f2cc839a44e in gd_mgmt_v3_unlock_timer_cbk () from /usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so #1 0x00007f2cd385a982 in gf_timer_proc () from /lib64/libglusterfs.so.0 #2 0x00007f2cd26abdd5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f2cd1f74b3d in clone () from /lib64/libc.so.6 (gdb) t a a bt Thread 13 (Thread 0x7f2cd3d34780 (LWP 18999)): #0 0x00007f2cd26acf47 in pthread_join () from /lib64/libpthread.so.0 #1 0x00007f2cd38aaaf8 in event_dispatch_epoll () from /lib64/libglusterfs.so.0 #2 0x000055bed2fd0247 in main (argc=5, argv=<optimized out>) at glusterfsd.c:2550 Thread 12 (Thread 0x7f2cc3574700 (LWP 19008)): #0 0x00007f2cd1f75113 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f2cd38aa392 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #2 0x00007f2cd26abdd5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f2cd1f74b3d in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f2c0efb5700 (LWP 6240)): #0 0x00007f2cd26afcf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f2cd3887e68 in syncenv_task () from /lib64/libglusterfs.so.0 #2 0x00007f2cd3888d30 in syncenv_processor () from /lib64/libglusterfs.so.0 #3 0x00007f2cd26abdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f2cd1f74b3d in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f2c0e7b4700 (LWP 6241)): #0 0x00007f2cd26afcf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f2cd3887e68 in syncenv_task () from /lib64/libglusterfs.so.0 #2 0x00007f2cd3888d30 in syncenv_processor () from /lib64/libglusterfs.so.0 #3 0x00007f2cd26abdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f2cd1f74b3d in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f2c11fbb700 (LWP 3257)): #0 0x00007f2cd26afcf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f2cd3887e68 in syncenv_task () from /lib64/libglusterfs.so.0 #2 0x00007f2cd3888d30 in syncenv_processor () from /lib64/libglusterfs.so.0 #3 0x00007f2cd26abdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f2cd1f74b3d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f2cca64b700 (LWP 19001)): #0 0x00007f2cd26b3411 in sigwait () from /lib64/libpthread.so.0 #1 0x000055bed2fd352b in glusterfs_sigwaiter (arg=<optimized out>) at glusterfsd.c:2137 #2 0x00007f2cd26abdd5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f2cd1f74b3d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f2cc3d75700 (LWP 19007)): #0 0x00007f2cd26af945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f2cc839614b in hooks_worker () from /usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so #2 0x00007f2cd26abdd5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f2cd1f74b3d in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f2c0f7b6700 (LWP 6239)): #0 0x00007f2cd26afcf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f2cd3887e68 in syncenv_task () from /lib64/libglusterfs.so.0 #2 0x00007f2cd3888d30 in syncenv_processor () from /lib64/libglusterfs.so.0 #3 0x00007f2cd26abdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f2cd1f74b3d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f2cc9e4a700 (LWP 19002)): #0 0x00007f2cd1f3b4fd in nanosleep () from /lib64/libc.so.6 #1 0x00007f2cd1f3b394 in sleep () from /lib64/libc.so.6 #2 0x00007f2cd387520d in pool_sweeper () from /lib64/libglusterfs.so.0 #3 0x00007f2cd26abdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f2cd1f74b3d in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f2c117ba700 (LWP 3258)): #0 0x00007f2cd26afcf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 ---Type <return> to continue, or q <return> to quit--- #1 0x00007f2cd3887e68 in syncenv_task () from /lib64/libglusterfs.so.0 #2 0x00007f2cd3888d30 in syncenv_processor () from /lib64/libglusterfs.so.0 #3 0x00007f2cd26abdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f2cd1f74b3d in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f2c0ffb7700 (LWP 6238)): #0 0x00007f2cd26afcf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f2cd3887e68 in syncenv_task () from /lib64/libglusterfs.so.0 #2 0x00007f2cd3888d30 in syncenv_processor () from /lib64/libglusterfs.so.0 #3 0x00007f2cd26abdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f2cd1f74b3d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f2c107b8700 (LWP 6237)): #0 0x00007f2cd26afcf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f2cd3887e68 in syncenv_task () from /lib64/libglusterfs.so.0 #2 0x00007f2cd3888d30 in syncenv_processor () from /lib64/libglusterfs.so.0 #3 0x00007f2cd26abdd5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f2cd1f74b3d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f2ccae4c700 (LWP 19000)): #0 0x00007f2cc839a44e in gd_mgmt_v3_unlock_timer_cbk () from /usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so #1 0x00007f2cd385a982 in gf_timer_proc () from /lib64/libglusterfs.so.0 #2 0x00007f2cd26abdd5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f2cd1f74b3d in clone () from /lib64/libc.so.6 Version-Release number of selected component (if applicable): How reproducible: -------------- hit it once Steps to Reproduce: 1.had a 6 node cluster 2. run script to create and start volumes in batches and in iterations (Script will be attached) 3. above crash was hit
sosreports @ http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/nchilaka/bug.1599220/ some sosreports may be missing due to glusterd taking huge time due to number of volumes core is in rhs-client18
also each node has test-logs directory with top command o/p
Let us pull in https://review.gluster.org/21228 to avoid this crash anyways as a bullet proof fix. As I mentioned earlier, the test done here is not within a supported configuration.
Please let us know for verifying this bug, should we be still testing with creations of about 2000 volumes? Is there anything else specific to be tested with the fix.
(In reply to nchilaka from comment #15) > Please let us know for verifying this bug, should we be still testing with > creations of about 2000 volumes? That should be sufficient. > Is there anything else specific to be tested with the fix. Nope. Thanks, Sanju
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0263