Description of problem: I try to generate glusterdump file with SIGUSR1 to find out maximum op-version for the glusterd on this host. Glusterd crashes upon receiving SIGUSR1 Version-Release number of selected component (if applicable): # cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=14.04 DISTRIB_CODENAME=trusty DISTRIB_DESCRIPTION="Ubuntu 14.04.4 LTS" # dpkg -l | grep gluster ii glusterfs-client 3.8.1-ubuntu1~trusty1 amd64 clustered file-system (client package) ii glusterfs-common 3.8.1-ubuntu1~trusty1 amd64 GlusterFS common libraries and translator modules ii glusterfs-server 3.8.1-ubuntu1~trusty1 amd64 clustered file-system (server package) How reproducible: Glusterd crashes each time I send SIGUSR1 Steps to Reproduce: 1. rm -f /var/run/gluster/glusterdump.* 2. service glusterfs-server start 3. kill -SIGUSR1 $(pidof glusterd) 4. ps -ef | grep glusterd ; ls /var/run/gluster | grep glusterdump Actual results: No glusterd in the ps output. No glusterdump file created. glusterd crashes with the following lines in the log: [2016-07-22 11:22:50.923915] I [MSGID: 106493] [glusterd-rpc-ops.c:691:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: c6654b8f-3ca8-424f-8d0f-7c0b7d8108cc pending frames: patchset: git://git.gluster.com/glusterfs.git signal received: 11 time of crash: 2016-07-22 11:23:09 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.8.1 /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_msg_backtrace_nomem+0x92)[0x7f21f4b87b02] /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_print_trace+0x31d)[0x7f21f4b9104d] /lib/x86_64-linux-gnu/libc.so.6(+0x36cb0)[0x7f21f3f81cb0] /lib/x86_64-linux-gnu/libc.so.6(_IO_vfprintf+0x1d03)[0x7f21f3f96943] /lib/x86_64-linux-gnu/libc.so.6(__vsnprintf_chk+0x95)[0x7f21f4054165] /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_proc_dump_add_section_fd+0x69)[0x7f21f4bb18a9] /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_proc_dump_add_section+0xa5)[0x7f21f4bb1ab5] /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x4ce3e)[0x7f21f4bb1e3e] /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_proc_dump_xlator_info+0x58)[0x7f21f4bb2658] /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_proc_dump_info+0x500)[0x7f21f4bb2d00] /usr/sbin/glusterd(glusterfs_sigwaiter+0xce)[0x55ffe99d2bae] /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7f21f4318184] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f21f404537d] --------- Expected results: Glusterd creates glusterdump file in /var/run/gluster Additional info:
Does this happen everytime? I've seen it earlier as well but couldn't reproduce.
coredump file?
A fix http://review.gluster.org/14987 is posted in master branch.
Created attachment 1182856 [details] coredump coredump for /usr/sbin/glusterd from glusterfs-server deb package, version 3.8.1-ubuntu1~trusty1, ppa: "deb http://ppa.launchpad.net/gluster/glusterfs-3.8/ubuntu trusty main"
Atin, Thank you for the quick reply! This is a release build without debug symbols. I've attached coredump. The backtrace for the crashed thread is: (gdb) bt #0 0x00007f591bfde943 in _IO_vfprintf_internal (s=s@entry=0x7f59196d01d0, format=<optimized out>, format@entry=0x7f591cc6aa10 "%s.%s - usage-type %s memusage", ap=ap@entry=0x7f59196d1398) at vfprintf.c:1661 #1 0x00007f591c09c165 in ___vsnprintf_chk (s=0x7f59196d0362 "mgmt/glusterd.management - usage-type ", maxlen=<optimized out>, flags=1, slen=<optimized out>, format=0x7f591cc6aa10 "%s.%s - usage-type %s memusage", args=0x7f59196d1398) at vsnprintf_chk.c:63 #2 0x00007f591cbf98a9 in gf_proc_dump_add_section_fd () from /usr/lib/x86_64-linux-gnu/libglusterfs.so.0 #3 0x00007f591cbf9ab5 in gf_proc_dump_add_section () from /usr/lib/x86_64-linux-gnu/libglusterfs.so.0 #4 0x00007f591cbf9e3e in ?? () from /usr/lib/x86_64-linux-gnu/libglusterfs.so.0 #5 0x00007f591cbfa658 in gf_proc_dump_xlator_info () from /usr/lib/x86_64-linux-gnu/libglusterfs.so.0 #6 0x00007f591cbfad00 in gf_proc_dump_info () from /usr/lib/x86_64-linux-gnu/libglusterfs.so.0 #7 0x0000560afcb5dbae in glusterfs_sigwaiter () #8 0x00007f591c360184 in start_thread (arg=0x7f59196d7700) at pthread_create.c:312 #9 0x00007f591c08d37d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 List of threads: (gdb) info threads Id Target Id Frame 7 Thread 0x7f59152df700 (LWP 12079) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 6 Thread 0x7f5919ed8700 (LWP 11999) 0x00007f591c367b9d in nanosleep () at ../sysdeps/unix/syscall-template.S:81 5 Thread 0x7f5914ade700 (LWP 12080) 0x00007f591c08da13 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 4 Thread 0x7f59186d5700 (LWP 12002) pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 3 Thread 0x7f591d0a4780 (LWP 11998) 0x00007f591c36165b in pthread_join (threadid=140020575758080, thread_return=0x0) at pthread_join.c:92 2 Thread 0x7f5918ed6700 (LWP 12001) pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 * 1 Thread 0x7f59196d7700 (LWP 12000) 0x00007f591bfde943 in _IO_vfprintf_internal (s=s@entry=0x7f59196d01d0, format=<optimized out>, format@entry=0x7f591cc6aa10 "%s.%s - usage-type %s memusage", ap=ap@entry=0x7f59196d1398) at vfprintf.c:1661
And yes, it happens every time on this specific node. Other nodes (Ubuntu 14.04.4/i686, CentOS 7/x86_64) process SIGUSR1 correctly.
*** This bug has been marked as a duplicate of bug 1364329 ***