Bug 1787463
| Summary: | Glusterd process is periodically crashing with a segmentation fault | ||
|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | Anthony Wingerter <awingerter> |
| Component: | glusterd | Assignee: | Sanju <srakonde> |
| Status: | CLOSED UPSTREAM | QA Contact: | |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 6 | CC: | bugs, pasik, srakonde |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-03-17 03:22:30 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Anthony Wingerter
2020-01-02 23:25:51 UTC
I have tried to look at the backtrace from the cores. Even though I installed release-6.1 I don't find any debug symbols. It looks like: Core was generated by `/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f442f3244a7 in ?? () [Current thread is 1 (LWP 21520)] (gdb) bt #0 0x00007f442f3244a7 in ?? () #1 0x4cce3ca800000001 in ?? () #2 0x0000000000018b1e in ?? () #3 0x00007f442f41faa8 in ?? () #4 0x00007f4400000000 in ?? () #5 0x00007f440c0174f0 in ?? () #6 0x00007f442f7b1b20 in ?? () #7 0x00007f441c4030c0 in ?? () #8 0x00007f442f7b1b90 in ?? () #9 0x00007f441c4030dc in ?? () #10 0x0000000000000007 in ?? () #11 0x0000562c75ecd4e0 in ?? () #12 0x00007f442f324db7 in ?? () #13 0x00007f4400000000 in ?? () #14 0x0000000000000000 in ?? () Can you please share output of "t a a bt" output? Thanks, Sanju Sanju, Thank you for the response. I am very unfamiliar with using gdb and collecting backtraces from the cores. Would it be possible for you to detail the configuration / collection steps needed? Thanks and best regards, -Anthony- Hi Anthony, 1. Take the core into gdb gdb glusterd <path to the corefile> 2. bt command gives you the backtrace of thread 1 and "t a a bt"(thread all apply backtrace) gives you backtrace of all threads. give "t a a bt" command at the gdb and collect the data. Hope that helps, Sanju Sanju, Thank you for the response. I apologize for getting back to you so late. Here is some data from one of the cores where glusterd crashed. [root@ch1c7ocvgl04 /]# gdb glusterd /core.7525 GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-115.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/sbin/glusterfsd...Reading symbols from /usr/sbin/glusterfsd...(no debugging symbols found)...done. (no debugging symbols found)...done. warning: core file may not match specified executable file. [New LWP 7657] [New LWP 7526] [New LWP 7529] [New LWP 7525] [New LWP 7527] [New LWP 7528] [New LWP 7531] [New LWP 7530] [New LWP 7656] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO'. Program terminated with signal 11, Segmentation fault. #0 0x00007fbac6a094a7 in glusterd_op_ac_brick_op_failed () from /usr/lib64/glusterfs/6.1/xlator/mgmt/glusterd.so Missing separate debuginfos, use: debuginfo-install glusterfs-server-6.1-1.el7.x86_64 (gdb) t a a bt Thread 9 (Thread 0x7fbac3a77700 (LWP 7656)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007fbac6aafddb in hooks_worker () from /usr/lib64/glusterfs/6.1/xlator/mgmt/glusterd.so #2 0x00007fbad16fedd5 in start_thread (arg=0x7fbac3a77700) at pthread_create.c:307 #3 0x00007fbad0fc5ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 8 (Thread 0x7fbac7e99700 (LWP 7530)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007fbad28ff810 in syncenv_task () from /lib64/libglusterfs.so.0 #2 0x00007fbad29006c0 in syncenv_processor () from /lib64/libglusterfs.so.0 #3 0x00007fbad16fedd5 in start_thread (arg=0x7fbac7e99700) at pthread_create.c:307 #4 0x00007fbad0fc5ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 7 (Thread 0x7fbac7698700 (LWP 7531)): #0 0x00007fbad0fbcf73 in select () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007fbad293e7e4 in runner () from /lib64/libglusterfs.so.0 #2 0x00007fbad16fedd5 in start_thread (arg=0x7fbac7698700) at pthread_create.c:307 #3 0x00007fbad0fc5ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 6 (Thread 0x7fbac8e9b700 (LWP 7528)): #0 0x00007fbad0f8ce2d in nanosleep () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007fbad0f8ccc4 in __sleep (seconds=0) at ../sysdeps/unix/sysv/linux/sleep.c:137 #2 0x00007fbad28eb54d in pool_sweeper () from /lib64/libglusterfs.so.0 #3 0x00007fbad16fedd5 in start_thread (arg=0x7fbac8e9b700) at pthread_create.c:307 #4 0x00007fbad0fc5ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 5 (Thread 0x7fbac969c700 (LWP 7527)): #0 0x00007fbad1706361 in do_sigwait (sig=0x7fbac969be1c, set=<optimized out>) at ../sysdeps/unix/sysv/linux/sigwait.c:60 #1 __sigwait (set=0x7fbac969be20, sig=0x7fbac969be1c) at ../sysdeps/unix/sysv/linux/sigwait.c:95 #2 0x000055b5e9cda1bb in glusterfs_sigwaiter () #3 0x00007fbad16fedd5 in start_thread (arg=0x7fbac969c700) at pthread_create.c:307 #4 0x00007fbad0fc5ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 4 (Thread 0x7fbad2dbe780 (LWP 7525)): #0 0x00007fbad16fff47 in pthread_join (threadid=140440114784000, thread_return=0x0) at pthread_join.c:90 #1 0x00007fbad2923478 in event_dispatch_epoll () from /lib64/libglusterfs.so.0 #2 0x000055b5e9cd6735 in main () Thread 3 (Thread 0x7fbac869a700 (LWP 7529)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007fbad28ff810 in syncenv_task () from /lib64/libglusterfs.so.0 #2 0x00007fbad29006c0 in syncenv_processor () from /lib64/libglusterfs.so.0 #3 0x00007fbad16fedd5 in start_thread (arg=0x7fbac869a700) at pthread_create.c:307 #4 0x00007fbad0fc5ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 ---Type <return> to continue, or q <return> to quit--- Thread 2 (Thread 0x7fbac9e9d700 (LWP 7526)): #0 0x00007fbad1705e3d in nanosleep () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007fbad28cdf76 in gf_timer_proc () from /lib64/libglusterfs.so.0 #2 0x00007fbad16fedd5 in start_thread (arg=0x7fbac9e9d700) at pthread_create.c:307 #3 0x00007fbad0fc5ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 1 (Thread 0x7fbac3276700 (LWP 7657)): #0 0x00007fbac6a094a7 in glusterd_op_ac_brick_op_failed () from /usr/lib64/glusterfs/6.1/xlator/mgmt/glusterd.so #1 0x00007fbac6a09db7 in glusterd_op_sm () from /usr/lib64/glusterfs/6.1/xlator/mgmt/glusterd.so #2 0x00007fbac6a419dc in glusterd_mgmt_v3_lock_peers_cbk_fn () from /usr/lib64/glusterfs/6.1/xlator/mgmt/glusterd.so #3 0x00007fbac6a40faa in glusterd_big_locked_cbk () from /usr/lib64/glusterfs/6.1/xlator/mgmt/glusterd.so #4 0x00007fbad2669021 in rpc_clnt_handle_reply () from /lib64/libgfrpc.so.0 #5 0x00007fbad2669387 in rpc_clnt_notify () from /lib64/libgfrpc.so.0 #6 0x00007fbad26659f3 in rpc_transport_notify () from /lib64/libgfrpc.so.0 #7 0x00007fbac5c0b875 in socket_event_handler () from /usr/lib64/glusterfs/6.1/rpc-transport/socket.so #8 0x00007fbad2924286 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #9 0x00007fbad16fedd5 in start_thread (arg=0x7fbac3276700) at pthread_create.c:307 #10 0x00007fbad0fc5ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Hi Anthony, Sorry for delayed response on this bug. Can you please install the debuginfo package related to glusterfs and then provide the back trace? Thanks, Sanju This bug is moved to https://github.com/gluster/glusterfs/issues/1106, and will be tracked there from now on. Visit GitHub issues URL for further details The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days |