Description of problem: The gluster-fuse 5.3-1 process crashes with a segmentation fault error 11 after high usage on the mount. The system is a 4 node distributed volume with 2 bricks per node. Info and backtrace below. Client has volume mounted with default options. gluster volume info ------------------------ Volume Name: tank Type: Distribute Volume ID: 9582685f-07fa-41fd-b9fc-ebab3a6989cf Status: Started Snapshot Count: 0 Number of Bricks: 8 Transport-type: tcp Bricks: Brick1: swp-gluster-01:/tank/volume1/brick Brick2: swp-gluster-02:/tank/volume1/brick Brick3: swp-gluster-03:/tank/volume1/brick Brick4: swp-gluster-04:/tank/volume1/brick Brick5: swp-gluster-01:/tank/volume2/brick Brick6: swp-gluster-02:/tank/volume2/brick Brick7: swp-gluster-03:/tank/volume2/brick Brick8: swp-gluster-04:/tank/volume2/brick Options Reconfigured: features.inode-quota: on features.quota: on nfs.disable: on features.quota-deem-statfs: on server.event-threads: 16 client.event-threads: 16 cluster.lookup-optimize: on cluster.readdir-optimize: on features.cache-invalidation: on features.cache-invalidation-timeout: 600 network.inode-lru-limit: 200000 performance.cache-samba-metadata: on performance.write-behind: on performance.client-io-threads: off performance.flush-behind: on performance.io-cache: off performance.io-thread-count: 64 performance.nl-cache: on performance.nl-cache-timeout: 600 performance.open-behind: on performance.quick-read: on performance.rda-request-size: 131072 performance.read-ahead-page-count: 16 performance.read-ahead: on performance.stat-prefetch: on performance.write-behind-window-size: 32MB storage.batch-fsync-delay-usec: 0 performance.md-cache-timeout: 600 cluster.enable-shared-storage: disable Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/glusterfs --process-name fuse --volfile-server=nfs.gluster.research.p'. Program terminated with signal 11, Segmentation fault. #0 __GI___pthread_mutex_lock (mutex=0x7f407ed19e78) at ../nptl/pthread_mutex_lock.c:66 66 unsigned int type = PTHREAD_MUTEX_TYPE_ELISION (mutex); Missing separate debuginfos, use: debuginfo-install glusterfs-fuse-5.3-1.el7.x86_64 (gdb) bt #0 __GI___pthread_mutex_lock (mutex=0x7f407ed19e78) at ../nptl/pthread_mutex_lock.c:66 #1 0x00007f0fce4bdc3c in __gf_free () from /lib64/libglusterfs.so.0 #2 0x00007f0fbb591c71 in ios_stat_add_to_list () from /usr/lib64/glusterfs/5.3/xlator/debug/io-stats.so #3 0x00007f0fbb59a6a1 in io_stats_writev_cbk () from /usr/lib64/glusterfs/5.3/xlator/debug/io-stats.so #4 0x00007f0fbb7c47b1 in mdc_writev_cbk () from /usr/lib64/glusterfs/5.3/xlator/performance/md-cache.so #5 0x00007f0fce51ddc8 in default_writev_cbk () from /lib64/libglusterfs.so.0 #6 0x00007f0fbbdf6f99 in qr_writev_cbk () from /usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so #7 0x00007f0fc001134a in rda_writev_cbk () from /usr/lib64/glusterfs/5.3/xlator/performance/readdir-ahead.so #8 0x00007f0fc021f751 in ra_writev_cbk () from /usr/lib64/glusterfs/5.3/xlator/performance/read-ahead.so #9 0x00007f0fc0431571 in wb_do_unwinds () from /usr/lib64/glusterfs/5.3/xlator/performance/write-behind.so #10 0x00007f0fc04329c0 in wb_process_queue () from /usr/lib64/glusterfs/5.3/xlator/performance/write-behind.so #11 0x00007f0fc0433377 in wb_writev () from /usr/lib64/glusterfs/5.3/xlator/performance/write-behind.so #12 0x00007f0fc021fca3 in ra_writev () from /usr/lib64/glusterfs/5.3/xlator/performance/read-ahead.so #13 0x00007f0fc000d10a in rda_writev () from /usr/lib64/glusterfs/5.3/xlator/performance/readdir-ahead.so #14 0x00007f0fbbdf5c0e in qr_writev () from /usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so #15 0x00007f0fce53e110 in default_writev_resume () from /lib64/libglusterfs.so.0 #16 0x00007f0fce4ba25a in call_resume_wind () from /lib64/libglusterfs.so.0 #17 0x00007f0fce4ba445 in call_resume () from /lib64/libglusterfs.so.0 #18 0x00007f0fbbbe8495 in ob_wake_cbk () from /usr/lib64/glusterfs/5.3/xlator/performance/open-behind.so #19 0x00007f0fce51c1eb in default_open_cbk () from /lib64/libglusterfs.so.0 #20 0x00007f0fc021abf6 in ra_open_cbk () from /usr/lib64/glusterfs/5.3/xlator/performance/read-ahead.so #21 0x00007f0fc06baa01 in dht_open_cbk () from /usr/lib64/glusterfs/5.3/xlator/cluster/distribute.so #22 0x00007f0fc0964bdd in client4_0_open_cbk () from /usr/lib64/glusterfs/5.3/xlator/protocol/client.so #23 0x00007f0fce25ec70 in rpc_clnt_handle_reply () from /lib64/libgfrpc.so.0 #24 0x00007f0fce25f043 in rpc_clnt_notify () from /lib64/libgfrpc.so.0 #25 0x00007f0fce25af23 in rpc_transport_notify () from /lib64/libgfrpc.so.0 #26 0x00007f0fc2e4837b in socket_event_handler () from /usr/lib64/glusterfs/5.3/rpc-transport/socket.so #27 0x00007f0fce4f6a49 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #28 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0fabfff700) at pthread_create.c:308 #29 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 gdb glusterd core.glfs_epoll005.1548850256.17447 Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/glusterfs --process-name fuse --volfile-server=nfs.gluster.research.p'. Program terminated with signal 11, Segmentation fault. #0 __GI___pthread_mutex_lock (mutex=0x7f407ed19e78) at ../nptl/pthread_mutex_lock.c:66 66 unsigned int type = PTHREAD_MUTEX_TYPE_ELISION (mutex); Missing separate debuginfos, use: debuginfo-install glusterfs-fuse-5.3-1.el7.x86_64 (gdb) t a a bt Thread 25 (Thread 0x7f0fa9ffb700 (LWP 17466)): #0 0x00007f0fccbbf183 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f0fce4f6790 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #2 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0fa9ffb700) at pthread_create.c:308 #3 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 24 (Thread 0x7f0fc385c700 (LWP 17452)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f0fce4d1cf8 in syncenv_task () from /lib64/libglusterfs.so.0 #2 0x00007f0fce4d2c40 in syncenv_processor () from /lib64/libglusterfs.so.0 #3 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0fc385c700) at pthread_create.c:308 #4 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 23 (Thread 0x7f0f8dffb700 (LWP 17475)): #0 0x00007f0fccbb5660 in __libc_readv (fd=8, vector=0x7f0f8dffae70, count=2) at ../sysdeps/unix/sysv/linux/readv.c:56 #1 0x00007f0fce4bfe49 in sys_readv () from /lib64/libglusterfs.so.0 #2 0x00007f0fc5880c60 in fuse_thread_proc () from /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so #3 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0f8dffb700) at pthread_create.c:308 #4 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 22 (Thread 0x7f0fc405d700 (LWP 17451)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f0fce4d1cf8 in syncenv_task () from /lib64/libglusterfs.so.0 #2 0x00007f0fce4d2c40 in syncenv_processor () from /lib64/libglusterfs.so.0 #3 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0fc405d700) at pthread_create.c:308 #4 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 21 (Thread 0x7f0fa97fa700 (LWP 17467)): #0 0x00007f0fccbbf183 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f0fce4f6790 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #2 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0fa97fa700) at pthread_create.c:308 #3 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 20 (Thread 0x7f0fb8a6a700 (LWP 17461)): #0 0x00007f0fccbbf183 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f0fce4f6790 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #2 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0fb8a6a700) at pthread_create.c:308 #3 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 19 (Thread 0x7f0fb9a6c700 (LWP 17459)): #0 0x00007f0fccbbf183 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f0fce4f6790 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #2 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0fb9a6c700) at pthread_create.c:308 #3 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 18 (Thread 0x7f0fc485e700 (LWP 17450)): #0 0x00007f0fccb8556d in nanosleep () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f0fccb85404 in __sleep (seconds=0) at ../sysdeps/unix/sysv/linux/sleep.c:137 #2 0x00007f0fce4bde7d in pool_sweeper () from /lib64/libglusterfs.so.0 #3 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0fc485e700) at pthread_create.c:308 #4 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 17 (Thread 0x7f0fce98b780 (LWP 17447)): #0 0x00007f0fcd2f6f97 in pthread_join (threadid=139705642981120, thread_return=0x0) at pthread_join.c:92 #1 0x00007f0fce4f5db8 in event_dispatch_epoll () from /lib64/libglusterfs.so.0 #2 0x0000561e9b26056b in main () Thread 16 (Thread 0x7f0f8e7fc700 (LWP 17472)): #0 0x00007f0fccbbf183 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f0fce4f6790 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #2 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0f8e7fc700) at pthread_create.c:308 #3 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 15 (Thread 0x7f0f8effd700 (LWP 17471)): #0 0x00007f0fccbbf183 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f0fce4f6790 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #2 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0f8effd700) at pthread_create.c:308 #3 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 14 (Thread 0x7f0f8d7fa700 (LWP 17476)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f0fc5869dbb in notify_kernel_loop () from /usr/lib64/glusterfs/5.3/xlator/mount/fuse.so #2 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0f8d7fa700) at pthread_create.c:308 #3 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 ---Type <return> to continue, or q <return> to quit--- Thread 13 (Thread 0x7f0fa8ff9700 (LWP 17468)): #0 0x00007f0fccbbf183 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f0fce4f6790 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #2 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0fa8ff9700) at pthread_create.c:308 #3 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 12 (Thread 0x7f0fbaa6e700 (LWP 17457)): #0 0x00007f0fccbb5c73 in select () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f0fce510d3f in runner () from /lib64/libglusterfs.so.0 #2 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0fbaa6e700) at pthread_create.c:308 #3 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 11 (Thread 0x7f0fb926b700 (LWP 17460)): #0 0x00007f0fccbbf183 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f0fce4f6790 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #2 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0fb926b700) at pthread_create.c:308 #3 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 10 (Thread 0x7f0f8ffff700 (LWP 17469)): #0 0x00007f0fccbbf183 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f0fce4f6790 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #2 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0f8ffff700) at pthread_create.c:308 #3 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 9 (Thread 0x7f0fba26d700 (LWP 17458)): #0 0x00007f0fccbbf183 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f0fce4f6790 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #2 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0fba26d700) at pthread_create.c:308 #3 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 8 (Thread 0x7f0fc5860700 (LWP 17448)): #0 0x00007f0fcd2fcf3d in nanosleep () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f0fce4a0f76 in gf_timer_proc () from /lib64/libglusterfs.so.0 #2 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0fc5860700) at pthread_create.c:308 #3 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 7 (Thread 0x7f0f8f7fe700 (LWP 17470)): #0 0x00007f0fccbbf183 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f0fce4f6790 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #2 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0f8f7fe700) at pthread_create.c:308 #3 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 6 (Thread 0x7f0fc1391700 (LWP 17456)): #0 0x00007f0fccbbf183 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f0fce4f6790 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #2 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0fc1391700) at pthread_create.c:308 #3 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 5 (Thread 0x7f0faaffd700 (LWP 17464)): #0 0x00007f0fccbbf183 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f0fce4f6790 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #2 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0faaffd700) at pthread_create.c:308 #3 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 4 (Thread 0x7f0fc505f700 (LWP 17449)): #0 0x00007f0fcd2fd461 in do_sigwait (sig=0x7f0fc505ee1c, set=<optimized out>) at ../sysdeps/unix/sysv/linux/sigwait.c:61 #1 __sigwait (set=0x7f0fc505ee20, sig=0x7f0fc505ee1c) at ../sysdeps/unix/sysv/linux/sigwait.c:99 #2 0x0000561e9b263e2b in glusterfs_sigwaiter () #3 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0fc505f700) at pthread_create.c:308 #4 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 3 (Thread 0x7f0fab7fe700 (LWP 17463)): #0 0x00007f0fccbbf183 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f0fce4f6790 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #2 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0fab7fe700) at pthread_create.c:308 #3 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 2 (Thread 0x7f0faa7fc700 (LWP 17465)): #0 0x00007f0fccbbf183 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f0fce4f6790 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #2 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0faa7fc700) at pthread_create.c:308 #3 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 1 (Thread 0x7f0fabfff700 (LWP 17462)): #0 __GI___pthread_mutex_lock (mutex=0x7f407ed19e78) at ../nptl/pthread_mutex_lock.c:66 #1 0x00007f0fce4bdc3c in __gf_free () from /lib64/libglusterfs.so.0 #2 0x00007f0fbb591c71 in ios_stat_add_to_list () from /usr/lib64/glusterfs/5.3/xlator/debug/io-stats.so #3 0x00007f0fbb59a6a1 in io_stats_writev_cbk () from /usr/lib64/glusterfs/5.3/xlator/debug/io-stats.so #4 0x00007f0fbb7c47b1 in mdc_writev_cbk () from /usr/lib64/glusterfs/5.3/xlator/performance/md-cache.so #5 0x00007f0fce51ddc8 in default_writev_cbk () from /lib64/libglusterfs.so.0 #6 0x00007f0fbbdf6f99 in qr_writev_cbk () from /usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so #7 0x00007f0fc001134a in rda_writev_cbk () from /usr/lib64/glusterfs/5.3/xlator/performance/readdir-ahead.so #8 0x00007f0fc021f751 in ra_writev_cbk () from /usr/lib64/glusterfs/5.3/xlator/performance/read-ahead.so #9 0x00007f0fc0431571 in wb_do_unwinds () from /usr/lib64/glusterfs/5.3/xlator/performance/write-behind.so #10 0x00007f0fc04329c0 in wb_process_queue () from /usr/lib64/glusterfs/5.3/xlator/performance/write-behind.so #11 0x00007f0fc0433377 in wb_writev () from /usr/lib64/glusterfs/5.3/xlator/performance/write-behind.so #12 0x00007f0fc021fca3 in ra_writev () from /usr/lib64/glusterfs/5.3/xlator/performance/read-ahead.so #13 0x00007f0fc000d10a in rda_writev () from /usr/lib64/glusterfs/5.3/xlator/performance/readdir-ahead.so #14 0x00007f0fbbdf5c0e in qr_writev () from /usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so #15 0x00007f0fce53e110 in default_writev_resume () from /lib64/libglusterfs.so.0 #16 0x00007f0fce4ba25a in call_resume_wind () from /lib64/libglusterfs.so.0 #17 0x00007f0fce4ba445 in call_resume () from /lib64/libglusterfs.so.0 #18 0x00007f0fbbbe8495 in ob_wake_cbk () from /usr/lib64/glusterfs/5.3/xlator/performance/open-behind.so #19 0x00007f0fce51c1eb in default_open_cbk () from /lib64/libglusterfs.so.0 #20 0x00007f0fc021abf6 in ra_open_cbk () from /usr/lib64/glusterfs/5.3/xlator/performance/read-ahead.so #21 0x00007f0fc06baa01 in dht_open_cbk () from /usr/lib64/glusterfs/5.3/xlator/cluster/distribute.so #22 0x00007f0fc0964bdd in client4_0_open_cbk () from /usr/lib64/glusterfs/5.3/xlator/protocol/client.so #23 0x00007f0fce25ec70 in rpc_clnt_handle_reply () from /lib64/libgfrpc.so.0 #24 0x00007f0fce25f043 in rpc_clnt_notify () from /lib64/libgfrpc.so.0 #25 0x00007f0fce25af23 in rpc_transport_notify () from /lib64/libgfrpc.so.0 #26 0x00007f0fc2e4837b in socket_event_handler () from /usr/lib64/glusterfs/5.3/rpc-transport/socket.so #27 0x00007f0fce4f6a49 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #28 0x00007f0fcd2f5e25 in start_thread (arg=0x7f0fabfff700) at pthread_create.c:308 #29 0x00007f0fccbbebad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
*** This bug has been marked as a duplicate of bug 1671556 ***