Description of problem: I'm using a volume with two replicas of the 3.6.9 version of GlusterFS. The volume on which the issue occurs is not very active and at one point a process dies suddenly. This issue has also caused core dumps, and what we found at the time of the problem is as follows: - /var/log/messages (Brick log is also the same) Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: pending frames: Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: pending frames: Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: patchset: git://git.gluster.com/glusterfs.git Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: signal received: 6 Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: time of crash: Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: 2019-03-24 09:15:40 Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: configuration details: Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: argp 1 Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: backtrace 1 Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: dlfcn 1 Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: libpthread 1 Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: llistxattr 1 Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: setfsid 1 Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: spinlock 1 Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: epoll.h 1 Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: xattr.h 1 Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: st_atim.tv_nsec 1 Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: package-string: glusterfs 3.6.9 Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: --------- Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: patchset: git://git.gluster.com/glusterfs.git Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: signal received: 6 Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: time of crash: Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: 2019-03-24 09:15:40 Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: configuration details: Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: argp 1 Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: backtrace 1 Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: dlfcn 1 Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: libpthread 1 Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: llistxattr 1 Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: setfsid 1 Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: spinlock 1 Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: epoll.h 1 Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: xattr.h 1 Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: st_atim.tv_nsec 1 Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: package-string: glusterfs 3.6.9 Mar 24 18:15:40 P-NAS8 var-lib-glusterFS8-8[119226]: --------- Mar 24 18:15:40 P-NAS8 kernel: audit_printk_skb: 57 callbacks suppressed Mar 24 18:15:40 P-NAS8 kernel: type=1701 audit(1553418940.165:27816716): auid=1002 uid=0 gid=0 ses=3174727 pid=127312 comm="glusterfsd" reason="memory violation" sig=6 Mar 24 18:15:40 P-NAS8 systemd-logind: Removed session 3174727. Mar 24 18:15:40 P-NAS8 kernel: audit_printk_skb: 57 callbacks suppressed Mar 24 18:15:40 P-NAS8 kernel: type=1701 audit(1553418940.165:27816716): auid=1002 uid=0 gid=0 ses=3174727 pid=127312 comm="glusterfsd" reason="memory violation" sig=6 - CoreDump [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/glusterfsd -s p-tview-nas8 --volfile-id repl_dist_vol'. Program terminated with signal 6, Aborted. #0 0x00007fb6da9895f7 in raise () from /lib64/libc.so.6 Missing separate debuginfos, use: debuginfo-install glusterfs-3.6.9-1.el7.x86_64 (gdb) bt #0 0x00007fb6da9895f7 in raise () from /lib64/libc.so.6 #1 0x00007fb6da98ace8 in abort () from /lib64/libc.so.6 #2 0x00007fb6da9c9317 in __libc_message () from /lib64/libc.so.6 #3 0x00007fb6da9d1023 in _int_free () from /lib64/libc.so.6 #4 0x00007fb6db968d29 in dict_destroy () from /lib64/libglusterfs.so.0 #5 0x00007fb6db99776d in call_stub_destroy () from /lib64/libglusterfs.so.0 #6 0x00007fb6ca286333 in iot_worker () from /usr/lib64/glusterfs/3.6.9/xlator/performance/io-threads.so #7 0x00007fb6db103dc5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fb6daa4a28d in clone () from /lib64/libc.so.6 - free total used free shared buff/cache available Mem: 31G 21G 1.1G 4.2G 9.0G 5.1G Swap: 15G 173M 15G Unfortunately, I didn't know the exact cause here. Is there any other good way to determine the cause? I look forward to your help. Version-Release number of selected component (if applicable): glusterfs-3.6.9 (community version) How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
> I'm using a volume with two replicas of the 3.6.9 version of GlusterFS. Is it possible to upgrade version of glusterfs? Current glusterfs versions are at least 2+ years from that version, and we have fixed a **lot** of memory violation errors (coverity/clang-scan etc). We had not seen any issues similar to this strace in a long time. Let us know how the upgrade goes, and ping us if you need help with any upgrade issues.
Hi Jeong, I'm closing this bug as gluster 3.6 was EOL'd long back. Please feel free to re-open the bug if issue persists in any of the current supported releases.