Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1648367

Summary: crash seen while running regression, intermittently.
Product: [Community] GlusterFS Reporter: Shyamsundar <srangana>
Component: loggingAssignee: Shyamsundar <srangana>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: high Docs Contact:
Priority: medium    
Version: 4.1CC: atumball, bugs, nbalacha
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-4.1.6 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1600878 Environment:
Last Closed: 2018-11-29 15:27:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1600878    
Bug Blocks:    

Description Shyamsundar 2018-11-09 14:40:24 UTC 
+++ This bug was initially created as a clone of Bug #1600878 +++

Description of problem:


Core was generated by `python2 /build/install/libexec/glusterfs/python/syncdaemon/gsyncd.py agent mast'.
08:35:25 Program terminated with signal 11, Segmentation fault.
08:35:25 #0  0x00007f4ebb491c5c in ?? ()
08:35:25 exe = 'python2 /build/install/libexec/glusterfs/python/syncdaemon/gsyncd.py agent mast'
08:35:25 ++ gdb -ex 'core-file /glusterepoll3-4825.core' -ex 'set pagination off' -ex 'info proc exe' -ex q
08:35:25 ++ cut -d ' ' -f1
08:35:25 ++ tail -1
08:35:25 ++ cut -d ''\''' -f2
08:35:25 + executable_name=python2
08:35:25 ++ which python2
08:35:25 + executable_path=/bin/python2
08:35:25 + set +x
08:35:25 
08:35:25 =========================================================
08:35:25               Start printing backtrace
08:35:25          program name : /bin/python2
08:35:25          corefile     : /glusterepoll3-4825.core
08:35:25 =========================================================
08:35:25 
08:35:25 warning: core file may not match specified executable file.
08:35:25 [New LWP 5113]
08:35:25 [New LWP 4825]
08:35:25 [New LWP 5109]
08:35:25 [New LWP 5111]
08:35:25 [New LWP 4841]
08:35:25 [New LWP 5107]
08:35:25 [New LWP 5106]
08:35:25 [New LWP 5105]
08:35:25 [New LWP 4836]
08:35:25 [New LWP 5110]
08:35:25 [New LWP 5108]
08:35:25 [New LWP 5112]
08:35:25 [Thread debugging using libthread_db enabled]
08:35:25 Using host libthread_db library "/lib64/libthread_db.so.1".
08:35:25 Core was generated by `python2 /build/install/libexec/glusterfs/python/syncdaemon/gsyncd.py agent mast'.
08:35:25 Program terminated with signal 11, Segmentation fault.
08:35:25 #0  0x00007f4ebb491c5c in gf_time_fmt (dst=0x7f4eb1ff9a90 "", sz_dst=256, utime=1531470915, fmt=0) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/common-utils.h:714
08:35:25 714	                strftime (dst, sz_dst, fmts[fmt], &tm);
08:35:25 
08:35:25 Thread 12 (Thread 0x7f4eb27fc700 (LWP 5112)):
08:35:25 #0  0x00007f4ecc07c183 in epoll_wait () from /lib64/libc.so.6
08:35:25 No symbol table info available.
08:35:25 #1  0x00007f4ebb50bd43 in event_dispatch_epoll_worker (data=0x7f4eac000ef0) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/event-epoll.c:653
08:35:25         event = {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}
08:35:25         ret = -1
08:35:25         ev_data = 0x7f4eac000ef0
08:35:25         event_pool = 0x7f4eb4057060
08:35:25         myindex = 3
08:35:25         timetodie = 0
08:35:25         __FUNCTION__ = "event_dispatch_epoll_worker"
08:35:25 #2  0x00007f4ecca5ae25 in start_thread () from /lib64/libpthread.so.0
08:35:25 No symbol table info available.
08:35:25 #3  0x00007f4ecc07bbad in clone () from /lib64/libc.so.6
08:35:25 No symbol table info available.
08:35:25 
08:35:25 Thread 11 (Thread 0x7f4eb8d58700 (LWP 5108)):
08:35:25 #0  0x00007f4ecca5bf97 in pthread_join () from /lib64/libpthread.so.0
08:35:25 No symbol table info available.
08:35:25 #1  0x00007f4ebb50bfd7 in event_dispatch_epoll (event_pool=0x7f4eb4057060) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/event-epoll.c:750
08:35:25         i = 4
08:35:25         t_id = 139975970502400
08:35:25         pollercount = 4
08:35:25         ret = 0
08:35:25         ev_data = 0x7f4eac0011d0
08:35:25         thread_name = "epoll3\000\000"
08:35:25         __FUNCTION__ = "event_dispatch_epoll"
08:35:25 #2  0x00007f4ebb4c972f in event_dispatch (event_pool=0x7f4eb4057060) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/event.c:124
08:35:25         ret = -1
08:35:25         __FUNCTION__ = "event_dispatch"
08:35:25 #3  0x00007f4ebbbf2bae in changelog_rpc_poller (arg=0x7f4eb40069f0) at /home/jenkins/root/workspace/centos7-regression/xlators/features/changelog/src/changelog-rpc-common.c:31
08:35:25         this = 0x7f4eb40069f0
08:35:25 #4  0x00007f4ecca5ae25 in start_thread () from /lib64/libpthread.so.0
08:35:25 No symbol table info available.
08:35:25 #5  0x00007f4ecc07bbad in clone () from /lib64/libc.so.6
08:35:25 No symbol table info available.
08:35:25 
08:35:25 Thread 10 (Thread 0x7f4eb37fe700 (LWP 5110)):
08:35:25 #0  0x00007f4ecc07c183 in epoll_wait () from /lib64/libc.so.6
08:35:25 No symbol table info available.
08:35:25 #1  0x00007f4ebb50bd43 in event_dispatch_epoll_worker (data=0x7f4eac000930) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/event-epoll.c:653
08:35:25         event = {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}
08:35:25         ret = -1
08:35:25         ev_data = 0x7f4eac000930
08:35:25         event_pool = 0x7f4eb4057060
08:35:25         myindex = 1
08:35:25         timetodie = 0
08:35:25         __FUNCTION__ = "event_dispatch_epoll_worker"
08:35:25 #2  0x00007f4ecca5ae25 in start_thread () from /lib64/libpthread.so.0
08:35:25 No symbol table info available.
08:35:25 #3  0x00007f4ecc07bbad in clone () from /lib64/libc.so.6
08:35:25 No symbol table info available.
08:35:25 
08:35:25 Thread 9 (Thread 0x7f4ec140f700 (LWP 4836)):
08:35:25 #0  0x00007f4ecc06cc8d in read () from /lib64/libc.so.6
08:35:25 No symbol table info available.
08:35:25 #1  0x00007f4ecbff8b24 in __GI__IO_file_underflow () from /lib64/libc.so.6
08:35:25 No symbol table info available.
08:35:25 #2  0x00007f4ecbff75d8 in __GI__IO_file_xsgetn () from /lib64/libc.so.6
08:35:25 No symbol table info available.
08:35:25 #3  0x00007f4ecbfebfef in fread () from /lib64/libc.so.6
08:35:25 No symbol table info available.
08:35:25 #4  0x00007f4ec2143533 in read_file () from /usr/lib64/python2.7/lib-dynload/cPickle.so
08:35:25 No symbol table info available.
08:35:25 #5  0x00007f4ec21486ae in load () from /usr/lib64/python2.7/lib-dynload/cPickle.so
08:35:25 No symbol table info available.
08:35:25 #6  0x00007f4ec2149c2d in cpm_load () from /usr/lib64/python2.7/lib-dynload/cPickle.so
08:35:25 No symbol table info available.
08:35:25 #7  0x00007f4eccd53c62 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #8  0x00007f4eccd536bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #9  0x00007f4eccd536bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #10 0x00007f4eccd5603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #11 0x00007f4ecccdf978 in function_call () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #12 0x00007f4ecccbaa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #13 0x00007f4eccd4e6fd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #14 0x00007f4eccd5603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #15 0x00007f4ecccdfa6d in function_call () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #16 0x00007f4ecccbaa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #17 0x00007f4eccd4e6fd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #18 0x00007f4eccd536bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #19 0x00007f4eccd536bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #20 0x00007f4eccd5603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #21 0x00007f4ecccdf978 in function_call () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #22 0x00007f4ecccbaa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #23 0x00007f4ecccc9a55 in instancemethod_call () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #24 0x00007f4ecccbaa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #25 0x00007f4eccd4c8f7 in PyEval_CallObjectWithKeywords () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #26 0x00007f4eccd84822 in t_bootstrap () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #27 0x00007f4ecca5ae25 in start_thread () from /lib64/libpthread.so.0
08:35:25 No symbol table info available.
08:35:25 #28 0x00007f4ecc07bbad in clone () from /lib64/libc.so.6
08:35:25 No symbol table info available.
08:35:25 
08:35:25 Thread 8 (Thread 0x7f4eba55b700 (LWP 5105)):
08:35:25 #0  0x00007f4ecca5ed42 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
08:35:25 No symbol table info available.
08:35:25 #1  0x00007f4ebb4e2710 in syncenv_task (proc=0x7f4eb405c5c0) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/syncop.c:603
08:35:25         env = 0x7f4eb405c5c0
08:35:25         task = 0x0
08:35:25         sleep_till = {tv_sec = 1531471515, tv_nsec = 0}
08:35:25         ret = 0
08:35:25 #2  0x00007f4ebb4e29ab in syncenv_processor (thdata=0x7f4eb405c5c0) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/syncop.c:695
08:35:25         env = 0x7f4eb405c5c0
08:35:25         proc = 0x7f4eb405c5c0
08:35:25         task = 0x0
08:35:25 #3  0x00007f4ecca5ae25 in start_thread () from /lib64/libpthread.so.0
08:35:25 No symbol table info available.
08:35:25 #4  0x00007f4ecc07bbad in clone () from /lib64/libc.so.6
08:35:25 No symbol table info available.
08:35:25 
08:35:25 Thread 7 (Thread 0x7f4eb9d5a700 (LWP 5106)):
08:35:25 #0  0x00007f4ecca5ed42 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
08:35:25 No symbol table info available.
08:35:25 #1  0x00007f4ebb4e2710 in syncenv_task (proc=0x7f4eb405c980) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/syncop.c:603
08:35:25         env = 0x7f4eb405c5c0
08:35:25         task = 0x0
08:35:25         sleep_till = {tv_sec = 1531471515, tv_nsec = 0}
08:35:25         ret = 0
08:35:25 #2  0x00007f4ebb4e29ab in syncenv_processor (thdata=0x7f4eb405c980) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/syncop.c:695
08:35:25         env = 0x7f4eb405c5c0
08:35:25         proc = 0x7f4eb405c980
08:35:25         task = 0x0
08:35:25 #3  0x00007f4ecca5ae25 in start_thread () from /lib64/libpthread.so.0
08:35:25 No symbol table info available.
08:35:25 #4  0x00007f4ecc07bbad in clone () from /lib64/libc.so.6
08:35:25 No symbol table info available.
08:35:25 
08:35:25 Thread 6 (Thread 0x7f4eb9559700 (LWP 5107)):
08:35:25 #0  0x00007f4ecc04256d in nanosleep () from /lib64/libc.so.6
08:35:25 No symbol table info available.
08:35:25 #1  0x00007f4ecc042404 in sleep () from /lib64/libc.so.6
08:35:25 No symbol table info available.
08:35:25 #2  0x00007f4ebb4caef3 in pool_sweeper (arg=0x0) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/mem-pool.c:470
08:35:25         state = {death_row = {next = 0x0, prev = 0x0}, cold_lists = {0x0 <repeats 1024 times>}, n_cold_lists = 0}
08:35:25         pool_list = 0x0
08:35:25         next_pl = 0x0
08:35:25         pt_pool = 0x0
08:35:25         i = 0
08:35:25         begin_time = {tv_sec = 0, tv_usec = 0}
08:35:25         end_time = {tv_sec = 0, tv_usec = 0}
08:35:25         elapsed = {tv_sec = 0, tv_usec = 0}
08:35:25 #3  0x00007f4ecca5ae25 in start_thread () from /lib64/libpthread.so.0
08:35:25 No symbol table info available.
08:35:25 #4  0x00007f4ecc07bbad in clone () from /lib64/libc.so.6
08:35:25 No symbol table info available.
08:35:25 
08:35:25 Thread 5 (Thread 0x7f4ec0c0e700 (LWP 4841)):
08:35:25 #0  0x00007f4ecc00c8c1 in __mempcpy_sse2 () from /lib64/libc.so.6
08:35:25 No symbol table info available.
08:35:25 #1  0x00007f4ecbff9e3c in __GI__IO_default_xsputn () from /lib64/libc.so.6
08:35:25 No symbol table info available.
08:35:25 #2  0x00007f4ecbff7862 in __GI__IO_file_xsputn () from /lib64/libc.so.6
08:35:25 No symbol table info available.
08:35:25 #3  0x00007f4ecbfc9ed3 in vfprintf () from /lib64/libc.so.6
08:35:25 No symbol table info available.
08:35:25 #4  0x00007f4ecbfd0297 in fprintf () from /lib64/libc.so.6
08:35:25 No symbol table info available.
08:35:25 #5  0x00007f4ebb494bd1 in gf_log_glusterlog (ctx=0x7f4eb4021e10, domain=0x7f4eb4008760 "gfchangelog", file=0x7f4ebbbf37fb "gf-changelog.c", function=0x7f4ebbbf39d0 <__FUNCTION__.21335> "gf_changelog_register_generic", line=562, level=GF_LOG_INFO, errnum=0, msgid=132028, appmsgstr=0x7f4ec0c0c498, callstr=0x0, tv=..., graph_id=0, fmt=gf_logformat_withmsgid) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/logging.c:1457
08:35:25         timestr = "2018-07-13 08:35:15.126391", '\000' <repeats 229 times>
08:35:25         header = 0x7f4eb4065da0 "[2018-07-13 08:35:15.126391] I [MSGID: 132028] [gf-changelog.c:562:gf_changelog_register_generic] 0-gfchangelog: "
08:35:25         footer = 0x0
08:35:25         msg = 0x7f4eb4065e80 "[2018-07-13 08:35:15.126391] I [MSGID: 132028] [gf-changelog.c:562:gf_changelog_register_generic] 0-gfchangelog: Registering brick\tbrick=/d/backends/master4\tnotify_filter=1"
08:35:25         hlen = 113
08:35:25         flen = 0
08:35:25         mlen = 59
08:35:25         ret = 113
08:35:25 #6  0x00007f4ebb4956a1 in gf_log_print_plain_fmt (ctx=0x7f4eb4021e10, domain=0x7f4eb4008760 "gfchangelog", file=0x7f4ebbbf37fb "gf-changelog.c", function=0x7f4ebbbf39d0 <__FUNCTION__.21335> "gf_changelog_register_generic", line=562, level=GF_LOG_INFO, errnum=0, msgid=132028, appmsgstr=0x7f4ec0c0c498, callstr=0x0, tv=..., graph_id=0, fmt=gf_logformat_withmsgid) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/logging.c:1690
08:35:25         ret = -1
08:35:25         logger = gf_logger_glusterlog
08:35:25 #7  0x00007f4ebb49622b in _gf_msg_internal (domain=0x7f4eb4008760 "gfchangelog", file=0x7f4ebbbf37a8 "/home/jenkins/root/workspace/centos7-regression/xlators/features/changelog/lib/src/gf-changelog.c", function=0x7f4ebbbf39d0 <__FUNCTION__.21335> "gf_changelog_register_generic", line=562, level=GF_LOG_INFO, errnum=0, msgid=132028, appmsgstr=0x7f4ec0c0c498, callstr=0x0, graph_id=0) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/logging.c:2028
08:35:25         ret = 0
08:35:25         size = 5
08:35:25         basename = 0x7f4ebbbf37fb "gf-changelog.c"
08:35:25         this = 0x7f4eb40069f0
08:35:25         ctx = 0x7f4eb4021e10
08:35:25         iter = 0x7f4eb4022108
08:35:25         buf_tmp = 0x7f4eb4022108
08:35:25         buf_new = 0x7f4eb4065ae8
08:35:25         first = 0x0
08:35:25         tv = {tv_sec = 1531470915, tv_usec = 126391}
08:35:25         found = false
08:35:25         flush_lru = false
08:35:25         flush_logged_msg = true
08:35:25         __FUNCTION__ = "_gf_msg_internal"
08:35:25 #8  0x00007f4ebb4965ff in _gf_msg (domain=0x7f4eb4008760 "gfchangelog", file=0x7f4ebbbf37a8 "/home/jenkins/root/workspace/centos7-regression/xlators/features/changelog/lib/src/gf-changelog.c", function=0x7f4ebbbf39d0 <__FUNCTION__.21335> "gf_changelog_register_generic", line=562, level=GF_LOG_INFO, errnum=0, trace=0, msgid=132028, fmt=0x7f4ebb56cf77 "%s") at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/logging.c:2107
08:35:25         ret = 59
08:35:25         msgstr = 0x7f4eb4065830 "Registering brick\tbrick=/d/backends/master4\tnotify_filter=1"
08:35:25         ap = {{gp_offset = 48, fp_offset = 48, overflow_arg_area = 0x7f4ec0c0c5a0, reg_save_area = 0x7f4ec0c0c4c0}}
08:35:25         this = 0x7f4eb40069f0
08:35:25         ctx = 0x7f4eb4021e10
08:35:25         callstr = '\000' <repeats 4095 times>
08:35:25         passcallstr = 0
08:35:25         log_inited = 1
08:35:25         __PRETTY_FUNCTION__ = "_gf_msg"
08:35:25 #9  0x00007f4ebb497d5a in _gf_smsg (domain=0x7f4eb4008760 "gfchangelog", file=0x7f4ebbbf37a8 "/home/jenkins/root/workspace/centos7-regression/xlators/features/changelog/lib/src/gf-changelog.c", function=0x7f4ebbbf39d0 <__FUNCTION__.21335> "gf_changelog_register_generic", line=562, level=GF_LOG_INFO, errnum=0, trace=0, msgid=132028, event=0x7f4ebbbf390f "Registering brick") at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/logging.c:2618
08:35:25         valist = {{gp_offset = 48, fp_offset = 48, overflow_arg_area = 0x7f4ec0c0c700, reg_save_area = 0x7f4ec0c0c600}}
08:35:25         msg = 0x7f4eb40658f0 "Registering brick\tbrick=/d/backends/master4\tnotify_filter=1"
08:35:25         ret = 59
08:35:25 #10 0x00007f4ebbbebda2 in gf_changelog_register_generic (bricks=0x7f4ec0c0c790, count=0, ordered=1, logfile=0x7f4ec1429dd4 "/var/log/glusterfs/geo-replication/master_127.0.0.1_slave/changes-d-backends-master4.log", lvl=7, xl=0x0) at /home/jenkins/root/workspace/centos7-regression/xlators/features/changelog/lib/src/gf-changelog.c:557
08:35:25         ret = 0
08:35:25         this = 0x7f4eb40069f0
08:35:25         old_this = 0x0
08:35:25         brick = 0x7f4ec0c0c790
08:35:25         need_order = true
08:35:25         __FUNCTION__ = "gf_changelog_register_generic"
08:35:25 #11 0x00007f4ebbbebf41 in gf_changelog_register (brick_path=0x7f4ec1438214 "/d/backends/master4", scratch_dir=0x7f4ec1acdd04 "/var/lib/misc/gluster/gsyncd/master_127.0.0.1_slave/d-backends-master4", log_file=0x7f4ec1429dd4 "/var/log/glusterfs/geo-replication/master_127.0.0.1_slave/changes-d-backends-master4.log", log_level=7, max_reconnects=5) at /home/jenkins/root/workspace/centos7-regression/xlators/features/changelog/lib/src/gf-changelog.c:633
08:35:25         brick = {brick_path = 0x7f4ec1438214 "/d/backends/master4", filter = 1, init = 0x7f4ebbbee3ad <gf_changelog_journal_init>, fini = 0x7f4ebbbee340 <gf_changelog_journal_fini>, callback = 0x7f4ebbbed929 <gf_changelog_handle_journal>, connected = 0x7f4ebbbed9d5 <gf_changelog_journal_connect>, disconnected = 0x7f4ebbbed97d <gf_changelog_journal_disconnect>, ptr = 0x7f4ec1acdd04}
08:35:25 #12 0x00007f4ebbdfddcc in ffi_call_unix64 () from /lib64/libffi.so.6
08:35:25 No symbol table info available.
08:35:25 #13 0x00007f4ebbdfd6f5 in ffi_call () from /lib64/libffi.so.6
08:35:25 No symbol table info available.
08:35:25 #14 0x00007f4ec0200cd9 in _ctypes_callproc () from /usr/lib64/python2.7/lib-dynload/_ctypes.so
08:35:25 No symbol table info available.
08:35:25 #15 0x00007f4ec01fa9e5 in PyCFuncPtr_call () from /usr/lib64/python2.7/lib-dynload/_ctypes.so
08:35:25 No symbol table info available.
08:35:25 #16 0x00007f4ecccbaa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #17 0x00007f4eccd4f236 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #18 0x00007f4eccd5603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #19 0x00007f4eccd5353c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #20 0x00007f4eccd5603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #21 0x00007f4ecccdf978 in function_call () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #22 0x00007f4ecccbaa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #23 0x00007f4eccd4e6fd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #24 0x00007f4eccd5603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #25 0x00007f4ecccdf978 in function_call () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #26 0x00007f4ecccbaa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #27 0x00007f4eccd4e6fd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #28 0x00007f4eccd5603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #29 0x00007f4ecccdfa6d in function_call () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #30 0x00007f4ecccbaa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #31 0x00007f4eccd4e6fd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #32 0x00007f4eccd536bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #33 0x00007f4eccd536bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #34 0x00007f4eccd5603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #35 0x00007f4ecccdf978 in function_call () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #36 0x00007f4ecccbaa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #37 0x00007f4ecccc9a55 in instancemethod_call () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #38 0x00007f4ecccbaa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #39 0x00007f4eccd4c8f7 in PyEval_CallObjectWithKeywords () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #40 0x00007f4eccd84822 in t_bootstrap () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #41 0x00007f4ecca5ae25 in start_thread () from /lib64/libpthread.so.0
08:35:25 No symbol table info available.
08:35:25 #42 0x00007f4ecc07bbad in clone () from /lib64/libc.so.6
08:35:25 No symbol table info available.
08:35:25 
08:35:25 Thread 4 (Thread 0x7f4eb2ffd700 (LWP 5111)):
08:35:25 #0  0x00007f4ecc07c183 in epoll_wait () from /lib64/libc.so.6
08:35:25 No symbol table info available.
08:35:25 #1  0x00007f4ebb50bd43 in event_dispatch_epoll_worker (data=0x7f4eac000c10) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/event-epoll.c:653
08:35:25         event = {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}
08:35:25         ret = -1
08:35:25         ev_data = 0x7f4eac000c10
08:35:25         event_pool = 0x7f4eb4057060
08:35:25         myindex = 2
08:35:25         timetodie = 0
08:35:25         __FUNCTION__ = "event_dispatch_epoll_worker"
08:35:25 #2  0x00007f4ecca5ae25 in start_thread () from /lib64/libpthread.so.0
08:35:25 No symbol table info available.
08:35:25 #3  0x00007f4ecc07bbad in clone () from /lib64/libc.so.6
08:35:25 No symbol table info available.
08:35:25 
08:35:25 Thread 3 (Thread 0x7f4eb3fff700 (LWP 5109)):
08:35:25 #0  0x00007f4ecca5e995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
08:35:25 No symbol table info available.
08:35:25 #1  0x00007f4ebbbf19d0 in gf_changelog_connection_janitor (arg=0x7f4eb40069f0) at /home/jenkins/root/workspace/centos7-regression/xlators/features/changelog/lib/src/gf-changelog-reborp.c:47
08:35:25         ret = 0
08:35:25         this = 0x7f4eb40069f0
08:35:25         priv = 0x7f4eb4064b20
08:35:25         entry = 0x0
08:35:25         event = 0x0
08:35:25         ev = 0x0
08:35:25         drained = 0
08:35:25         __FUNCTION__ = "gf_changelog_connection_janitor"
08:35:25 #2  0x00007f4ecca5ae25 in start_thread () from /lib64/libpthread.so.0
08:35:25 No symbol table info available.
08:35:25 #3  0x00007f4ecc07bbad in clone () from /lib64/libc.so.6
08:35:25 No symbol table info available.
08:35:25 
08:35:25 Thread 2 (Thread 0x7f4ecd24c740 (LWP 4825)):
08:35:25 #0  0x00007f4ecc072c73 in select () from /lib64/libc.so.6
08:35:25 No symbol table info available.
08:35:25 #1  0x00007f4ec189f17e in select_select () from /usr/lib64/python2.7/lib-dynload/selectmodule.so
08:35:25 No symbol table info available.
08:35:25 #2  0x00007f4eccd5420a in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #3  0x00007f4eccd5603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #4  0x00007f4ecccdf978 in function_call () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #5  0x00007f4ecccbaa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #6  0x00007f4eccd4e6fd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #7  0x00007f4eccd5603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #8  0x00007f4eccd5353c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #9  0x00007f4eccd5603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #10 0x00007f4ecccdf978 in function_call () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #11 0x00007f4ecccbaa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #12 0x00007f4ecccc9a55 in instancemethod_call () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #13 0x00007f4ecccbaa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #14 0x00007f4eccd11a87 in slot_tp_init () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #15 0x00007f4eccd1079f in type_call () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #16 0x00007f4ecccbaa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #17 0x00007f4eccd4f236 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #18 0x00007f4eccd536bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #19 0x00007f4eccd5603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #20 0x00007f4eccd5353c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #21 0x00007f4eccd536bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #22 0x00007f4eccd5603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #23 0x00007f4eccd56142 in PyEval_EvalCode () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #24 0x00007f4eccd6f57f in run_mod () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #25 0x00007f4eccd7073e in PyRun_FileExFlags () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #26 0x00007f4eccd719c9 in PyRun_SimpleFileExFlags () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #27 0x00007f4eccd82b7f in Py_Main () from /lib64/libpython2.7.so.1.0
08:35:25 No symbol table info available.
08:35:25 #28 0x00007f4ecbf9f445 in __libc_start_main () from /lib64/libc.so.6
08:35:25 No symbol table info available.
08:35:25 #29 0x000000000040066e in ?? ()
08:35:25 No symbol table info available.
08:35:25 #30 0x00007ffcbe86e3e8 in ?? ()
08:35:25 No symbol table info available.
08:35:25 #31 0x000000000000001c in ?? ()
08:35:25 No symbol table info available.
08:35:25 #32 0x000000000000000f in ?? ()
08:35:25 No symbol table info available.
08:35:25 #33 0x00007ffcbe86fe28 in ?? ()
08:35:25 No symbol table info available.
08:35:25 #34 0x00007ffcbe86fe30 in ?? ()
08:35:25 No symbol table info available.
08:35:25 #35 0x00007ffcbe86fe6d in ?? ()
08:35:25 No symbol table info available.
08:35:25 #36 0x00007ffcbe86fe73 in ?? ()
08:35:25 No symbol table info available.
08:35:25 #37 0x00007ffcbe86fe7a in ?? ()
08:35:25 No symbol table info available.
08:35:25 #38 0x00007ffcbe86fe8b in ?? ()
08:35:25 No symbol table info available.
08:35:25 #39 0x00007ffcbe86fe98 in ?? ()
08:35:25 No symbol table info available.
08:35:25 #40 0x00007ffcbe86feac in ?? ()
08:35:25 No symbol table info available.
08:35:25 #41 0x00007ffcbe86feb9 in ?? ()
08:35:25 No symbol table info available.
08:35:25 #42 0x00007ffcbe86fed6 in ?? ()
08:35:25 No symbol table info available.
08:35:25 #43 0x00007ffcbe86fee6 in ?? ()
08:35:25 No symbol table info available.
08:35:25 #44 0x00007ffcbe86ff0b in ?? ()
08:35:25 No symbol table info available.
08:35:25 #45 0x00007ffcbe86ff16 in ?? ()
08:35:25 No symbol table info available.
08:35:25 #46 0x00007ffcbe86ff3b in ?? ()
08:35:25 No symbol table info available.
08:35:25 #47 0x00007ffcbe86ff44 in ?? ()
08:35:25 No symbol table info available.
08:35:25 #48 0x0000000000000000 in ?? ()
08:35:25 No symbol table info available.
08:35:25 
08:35:25 Thread 1 (Thread 0x7f4eb1ffb700 (LWP 5113)):
08:35:25 #0  0x00007f4ebb491c5c in gf_time_fmt (dst=0x7f4eb1ff9a90 "", sz_dst=256, utime=1531470915, fmt=0) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/common-utils.h:714
08:35:25         timefmt_last = gf_timefmt_last
08:35:25         fmts = 0x7f4ebb796500 <__gf_timefmts>
08:35:25         zeros = 0x7f4ebb796540 <__gf_zerotimes>
08:35:25         tm = {tm_sec = 15, tm_min = 35, tm_hour = 8, tm_mday = 13, tm_mon = 6, tm_year = 118, tm_wday = 5, tm_yday = 193, tm_isdst = 0, tm_gmtoff = 0, tm_zone = 0x7f4ecc105714 "GMT"}
08:35:25         res = 0x7f4eb1ff99d0
08:35:25         localtime = 0
08:35:25 #1  0x00007f4ebb4947ef in gf_log_glusterlog (ctx=0x7f4eb4021e10, domain=0x7f4ebb577aaf "epoll", file=0x7f4ebb577aa1 "event-epoll.c", function=0x7f4ebb577e50 <__FUNCTION__.10680> "event_dispatch_epoll_worker", line=617, level=GF_LOG_INFO, errnum=0, msgid=101190, appmsgstr=0x7f4eb1ffadc8, callstr=0x0, tv=..., graph_id=0, fmt=gf_logformat_withmsgid) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/logging.c:1389
08:35:25         timestr = '\000' <repeats 255 times>
08:35:25         header = 0x0
08:35:25         footer = 0x0
08:35:25         msg = 0x0
08:35:25         hlen = 0
08:35:25         flen = 0
08:35:25         mlen = 0
08:35:25         ret = 0
08:35:25 #2  0x00007f4ebb4956a1 in gf_log_print_plain_fmt (ctx=0x7f4eb4021e10, domain=0x7f4ebb577aaf "epoll", file=0x7f4ebb577aa1 "event-epoll.c", function=0x7f4ebb577e50 <__FUNCTION__.10680> "event_dispatch_epoll_worker", line=617, level=GF_LOG_INFO, errnum=0, msgid=101190, appmsgstr=0x7f4eb1ffadc8, callstr=0x0, tv=..., graph_id=0, fmt=gf_logformat_withmsgid) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/logging.c:1690
08:35:25         ret = -1
08:35:25         logger = gf_logger_glusterlog
08:35:25 #3  0x00007f4ebb49622b in _gf_msg_internal (domain=0x7f4ebb577aaf "epoll", file=0x7f4ebb577a60 "/home/jenkins/root/workspace/centos7-regression/libglusterfs/src/event-epoll.c", function=0x7f4ebb577e50 <__FUNCTION__.10680> "event_dispatch_epoll_worker", line=617, level=GF_LOG_INFO, errnum=0, msgid=101190, appmsgstr=0x7f4eb1ffadc8, callstr=0x0, graph_id=0) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/logging.c:2028
08:35:25         ret = 0
08:35:25         size = 5
08:35:25         basename = 0x7f4ebb577aa1 "event-epoll.c"
08:35:25         this = 0x7f4ebb79b2c0 <global_xlator>
08:35:25         ctx = 0x7f4eb4021e10
08:35:25         iter = 0x7f4eb4022108
08:35:25         buf_tmp = 0x7f4eb4065ae8
08:35:25         buf_new = 0x7f4e94000b18
08:35:25         first = 0x7f4eb4065ae8
08:35:25         tv = {tv_sec = 1531470915, tv_usec = 126410}
08:35:25         found = false
08:35:25         flush_lru = false
08:35:25         flush_logged_msg = true
08:35:25         __FUNCTION__ = "_gf_msg_internal"
08:35:25 #4  0x00007f4ebb4965ff in _gf_msg (domain=0x7f4ebb577aaf "epoll", file=0x7f4ebb577a60 "/home/jenkins/root/workspace/centos7-regression/libglusterfs/src/event-epoll.c", function=0x7f4ebb577e50 <__FUNCTION__.10680> "event_dispatch_epoll_worker", line=617, level=GF_LOG_INFO, errnum=0, trace=0, msgid=101190, fmt=0x7f4ebb577c6b "Started thread with index %d") at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/logging.c:2107
08:35:25         ret = 27
08:35:25         msgstr = 0x7f4e94000950 "Started thread with index 4"
08:35:25         ap = {{gp_offset = 48, fp_offset = 48, overflow_arg_area = 0x7f4eb1ffaed0, reg_save_area = 0x7f4eb1ffadf0}}
08:35:25         this = 0x7f4ebb79b2c0 <global_xlator>
08:35:25         ctx = 0x7f4eb4021e10
08:35:25         callstr = '\000' <repeats 4095 times>
08:35:25         passcallstr = 0
08:35:25         log_inited = 1
08:35:25         __PRETTY_FUNCTION__ = "_gf_msg"
08:35:25 #5  0x00007f4ebb50bbff in event_dispatch_epoll_worker (data=0x7f4eac0011d0) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/event-epoll.c:616
08:35:25         event = {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}
08:35:25         ret = -1
08:35:25         ev_data = 0x7f4eac0011d0
08:35:25         event_pool = 0x7f4eb4057060
08:35:25         myindex = 4
08:35:25         timetodie = 0
08:35:25         __FUNCTION__ = "event_dispatch_epoll_worker"
08:35:25 #6  0x00007f4ecca5ae25 in start_thread () from /lib64/libpthread.so.0
08:35:25 No symbol table info available.
08:35:25 #7  0x00007f4ecc07bbad in clone () from /lib64/libc.so.6
08:35:25 No symbol table info available.
08:35:25 =========================================================
08:35:25               Finish backtrace
08:35:25          program name : /bin/python2
08:35:25          corefile     : /glusterepoll3-4825.core
08:35:25 =========================================================
08:35:25 

Version-Release number of selected component (if applicable):
master

How reproducible:
1/10

Steps to Reproduce:
1. run regression on upstream master

Actual results:
Sometimes the above backtraces are seen

Expected results:
No crashes
Comment 1 Shyamsundar 2018-11-09 14:41:28 UTC 
Instance of the same on release-4.1 is here: https://build.gluster.org/job/centos7-regression/3616/consoleFull

Opened the bug to backport: https://review.gluster.org/c/glusterfs/+/20504
Comment 2 Worker Ant 2018-11-09 14:43:33 UTC 
REVIEW: https://review.gluster.org/21612 (logging: check for fmts not being NULL) posted (#1) for review on release-4.1 by Shyamsundar Ranganathan
Comment 3 Worker Ant 2018-11-13 13:55:10 UTC 
REVIEW: https://review.gluster.org/21612 (logging: check for fmts not being NULL) posted (#2) for review on release-4.1 by Shyamsundar Ranganathan
Comment 4 Shyamsundar 2018-11-29 15:27:59 UTC 
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-4.1.6, please open a new bug report.

glusterfs-4.1.6 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] https://lists.gluster.org/pipermail/announce/2018-November/000116.html
[2] https://www.gluster.org/pipermail/gluster-users/