Bug 1600878
| Summary: | crash seen while running regression, intermittently. | |||
|---|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | Amar Tumballi <atumball> | |
| Component: | logging | Assignee: | bugs <bugs> | |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | ||
| Severity: | high | Docs Contact: | ||
| Priority: | medium | |||
| Version: | mainline | CC: | bugs, nbalacha | |
| Target Milestone: | --- | |||
| Target Release: | --- | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | glusterfs-5.0 | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1648367 (view as bug list) | Environment: | ||
| Last Closed: | 2018-10-23 15:14:36 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1648367 | |||
REVIEW: https://review.gluster.org/20504 (logging: check for fmts not being NULL) posted (#1) for review on master by Amar Tumballi COMMIT: https://review.gluster.org/20504 committed in master by "Amar Tumballi" <amarts> with a commit message- logging: check for fmts not being NULL this fix is just a review of possible SIGSEGV issues in line 714 as per crash report at the bug: ``` 08:35:25 Program terminated with signal 11, Segmentation fault. 08:35:25 #0 0x00007f4ebb491c5c in gf_time_fmt (dst=0x7f4eb1ff9a90 "", sz_dst=256, utime=1531470915, fmt=0) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/common-utils.h:714 ``` fixes: bz#1600878 Change-Id: I160c391f8ac1a3456e59103d293b24e0e3fae718 Signed-off-by: Amar Tumballi <amarts> This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-5.0, please open a new bug report. glusterfs-5.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] https://lists.gluster.org/pipermail/announce/2018-October/000115.html [2] https://www.gluster.org/pipermail/gluster-users/ |
Description of problem: Core was generated by `python2 /build/install/libexec/glusterfs/python/syncdaemon/gsyncd.py agent mast'. 08:35:25 Program terminated with signal 11, Segmentation fault. 08:35:25 #0 0x00007f4ebb491c5c in ?? () 08:35:25 exe = 'python2 /build/install/libexec/glusterfs/python/syncdaemon/gsyncd.py agent mast' 08:35:25 ++ gdb -ex 'core-file /glusterepoll3-4825.core' -ex 'set pagination off' -ex 'info proc exe' -ex q 08:35:25 ++ cut -d ' ' -f1 08:35:25 ++ tail -1 08:35:25 ++ cut -d ''\''' -f2 08:35:25 + executable_name=python2 08:35:25 ++ which python2 08:35:25 + executable_path=/bin/python2 08:35:25 + set +x 08:35:25 08:35:25 ========================================================= 08:35:25 Start printing backtrace 08:35:25 program name : /bin/python2 08:35:25 corefile : /glusterepoll3-4825.core 08:35:25 ========================================================= 08:35:25 08:35:25 warning: core file may not match specified executable file. 08:35:25 [New LWP 5113] 08:35:25 [New LWP 4825] 08:35:25 [New LWP 5109] 08:35:25 [New LWP 5111] 08:35:25 [New LWP 4841] 08:35:25 [New LWP 5107] 08:35:25 [New LWP 5106] 08:35:25 [New LWP 5105] 08:35:25 [New LWP 4836] 08:35:25 [New LWP 5110] 08:35:25 [New LWP 5108] 08:35:25 [New LWP 5112] 08:35:25 [Thread debugging using libthread_db enabled] 08:35:25 Using host libthread_db library "/lib64/libthread_db.so.1". 08:35:25 Core was generated by `python2 /build/install/libexec/glusterfs/python/syncdaemon/gsyncd.py agent mast'. 08:35:25 Program terminated with signal 11, Segmentation fault. 08:35:25 #0 0x00007f4ebb491c5c in gf_time_fmt (dst=0x7f4eb1ff9a90 "", sz_dst=256, utime=1531470915, fmt=0) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/common-utils.h:714 08:35:25 714 strftime (dst, sz_dst, fmts[fmt], &tm); 08:35:25 08:35:25 Thread 12 (Thread 0x7f4eb27fc700 (LWP 5112)): 08:35:25 #0 0x00007f4ecc07c183 in epoll_wait () from /lib64/libc.so.6 08:35:25 No symbol table info available. 08:35:25 #1 0x00007f4ebb50bd43 in event_dispatch_epoll_worker (data=0x7f4eac000ef0) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/event-epoll.c:653 08:35:25 event = {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}} 08:35:25 ret = -1 08:35:25 ev_data = 0x7f4eac000ef0 08:35:25 event_pool = 0x7f4eb4057060 08:35:25 myindex = 3 08:35:25 timetodie = 0 08:35:25 __FUNCTION__ = "event_dispatch_epoll_worker" 08:35:25 #2 0x00007f4ecca5ae25 in start_thread () from /lib64/libpthread.so.0 08:35:25 No symbol table info available. 08:35:25 #3 0x00007f4ecc07bbad in clone () from /lib64/libc.so.6 08:35:25 No symbol table info available. 08:35:25 08:35:25 Thread 11 (Thread 0x7f4eb8d58700 (LWP 5108)): 08:35:25 #0 0x00007f4ecca5bf97 in pthread_join () from /lib64/libpthread.so.0 08:35:25 No symbol table info available. 08:35:25 #1 0x00007f4ebb50bfd7 in event_dispatch_epoll (event_pool=0x7f4eb4057060) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/event-epoll.c:750 08:35:25 i = 4 08:35:25 t_id = 139975970502400 08:35:25 pollercount = 4 08:35:25 ret = 0 08:35:25 ev_data = 0x7f4eac0011d0 08:35:25 thread_name = "epoll3\000\000" 08:35:25 __FUNCTION__ = "event_dispatch_epoll" 08:35:25 #2 0x00007f4ebb4c972f in event_dispatch (event_pool=0x7f4eb4057060) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/event.c:124 08:35:25 ret = -1 08:35:25 __FUNCTION__ = "event_dispatch" 08:35:25 #3 0x00007f4ebbbf2bae in changelog_rpc_poller (arg=0x7f4eb40069f0) at /home/jenkins/root/workspace/centos7-regression/xlators/features/changelog/src/changelog-rpc-common.c:31 08:35:25 this = 0x7f4eb40069f0 08:35:25 #4 0x00007f4ecca5ae25 in start_thread () from /lib64/libpthread.so.0 08:35:25 No symbol table info available. 08:35:25 #5 0x00007f4ecc07bbad in clone () from /lib64/libc.so.6 08:35:25 No symbol table info available. 08:35:25 08:35:25 Thread 10 (Thread 0x7f4eb37fe700 (LWP 5110)): 08:35:25 #0 0x00007f4ecc07c183 in epoll_wait () from /lib64/libc.so.6 08:35:25 No symbol table info available. 08:35:25 #1 0x00007f4ebb50bd43 in event_dispatch_epoll_worker (data=0x7f4eac000930) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/event-epoll.c:653 08:35:25 event = {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}} 08:35:25 ret = -1 08:35:25 ev_data = 0x7f4eac000930 08:35:25 event_pool = 0x7f4eb4057060 08:35:25 myindex = 1 08:35:25 timetodie = 0 08:35:25 __FUNCTION__ = "event_dispatch_epoll_worker" 08:35:25 #2 0x00007f4ecca5ae25 in start_thread () from /lib64/libpthread.so.0 08:35:25 No symbol table info available. 08:35:25 #3 0x00007f4ecc07bbad in clone () from /lib64/libc.so.6 08:35:25 No symbol table info available. 08:35:25 08:35:25 Thread 9 (Thread 0x7f4ec140f700 (LWP 4836)): 08:35:25 #0 0x00007f4ecc06cc8d in read () from /lib64/libc.so.6 08:35:25 No symbol table info available. 08:35:25 #1 0x00007f4ecbff8b24 in __GI__IO_file_underflow () from /lib64/libc.so.6 08:35:25 No symbol table info available. 08:35:25 #2 0x00007f4ecbff75d8 in __GI__IO_file_xsgetn () from /lib64/libc.so.6 08:35:25 No symbol table info available. 08:35:25 #3 0x00007f4ecbfebfef in fread () from /lib64/libc.so.6 08:35:25 No symbol table info available. 08:35:25 #4 0x00007f4ec2143533 in read_file () from /usr/lib64/python2.7/lib-dynload/cPickle.so 08:35:25 No symbol table info available. 08:35:25 #5 0x00007f4ec21486ae in load () from /usr/lib64/python2.7/lib-dynload/cPickle.so 08:35:25 No symbol table info available. 08:35:25 #6 0x00007f4ec2149c2d in cpm_load () from /usr/lib64/python2.7/lib-dynload/cPickle.so 08:35:25 No symbol table info available. 08:35:25 #7 0x00007f4eccd53c62 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #8 0x00007f4eccd536bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #9 0x00007f4eccd536bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #10 0x00007f4eccd5603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #11 0x00007f4ecccdf978 in function_call () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #12 0x00007f4ecccbaa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #13 0x00007f4eccd4e6fd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #14 0x00007f4eccd5603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #15 0x00007f4ecccdfa6d in function_call () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #16 0x00007f4ecccbaa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #17 0x00007f4eccd4e6fd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #18 0x00007f4eccd536bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #19 0x00007f4eccd536bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #20 0x00007f4eccd5603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #21 0x00007f4ecccdf978 in function_call () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #22 0x00007f4ecccbaa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #23 0x00007f4ecccc9a55 in instancemethod_call () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #24 0x00007f4ecccbaa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #25 0x00007f4eccd4c8f7 in PyEval_CallObjectWithKeywords () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #26 0x00007f4eccd84822 in t_bootstrap () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #27 0x00007f4ecca5ae25 in start_thread () from /lib64/libpthread.so.0 08:35:25 No symbol table info available. 08:35:25 #28 0x00007f4ecc07bbad in clone () from /lib64/libc.so.6 08:35:25 No symbol table info available. 08:35:25 08:35:25 Thread 8 (Thread 0x7f4eba55b700 (LWP 5105)): 08:35:25 #0 0x00007f4ecca5ed42 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 08:35:25 No symbol table info available. 08:35:25 #1 0x00007f4ebb4e2710 in syncenv_task (proc=0x7f4eb405c5c0) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/syncop.c:603 08:35:25 env = 0x7f4eb405c5c0 08:35:25 task = 0x0 08:35:25 sleep_till = {tv_sec = 1531471515, tv_nsec = 0} 08:35:25 ret = 0 08:35:25 #2 0x00007f4ebb4e29ab in syncenv_processor (thdata=0x7f4eb405c5c0) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/syncop.c:695 08:35:25 env = 0x7f4eb405c5c0 08:35:25 proc = 0x7f4eb405c5c0 08:35:25 task = 0x0 08:35:25 #3 0x00007f4ecca5ae25 in start_thread () from /lib64/libpthread.so.0 08:35:25 No symbol table info available. 08:35:25 #4 0x00007f4ecc07bbad in clone () from /lib64/libc.so.6 08:35:25 No symbol table info available. 08:35:25 08:35:25 Thread 7 (Thread 0x7f4eb9d5a700 (LWP 5106)): 08:35:25 #0 0x00007f4ecca5ed42 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 08:35:25 No symbol table info available. 08:35:25 #1 0x00007f4ebb4e2710 in syncenv_task (proc=0x7f4eb405c980) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/syncop.c:603 08:35:25 env = 0x7f4eb405c5c0 08:35:25 task = 0x0 08:35:25 sleep_till = {tv_sec = 1531471515, tv_nsec = 0} 08:35:25 ret = 0 08:35:25 #2 0x00007f4ebb4e29ab in syncenv_processor (thdata=0x7f4eb405c980) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/syncop.c:695 08:35:25 env = 0x7f4eb405c5c0 08:35:25 proc = 0x7f4eb405c980 08:35:25 task = 0x0 08:35:25 #3 0x00007f4ecca5ae25 in start_thread () from /lib64/libpthread.so.0 08:35:25 No symbol table info available. 08:35:25 #4 0x00007f4ecc07bbad in clone () from /lib64/libc.so.6 08:35:25 No symbol table info available. 08:35:25 08:35:25 Thread 6 (Thread 0x7f4eb9559700 (LWP 5107)): 08:35:25 #0 0x00007f4ecc04256d in nanosleep () from /lib64/libc.so.6 08:35:25 No symbol table info available. 08:35:25 #1 0x00007f4ecc042404 in sleep () from /lib64/libc.so.6 08:35:25 No symbol table info available. 08:35:25 #2 0x00007f4ebb4caef3 in pool_sweeper (arg=0x0) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/mem-pool.c:470 08:35:25 state = {death_row = {next = 0x0, prev = 0x0}, cold_lists = {0x0 <repeats 1024 times>}, n_cold_lists = 0} 08:35:25 pool_list = 0x0 08:35:25 next_pl = 0x0 08:35:25 pt_pool = 0x0 08:35:25 i = 0 08:35:25 begin_time = {tv_sec = 0, tv_usec = 0} 08:35:25 end_time = {tv_sec = 0, tv_usec = 0} 08:35:25 elapsed = {tv_sec = 0, tv_usec = 0} 08:35:25 #3 0x00007f4ecca5ae25 in start_thread () from /lib64/libpthread.so.0 08:35:25 No symbol table info available. 08:35:25 #4 0x00007f4ecc07bbad in clone () from /lib64/libc.so.6 08:35:25 No symbol table info available. 08:35:25 08:35:25 Thread 5 (Thread 0x7f4ec0c0e700 (LWP 4841)): 08:35:25 #0 0x00007f4ecc00c8c1 in __mempcpy_sse2 () from /lib64/libc.so.6 08:35:25 No symbol table info available. 08:35:25 #1 0x00007f4ecbff9e3c in __GI__IO_default_xsputn () from /lib64/libc.so.6 08:35:25 No symbol table info available. 08:35:25 #2 0x00007f4ecbff7862 in __GI__IO_file_xsputn () from /lib64/libc.so.6 08:35:25 No symbol table info available. 08:35:25 #3 0x00007f4ecbfc9ed3 in vfprintf () from /lib64/libc.so.6 08:35:25 No symbol table info available. 08:35:25 #4 0x00007f4ecbfd0297 in fprintf () from /lib64/libc.so.6 08:35:25 No symbol table info available. 08:35:25 #5 0x00007f4ebb494bd1 in gf_log_glusterlog (ctx=0x7f4eb4021e10, domain=0x7f4eb4008760 "gfchangelog", file=0x7f4ebbbf37fb "gf-changelog.c", function=0x7f4ebbbf39d0 <__FUNCTION__.21335> "gf_changelog_register_generic", line=562, level=GF_LOG_INFO, errnum=0, msgid=132028, appmsgstr=0x7f4ec0c0c498, callstr=0x0, tv=..., graph_id=0, fmt=gf_logformat_withmsgid) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/logging.c:1457 08:35:25 timestr = "2018-07-13 08:35:15.126391", '\000' <repeats 229 times> 08:35:25 header = 0x7f4eb4065da0 "[2018-07-13 08:35:15.126391] I [MSGID: 132028] [gf-changelog.c:562:gf_changelog_register_generic] 0-gfchangelog: " 08:35:25 footer = 0x0 08:35:25 msg = 0x7f4eb4065e80 "[2018-07-13 08:35:15.126391] I [MSGID: 132028] [gf-changelog.c:562:gf_changelog_register_generic] 0-gfchangelog: Registering brick\tbrick=/d/backends/master4\tnotify_filter=1" 08:35:25 hlen = 113 08:35:25 flen = 0 08:35:25 mlen = 59 08:35:25 ret = 113 08:35:25 #6 0x00007f4ebb4956a1 in gf_log_print_plain_fmt (ctx=0x7f4eb4021e10, domain=0x7f4eb4008760 "gfchangelog", file=0x7f4ebbbf37fb "gf-changelog.c", function=0x7f4ebbbf39d0 <__FUNCTION__.21335> "gf_changelog_register_generic", line=562, level=GF_LOG_INFO, errnum=0, msgid=132028, appmsgstr=0x7f4ec0c0c498, callstr=0x0, tv=..., graph_id=0, fmt=gf_logformat_withmsgid) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/logging.c:1690 08:35:25 ret = -1 08:35:25 logger = gf_logger_glusterlog 08:35:25 #7 0x00007f4ebb49622b in _gf_msg_internal (domain=0x7f4eb4008760 "gfchangelog", file=0x7f4ebbbf37a8 "/home/jenkins/root/workspace/centos7-regression/xlators/features/changelog/lib/src/gf-changelog.c", function=0x7f4ebbbf39d0 <__FUNCTION__.21335> "gf_changelog_register_generic", line=562, level=GF_LOG_INFO, errnum=0, msgid=132028, appmsgstr=0x7f4ec0c0c498, callstr=0x0, graph_id=0) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/logging.c:2028 08:35:25 ret = 0 08:35:25 size = 5 08:35:25 basename = 0x7f4ebbbf37fb "gf-changelog.c" 08:35:25 this = 0x7f4eb40069f0 08:35:25 ctx = 0x7f4eb4021e10 08:35:25 iter = 0x7f4eb4022108 08:35:25 buf_tmp = 0x7f4eb4022108 08:35:25 buf_new = 0x7f4eb4065ae8 08:35:25 first = 0x0 08:35:25 tv = {tv_sec = 1531470915, tv_usec = 126391} 08:35:25 found = false 08:35:25 flush_lru = false 08:35:25 flush_logged_msg = true 08:35:25 __FUNCTION__ = "_gf_msg_internal" 08:35:25 #8 0x00007f4ebb4965ff in _gf_msg (domain=0x7f4eb4008760 "gfchangelog", file=0x7f4ebbbf37a8 "/home/jenkins/root/workspace/centos7-regression/xlators/features/changelog/lib/src/gf-changelog.c", function=0x7f4ebbbf39d0 <__FUNCTION__.21335> "gf_changelog_register_generic", line=562, level=GF_LOG_INFO, errnum=0, trace=0, msgid=132028, fmt=0x7f4ebb56cf77 "%s") at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/logging.c:2107 08:35:25 ret = 59 08:35:25 msgstr = 0x7f4eb4065830 "Registering brick\tbrick=/d/backends/master4\tnotify_filter=1" 08:35:25 ap = {{gp_offset = 48, fp_offset = 48, overflow_arg_area = 0x7f4ec0c0c5a0, reg_save_area = 0x7f4ec0c0c4c0}} 08:35:25 this = 0x7f4eb40069f0 08:35:25 ctx = 0x7f4eb4021e10 08:35:25 callstr = '\000' <repeats 4095 times> 08:35:25 passcallstr = 0 08:35:25 log_inited = 1 08:35:25 __PRETTY_FUNCTION__ = "_gf_msg" 08:35:25 #9 0x00007f4ebb497d5a in _gf_smsg (domain=0x7f4eb4008760 "gfchangelog", file=0x7f4ebbbf37a8 "/home/jenkins/root/workspace/centos7-regression/xlators/features/changelog/lib/src/gf-changelog.c", function=0x7f4ebbbf39d0 <__FUNCTION__.21335> "gf_changelog_register_generic", line=562, level=GF_LOG_INFO, errnum=0, trace=0, msgid=132028, event=0x7f4ebbbf390f "Registering brick") at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/logging.c:2618 08:35:25 valist = {{gp_offset = 48, fp_offset = 48, overflow_arg_area = 0x7f4ec0c0c700, reg_save_area = 0x7f4ec0c0c600}} 08:35:25 msg = 0x7f4eb40658f0 "Registering brick\tbrick=/d/backends/master4\tnotify_filter=1" 08:35:25 ret = 59 08:35:25 #10 0x00007f4ebbbebda2 in gf_changelog_register_generic (bricks=0x7f4ec0c0c790, count=0, ordered=1, logfile=0x7f4ec1429dd4 "/var/log/glusterfs/geo-replication/master_127.0.0.1_slave/changes-d-backends-master4.log", lvl=7, xl=0x0) at /home/jenkins/root/workspace/centos7-regression/xlators/features/changelog/lib/src/gf-changelog.c:557 08:35:25 ret = 0 08:35:25 this = 0x7f4eb40069f0 08:35:25 old_this = 0x0 08:35:25 brick = 0x7f4ec0c0c790 08:35:25 need_order = true 08:35:25 __FUNCTION__ = "gf_changelog_register_generic" 08:35:25 #11 0x00007f4ebbbebf41 in gf_changelog_register (brick_path=0x7f4ec1438214 "/d/backends/master4", scratch_dir=0x7f4ec1acdd04 "/var/lib/misc/gluster/gsyncd/master_127.0.0.1_slave/d-backends-master4", log_file=0x7f4ec1429dd4 "/var/log/glusterfs/geo-replication/master_127.0.0.1_slave/changes-d-backends-master4.log", log_level=7, max_reconnects=5) at /home/jenkins/root/workspace/centos7-regression/xlators/features/changelog/lib/src/gf-changelog.c:633 08:35:25 brick = {brick_path = 0x7f4ec1438214 "/d/backends/master4", filter = 1, init = 0x7f4ebbbee3ad <gf_changelog_journal_init>, fini = 0x7f4ebbbee340 <gf_changelog_journal_fini>, callback = 0x7f4ebbbed929 <gf_changelog_handle_journal>, connected = 0x7f4ebbbed9d5 <gf_changelog_journal_connect>, disconnected = 0x7f4ebbbed97d <gf_changelog_journal_disconnect>, ptr = 0x7f4ec1acdd04} 08:35:25 #12 0x00007f4ebbdfddcc in ffi_call_unix64 () from /lib64/libffi.so.6 08:35:25 No symbol table info available. 08:35:25 #13 0x00007f4ebbdfd6f5 in ffi_call () from /lib64/libffi.so.6 08:35:25 No symbol table info available. 08:35:25 #14 0x00007f4ec0200cd9 in _ctypes_callproc () from /usr/lib64/python2.7/lib-dynload/_ctypes.so 08:35:25 No symbol table info available. 08:35:25 #15 0x00007f4ec01fa9e5 in PyCFuncPtr_call () from /usr/lib64/python2.7/lib-dynload/_ctypes.so 08:35:25 No symbol table info available. 08:35:25 #16 0x00007f4ecccbaa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #17 0x00007f4eccd4f236 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #18 0x00007f4eccd5603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #19 0x00007f4eccd5353c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #20 0x00007f4eccd5603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #21 0x00007f4ecccdf978 in function_call () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #22 0x00007f4ecccbaa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #23 0x00007f4eccd4e6fd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #24 0x00007f4eccd5603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #25 0x00007f4ecccdf978 in function_call () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #26 0x00007f4ecccbaa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #27 0x00007f4eccd4e6fd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #28 0x00007f4eccd5603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #29 0x00007f4ecccdfa6d in function_call () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #30 0x00007f4ecccbaa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #31 0x00007f4eccd4e6fd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #32 0x00007f4eccd536bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #33 0x00007f4eccd536bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #34 0x00007f4eccd5603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #35 0x00007f4ecccdf978 in function_call () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #36 0x00007f4ecccbaa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #37 0x00007f4ecccc9a55 in instancemethod_call () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #38 0x00007f4ecccbaa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #39 0x00007f4eccd4c8f7 in PyEval_CallObjectWithKeywords () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #40 0x00007f4eccd84822 in t_bootstrap () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #41 0x00007f4ecca5ae25 in start_thread () from /lib64/libpthread.so.0 08:35:25 No symbol table info available. 08:35:25 #42 0x00007f4ecc07bbad in clone () from /lib64/libc.so.6 08:35:25 No symbol table info available. 08:35:25 08:35:25 Thread 4 (Thread 0x7f4eb2ffd700 (LWP 5111)): 08:35:25 #0 0x00007f4ecc07c183 in epoll_wait () from /lib64/libc.so.6 08:35:25 No symbol table info available. 08:35:25 #1 0x00007f4ebb50bd43 in event_dispatch_epoll_worker (data=0x7f4eac000c10) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/event-epoll.c:653 08:35:25 event = {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}} 08:35:25 ret = -1 08:35:25 ev_data = 0x7f4eac000c10 08:35:25 event_pool = 0x7f4eb4057060 08:35:25 myindex = 2 08:35:25 timetodie = 0 08:35:25 __FUNCTION__ = "event_dispatch_epoll_worker" 08:35:25 #2 0x00007f4ecca5ae25 in start_thread () from /lib64/libpthread.so.0 08:35:25 No symbol table info available. 08:35:25 #3 0x00007f4ecc07bbad in clone () from /lib64/libc.so.6 08:35:25 No symbol table info available. 08:35:25 08:35:25 Thread 3 (Thread 0x7f4eb3fff700 (LWP 5109)): 08:35:25 #0 0x00007f4ecca5e995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 08:35:25 No symbol table info available. 08:35:25 #1 0x00007f4ebbbf19d0 in gf_changelog_connection_janitor (arg=0x7f4eb40069f0) at /home/jenkins/root/workspace/centos7-regression/xlators/features/changelog/lib/src/gf-changelog-reborp.c:47 08:35:25 ret = 0 08:35:25 this = 0x7f4eb40069f0 08:35:25 priv = 0x7f4eb4064b20 08:35:25 entry = 0x0 08:35:25 event = 0x0 08:35:25 ev = 0x0 08:35:25 drained = 0 08:35:25 __FUNCTION__ = "gf_changelog_connection_janitor" 08:35:25 #2 0x00007f4ecca5ae25 in start_thread () from /lib64/libpthread.so.0 08:35:25 No symbol table info available. 08:35:25 #3 0x00007f4ecc07bbad in clone () from /lib64/libc.so.6 08:35:25 No symbol table info available. 08:35:25 08:35:25 Thread 2 (Thread 0x7f4ecd24c740 (LWP 4825)): 08:35:25 #0 0x00007f4ecc072c73 in select () from /lib64/libc.so.6 08:35:25 No symbol table info available. 08:35:25 #1 0x00007f4ec189f17e in select_select () from /usr/lib64/python2.7/lib-dynload/selectmodule.so 08:35:25 No symbol table info available. 08:35:25 #2 0x00007f4eccd5420a in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #3 0x00007f4eccd5603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #4 0x00007f4ecccdf978 in function_call () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #5 0x00007f4ecccbaa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #6 0x00007f4eccd4e6fd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #7 0x00007f4eccd5603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #8 0x00007f4eccd5353c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #9 0x00007f4eccd5603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #10 0x00007f4ecccdf978 in function_call () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #11 0x00007f4ecccbaa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #12 0x00007f4ecccc9a55 in instancemethod_call () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #13 0x00007f4ecccbaa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #14 0x00007f4eccd11a87 in slot_tp_init () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #15 0x00007f4eccd1079f in type_call () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #16 0x00007f4ecccbaa63 in PyObject_Call () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #17 0x00007f4eccd4f236 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #18 0x00007f4eccd536bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #19 0x00007f4eccd5603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #20 0x00007f4eccd5353c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #21 0x00007f4eccd536bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #22 0x00007f4eccd5603d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #23 0x00007f4eccd56142 in PyEval_EvalCode () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #24 0x00007f4eccd6f57f in run_mod () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #25 0x00007f4eccd7073e in PyRun_FileExFlags () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #26 0x00007f4eccd719c9 in PyRun_SimpleFileExFlags () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #27 0x00007f4eccd82b7f in Py_Main () from /lib64/libpython2.7.so.1.0 08:35:25 No symbol table info available. 08:35:25 #28 0x00007f4ecbf9f445 in __libc_start_main () from /lib64/libc.so.6 08:35:25 No symbol table info available. 08:35:25 #29 0x000000000040066e in ?? () 08:35:25 No symbol table info available. 08:35:25 #30 0x00007ffcbe86e3e8 in ?? () 08:35:25 No symbol table info available. 08:35:25 #31 0x000000000000001c in ?? () 08:35:25 No symbol table info available. 08:35:25 #32 0x000000000000000f in ?? () 08:35:25 No symbol table info available. 08:35:25 #33 0x00007ffcbe86fe28 in ?? () 08:35:25 No symbol table info available. 08:35:25 #34 0x00007ffcbe86fe30 in ?? () 08:35:25 No symbol table info available. 08:35:25 #35 0x00007ffcbe86fe6d in ?? () 08:35:25 No symbol table info available. 08:35:25 #36 0x00007ffcbe86fe73 in ?? () 08:35:25 No symbol table info available. 08:35:25 #37 0x00007ffcbe86fe7a in ?? () 08:35:25 No symbol table info available. 08:35:25 #38 0x00007ffcbe86fe8b in ?? () 08:35:25 No symbol table info available. 08:35:25 #39 0x00007ffcbe86fe98 in ?? () 08:35:25 No symbol table info available. 08:35:25 #40 0x00007ffcbe86feac in ?? () 08:35:25 No symbol table info available. 08:35:25 #41 0x00007ffcbe86feb9 in ?? () 08:35:25 No symbol table info available. 08:35:25 #42 0x00007ffcbe86fed6 in ?? () 08:35:25 No symbol table info available. 08:35:25 #43 0x00007ffcbe86fee6 in ?? () 08:35:25 No symbol table info available. 08:35:25 #44 0x00007ffcbe86ff0b in ?? () 08:35:25 No symbol table info available. 08:35:25 #45 0x00007ffcbe86ff16 in ?? () 08:35:25 No symbol table info available. 08:35:25 #46 0x00007ffcbe86ff3b in ?? () 08:35:25 No symbol table info available. 08:35:25 #47 0x00007ffcbe86ff44 in ?? () 08:35:25 No symbol table info available. 08:35:25 #48 0x0000000000000000 in ?? () 08:35:25 No symbol table info available. 08:35:25 08:35:25 Thread 1 (Thread 0x7f4eb1ffb700 (LWP 5113)): 08:35:25 #0 0x00007f4ebb491c5c in gf_time_fmt (dst=0x7f4eb1ff9a90 "", sz_dst=256, utime=1531470915, fmt=0) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/common-utils.h:714 08:35:25 timefmt_last = gf_timefmt_last 08:35:25 fmts = 0x7f4ebb796500 <__gf_timefmts> 08:35:25 zeros = 0x7f4ebb796540 <__gf_zerotimes> 08:35:25 tm = {tm_sec = 15, tm_min = 35, tm_hour = 8, tm_mday = 13, tm_mon = 6, tm_year = 118, tm_wday = 5, tm_yday = 193, tm_isdst = 0, tm_gmtoff = 0, tm_zone = 0x7f4ecc105714 "GMT"} 08:35:25 res = 0x7f4eb1ff99d0 08:35:25 localtime = 0 08:35:25 #1 0x00007f4ebb4947ef in gf_log_glusterlog (ctx=0x7f4eb4021e10, domain=0x7f4ebb577aaf "epoll", file=0x7f4ebb577aa1 "event-epoll.c", function=0x7f4ebb577e50 <__FUNCTION__.10680> "event_dispatch_epoll_worker", line=617, level=GF_LOG_INFO, errnum=0, msgid=101190, appmsgstr=0x7f4eb1ffadc8, callstr=0x0, tv=..., graph_id=0, fmt=gf_logformat_withmsgid) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/logging.c:1389 08:35:25 timestr = '\000' <repeats 255 times> 08:35:25 header = 0x0 08:35:25 footer = 0x0 08:35:25 msg = 0x0 08:35:25 hlen = 0 08:35:25 flen = 0 08:35:25 mlen = 0 08:35:25 ret = 0 08:35:25 #2 0x00007f4ebb4956a1 in gf_log_print_plain_fmt (ctx=0x7f4eb4021e10, domain=0x7f4ebb577aaf "epoll", file=0x7f4ebb577aa1 "event-epoll.c", function=0x7f4ebb577e50 <__FUNCTION__.10680> "event_dispatch_epoll_worker", line=617, level=GF_LOG_INFO, errnum=0, msgid=101190, appmsgstr=0x7f4eb1ffadc8, callstr=0x0, tv=..., graph_id=0, fmt=gf_logformat_withmsgid) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/logging.c:1690 08:35:25 ret = -1 08:35:25 logger = gf_logger_glusterlog 08:35:25 #3 0x00007f4ebb49622b in _gf_msg_internal (domain=0x7f4ebb577aaf "epoll", file=0x7f4ebb577a60 "/home/jenkins/root/workspace/centos7-regression/libglusterfs/src/event-epoll.c", function=0x7f4ebb577e50 <__FUNCTION__.10680> "event_dispatch_epoll_worker", line=617, level=GF_LOG_INFO, errnum=0, msgid=101190, appmsgstr=0x7f4eb1ffadc8, callstr=0x0, graph_id=0) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/logging.c:2028 08:35:25 ret = 0 08:35:25 size = 5 08:35:25 basename = 0x7f4ebb577aa1 "event-epoll.c" 08:35:25 this = 0x7f4ebb79b2c0 <global_xlator> 08:35:25 ctx = 0x7f4eb4021e10 08:35:25 iter = 0x7f4eb4022108 08:35:25 buf_tmp = 0x7f4eb4065ae8 08:35:25 buf_new = 0x7f4e94000b18 08:35:25 first = 0x7f4eb4065ae8 08:35:25 tv = {tv_sec = 1531470915, tv_usec = 126410} 08:35:25 found = false 08:35:25 flush_lru = false 08:35:25 flush_logged_msg = true 08:35:25 __FUNCTION__ = "_gf_msg_internal" 08:35:25 #4 0x00007f4ebb4965ff in _gf_msg (domain=0x7f4ebb577aaf "epoll", file=0x7f4ebb577a60 "/home/jenkins/root/workspace/centos7-regression/libglusterfs/src/event-epoll.c", function=0x7f4ebb577e50 <__FUNCTION__.10680> "event_dispatch_epoll_worker", line=617, level=GF_LOG_INFO, errnum=0, trace=0, msgid=101190, fmt=0x7f4ebb577c6b "Started thread with index %d") at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/logging.c:2107 08:35:25 ret = 27 08:35:25 msgstr = 0x7f4e94000950 "Started thread with index 4" 08:35:25 ap = {{gp_offset = 48, fp_offset = 48, overflow_arg_area = 0x7f4eb1ffaed0, reg_save_area = 0x7f4eb1ffadf0}} 08:35:25 this = 0x7f4ebb79b2c0 <global_xlator> 08:35:25 ctx = 0x7f4eb4021e10 08:35:25 callstr = '\000' <repeats 4095 times> 08:35:25 passcallstr = 0 08:35:25 log_inited = 1 08:35:25 __PRETTY_FUNCTION__ = "_gf_msg" 08:35:25 #5 0x00007f4ebb50bbff in event_dispatch_epoll_worker (data=0x7f4eac0011d0) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/event-epoll.c:616 08:35:25 event = {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}} 08:35:25 ret = -1 08:35:25 ev_data = 0x7f4eac0011d0 08:35:25 event_pool = 0x7f4eb4057060 08:35:25 myindex = 4 08:35:25 timetodie = 0 08:35:25 __FUNCTION__ = "event_dispatch_epoll_worker" 08:35:25 #6 0x00007f4ecca5ae25 in start_thread () from /lib64/libpthread.so.0 08:35:25 No symbol table info available. 08:35:25 #7 0x00007f4ecc07bbad in clone () from /lib64/libc.so.6 08:35:25 No symbol table info available. 08:35:25 ========================================================= 08:35:25 Finish backtrace 08:35:25 program name : /bin/python2 08:35:25 corefile : /glusterepoll3-4825.core 08:35:25 ========================================================= 08:35:25 Version-Release number of selected component (if applicable): master How reproducible: 1/10 Steps to Reproduce: 1. run regression on upstream master Actual results: Sometimes the above backtraces are seen Expected results: No crashes Additional info: