Description of problem: After upgrade, geo-rep crashes in gf_history_changelog. bt >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (gdb) bt #0 0x0000003b6a232925 in raise () from /lib64/libc.so.6 #1 0x0000003b6a234105 in abort () from /lib64/libc.so.6 #2 0x0000003b6a270837 in __libc_message () from /lib64/libc.so.6 #3 0x0000003b6a276166 in malloc_printerr () from /lib64/libc.so.6 #4 0x0000003b6aa0884d in pthread_attr_destroy () from /lib64/libpthread.so.0 #5 0x00007ff3fe13905a in gf_history_changelog (changelog_dir=<value optimized out>, start=<value optimized out>, end=1403167635, n_parallel=<value optimized out>, actual_end=0x174ec70) at gf-history-changelog.c:921 #6 0x00007ff40034fdac in ffi_call_unix64 () from /usr/lib64/libffi.so.5 #7 0x00007ff40034fb34 in ffi_call () from /usr/lib64/libffi.so.5 #8 0x00007ff400563002 in _CallProc () from /usr/lib64/python2.6/lib-dynload/_ctypes.so #9 0x00007ff40055c3a2 in ?? () from /usr/lib64/python2.6/lib-dynload/_ctypes.so #10 0x0000003b6ae43c63 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 #11 0x0000003b6aed4f74 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 #12 0x0000003b6aed6b8f in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 #13 0x0000003b6aed7657 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0 #14 0x0000003b6ae6acb0 in ?? () from /usr/lib64/libpython2.6.so.1.0 #15 0x0000003b6ae43c63 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 #16 0x0000003b6aed4470 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 #17 0x0000003b6aed7657 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0 #18 0x0000003b6ae6acb0 in ?? () from /usr/lib64/libpython2.6.so.1.0 #19 0x0000003b6ae43c63 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 #20 0x0000003b6aed4470 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 #21 0x0000003b6aed7657 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0 #22 0x0000003b6ae6adad in ?? () from /usr/lib64/libpython2.6.so.1.0 #23 0x0000003b6ae43c63 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 #24 0x0000003b6aed4470 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 #25 0x0000003b6aed6b8f in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 #26 0x0000003b6aed6b8f in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 #27 0x0000003b6aed7657 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0 #28 0x0000003b6ae6acb0 in ?? () from /usr/lib64/libpython2.6.so.1.0 #29 0x0000003b6ae43c63 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 #30 0x0000003b6ae566af in ?? () from /usr/lib64/libpython2.6.so.1.0 #31 0x0000003b6ae43c63 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0 #32 0x0000003b6aecfc93 in PyEval_CallObjectWithKeywords () from /usr/lib64/libpython2.6.so.1.0 #33 0x0000003b6af017ba in ?? () from /usr/lib64/libpython2.6.so.1.0 #34 0x0000003b6aa079d1 in start_thread () from /lib64/libpthread.so.0 #35 0x0000003b6a2e8b7d in clone () from /lib64/libc.so.6 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Version-Release number of selected component (if applicable): upgrade from 3.4.0.59rhs to 3.6.0.19rhs How reproducible: Didn't try to reproduce. Steps to Reproduce: 1. setup a geo-rep with the build 3.4.0.33rhs between master(6x2) and slave(6x2) 2. keep creating data on master. 3. start upgrade of one slave and one master using the upgrade steps a. Kill all the gsync and gluster processes. You can use the following commands to do the same. ps -aef | grep gluster | grep gsync | awk '{print $2}' | xargs kill -9 pkill glusterfsd pkill glusterfs pkill glusterd b. Upgrade the node with RHS-3.0 c. Reboot the node 4. First upgrade slave node mentioned in the geo-rep command. 4. then, upgrade all the master and slave nodes. Actual results: geo-rep crashes in gf_history_changelog Expected results: geo-rep shouldn't crash. Additional info:
https://code.engineering.redhat.com/gerrit/#/c/27312/
Tested on glusterfs-3.6.0.22 and no crash is seen. Moving the bug to verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHEA-2014-1278.html