Description of problem: ============================= Bitd crashed Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: ======================= 1.Create EC(4+2) volume and do some IO 2.After some time kill few of the brick process and do some IO and after some time bitd crashed 3. Actual results: Expected results: Additional info: ===================== [root@rhs-client37 /]# gdb /usr/sbin/glusterfs core.3549 GNU gdb (GDB) Red Hat Enterprise Linux (7.2-75.el6) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/sbin/glusterfs...Reading symbols from /usr/lib/debug/usr/sbin/glusterfsd.debug...done. done. [New Thread 3554] [New Thread 3566] [New Thread 3570] [New Thread 3569] [New Thread 3568] [New Thread 3571] [New Thread 3574] [New Thread 3572] [New Thread 3573] [New Thread 3575] [New Thread 3567] [New Thread 3552] [New Thread 3578] [New Thread 3550] [New Thread 3551] [New Thread 3576] [New Thread 3553] [New Thread 3549] Missing separate debuginfo for /usr/lib64/libglusterfs.so.0 Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/46/8af909651358455ffde7c9ace19d1ffd2021bf Missing separate debuginfo for /usr/lib64/libgfrpc.so.0 Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/ef/e88ad4c332410ffbb8035abe7089b1a7f6fa6a Missing separate debuginfo for /usr/lib64/libgfxdr.so.0 Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/1c/25b0c0c52dd23c9764b4640e3df562f2f82a41 Missing separate debuginfo for /usr/lib64/glusterfs/3.7.0beta1/rpc-transport/socket.so Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/de/790f689014347d257f02534f477020ff8d7151 Missing separate debuginfo for /usr/lib64/glusterfs/3.7.0beta1/xlator/protocol/client.so Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/e5/1750f399f1029a613f76c1761007aa379b1645 Missing separate debuginfo for /usr/lib64/glusterfs/3.7.0beta1/xlator/features/bit-rot.so Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/14/5b71000b3c7930e2d7713dc2747eefe175f9e9 Missing separate debuginfo for /usr/lib64/libgfchangelog.so.0 Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/29/450f8474b5833e0b0b12acdbbd4141253dbebd warning: .dynamic section for "/usr/lib64/libglusterfs.so.0" is not at the expected address (wrong library or version mismatch?) Missing separate debuginfo for Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/08/e42c6c3d2cd1e5d68a43b717c9eb3d310f2df0 Reading symbols from /usr/lib64/libglusterfs.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libglusterfs.so.0.0.1.debug...done. done. Loaded symbols for /usr/lib64/libglusterfs.so.0 Reading symbols from /usr/lib64/libgfrpc.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libgfrpc.so.0.0.1.debug...done. done. Loaded symbols for /usr/lib64/libgfrpc.so.0 Reading symbols from /usr/lib64/libgfxdr.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libgfxdr.so.0.0.1.debug...done. done. Loaded symbols for /usr/lib64/libgfxdr.so.0 Reading symbols from /lib64/libz.so.1.2.3...Reading symbols from /usr/lib/debug/lib64/libz.so.1.2.3.debug...done. done. Loaded symbols for /lib64/libz.so.1.2.3 Reading symbols from /lib64/libuuid.so.1.3.0...Reading symbols from /usr/lib/debug/lib64/libuuid.so.1.3.0.debug...done. done. Loaded symbols for /lib64/libuuid.so.1.3.0 Reading symbols from /lib64/librt-2.12.so...Reading symbols from /usr/lib/debug/lib64/librt-2.12.so.debug...done. done. Loaded symbols for /lib64/librt-2.12.so Reading symbols from /lib64/libpthread-2.12.so...Reading symbols from /usr/lib/debug/lib64/libpthread-2.12.so.debug...done. [Thread debugging using libthread_db enabled] done. Loaded symbols for /lib64/libpthread-2.12.so Reading symbols from /usr/lib64/libcrypto.so.1.0.1e...Reading symbols from /usr/lib/debug/usr/lib64/libcrypto.so.1.0.1e.debug...done. done. Loaded symbols for /usr/lib64/libcrypto.so.1.0.1e Reading symbols from /lib64/libc-2.12.so...Reading symbols from /usr/lib/debug/lib64/libc-2.12.so.debug...done. done. Loaded symbols for /lib64/libc-2.12.so Reading symbols from /lib64/libdl-2.12.so...Reading symbols from /usr/lib/debug/lib64/libdl-2.12.so.debug...done. done. Loaded symbols for /lib64/libdl-2.12.so Reading symbols from /lib64/ld-2.12.so...Reading symbols from /usr/lib/debug/lib64/ld-2.12.so.debug...done. done. Loaded symbols for /lib64/ld-2.12.so Reading symbols from /usr/lib64/libssl.so.1.0.1e...Reading symbols from /usr/lib/debug/usr/lib64/libssl.so.1.0.1e.debug...done. done. Loaded symbols for /usr/lib64/libssl.so.1.0.1e Reading symbols from /lib64/libgssapi_krb5.so.2.2...Reading symbols from /usr/lib/debug/lib64/libgssapi_krb5.so.2.2.debug...done. done. Loaded symbols for /lib64/libgssapi_krb5.so.2.2 Reading symbols from /lib64/libkrb5.so.3.3...Reading symbols from /usr/lib/debug/lib64/libkrb5.so.3.3.debug...done. done. Loaded symbols for /lib64/libkrb5.so.3.3 Reading symbols from /lib64/libcom_err.so.2.1...Reading symbols from /usr/lib/debug/lib64/libcom_err.so.2.1.debug...done. done. Loaded symbols for /lib64/libcom_err.so.2.1 Reading symbols from /lib64/libk5crypto.so.3.1...Reading symbols from /usr/lib/debug/lib64/libk5crypto.so.3.1.debug...done. done. Loaded symbols for /lib64/libk5crypto.so.3.1 Reading symbols from /lib64/libkrb5support.so.0.1...Reading symbols from /usr/lib/debug/lib64/libkrb5support.so.0.1.debug...done. done. Loaded symbols for /lib64/libkrb5support.so.0.1 Reading symbols from /lib64/libkeyutils.so.1.3...Reading symbols from /usr/lib/debug/lib64/libkeyutils.so.1.3.debug...done. done. Loaded symbols for /lib64/libkeyutils.so.1.3 Reading symbols from /lib64/libresolv-2.12.so...Reading symbols from /usr/lib/debug/lib64/libresolv-2.12.so.debug...done. done. Loaded symbols for /lib64/libresolv-2.12.so Reading symbols from /lib64/libselinux.so.1...Reading symbols from /usr/lib/debug/lib64/libselinux.so.1.debug...done. done. Loaded symbols for /lib64/libselinux.so.1 Reading symbols from /lib64/libnss_files-2.12.so...Reading symbols from /usr/lib/debug/lib64/libnss_files-2.12.so.debug...done. done. Loaded symbols for /lib64/libnss_files-2.12.so Reading symbols from /usr/lib64/libgfchangelog.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libgfchangelog.so.0.0.1.debug...done. done. Loaded symbols for /usr/lib64/libgfchangelog.so.0 Reading symbols from /lib64/libgcc_s-4.4.7-20120601.so.1...Reading symbols from /usr/lib/debug/lib64/libgcc_s-4.4.7-20120601.so.1.debug...done. done. Loaded symbols for /lib64/libgcc_s-4.4.7-20120601.so.1 Core was generated by `/usr/sbin/glusterfs -s localhost --volfile-id gluster/bitd -p /var/lib/glusterd'. Program terminated with signal 7, Bus error. #0 0x00007f809bdfbc54 in gf_changelog_reborp_rpcsvc_notify (rpc=0x7f8068002880, mydata=0x7f8068001850, event=RPCSVC_EVENT_TRANSPORT_DESTROY, data=0xffffffffffffff58) at gf-changelog-reborp.c:179 179 } (gdb) bt #0 0x00007f809bdfbc54 in gf_changelog_reborp_rpcsvc_notify (rpc=0x7f8068002880, mydata=0x7f8068001850, event=RPCSVC_EVENT_TRANSPORT_DESTROY, data=0xffffffffffffff58) at gf-changelog-reborp.c:179 #1 0x00007f8068002880 in ?? () #2 0x00007f809c13d610 in ?? () #3 0x00007f8068008c00 in ?? () #4 0x00007f809c13d610 in ?? () #5 0x00007f8068009e30 in ?? () #6 0x000000326fe09f94 in rpcsvc_program_notify (trans=<value optimized out>, mydata=<value optimized out>, event=<value optimized out>, data=0x7f80a0e91c74) at rpcsvc.c:340 #7 rpcsvc_accept (trans=<value optimized out>, mydata=<value optimized out>, event=<value optimized out>, data=0x7f80a0e91c74) at rpcsvc.c:363 #8 rpcsvc_notify (trans=<value optimized out>, mydata=<value optimized out>, event=<value optimized out>, data=0x7f80a0e91c74) at rpcsvc.c:788 #9 0x000000326fe0b8e8 in rpc_transport_notify (this=<value optimized out>, event=<value optimized out>, data=<value optimized out>) at rpc-transport.c:543 #10 0x00007f80a10c0632 in ?? () #11 0x00007f809c104410 in ?? () #12 0x00007f809c104408 in ?? () #13 0x00007f809c13e140 in ?? () #14 0x00007f809c13e248 in ?? () #15 0x000000109c1037ec in ?? () #16 0x00000000009a4010 in ?? () #17 0x00007f8000000000 in ?? () #18 0x0000000202f3d8bc in ?? () #19 0x0000000000000001 in ?? () #20 0x0000000000000000 in ?? ()
This seems to still exist even with the fix for BZ #1207146 (or more closely #1215161). I'll send a patch for this soon.
*** Bug 1221620 has been marked as a duplicate of this bug. ***
Unable to reproduce this now. Can we perform a round of test and see if this still exists.
Crash found again in NetBSD machine: (gdb) bt #0 0xb9a91d41 in gf_changelog_reborp_rpcsvc_notify (rpc=0xbb124870, mydata=0xbb105830, event=RPCSVC_EVENT_ACCEPT, data=0xb859f030) at /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/xlators/features/changelog/lib/src/gf-changelog-reborp.c:106 #1 0xbb6fec47 in rpcsvc_program_notify (listener=0xbb1c81e0, event=RPCSVC_EVENT_ACCEPT, data=0xb859f030) at /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-lib/src/rpcsvc.c:340 #2 0xbb6fecc5 in rpcsvc_accept (svc=0xbb124870, listen_trans=0xbb1d6030, new_trans=0xb859f030) at /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-lib/src/rpcsvc.c:363 #3 0xbb6ffa9b in rpcsvc_notify (trans=0xbb1d6030, mydata=0xbb124870, event=RPC_TRANSPORT_ACCEPT, data=0xb859f030) at /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-lib/src/rpcsvc.c:79\ 0 #4 0xbb705088 in rpc_transport_notify (this=0xbb1d6030, event=RPC_TRANSPORT_ACCEPT, data=0xb859f030) at /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-lib/src/rpc-transport.c:546 #5 0xbb26cb95 in socket_server_event_handler (fd=16, idx=3, data=0xbb1d6030, poll_in=1, poll_out=0, poll_err=0) at /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-transport/socket/src/\ socket.c:2770 #6 0xbb7a5842 in event_dispatch_poll_handler (event_pool=0xbb143030, ufds=0xb990a0b0, i=3) at /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/libglusterfs/src/event-poll.c:393 #7 0xbb7a5b89 in event_dispatch_poll (event_pool=0xbb143030) at /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/libglusterfs/src/event-poll.c:489 #8 0xbb774b49 in event_dispatch (event_pool=0xbb143030) at /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/libglusterfs/src/event.c:128 #9 0x08050e20 in main (argc=12, argv=0xbf7feaac) at /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/glusterfsd/src/glusterfsd.c:2350
Not seen this in any latest releases.