Hide Forgot
It is duplicate of 2497. The loc is null according to the log: [2011-07-27 12:11:00.502802] I [afr-open.c:435:afr_openfd_sh] 0-podo-replicate-0: data self-heal triggered. path: (null), reason: Replicate up down flush, data lock is held. This issue is fixed in 3.1.5. *** This bug has been marked as a duplicate of bug 2497 ***
Client is a postfix mail server. having some errors when using smtp and imap. today client glusterfs crash. this is some client log: 1: volume podo-client-0 2: type protocol/client 3: option remote-host 10.101.120.11 4: option remote-subvolume /home/podo 5: option transport-type tcp 6: end-volume 7: 8: volume podo-client-1 9: type protocol/client 10: option remote-host 10.101.120.12 11: option remote-subvolume /home/podo 12: option transport-type tcp 13: end-volume 14: 15: volume podo-replicate-0 16: type cluster/replicate 17: option favorite-child podo-client-0 18: subvolumes podo-client-0 podo-client-1 19: end-volume 20: 21: volume podo-write-behind 22: type performance/write-behind 23: subvolumes podo-replicate-0 24: end-volume 25: 26: volume podo-read-ahead 27: type performance/read-ahead 28: subvolumes podo-write-behind 29: end-volume 30: 31: volume podo-io-cache 32: type performance/io-cache 33: subvolumes podo-read-ahead 34: end-volume 35: 36: volume podo-quick-read 37: type performance/quick-read 38: subvolumes podo-io-cache 39: end-volume 40: 41: volume podo-stat-prefetch 42: type performance/stat-prefetch 43: subvolumes podo-quick-read 44: end-volume 45: 46: volume podo 47: type debug/io-stats 48: subvolumes podo-stat-prefetch 49: end-volume +------------------------------------------------------------------------------+ [2011-07-27 12:10:50.98618] E [afr-self-heal-common.c:1214:sh_missing_entries_create] 0-podo-replicate-0: no missing files - /1003/7/88/Maildir/.&ZbCNJmI3-.INBOX/dovecot.index.log. proceeding to metadata check [2011-07-27 12:10:50.101202] E [afr-common.c:110:afr_set_split_brain] 0-podo-replicate-0: invalid argument: inode [2011-07-27 12:10:50.101307] I [afr-self-heal-common.c:1527:afr_self_heal_completion_cbk] 0-podo-replicate-0: background data self-heal completed on /1003/7/88/Maildir/.&ZbCNJmI3-.INBOX/dovecot.index.log [2011-07-27 12:10:51.553537] I [afr-common.c:716:afr_lookup_done] 0-podo-replicate-0: background meta-data entry self-heal triggered. path: /1003/7/79/Maildir [2011-07-27 12:10:51.566743] I [afr-self-heal-common.c:1527:afr_self_heal_completion_cbk] 0-podo-replicate-0: background meta-data entry self-heal completed on /1003/7/79/Maildir [2011-07-27 12:10:51.572941] I [afr-common.c:613:afr_lookup_self_heal_check] 0-podo-replicate-0: size differs for /1003/7/79/Maildir/dovecot.index.log [2011-07-27 12:10:51.572979] I [afr-common.c:716:afr_lookup_done] 0-podo-replicate-0: background data self-heal triggered. path: /1003/7/79/Maildir/dovecot.index.log [2011-07-27 12:10:51.577881] I [afr-self-heal-common.c:1527:afr_self_heal_completion_cbk] 0-podo-replicate-0: background data self-heal completed on /1003/7/79/Maildir/dovecot.index.log [2011-07-27 12:10:59.159664] I [afr-common.c:716:afr_lookup_done] 0-podo-replicate-0: background meta-data entry self-heal triggered. path: /1003/7/88/Maildir/.&ZbCNJmI3-.INBOX [2011-07-27 12:10:59.165930] I [afr-self-heal-common.c:1527:afr_self_heal_completion_cbk] 0-podo-replicate-0: background meta-data entry self-heal completed on /1003/7/88/Maildir/.&ZbCNJmI3-.INBOX [2011-07-27 12:11:00.502107] W [fuse-bridge.c:2577:fuse_getxattr] 0-glusterfs-fuse: 32770407: GETXATTR (null)/139671323265676 (security.capability) (fuse_loc_fill() failed) [2011-07-27 12:11:00.502401] W [fuse-bridge.c:2577:fuse_getxattr] 0-glusterfs-fuse: 32770410: GETXATTR (null)/139671323265676 (security.capability) (fuse_loc_fill() failed) [2011-07-27 12:11:00.502802] I [afr-open.c:435:afr_openfd_sh] 0-podo-replicate-0: data self-heal triggered. path: (null), reason: Replicate up down flush, data lock is held pending frames: frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) frame : type(1) op(STAT) patchset: v3.1.3 signal received: 11 time of crash: 2011-07-27 12:11:00 configuration details: argp 1 backtrace 1 dlfcn 1 fdatasync 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.1.3 /lib64/libc.so.6[0x39b7e32a20] /lib64/libc.so.6[0x39b7f2148f] /opt/glusterfs/3.1.3/lib64/libglusterfs.so.0(loc_copy+0x55)[0x7f07cc0714e5] /opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/cluster/replicate.so(afr_self_heal+0x441)[0x7f07c8d97261] /opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/cluster/replicate.so(afr_openfd_sh+0x114)[0x7f07c8d8d994] /opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/cluster/replicate.so(afr_internal_lock_finish+0x84)[0x7f07c8d8f874] /opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/cluster/replicate.so(afr_post_blocking_inodelk_cbk+0x4c)[0x7f07c8d8febc] /opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/cluster/replicate.so(afr_lock_blocking+0x33f)[0x7f07c8da74bf] /opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/cluster/replicate.so(+0x37c23)[0x7f07c8da7c23] /opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/cluster/replicate.so(+0x37e14)[0x7f07c8da7e14] /opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/protocol/client.so(client3_1_finodelk_cbk+0x95)[0x7f07c8fd8d35] /opt/glusterfs/3.1.3/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa2)[0x7f07cbe50162] /opt/glusterfs/3.1.3/lib64/libgfrpc.so.0(rpc_clnt_notify+0x8d)[0x7f07cbe5035d] /opt/glusterfs/3.1.3/lib64/libgfrpc.so.0(rpc_transport_notify+0x2c)[0x7f07cbe4b93c] /opt/glusterfs/3.1.3/lib64/libgfrpc.so.0(rpc_clnt_notify+0x8d)[0x7f07cbe5035d] /opt/glusterfs/3.1.3/lib64/libgfrpc.so.0(rpc_transport_notify+0x2c)[0x7f07cbe4b93c] /opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/rpc-transport/socket.so(socket_event_poll_in+0x3f)[0x7f07c9e0af9f] /opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/rpc-transport/socket.so(socket_event_handler+0x170)[0x7f07c9e0b130] /opt/glusterfs/3.1.3/lib64/libglusterfs.so.0(+0x37b07)[0x7f07cc08fb07] /opt/glusterfs/3.1.3/sbin/glusterfs(main+0x39b)[0x4049ab] /lib64/libc.so.6(__libc_start_main+0xfd)[0x39b7e1ec5d] /opt/glusterfs/3.1.3/sbin/glusterfs[0x402ec9] ---------