Bug 764993 (GLUSTER-3261)

Summary: client glusterfs crash
Product: [Community] GlusterFS Reporter: visonge <visonge>
Component: locksAssignee: Pranith Kumar K <pkarampu>
Status: CLOSED DUPLICATE QA Contact:
Severity: high Docs Contact:
Priority: medium    
Version: 3.1.3CC: gluster-bugs
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Pranith Kumar K 2011-07-27 06:40:54 UTC 
It is duplicate of 2497. The loc is null according to the log:
[2011-07-27 12:11:00.502802] I [afr-open.c:435:afr_openfd_sh]
0-podo-replicate-0:  data self-heal triggered. path: (null), reason: Replicate
up down flush, data lock is held.

This issue is fixed in 3.1.5.

*** This bug has been marked as a duplicate of bug 2497 ***
Comment 1 visonge 2011-07-27 09:24:46 UTC 
Client is a postfix mail server. having some errors when using smtp and imap. 
today client glusterfs  crash.

this is some client log:
  1: volume podo-client-0
  2:     type protocol/client
  3:     option remote-host 10.101.120.11
  4:     option remote-subvolume /home/podo
  5:     option transport-type tcp
  6: end-volume
  7:
  8: volume podo-client-1
  9:     type protocol/client
 10:     option remote-host 10.101.120.12
 11:     option remote-subvolume /home/podo
 12:     option transport-type tcp
 13: end-volume
14:
 15: volume podo-replicate-0
 16:     type cluster/replicate
 17:     option favorite-child podo-client-0
 18:     subvolumes podo-client-0 podo-client-1
 19: end-volume
 20:
 21: volume podo-write-behind
 22:     type performance/write-behind
 23:     subvolumes podo-replicate-0
 24: end-volume
 25:
 26: volume podo-read-ahead
 27:     type performance/read-ahead
 28:     subvolumes podo-write-behind
 29: end-volume
 30:
 31: volume podo-io-cache
 32:     type performance/io-cache
 33:     subvolumes podo-read-ahead
 34: end-volume
 35:
 36: volume podo-quick-read
 37:     type performance/quick-read
 38:     subvolumes podo-io-cache
 39: end-volume
40:
 41: volume podo-stat-prefetch
 42:     type performance/stat-prefetch
 43:     subvolumes podo-quick-read
 44: end-volume
 45:
 46: volume podo
 47:     type debug/io-stats
 48:     subvolumes podo-stat-prefetch
 49: end-volume

+------------------------------------------------------------------------------+


[2011-07-27 12:10:50.98618] E [afr-self-heal-common.c:1214:sh_missing_entries_create] 0-podo-replicate-0: no missing files - /1003/7/88/Maildir/.&ZbCNJmI3-.INBOX/dovecot.index.log. proceeding to metadata check
[2011-07-27 12:10:50.101202] E [afr-common.c:110:afr_set_split_brain] 0-podo-replicate-0: invalid argument: inode
[2011-07-27 12:10:50.101307] I [afr-self-heal-common.c:1527:afr_self_heal_completion_cbk] 0-podo-replicate-0: background  data self-heal completed on /1003/7/88/Maildir/.&ZbCNJmI3-.INBOX/dovecot.index.log
[2011-07-27 12:10:51.553537] I [afr-common.c:716:afr_lookup_done] 0-podo-replicate-0: background  meta-data entry self-heal triggered. path: /1003/7/79/Maildir
[2011-07-27 12:10:51.566743] I [afr-self-heal-common.c:1527:afr_self_heal_completion_cbk] 0-podo-replicate-0: background  meta-data entry self-heal completed on /1003/7/79/Maildir
[2011-07-27 12:10:51.572941] I [afr-common.c:613:afr_lookup_self_heal_check] 0-podo-replicate-0: size differs for /1003/7/79/Maildir/dovecot.index.log
[2011-07-27 12:10:51.572979] I [afr-common.c:716:afr_lookup_done] 0-podo-replicate-0: background  data self-heal triggered. path: /1003/7/79/Maildir/dovecot.index.log
[2011-07-27 12:10:51.577881] I [afr-self-heal-common.c:1527:afr_self_heal_completion_cbk] 0-podo-replicate-0: background  data self-heal completed on /1003/7/79/Maildir/dovecot.index.log
[2011-07-27 12:10:59.159664] I [afr-common.c:716:afr_lookup_done] 0-podo-replicate-0: background  meta-data entry self-heal triggered. path: /1003/7/88/Maildir/.&ZbCNJmI3-.INBOX
[2011-07-27 12:10:59.165930] I [afr-self-heal-common.c:1527:afr_self_heal_completion_cbk] 0-podo-replicate-0: background  meta-data entry self-heal completed on /1003/7/88/Maildir/.&ZbCNJmI3-.INBOX
[2011-07-27 12:11:00.502107] W [fuse-bridge.c:2577:fuse_getxattr] 0-glusterfs-fuse: 32770407: GETXATTR (null)/139671323265676 (security.capability) (fuse_loc_fill() failed)
[2011-07-27 12:11:00.502401] W [fuse-bridge.c:2577:fuse_getxattr] 0-glusterfs-fuse: 32770410: GETXATTR (null)/139671323265676 (security.capability) (fuse_loc_fill() failed)
[2011-07-27 12:11:00.502802] I [afr-open.c:435:afr_openfd_sh] 0-podo-replicate-0:  data self-heal triggered. path: (null), reason: Replicate up down flush, data lock is held

pending frames:
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)
frame : type(1) op(STAT)

patchset: v3.1.3
signal received: 11
time of crash: 2011-07-27 12:11:00
configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.1.3
/lib64/libc.so.6[0x39b7e32a20]
/lib64/libc.so.6[0x39b7f2148f]
/opt/glusterfs/3.1.3/lib64/libglusterfs.so.0(loc_copy+0x55)[0x7f07cc0714e5]
/opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/cluster/replicate.so(afr_self_heal+0x441)[0x7f07c8d97261]
/opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/cluster/replicate.so(afr_openfd_sh+0x114)[0x7f07c8d8d994]
/opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/cluster/replicate.so(afr_internal_lock_finish+0x84)[0x7f07c8d8f874]
/opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/cluster/replicate.so(afr_post_blocking_inodelk_cbk+0x4c)[0x7f07c8d8febc]
/opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/cluster/replicate.so(afr_lock_blocking+0x33f)[0x7f07c8da74bf]
/opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/cluster/replicate.so(+0x37c23)[0x7f07c8da7c23]
/opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/cluster/replicate.so(+0x37e14)[0x7f07c8da7e14]
/opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/protocol/client.so(client3_1_finodelk_cbk+0x95)[0x7f07c8fd8d35]
/opt/glusterfs/3.1.3/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa2)[0x7f07cbe50162]
/opt/glusterfs/3.1.3/lib64/libgfrpc.so.0(rpc_clnt_notify+0x8d)[0x7f07cbe5035d]
/opt/glusterfs/3.1.3/lib64/libgfrpc.so.0(rpc_transport_notify+0x2c)[0x7f07cbe4b93c]
/opt/glusterfs/3.1.3/lib64/libgfrpc.so.0(rpc_clnt_notify+0x8d)[0x7f07cbe5035d]
/opt/glusterfs/3.1.3/lib64/libgfrpc.so.0(rpc_transport_notify+0x2c)[0x7f07cbe4b93c]
/opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/rpc-transport/socket.so(socket_event_poll_in+0x3f)[0x7f07c9e0af9f]
/opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/rpc-transport/socket.so(socket_event_handler+0x170)[0x7f07c9e0b130]
/opt/glusterfs/3.1.3/lib64/libglusterfs.so.0(+0x37b07)[0x7f07cc08fb07]
/opt/glusterfs/3.1.3/sbin/glusterfs(main+0x39b)[0x4049ab]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x39b7e1ec5d]
/opt/glusterfs/3.1.3/sbin/glusterfs[0x402ec9]
---------