Bug 762079 (GLUSTER-347)

Summary: Assertion failed: inode->ref
Product: [Community] GlusterFS Reporter: Vikas Gorur <vikas>
Component: coreAssignee: Anand Avati <aavati>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: medium Docs Contact:
Priority: low    
Version: mainlineCC: chrisw, gluster-bugs
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: RTNR Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Vikas Gorur 2009-10-29 10:35:15 UTC 
Backtrace:

[2009-10-29 17:49:51] T [fuse-bridge.c:1885:fuse_readv] glusterfs-fuse: 104322: READ (0xb621fa00, size=131072, offset=0)
[2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104321: STAT() /dbench-4.0/clients/client1/~dmtmp/WORD => 969438
[2009-10-29 17:49:51] T [fuse-bridge.c:1051:fuse_setattr] glusterfs-fuse: 104323: SETATTR (3055672112)/dbench-4.0/clients/client4/~dmtmp/WORD/~WRD0003.TMP
[2009-10-29 17:49:51] T [fuse-bridge.c:1909:fuse_writev_cbk] glusterfs-fuse: 104318: WRITE => 1/1,15359/0
[2009-10-29 17:49:51] T [fuse-bridge.c:2051:fuse_release] glusterfs-fuse: 104324: RELEASE 0x80b8a58 (FLUSH implied)
[2009-10-29 17:49:51] T [fuse-bridge.c:856:fuse_getattr] glusterfs-fuse: 104325: FGETATTR 134981624 (/dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP/0x80b8a58)
[2009-10-29 17:49:51] T [fuse-bridge.c:1842:fuse_readv_cbk] glusterfs-fuse: 104322: READ => 120756/131072,0/120756
[2009-10-29 17:49:51] T [fuse-bridge.c:2238:fuse_readdir] glusterfs-fuse: 104326: READDIR (0xb620ae18, size=4096, offset=0)
[2009-10-29 17:49:51] T [fuse-bridge.c:635:fuse_lookup] glusterfs-fuse: 104327: LOOKUP /dbench-4.0/clients/client1/~dmtmp/WORD/HANDOUT.DOC
[2009-10-29 17:49:51] D [inode.c:886:inode_path] fuse/inode: no dentry for non-root inode 975752
[2009-10-29 17:49:51] D [fuse-bridge.c:445:fuse_loc_fill] glusterfs-fuse: inode_path failed for 975752
[2009-10-29 17:49:51] T [fuse-bridge.c:856:fuse_getattr] glusterfs-fuse: 104328: FGETATTR 3055661960 ((null)/0xb621f510)
[2009-10-29 17:49:51] T [fuse-bridge.c:1956:fuse_write] glusterfs-fuse: 104329: WRITE (0xb621fa00, size=32, offset=47432)
[2009-10-29 17:49:51] T [fuse-bridge.c:934:fuse_setattr_cbk] glusterfs-fuse: 104323: SETATTR() /dbench-4.0/clients/client4/~dmtmp/WORD/~WRD0003.TMP => 975764
[2009-10-29 17:49:51] T [fuse-bridge.c:1051:fuse_setattr] glusterfs-fuse: 104330: SETATTR (3055672112)/dbench-4.0/clients/client4/~dmtmp/WORD/~WRD0003.TMP
[2009-10-29 17:49:51] T [fuse-bridge.c:2183:fuse_readdir_cbk] glusterfs-fuse: 104326: READDIR => 7/4096,0
[2009-10-29 17:49:51] T [fuse-bridge.c:562:fuse_entry_cbk] glusterfs-fuse: 104327: LOOKUP() /dbench-4.0/clients/client1/~dmtmp/WORD/HANDOUT.DOC => -1 (No such file or directory)
[2009-10-29 17:49:51] T [fuse-bridge.c:2238:fuse_readdir] glusterfs-fuse: 104331: READDIR (0xb620ae18, size=4096, offset=1856558593)
[2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104325: FSTAT() /dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP => 975880
[2009-10-29 17:49:51] T [fuse-bridge.c:2120:fuse_opendir] glusterfs-fuse: 104332: OPENDIR /dbench-4.0/clients/client1/~dmtmp/WORD
[2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104328: FSTAT() ERR => 975752
[2009-10-29 17:49:51] T [fuse-bridge.c:1164:fuse_err_cbk] glusterfs-fuse: 104324: FLUSH() ERR => 0
[2009-10-29 17:49:51] T [fuse-bridge.c:1815:fuse_open] glusterfs-fuse: 104333: OPEN /dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP
[2009-10-29 17:49:51] T [fuse-bridge.c:2183:fuse_readdir_cbk] glusterfs-fuse: 104331: READDIR => 0/4096,1856558593
[2009-10-29 17:49:51] T [fuse-bridge.c:1956:fuse_write] glusterfs-fuse: 104334: WRITE (0xb621f510, size=15360, offset=0)
[2009-10-29 17:49:51] T [fuse-bridge.c:934:fuse_setattr_cbk] glusterfs-fuse: 104330: SETATTR() /dbench-4.0/clients/client4/~dmtmp/WORD/~WRD0003.TMP => 975764
[2009-10-29 17:49:51] T [fuse-bridge.c:2256:fuse_releasedir] glusterfs-fuse: 104335: RELEASEDIR 0xb620ae18
[2009-10-29 17:49:51] T [fuse-bridge.c:847:fuse_getattr] glusterfs-fuse: 104336: GETATTR 3055622456 (/dbench-4.0/clients/client0/~dmtmp/WORD)
[2009-10-29 17:49:51] T [fuse-bridge.c:889:fuse_fd_cbk] glusterfs-fuse: 104332: OPENDIR() /dbench-4.0/clients/client1/~dmtmp/WORD => 0xb620aac8
[2009-10-29 17:49:51] T [fuse-bridge.c:635:fuse_lookup] glusterfs-fuse: 104337: LOOKUP /dbench-4.0/clients/client4/~dmtmp/WORD/~WRL0004.TMP
[2009-10-29 17:49:51] T [fuse-bridge.c:2238:fuse_readdir] glusterfs-fuse: 104338: READDIR (0xb620aac8, size=4096, offset=0)
[2009-10-29 17:49:51] T [fuse-bridge.c:889:fuse_fd_cbk] glusterfs-fuse: 104333: OPEN() /dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP => 0xb621adc0
[2009-10-29 17:49:51] T [fuse-bridge.c:1909:fuse_writev_cbk] glusterfs-fuse: 104329: WRITE => 32/32,47432/120756
[2009-10-29 17:49:51] T [fuse-bridge.c:1885:fuse_readv] glusterfs-fuse: 104339: READ (0xb621adc0, size=4096, offset=0)
[2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104336: STAT() /dbench-4.0/clients/client0/~dmtmp/WORD => 969518
[2009-10-29 17:49:51] T [fuse-bridge.c:1885:fuse_readv] glusterfs-fuse: 104340: READ (0xb621fa00, size=131072, offset=0)
[2009-10-29 17:49:51] T [fuse-bridge.c:635:fuse_lookup] glusterfs-fuse: 104341: LOOKUP /dbench-4.0/clients/client0/~dmtmp/WORD/HANDOUT.DOC
[2009-10-29 17:49:51] T [fuse-bridge.c:562:fuse_entry_cbk] glusterfs-fuse: 104337: LOOKUP() /dbench-4.0/clients/client4/~dmtmp/WORD/~WRL0004.TMP => -1 (No such file or directory)
[2009-10-29 17:49:51] T [fuse-bridge.c:2120:fuse_opendir] glusterfs-fuse: 104342: OPENDIR /dbench-4.0/clients/client4/~dmtmp/WORD
[2009-10-29 17:49:51] T [fuse-bridge.c:2183:fuse_readdir_cbk] glusterfs-fuse: 104338: READDIR => 6/4096,0
[2009-10-29 17:49:51] T [fuse-bridge.c:2238:fuse_readdir] glusterfs-fuse: 104343: READDIR (0xb620aac8, size=4096, offset=1856558593)
[2009-10-29 17:49:51] T [fuse-bridge.c:1842:fuse_readv_cbk] glusterfs-fuse: 104340: READ => 120756/131072,0/120756
[2009-10-29 17:49:51] T [fuse-bridge.c:1956:fuse_write] glusterfs-fuse: 104344: WRITE (0xb621fa00, size=1568, offset=116096)
[2009-10-29 17:49:51] T [fuse-bridge.c:562:fuse_entry_cbk] glusterfs-fuse: 104341: LOOKUP() /dbench-4.0/clients/client0/~dmtmp/WORD/HANDOUT.DOC => -1 (No such file or directory)
[2009-10-29 17:49:51] T [fuse-bridge.c:1595:fuse_rename] glusterfs-fuse: 104345: RENAME `/dbench-4.0/clients/client0/~dmtmp/WORD/~WRD0000.TMP (975690)' -> `/dbench-4.0/clients/client0/~dmtmp/WORD/HANDOUT.DOC (0)'
[2009-10-29 17:49:51] T [fuse-bridge.c:889:fuse_fd_cbk] glusterfs-fuse: 104342: OPENDIR() /dbench-4.0/clients/client4/~dmtmp/WORD => 0xb620ae18
[2009-10-29 17:49:51] T [fuse-bridge.c:1842:fuse_readv_cbk] glusterfs-fuse: 104339: READ => 4096/4096,0/15360
[2009-10-29 17:49:51] T [fuse-bridge.c:2238:fuse_readdir] glusterfs-fuse: 104346: READDIR (0xb620ae18, size=4096, offset=0)
[2009-10-29 17:49:51] T [fuse-bridge.c:2183:fuse_readdir_cbk] glusterfs-fuse: 104343: READDIR => 0/4096,1856558593
[2009-10-29 17:49:51] T [fuse-bridge.c:1909:fuse_writev_cbk] glusterfs-fuse: 104334: WRITE => 15360/15360,0/15360
[2009-10-29 17:49:51] T [fuse-bridge.c:856:fuse_getattr] glusterfs-fuse: 104347: FGETATTR 134981624 (/dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP/0xb621adc0)
[2009-10-29 17:49:51] T [fuse-bridge.c:2183:fuse_readdir_cbk] glusterfs-fuse: 104346: READDIR => 7/4096,0
[2009-10-29 17:49:51] T [fuse-bridge.c:2256:fuse_releasedir] glusterfs-fuse: 104348: RELEASEDIR 0xb620aac8
[2009-10-29 17:49:51] T [fuse-bridge.c:847:fuse_getattr] glusterfs-fuse: 104349: GETATTR 3055646864 (/dbench-4.0/clients/client1/~dmtmp/WORD)
[2009-10-29 17:49:51] T [fuse-bridge.c:1993:fuse_flush] glusterfs-fuse: 104350: FLUSH 0xb621f510
[2009-10-29 17:49:51] T [fuse-bridge.c:2238:fuse_readdir] glusterfs-fuse: 104351: READDIR (0xb620ae18, size=4096, offset=1856558593)
[2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104349: STAT() /dbench-4.0/clients/client1/~dmtmp/WORD => 969438
[2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104347: FSTAT() /dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP => 975880
[2009-10-29 17:49:51] T [fuse-bridge.c:2183:fuse_readdir_cbk] glusterfs-fuse: 104351: READDIR => 0/4096,1856558593
[2009-10-29 17:49:51] T [fuse-bridge.c:1909:fuse_writev_cbk] glusterfs-fuse: 104344: WRITE => 1568/1568,116096/120756
[2009-10-29 17:49:51] T [fuse-bridge.c:635:fuse_lookup] glusterfs-fuse: 104352: LOOKUP /dbench-4.0/clients/client1/~dmtmp/WORD/HANDOUT.DOC
[2009-10-29 17:49:51] T [fuse-bridge.c:1164:fuse_err_cbk] glusterfs-fuse: 104350: FLUSH() ERR => 0
[2009-10-29 17:49:51] T [fuse-bridge.c:1051:fuse_setattr] glusterfs-fuse: 104353: SETATTR (134981624)/dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP
[2009-10-29 17:49:51] T [fuse-bridge.c:1524:fuse_rename_cbk] glusterfs-fuse: 104345: /dbench-4.0/clients/client0/~dmtmp/WORD/~WRD0000.TMP -> /dbench-4.0/clients/client0/~dmtmp/WORD/HANDOUT.DOC => 0 (buf->st_ino=975690 , loc->ino=975690)
[2009-10-29 17:49:51] T [fuse-bridge.c:2256:fuse_releasedir] glusterfs-fuse: 104354: RELEASEDIR 0xb620ae18
[2009-10-29 17:49:51] T [fuse-bridge.c:562:fuse_entry_cbk] glusterfs-fuse: 104352: LOOKUP() /dbench-4.0/clients/client1/~dmtmp/WORD/HANDOUT.DOC => -1 (No such file or directory)
[2009-10-29 17:49:51] T [fuse-bridge.c:1993:fuse_flush] glusterfs-fuse: 104355: FLUSH 0x80a30b8
[2009-10-29 17:49:51] T [fuse-bridge.c:934:fuse_setattr_cbk] glusterfs-fuse: 104353: SETATTR() /dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP => 975880
[2009-10-29 17:49:51] T [fuse-bridge.c:1885:fuse_readv] glusterfs-fuse: 104356: READ (0xb621fa00, size=131072, offset=0)
[2009-10-29 17:49:51] T [fuse-bridge.c:2051:fuse_release] glusterfs-fuse: 104357: RELEASE 0xb621f510 (FLUSH implied)
[2009-10-29 17:49:51] D [inode.c:886:inode_path] fuse/inode: no dentry for non-root inode 975752
[2009-10-29 17:49:51] D [fuse-bridge.c:445:fuse_loc_fill] glusterfs-fuse: inode_path failed for 975752
[2009-10-29 17:49:51] T [fuse-bridge.c:856:fuse_getattr] glusterfs-fuse: 104358: FGETATTR 3055661960 ((null)/0xb621f510)
[2009-10-29 17:49:51] T [fuse-bridge.c:847:fuse_getattr] glusterfs-fuse: 104359: GETATTR 3055622456 (/dbench-4.0/clients/client0/~dmtmp/WORD)
[2009-10-29 17:49:51] T [fuse-bridge.c:2120:fuse_opendir] glusterfs-fuse: 104360: OPENDIR /dbench-4.0/clients/client1/~dmtmp/WORD
[2009-10-29 17:49:51] T [fuse-bridge.c:1051:fuse_setattr] glusterfs-fuse: 104361: SETATTR (134981624)/dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP
[2009-10-29 17:49:51] T [fuse-bridge.c:1842:fuse_readv_cbk] glusterfs-fuse: 104356: READ => 120756/131072,0/120756
[2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104359: STAT() /dbench-4.0/clients/client0/~dmtmp/WORD => 969518
[2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104358: FSTAT() ERR => 975752
[2009-10-29 17:49:51] T [fuse-bridge.c:889:fuse_fd_cbk] glusterfs-fuse: 104360: OPENDIR() /dbench-4.0/clients/client1/~dmtmp/WORD => 0xb6218ed8
[2009-10-29 17:49:51] T [fuse-bridge.c:847:fuse_getattr] glusterfs-fuse: 104362: GETATTR 134935672 (/dbench-4.0/clients/client0/~dmtmp/WORD/HANDOUT.DOC)
[2009-10-29 17:49:51] T [fuse-bridge.c:1956:fuse_write] glusterfs-fuse: 104363: WRITE (0xb621fa00, size=7612, offset=87496)
[2009-10-29 17:49:51] T [fuse-bridge.c:1164:fuse_err_cbk] glusterfs-fuse: 104355: FLUSH() ERR => 0
[2009-10-29 17:49:51] D [inode.c:886:inode_path] fuse/inode: no dentry for non-root inode 975752
[2009-10-29 17:49:51] T [fuse-bridge.c:1164:fuse_err_cbk] glusterfs-fuse: 104357: FLUSH() ERR => 0
[2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104362: STAT() /dbench-4.0/clients/client0/~dmtmp/WORD/HANDOUT.DOC => 975690
[2009-10-29 17:49:51] D [fuse-bridge.c:445:fuse_loc_fill] glusterfs-fuse: inode_path failed for 975752
[2009-10-29 17:49:51] W [fuse-bridge.c:1802:fuse_open] glusterfs-fuse: 104364: OPEN (null) (fuse_loc_fill() failed)
[2009-10-29 17:49:51] T [fuse-bridge.c:2238:fuse_readdir] glusterfs-fuse: 104365: READDIR (0xb6218ed8, size=4096, offset=0)
[2009-10-29 17:49:51] T [fuse-bridge.c:2051:fuse_release] glusterfs-fuse: 104366: RELEASE 0x80a30b8 (FLUSH implied)
[2009-10-29 17:49:51] T [fuse-bridge.c:847:fuse_getattr] glusterfs-fuse: 104367: GETATTR 3055620528 (/dbench-4.0/clients/client4/~dmtmp/WORD)
[2009-10-29 17:49:51] T [fuse-bridge.c:934:fuse_setattr_cbk] glusterfs-fuse: 104361: SETATTR() /dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP => 975880
[2009-10-29 17:49:51] T [fuse-bridge.c:1815:fuse_open] glusterfs-fuse: 104368: OPEN /dbench-4.0/clients/client0/~dmtmp/WORD/HANDOUT.DOC
[2009-10-29 17:49:51] T [fuse-bridge.c:1993:fuse_flush] glusterfs-fuse: 104369: FLUSH 0xb6412c48
[2009-10-29 17:49:51] T [fuse-bridge.c:635:fuse_lookup] glusterfs-fuse: 104370: LOOKUP /dbench-4.0/clients/client3/~dmtmp/WORD/~WRL0004.TMP
[2009-10-29 17:49:51] T [fuse-bridge.c:2183:fuse_readdir_cbk] glusterfs-fuse: 104365: READDIR => 6/4096,0
[2009-10-29 17:49:51] T [fuse-bridge.c:2238:fuse_readdir] glusterfs-fuse: 104371: READDIR (0xb6218ed8, size=4096, offset=1856558593)
[2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104367: STAT() /dbench-4.0/clients/client4/~dmtmp/WORD => 969362
[2009-10-29 17:49:51] T [fuse-bridge.c:856:fuse_getattr] glusterfs-fuse: 104372: FGETATTR 134934656 (/dbench-4.0/clients/client4/~dmtmp/WORD/HANDOUT.DOC/0x80a30b8)
[2009-10-29 17:49:51] T [fuse-bridge.c:889:fuse_fd_cbk] glusterfs-fuse: 104368: OPEN() /dbench-4.0/clients/client0/~dmtmp/WORD/HANDOUT.DOC => 0x80a2208
[2009-10-29 17:49:51] T [fuse-bridge.c:1885:fuse_readv] glusterfs-fuse: 104373: READ (0x80a2208, size=4096, offset=0)
[2009-10-29 17:49:51] T [fuse-bridge.c:2183:fuse_readdir_cbk] glusterfs-fuse: 104371: READDIR => 0/4096,1856558593
[2009-10-29 17:49:51] T [fuse-bridge.c:562:fuse_entry_cbk] glusterfs-fuse: 104370: LOOKUP() /dbench-4.0/clients/client3/~dmtmp/WORD/~WRL0004.TMP => -1 (No such file or directory)
[2009-10-29 17:49:51] T [fuse-bridge.c:1164:fuse_err_cbk] glusterfs-fuse: 104366: FLUSH() ERR => 0
[2009-10-29 17:49:51] T [fuse-bridge.c:1842:fuse_readv_cbk] glusterfs-fuse: 104373: READ => 4096/4096,0/15360
glusterfs: ../../../libglusterfs/src/inode.c:352: __inode_unref: Assertion `inode->ref' failed.

Program received signal SIGABRT, Aborted.
0xb7fe3410 in __kernel_vsyscall ()
(gdb) bt
#0  0xb7fe3410 in __kernel_vsyscall ()
#1  0xb7e40085 in raise () from /lib/tls/i686/cmov/libc.so.6
#2  0xb7e41a01 in abort () from /lib/tls/i686/cmov/libc.so.6
#3  0xb7e3910e in __assert_fail () from /lib/tls/i686/cmov/libc.so.6
#4  0xb7fa5908 in __inode_unref (inode=0xb621b388) at ../../../libglusterfs/src/inode.c:352
#5  0xb7fa59d9 in inode_unref (inode=0xb621b388) at ../../../libglusterfs/src/inode.c:391
#6  0xb7fb6423 in fd_destroy (fd=0xb621f510) at ../../../libglusterfs/src/fd.c:422
#7  0xb7fb6500 in fd_unref (fd=0xb621f510) at ../../../libglusterfs/src/fd.c:448
#8  0xb7571b38 in afr_local_cleanup (local=0xb6221a48, this=0x8056940)
    at ../../../../../xlators/cluster/afr/src/afr.c:269
#9  0xb75740f8 in afr_flush_done (frame=0xb621f408, this=0x8056940)
    at ../../../../../xlators/cluster/afr/src/afr.c:1043
#10 0xb758d9ce in afr_unlock_common_cbk (frame=0xb621f408, cookie=0xb621def0, 
    this=0x8056940, op_ret=0, op_errno=0)
    at ../../../../../xlators/cluster/afr/src/afr-transaction.c:398
#11 0xb75c8806 in client_finodelk_cbk (frame=0xb621def0, hdr=0xb620f400, hdrlen=32, 
    iobuf=0x0) at ../../../../../xlators/protocol/client/src/client-protocol.c:5165
#12 0xb75ca7e3 in protocol_client_interpret (this=0x8051cf0, trans=0x808f270, 
    hdr_p=0xb620f400 "", hdrlen=32, iobuf=0x0)
    at ../../../../../xlators/protocol/client/src/client-protocol.c:5969
#13 0xb75cb571 in protocol_client_pollin (this=0x8051cf0, trans=0x808f270)
    at ../../../../../xlators/protocol/client/src/client-protocol.c:6263
#14 0xb75cbbc0 in notify (this=0x8051cf0, event=2, data=0x808f270)
    at ../../../../../xlators/protocol/client/src/client-protocol.c:6382
#15 0xb7f97f82 in xlator_notify (xl=0x8051cf0, event=2, data=0x808f270)
    at ../../../libglusterfs/src/xlator.c:923
#16 0xb6d63663 in socket_event_poll_in (this=0x808f270)
    at ../../../../transport/socket/src/socket.c:713
#17 0xb6d63968 in socket_event_handler (fd=9, idx=0, data=0x808f270, poll_in=1, 
    poll_out=0, poll_err=0) at ../../../../transport/socket/src/socket.c:813
#18 0xb7fb918c in event_dispatch_epoll_handler (event_pool=0x80511d8, events=0x8090e90, 
    i=0) at ../../../libglusterfs/src/event.c:804
#19 0xb7fb935d in event_dispatch_epoll (event_pool=0x80511d8)
    at ../../../libglusterfs/src/event.c:867
#20 0xb7fb96b8 in event_dispatch (event_pool=0x80511d8)
    at ../../../libglusterfs/src/event.c:975
#21 0x0804dd2a in main (argc=7, argv=0xbffff864)
    at ../../../glusterfsd/src/glusterfsd.c:1379
Comment 1 Vikas Gorur 2009-10-29 10:36:26 UTC 
Additional info:

(gdb) p *fd
$36 = {pid = 32037, flags = 66, refcount = 0, flush_unique = 104350, inode_list = {next = 0xb621f524, prev = 0xb621f524}, inode = 0xb621b388, lock = 1, _ctx = 0xeeeeeeee}

(gdb) p *fd->inode
$38 = {table = 0x8057258, lock = 1, nlookup = 1, generation = 397994504831633667, 
  in_attic = 0, ref = 0, ino = 975752, st_mode = 33152, fd_list = {next = 0xb621b3b4, prev = 0xb621b3b4}, dentry_list = {next = 0xb621b3bc, prev = 0xb621b3bc}, hash = {next = 0xb621b3c4, prev = 0xb621b3c4}, list = {next = 0x8057298, prev = 0xb621d8b4}, _ctx = 0xb6214df0}
Comment 2 Anand Avati 2009-10-29 13:32:32 UTC 
PATCH: http://patches.gluster.com/patch/2068 in master (protocol/server, mount/fuse: Handle returned inode in create properly)
Comment 3 Vikas Gorur 2009-10-29 13:34:36 UTC 
Running dbench on an afr (2 server) configuration results in assertion failed: inode->ref

Relevant portion of the log: The guilty inode is 975752, corresponding to fd=0xb621f510

[2009-10-29 17:49:51] T [fuse-bridge.c:1675:fuse_create_cbk] glusterfs-fuse: 104309: CREATE() /dbench-4.0/clients/client2/~dmtmp/WORD/~WRD0003.TMP => 0xb621f510 (ino=975752)
[2009-10-29 17:49:51] T [fuse-bridge.c:1956:fuse_write] glusterfs-fuse: 104318: WRITE (0xb621f510, size=1, offset=15359)
[2009-10-29 17:49:51] T [fuse-bridge.c:1164:fuse_err_cbk] glusterfs-fuse: 104279: FLUSH() ERR => 0
[2009-10-29 17:49:51] T [fuse-bridge.c:562:fuse_entry_cbk] glusterfs-fuse: 104315: LOOKUP() /dbench-4.0/clients/client0/~dmtmp/WORD/HANDOUT.DOC => -1 (No such file or directory)
[2009-10-29 17:49:51] T [fuse-bridge.c:2120:fuse_opendir] glusterfs-fuse: 104319: OPENDIR /dbench-4.0/clients/client0/~dmtmp/WORD
[2009-10-29 17:49:51] T [fuse-bridge.c:1164:fuse_err_cbk] glusterfs-fuse: 104286: FLUSH() ERR => 0
[2009-10-29 17:49:51] T [fuse-bridge.c:1223:fuse_unlink_cbk] glusterfs-fuse: 104287: UNLINK() /dbench-4.0/clients/client1/~dmtmp/WORD/HANDOUT.DOC => 0
[2009-10-29 17:49:51] T [fuse-bridge.c:1164:fuse_err_cbk] glusterfs-fuse: 104314: FLUSH() ERR => 0
[2009-10-29 17:49:51] T [fuse-bridge.c:1909:fuse_writev_cbk] glusterfs-fuse: 104313: WRITE => 1531/1531,47504/120756
[2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104317: FSTAT() /dbench-4.0/clients/client4/~dmtmp/WORD/~WRD0003.TMP => 975764
[2009-10-29 17:49:51] T [fuse-bridge.c:1164:fuse_err_cbk] glusterfs-fuse: 104316: FLUSH() ERR => 0
[2009-10-29 17:49:51] T [fuse-bridge.c:889:fuse_fd_cbk] glusterfs-fuse: 104319: OPENDIR() /dbench-4.0/clients/client0/~dmtmp/WORD => 0xb620ae18
[2009-10-29 17:49:51] T [fuse-bridge.c:847:fuse_getattr] glusterfs-fuse: 104321: GETATTR 3055646864 (/dbench-4.0/clients/client1/~dmtmp/WORD)
[2009-10-29 17:49:51] T [fuse-bridge.c:1885:fuse_readv] glusterfs-fuse: 104322: READ (0xb621fa00, size=131072, offset=0)
[2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104321: STAT() /dbench-4.0/clients/client1/~dmtmp/WORD => 969438
[2009-10-29 17:49:51] T [fuse-bridge.c:1051:fuse_setattr] glusterfs-fuse: 104323: SETATTR (3055672112)/dbench-4.0/clients/client4/~dmtmp/WORD/~WRD0003.TMP
[2009-10-29 17:49:51] T [fuse-bridge.c:1909:fuse_writev_cbk] glusterfs-fuse: 104318: WRITE => 1/1,15359/0
[2009-10-29 17:49:51] T [fuse-bridge.c:2051:fuse_release] glusterfs-fuse: 104324: RELEASE 0x80b8a58 (FLUSH implied)
[2009-10-29 17:49:51] T [fuse-bridge.c:856:fuse_getattr] glusterfs-fuse: 104325: FGETATTR 134981624 (/dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP/0x80b8a58)
[2009-10-29 17:49:51] T [fuse-bridge.c:1842:fuse_readv_cbk] glusterfs-fuse: 104322: READ => 120756/131072,0/120756
[2009-10-29 17:49:51] T [fuse-bridge.c:2238:fuse_readdir] glusterfs-fuse: 104326: READDIR (0xb620ae18, size=4096, offset=0)
[2009-10-29 17:49:51] T [fuse-bridge.c:635:fuse_lookup] glusterfs-fuse: 104327: LOOKUP /dbench-4.0/clients/client1/~dmtmp/WORD/HANDOUT.DOC
[2009-10-29 17:49:51] D [inode.c:886:inode_path] fuse/inode: no dentry for non-root inode 975752
[2009-10-29 17:49:51] D [fuse-bridge.c:445:fuse_loc_fill] glusterfs-fuse: inode_path failed for 975752
[2009-10-29 17:49:51] T [fuse-bridge.c:856:fuse_getattr] glusterfs-fuse: 104328: FGETATTR 3055661960 ((null)/0xb621f510)
[2009-10-29 17:49:51] T [fuse-bridge.c:1956:fuse_write] glusterfs-fuse: 104329: WRITE (0xb621fa00, size=32, offset=47432)
[2009-10-29 17:49:51] T [fuse-bridge.c:934:fuse_setattr_cbk] glusterfs-fuse: 104323: SETATTR() /dbench-4.0/clients/client4/~dmtmp/WORD/~WRD0003.TMP => 975764
[2009-10-29 17:49:51] T [fuse-bridge.c:1051:fuse_setattr] glusterfs-fuse: 104330: SETATTR (3055672112)/dbench-4.0/clients/client4/~dmtmp/WORD/~WRD0003.TMP
[2009-10-29 17:49:51] T [fuse-bridge.c:2183:fuse_readdir_cbk] glusterfs-fuse: 104326: READDIR => 7/4096,0
[2009-10-29 17:49:51] T [fuse-bridge.c:562:fuse_entry_cbk] glusterfs-fuse: 104327: LOOKUP() /dbench-4.0/clients/client1/~dmtmp/WORD/HANDOUT.DOC => -1 (No such file or directory)
[2009-10-29 17:49:51] T [fuse-bridge.c:2238:fuse_readdir] glusterfs-fuse: 104331: READDIR (0xb620ae18, size=4096, offset=1856558593)
[2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104325: FSTAT() /dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP => 975880
[2009-10-29 17:49:51] T [fuse-bridge.c:2120:fuse_opendir] glusterfs-fuse: 104332: OPENDIR /dbench-4.0/clients/client1/~dmtmp/WORD
[2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104328: FSTAT() ERR => 975752
[2009-10-29 17:49:51] T [fuse-bridge.c:1164:fuse_err_cbk] glusterfs-fuse: 104324: FLUSH() ERR => 0
[2009-10-29 17:49:51] T [fuse-bridge.c:1815:fuse_open] glusterfs-fuse: 104333: OPEN /dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP
[2009-10-29 17:49:51] T [fuse-bridge.c:2183:fuse_readdir_cbk] glusterfs-fuse: 104331: READDIR => 0/4096,1856558593
[2009-10-29 17:49:51] T [fuse-bridge.c:1956:fuse_write] glusterfs-fuse: 104334: WRITE (0xb621f510, size=15360, offset=0)
[2009-10-29 17:49:51] T [fuse-bridge.c:934:fuse_setattr_cbk] glusterfs-fuse: 104330: SETATTR() /dbench-4.0/clients/client4/~dmtmp/WORD/~WRD0003.TMP => 975764
[2009-10-29 17:49:51] T [fuse-bridge.c:2256:fuse_releasedir] glusterfs-fuse: 104335: RELEASEDIR 0xb620ae18
[2009-10-29 17:49:51] T [fuse-bridge.c:847:fuse_getattr] glusterfs-fuse: 104336: GETATTR 3055622456 (/dbench-4.0/clients/client0/~dmtmp/WORD)
[2009-10-29 17:49:51] T [fuse-bridge.c:889:fuse_fd_cbk] glusterfs-fuse: 104332: OPENDIR() /dbench-4.0/clients/client1/~dmtmp/WORD => 0xb620aac8
[2009-10-29 17:49:51] T [fuse-bridge.c:635:fuse_lookup] glusterfs-fuse: 104337: LOOKUP /dbench-4.0/clients/client4/~dmtmp/WORD/~WRL0004.TMP
[2009-10-29 17:49:51] T [fuse-bridge.c:2238:fuse_readdir] glusterfs-fuse: 104338: READDIR (0xb620aac8, size=4096, offset=0)
[2009-10-29 17:49:51] T [fuse-bridge.c:889:fuse_fd_cbk] glusterfs-fuse: 104333: OPEN() /dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP => 0xb621adc0
[2009-10-29 17:49:51] T [fuse-bridge.c:1909:fuse_writev_cbk] glusterfs-fuse: 104329: WRITE => 32/32,47432/120756
[2009-10-29 17:49:51] T [fuse-bridge.c:1885:fuse_readv] glusterfs-fuse: 104339: READ (0xb621adc0, size=4096, offset=0)
[2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104336: STAT() /dbench-4.0/clients/client0/~dmtmp/WORD => 969518
[2009-10-29 17:49:51] T [fuse-bridge.c:1885:fuse_readv] glusterfs-fuse: 104340: READ (0xb621fa00, size=131072, offset=0)
[2009-10-29 17:49:51] T [fuse-bridge.c:635:fuse_lookup] glusterfs-fuse: 104341: LOOKUP /dbench-4.0/clients/client0/~dmtmp/WORD/HANDOUT.DOC
[2009-10-29 17:49:51] T [fuse-bridge.c:562:fuse_entry_cbk] glusterfs-fuse: 104337: LOOKUP() /dbench-4.0/clients/client4/~dmtmp/WORD/~WRL0004.TMP => -1 (No such file or directory)
[2009-10-29 17:49:51] T [fuse-bridge.c:2120:fuse_opendir] glusterfs-fuse: 104342: OPENDIR /dbench-4.0/clients/client4/~dmtmp/WORD
[2009-10-29 17:49:51] T [fuse-bridge.c:2183:fuse_readdir_cbk] glusterfs-fuse: 104338: READDIR => 6/4096,0
[2009-10-29 17:49:51] T [fuse-bridge.c:2238:fuse_readdir] glusterfs-fuse: 104343: READDIR (0xb620aac8, size=4096, offset=1856558593)
[2009-10-29 17:49:51] T [fuse-bridge.c:1842:fuse_readv_cbk] glusterfs-fuse: 104340: READ => 120756/131072,0/120756
[2009-10-29 17:49:51] T [fuse-bridge.c:1956:fuse_write] glusterfs-fuse: 104344: WRITE (0xb621fa00, size=1568, offset=116096)
[2009-10-29 17:49:51] T [fuse-bridge.c:562:fuse_entry_cbk] glusterfs-fuse: 104341: LOOKUP() /dbench-4.0/clients/client0/~dmtmp/WORD/HANDOUT.DOC => -1 (No such file or directory)
[2009-10-29 17:49:51] T [fuse-bridge.c:1595:fuse_rename] glusterfs-fuse: 104345: RENAME `/dbench-4.0/clients/client0/~dmtmp/WORD/~WRD0000.TMP (975690)' -> `/dbench-4.0/clients/client0/~dmtmp/WORD/HANDOUT.DOC (0)'
[2009-10-29 17:49:51] T [fuse-bridge.c:889:fuse_fd_cbk] glusterfs-fuse: 104342: OPENDIR() /dbench-4.0/clients/client4/~dmtmp/WORD => 0xb620ae18
[2009-10-29 17:49:51] T [fuse-bridge.c:1842:fuse_readv_cbk] glusterfs-fuse: 104339: READ => 4096/4096,0/15360
[2009-10-29 17:49:51] T [fuse-bridge.c:2238:fuse_readdir] glusterfs-fuse: 104346: READDIR (0xb620ae18, size=4096, offset=0)
[2009-10-29 17:49:51] T [fuse-bridge.c:2183:fuse_readdir_cbk] glusterfs-fuse: 104343: READDIR => 0/4096,1856558593
[2009-10-29 17:49:51] T [fuse-bridge.c:1909:fuse_writev_cbk] glusterfs-fuse: 104334: WRITE => 15360/15360,0/15360
[2009-10-29 17:49:51] T [fuse-bridge.c:856:fuse_getattr] glusterfs-fuse: 104347: FGETATTR 134981624 (/dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP/0xb621adc0)
[2009-10-29 17:49:51] T [fuse-bridge.c:2183:fuse_readdir_cbk] glusterfs-fuse: 104346: READDIR => 7/4096,0
[2009-10-29 17:49:51] T [fuse-bridge.c:2256:fuse_releasedir] glusterfs-fuse: 104348: RELEASEDIR 0xb620aac8
[2009-10-29 17:49:51] T [fuse-bridge.c:847:fuse_getattr] glusterfs-fuse: 104349: GETATTR 3055646864 (/dbench-4.0/clients/client1/~dmtmp/WORD)
[2009-10-29 17:49:51] T [fuse-bridge.c:1993:fuse_flush] glusterfs-fuse: 104350: FLUSH 0xb621f510
[2009-10-29 17:49:51] T [fuse-bridge.c:2238:fuse_readdir] glusterfs-fuse: 104351: READDIR (0xb620ae18, size=4096, offset=1856558593)
[2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104349: STAT() /dbench-4.0/clients/client1/~dmtmp/WORD => 969438
[2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104347: FSTAT() /dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP => 975880
[2009-10-29 17:49:51] T [fuse-bridge.c:2183:fuse_readdir_cbk] glusterfs-fuse: 104351: READDIR => 0/4096,1856558593
[2009-10-29 17:49:51] T [fuse-bridge.c:1909:fuse_writev_cbk] glusterfs-fuse: 104344: WRITE => 1568/1568,116096/120756
[2009-10-29 17:49:51] T [fuse-bridge.c:635:fuse_lookup] glusterfs-fuse: 104352: LOOKUP /dbench-4.0/clients/client1/~dmtmp/WORD/HANDOUT.DOC
[2009-10-29 17:49:51] T [fuse-bridge.c:1164:fuse_err_cbk] glusterfs-fuse: 104350: FLUSH() ERR => 0
[2009-10-29 17:49:51] T [fuse-bridge.c:1051:fuse_setattr] glusterfs-fuse: 104353: SETATTR (134981624)/dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP
[2009-10-29 17:49:51] T [fuse-bridge.c:1524:fuse_rename_cbk] glusterfs-fuse: 104345: /dbench-4.0/clients/client0/~dmtmp/WORD/~WRD0000.TMP -> /dbench-4.0/clients/client0/~dmtmp/WORD/HANDOUT.DOC => 0 (buf->st_ino=975690 , loc->ino=975690)
[2009-10-29 17:49:51] T [fuse-bridge.c:2256:fuse_releasedir] glusterfs-fuse: 104354: RELEASEDIR 0xb620ae18
[2009-10-29 17:49:51] T [fuse-bridge.c:562:fuse_entry_cbk] glusterfs-fuse: 104352: LOOKUP() /dbench-4.0/clients/client1/~dmtmp/WORD/HANDOUT.DOC => -1 (No such file or directory)
[2009-10-29 17:49:51] T [fuse-bridge.c:1993:fuse_flush] glusterfs-fuse: 104355: FLUSH 0x80a30b8
[2009-10-29 17:49:51] T [fuse-bridge.c:934:fuse_setattr_cbk] glusterfs-fuse: 104353: SETATTR() /dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP => 975880
[2009-10-29 17:49:51] T [fuse-bridge.c:1885:fuse_readv] glusterfs-fuse: 104356: READ (0xb621fa00, size=131072, offset=0)
[2009-10-29 17:49:51] T [fuse-bridge.c:2051:fuse_release] glusterfs-fuse: 104357: RELEASE 0xb621f510 (FLUSH implied)
[2009-10-29 17:49:51] D [inode.c:886:inode_path] fuse/inode: no dentry for non-root inode 975752
[2009-10-29 17:49:51] D [fuse-bridge.c:445:fuse_loc_fill] glusterfs-fuse: inode_path failed for 975752
[2009-10-29 17:49:51] T [fuse-bridge.c:856:fuse_getattr] glusterfs-fuse: 104358: FGETATTR 3055661960 ((null)/0xb621f510)
[2009-10-29 17:49:51] T [fuse-bridge.c:847:fuse_getattr] glusterfs-fuse: 104359: GETATTR 3055622456 (/dbench-4.0/clients/client0/~dmtmp/WORD)
[2009-10-29 17:49:51] T [fuse-bridge.c:2120:fuse_opendir] glusterfs-fuse: 104360: OPENDIR /dbench-4.0/clients/client1/~dmtmp/WORD
[2009-10-29 17:49:51] T [fuse-bridge.c:1051:fuse_setattr] glusterfs-fuse: 104361: SETATTR (134981624)/dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP
[2009-10-29 17:49:51] T [fuse-bridge.c:1842:fuse_readv_cbk] glusterfs-fuse: 104356: READ => 120756/131072,0/120756
[2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104359: STAT() /dbench-4.0/clients/client0/~dmtmp/WORD => 969518
[2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104358: FSTAT() ERR => 975752
[2009-10-29 17:49:51] T [fuse-bridge.c:889:fuse_fd_cbk] glusterfs-fuse: 104360: OPENDIR() /dbench-4.0/clients/client1/~dmtmp/WORD => 0xb6218ed8
[2009-10-29 17:49:51] T [fuse-bridge.c:847:fuse_getattr] glusterfs-fuse: 104362: GETATTR 134935672 (/dbench-4.0/clients/client0/~dmtmp/WORD/HANDOUT.DOC)
[2009-10-29 17:49:51] T [fuse-bridge.c:1956:fuse_write] glusterfs-fuse: 104363: WRITE (0xb621fa00, size=7612, offset=87496)
[2009-10-29 17:49:51] T [fuse-bridge.c:1164:fuse_err_cbk] glusterfs-fuse: 104355: FLUSH() ERR => 0
[2009-10-29 17:49:51] D [inode.c:886:inode_path] fuse/inode: no dentry for non-root inode 975752
[2009-10-29 17:49:51] T [fuse-bridge.c:1164:fuse_err_cbk] glusterfs-fuse: 104357: FLUSH() ERR => 0
[2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104362: STAT() /dbench-4.0/clients/client0/~dmtmp/WORD/HANDOUT.DOC => 975690
[2009-10-29 17:49:51] D [fuse-bridge.c:445:fuse_loc_fill] glusterfs-fuse: inode_path failed for 975752
[2009-10-29 17:49:51] W [fuse-bridge.c:1802:fuse_open] glusterfs-fuse: 104364: OPEN (null) (fuse_loc_fill() failed)
[2009-10-29 17:49:51] T [fuse-bridge.c:2238:fuse_readdir] glusterfs-fuse: 104365: READDIR (0xb6218ed8, size=4096, offset=0)
[2009-10-29 17:49:51] T [fuse-bridge.c:2051:fuse_release] glusterfs-fuse: 104366: RELEASE 0x80a30b8 (FLUSH implied)
[2009-10-29 17:49:51] T [fuse-bridge.c:847:fuse_getattr] glusterfs-fuse: 104367: GETATTR 3055620528 (/dbench-4.0/clients/client4/~dmtmp/WORD)
[2009-10-29 17:49:51] T [fuse-bridge.c:934:fuse_setattr_cbk] glusterfs-fuse: 104361: SETATTR() /dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP => 975880
[2009-10-29 17:49:51] T [fuse-bridge.c:1815:fuse_open] glusterfs-fuse: 104368: OPEN /dbench-4.0/clients/client0/~dmtmp/WORD/HANDOUT.DOC
[2009-10-29 17:49:51] T [fuse-bridge.c:1993:fuse_flush] glusterfs-fuse: 104369: FLUSH 0xb6412c48
[2009-10-29 17:49:51] T [fuse-bridge.c:635:fuse_lookup] glusterfs-fuse: 104370: LOOKUP /dbench-4.0/clients/client3/~dmtmp/WORD/~WRL0004.TMP
[2009-10-29 17:49:51] T [fuse-bridge.c:2183:fuse_readdir_cbk] glusterfs-fuse: 104365: READDIR => 6/4096,0
[2009-10-29 17:49:51] T [fuse-bridge.c:2238:fuse_readdir] glusterfs-fuse: 104371: READDIR (0xb6218ed8, size=4096, offset=1856558593)
[2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104367: STAT() /dbench-4.0/clients/client4/~dmtmp/WORD => 969362
[2009-10-29 17:49:51] T [fuse-bridge.c:856:fuse_getattr] glusterfs-fuse: 104372: FGETATTR 134934656 (/dbench-4.0/clients/client4/~dmtmp/WORD/HANDOUT.DOC/0x80a30b8)
[2009-10-29 17:49:51] T [fuse-bridge.c:889:fuse_fd_cbk] glusterfs-fuse: 104368: OPEN() /dbench-4.0/clients/client0/~dmtmp/WORD/HANDOUT.DOC => 0x80a2208
[2009-10-29 17:49:51] T [fuse-bridge.c:1885:fuse_readv] glusterfs-fuse: 104373: READ (0x80a2208, size=4096, offset=0)
[2009-10-29 17:49:51] T [fuse-bridge.c:2183:fuse_readdir_cbk] glusterfs-fuse: 104371: READDIR => 0/4096,1856558593
[2009-10-29 17:49:51] T [fuse-bridge.c:562:fuse_entry_cbk] glusterfs-fuse: 104370: LOOKUP() /dbench-4.0/clients/client3/~dmtmp/WORD/~WRL0004.TMP => -1 (No such file or directory)
[2009-10-29 17:49:51] T [fuse-bridge.c:1164:fuse_err_cbk] glusterfs-fuse: 104366: FLUSH() ERR => 0
[2009-10-29 17:49:51] T [fuse-bridge.c:1842:fuse_readv_cbk] glusterfs-fuse: 104373: READ => 4096/4096,0/15360
glusterfs: ../../../libglusterfs/src/inode.c:352: __inode_unref: Assertion `inode->ref' failed.