Hide Forgot
Backtrace: [2009-10-29 17:49:51] T [fuse-bridge.c:1885:fuse_readv] glusterfs-fuse: 104322: READ (0xb621fa00, size=131072, offset=0) [2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104321: STAT() /dbench-4.0/clients/client1/~dmtmp/WORD => 969438 [2009-10-29 17:49:51] T [fuse-bridge.c:1051:fuse_setattr] glusterfs-fuse: 104323: SETATTR (3055672112)/dbench-4.0/clients/client4/~dmtmp/WORD/~WRD0003.TMP [2009-10-29 17:49:51] T [fuse-bridge.c:1909:fuse_writev_cbk] glusterfs-fuse: 104318: WRITE => 1/1,15359/0 [2009-10-29 17:49:51] T [fuse-bridge.c:2051:fuse_release] glusterfs-fuse: 104324: RELEASE 0x80b8a58 (FLUSH implied) [2009-10-29 17:49:51] T [fuse-bridge.c:856:fuse_getattr] glusterfs-fuse: 104325: FGETATTR 134981624 (/dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP/0x80b8a58) [2009-10-29 17:49:51] T [fuse-bridge.c:1842:fuse_readv_cbk] glusterfs-fuse: 104322: READ => 120756/131072,0/120756 [2009-10-29 17:49:51] T [fuse-bridge.c:2238:fuse_readdir] glusterfs-fuse: 104326: READDIR (0xb620ae18, size=4096, offset=0) [2009-10-29 17:49:51] T [fuse-bridge.c:635:fuse_lookup] glusterfs-fuse: 104327: LOOKUP /dbench-4.0/clients/client1/~dmtmp/WORD/HANDOUT.DOC [2009-10-29 17:49:51] D [inode.c:886:inode_path] fuse/inode: no dentry for non-root inode 975752 [2009-10-29 17:49:51] D [fuse-bridge.c:445:fuse_loc_fill] glusterfs-fuse: inode_path failed for 975752 [2009-10-29 17:49:51] T [fuse-bridge.c:856:fuse_getattr] glusterfs-fuse: 104328: FGETATTR 3055661960 ((null)/0xb621f510) [2009-10-29 17:49:51] T [fuse-bridge.c:1956:fuse_write] glusterfs-fuse: 104329: WRITE (0xb621fa00, size=32, offset=47432) [2009-10-29 17:49:51] T [fuse-bridge.c:934:fuse_setattr_cbk] glusterfs-fuse: 104323: SETATTR() /dbench-4.0/clients/client4/~dmtmp/WORD/~WRD0003.TMP => 975764 [2009-10-29 17:49:51] T [fuse-bridge.c:1051:fuse_setattr] glusterfs-fuse: 104330: SETATTR (3055672112)/dbench-4.0/clients/client4/~dmtmp/WORD/~WRD0003.TMP [2009-10-29 17:49:51] T [fuse-bridge.c:2183:fuse_readdir_cbk] glusterfs-fuse: 104326: READDIR => 7/4096,0 [2009-10-29 17:49:51] T [fuse-bridge.c:562:fuse_entry_cbk] glusterfs-fuse: 104327: LOOKUP() /dbench-4.0/clients/client1/~dmtmp/WORD/HANDOUT.DOC => -1 (No such file or directory) [2009-10-29 17:49:51] T [fuse-bridge.c:2238:fuse_readdir] glusterfs-fuse: 104331: READDIR (0xb620ae18, size=4096, offset=1856558593) [2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104325: FSTAT() /dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP => 975880 [2009-10-29 17:49:51] T [fuse-bridge.c:2120:fuse_opendir] glusterfs-fuse: 104332: OPENDIR /dbench-4.0/clients/client1/~dmtmp/WORD [2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104328: FSTAT() ERR => 975752 [2009-10-29 17:49:51] T [fuse-bridge.c:1164:fuse_err_cbk] glusterfs-fuse: 104324: FLUSH() ERR => 0 [2009-10-29 17:49:51] T [fuse-bridge.c:1815:fuse_open] glusterfs-fuse: 104333: OPEN /dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP [2009-10-29 17:49:51] T [fuse-bridge.c:2183:fuse_readdir_cbk] glusterfs-fuse: 104331: READDIR => 0/4096,1856558593 [2009-10-29 17:49:51] T [fuse-bridge.c:1956:fuse_write] glusterfs-fuse: 104334: WRITE (0xb621f510, size=15360, offset=0) [2009-10-29 17:49:51] T [fuse-bridge.c:934:fuse_setattr_cbk] glusterfs-fuse: 104330: SETATTR() /dbench-4.0/clients/client4/~dmtmp/WORD/~WRD0003.TMP => 975764 [2009-10-29 17:49:51] T [fuse-bridge.c:2256:fuse_releasedir] glusterfs-fuse: 104335: RELEASEDIR 0xb620ae18 [2009-10-29 17:49:51] T [fuse-bridge.c:847:fuse_getattr] glusterfs-fuse: 104336: GETATTR 3055622456 (/dbench-4.0/clients/client0/~dmtmp/WORD) [2009-10-29 17:49:51] T [fuse-bridge.c:889:fuse_fd_cbk] glusterfs-fuse: 104332: OPENDIR() /dbench-4.0/clients/client1/~dmtmp/WORD => 0xb620aac8 [2009-10-29 17:49:51] T [fuse-bridge.c:635:fuse_lookup] glusterfs-fuse: 104337: LOOKUP /dbench-4.0/clients/client4/~dmtmp/WORD/~WRL0004.TMP [2009-10-29 17:49:51] T [fuse-bridge.c:2238:fuse_readdir] glusterfs-fuse: 104338: READDIR (0xb620aac8, size=4096, offset=0) [2009-10-29 17:49:51] T [fuse-bridge.c:889:fuse_fd_cbk] glusterfs-fuse: 104333: OPEN() /dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP => 0xb621adc0 [2009-10-29 17:49:51] T [fuse-bridge.c:1909:fuse_writev_cbk] glusterfs-fuse: 104329: WRITE => 32/32,47432/120756 [2009-10-29 17:49:51] T [fuse-bridge.c:1885:fuse_readv] glusterfs-fuse: 104339: READ (0xb621adc0, size=4096, offset=0) [2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104336: STAT() /dbench-4.0/clients/client0/~dmtmp/WORD => 969518 [2009-10-29 17:49:51] T [fuse-bridge.c:1885:fuse_readv] glusterfs-fuse: 104340: READ (0xb621fa00, size=131072, offset=0) [2009-10-29 17:49:51] T [fuse-bridge.c:635:fuse_lookup] glusterfs-fuse: 104341: LOOKUP /dbench-4.0/clients/client0/~dmtmp/WORD/HANDOUT.DOC [2009-10-29 17:49:51] T [fuse-bridge.c:562:fuse_entry_cbk] glusterfs-fuse: 104337: LOOKUP() /dbench-4.0/clients/client4/~dmtmp/WORD/~WRL0004.TMP => -1 (No such file or directory) [2009-10-29 17:49:51] T [fuse-bridge.c:2120:fuse_opendir] glusterfs-fuse: 104342: OPENDIR /dbench-4.0/clients/client4/~dmtmp/WORD [2009-10-29 17:49:51] T [fuse-bridge.c:2183:fuse_readdir_cbk] glusterfs-fuse: 104338: READDIR => 6/4096,0 [2009-10-29 17:49:51] T [fuse-bridge.c:2238:fuse_readdir] glusterfs-fuse: 104343: READDIR (0xb620aac8, size=4096, offset=1856558593) [2009-10-29 17:49:51] T [fuse-bridge.c:1842:fuse_readv_cbk] glusterfs-fuse: 104340: READ => 120756/131072,0/120756 [2009-10-29 17:49:51] T [fuse-bridge.c:1956:fuse_write] glusterfs-fuse: 104344: WRITE (0xb621fa00, size=1568, offset=116096) [2009-10-29 17:49:51] T [fuse-bridge.c:562:fuse_entry_cbk] glusterfs-fuse: 104341: LOOKUP() /dbench-4.0/clients/client0/~dmtmp/WORD/HANDOUT.DOC => -1 (No such file or directory) [2009-10-29 17:49:51] T [fuse-bridge.c:1595:fuse_rename] glusterfs-fuse: 104345: RENAME `/dbench-4.0/clients/client0/~dmtmp/WORD/~WRD0000.TMP (975690)' -> `/dbench-4.0/clients/client0/~dmtmp/WORD/HANDOUT.DOC (0)' [2009-10-29 17:49:51] T [fuse-bridge.c:889:fuse_fd_cbk] glusterfs-fuse: 104342: OPENDIR() /dbench-4.0/clients/client4/~dmtmp/WORD => 0xb620ae18 [2009-10-29 17:49:51] T [fuse-bridge.c:1842:fuse_readv_cbk] glusterfs-fuse: 104339: READ => 4096/4096,0/15360 [2009-10-29 17:49:51] T [fuse-bridge.c:2238:fuse_readdir] glusterfs-fuse: 104346: READDIR (0xb620ae18, size=4096, offset=0) [2009-10-29 17:49:51] T [fuse-bridge.c:2183:fuse_readdir_cbk] glusterfs-fuse: 104343: READDIR => 0/4096,1856558593 [2009-10-29 17:49:51] T [fuse-bridge.c:1909:fuse_writev_cbk] glusterfs-fuse: 104334: WRITE => 15360/15360,0/15360 [2009-10-29 17:49:51] T [fuse-bridge.c:856:fuse_getattr] glusterfs-fuse: 104347: FGETATTR 134981624 (/dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP/0xb621adc0) [2009-10-29 17:49:51] T [fuse-bridge.c:2183:fuse_readdir_cbk] glusterfs-fuse: 104346: READDIR => 7/4096,0 [2009-10-29 17:49:51] T [fuse-bridge.c:2256:fuse_releasedir] glusterfs-fuse: 104348: RELEASEDIR 0xb620aac8 [2009-10-29 17:49:51] T [fuse-bridge.c:847:fuse_getattr] glusterfs-fuse: 104349: GETATTR 3055646864 (/dbench-4.0/clients/client1/~dmtmp/WORD) [2009-10-29 17:49:51] T [fuse-bridge.c:1993:fuse_flush] glusterfs-fuse: 104350: FLUSH 0xb621f510 [2009-10-29 17:49:51] T [fuse-bridge.c:2238:fuse_readdir] glusterfs-fuse: 104351: READDIR (0xb620ae18, size=4096, offset=1856558593) [2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104349: STAT() /dbench-4.0/clients/client1/~dmtmp/WORD => 969438 [2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104347: FSTAT() /dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP => 975880 [2009-10-29 17:49:51] T [fuse-bridge.c:2183:fuse_readdir_cbk] glusterfs-fuse: 104351: READDIR => 0/4096,1856558593 [2009-10-29 17:49:51] T [fuse-bridge.c:1909:fuse_writev_cbk] glusterfs-fuse: 104344: WRITE => 1568/1568,116096/120756 [2009-10-29 17:49:51] T [fuse-bridge.c:635:fuse_lookup] glusterfs-fuse: 104352: LOOKUP /dbench-4.0/clients/client1/~dmtmp/WORD/HANDOUT.DOC [2009-10-29 17:49:51] T [fuse-bridge.c:1164:fuse_err_cbk] glusterfs-fuse: 104350: FLUSH() ERR => 0 [2009-10-29 17:49:51] T [fuse-bridge.c:1051:fuse_setattr] glusterfs-fuse: 104353: SETATTR (134981624)/dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP [2009-10-29 17:49:51] T [fuse-bridge.c:1524:fuse_rename_cbk] glusterfs-fuse: 104345: /dbench-4.0/clients/client0/~dmtmp/WORD/~WRD0000.TMP -> /dbench-4.0/clients/client0/~dmtmp/WORD/HANDOUT.DOC => 0 (buf->st_ino=975690 , loc->ino=975690) [2009-10-29 17:49:51] T [fuse-bridge.c:2256:fuse_releasedir] glusterfs-fuse: 104354: RELEASEDIR 0xb620ae18 [2009-10-29 17:49:51] T [fuse-bridge.c:562:fuse_entry_cbk] glusterfs-fuse: 104352: LOOKUP() /dbench-4.0/clients/client1/~dmtmp/WORD/HANDOUT.DOC => -1 (No such file or directory) [2009-10-29 17:49:51] T [fuse-bridge.c:1993:fuse_flush] glusterfs-fuse: 104355: FLUSH 0x80a30b8 [2009-10-29 17:49:51] T [fuse-bridge.c:934:fuse_setattr_cbk] glusterfs-fuse: 104353: SETATTR() /dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP => 975880 [2009-10-29 17:49:51] T [fuse-bridge.c:1885:fuse_readv] glusterfs-fuse: 104356: READ (0xb621fa00, size=131072, offset=0) [2009-10-29 17:49:51] T [fuse-bridge.c:2051:fuse_release] glusterfs-fuse: 104357: RELEASE 0xb621f510 (FLUSH implied) [2009-10-29 17:49:51] D [inode.c:886:inode_path] fuse/inode: no dentry for non-root inode 975752 [2009-10-29 17:49:51] D [fuse-bridge.c:445:fuse_loc_fill] glusterfs-fuse: inode_path failed for 975752 [2009-10-29 17:49:51] T [fuse-bridge.c:856:fuse_getattr] glusterfs-fuse: 104358: FGETATTR 3055661960 ((null)/0xb621f510) [2009-10-29 17:49:51] T [fuse-bridge.c:847:fuse_getattr] glusterfs-fuse: 104359: GETATTR 3055622456 (/dbench-4.0/clients/client0/~dmtmp/WORD) [2009-10-29 17:49:51] T [fuse-bridge.c:2120:fuse_opendir] glusterfs-fuse: 104360: OPENDIR /dbench-4.0/clients/client1/~dmtmp/WORD [2009-10-29 17:49:51] T [fuse-bridge.c:1051:fuse_setattr] glusterfs-fuse: 104361: SETATTR (134981624)/dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP [2009-10-29 17:49:51] T [fuse-bridge.c:1842:fuse_readv_cbk] glusterfs-fuse: 104356: READ => 120756/131072,0/120756 [2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104359: STAT() /dbench-4.0/clients/client0/~dmtmp/WORD => 969518 [2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104358: FSTAT() ERR => 975752 [2009-10-29 17:49:51] T [fuse-bridge.c:889:fuse_fd_cbk] glusterfs-fuse: 104360: OPENDIR() /dbench-4.0/clients/client1/~dmtmp/WORD => 0xb6218ed8 [2009-10-29 17:49:51] T [fuse-bridge.c:847:fuse_getattr] glusterfs-fuse: 104362: GETATTR 134935672 (/dbench-4.0/clients/client0/~dmtmp/WORD/HANDOUT.DOC) [2009-10-29 17:49:51] T [fuse-bridge.c:1956:fuse_write] glusterfs-fuse: 104363: WRITE (0xb621fa00, size=7612, offset=87496) [2009-10-29 17:49:51] T [fuse-bridge.c:1164:fuse_err_cbk] glusterfs-fuse: 104355: FLUSH() ERR => 0 [2009-10-29 17:49:51] D [inode.c:886:inode_path] fuse/inode: no dentry for non-root inode 975752 [2009-10-29 17:49:51] T [fuse-bridge.c:1164:fuse_err_cbk] glusterfs-fuse: 104357: FLUSH() ERR => 0 [2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104362: STAT() /dbench-4.0/clients/client0/~dmtmp/WORD/HANDOUT.DOC => 975690 [2009-10-29 17:49:51] D [fuse-bridge.c:445:fuse_loc_fill] glusterfs-fuse: inode_path failed for 975752 [2009-10-29 17:49:51] W [fuse-bridge.c:1802:fuse_open] glusterfs-fuse: 104364: OPEN (null) (fuse_loc_fill() failed) [2009-10-29 17:49:51] T [fuse-bridge.c:2238:fuse_readdir] glusterfs-fuse: 104365: READDIR (0xb6218ed8, size=4096, offset=0) [2009-10-29 17:49:51] T [fuse-bridge.c:2051:fuse_release] glusterfs-fuse: 104366: RELEASE 0x80a30b8 (FLUSH implied) [2009-10-29 17:49:51] T [fuse-bridge.c:847:fuse_getattr] glusterfs-fuse: 104367: GETATTR 3055620528 (/dbench-4.0/clients/client4/~dmtmp/WORD) [2009-10-29 17:49:51] T [fuse-bridge.c:934:fuse_setattr_cbk] glusterfs-fuse: 104361: SETATTR() /dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP => 975880 [2009-10-29 17:49:51] T [fuse-bridge.c:1815:fuse_open] glusterfs-fuse: 104368: OPEN /dbench-4.0/clients/client0/~dmtmp/WORD/HANDOUT.DOC [2009-10-29 17:49:51] T [fuse-bridge.c:1993:fuse_flush] glusterfs-fuse: 104369: FLUSH 0xb6412c48 [2009-10-29 17:49:51] T [fuse-bridge.c:635:fuse_lookup] glusterfs-fuse: 104370: LOOKUP /dbench-4.0/clients/client3/~dmtmp/WORD/~WRL0004.TMP [2009-10-29 17:49:51] T [fuse-bridge.c:2183:fuse_readdir_cbk] glusterfs-fuse: 104365: READDIR => 6/4096,0 [2009-10-29 17:49:51] T [fuse-bridge.c:2238:fuse_readdir] glusterfs-fuse: 104371: READDIR (0xb6218ed8, size=4096, offset=1856558593) [2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104367: STAT() /dbench-4.0/clients/client4/~dmtmp/WORD => 969362 [2009-10-29 17:49:51] T [fuse-bridge.c:856:fuse_getattr] glusterfs-fuse: 104372: FGETATTR 134934656 (/dbench-4.0/clients/client4/~dmtmp/WORD/HANDOUT.DOC/0x80a30b8) [2009-10-29 17:49:51] T [fuse-bridge.c:889:fuse_fd_cbk] glusterfs-fuse: 104368: OPEN() /dbench-4.0/clients/client0/~dmtmp/WORD/HANDOUT.DOC => 0x80a2208 [2009-10-29 17:49:51] T [fuse-bridge.c:1885:fuse_readv] glusterfs-fuse: 104373: READ (0x80a2208, size=4096, offset=0) [2009-10-29 17:49:51] T [fuse-bridge.c:2183:fuse_readdir_cbk] glusterfs-fuse: 104371: READDIR => 0/4096,1856558593 [2009-10-29 17:49:51] T [fuse-bridge.c:562:fuse_entry_cbk] glusterfs-fuse: 104370: LOOKUP() /dbench-4.0/clients/client3/~dmtmp/WORD/~WRL0004.TMP => -1 (No such file or directory) [2009-10-29 17:49:51] T [fuse-bridge.c:1164:fuse_err_cbk] glusterfs-fuse: 104366: FLUSH() ERR => 0 [2009-10-29 17:49:51] T [fuse-bridge.c:1842:fuse_readv_cbk] glusterfs-fuse: 104373: READ => 4096/4096,0/15360 glusterfs: ../../../libglusterfs/src/inode.c:352: __inode_unref: Assertion `inode->ref' failed. Program received signal SIGABRT, Aborted. 0xb7fe3410 in __kernel_vsyscall () (gdb) bt #0 0xb7fe3410 in __kernel_vsyscall () #1 0xb7e40085 in raise () from /lib/tls/i686/cmov/libc.so.6 #2 0xb7e41a01 in abort () from /lib/tls/i686/cmov/libc.so.6 #3 0xb7e3910e in __assert_fail () from /lib/tls/i686/cmov/libc.so.6 #4 0xb7fa5908 in __inode_unref (inode=0xb621b388) at ../../../libglusterfs/src/inode.c:352 #5 0xb7fa59d9 in inode_unref (inode=0xb621b388) at ../../../libglusterfs/src/inode.c:391 #6 0xb7fb6423 in fd_destroy (fd=0xb621f510) at ../../../libglusterfs/src/fd.c:422 #7 0xb7fb6500 in fd_unref (fd=0xb621f510) at ../../../libglusterfs/src/fd.c:448 #8 0xb7571b38 in afr_local_cleanup (local=0xb6221a48, this=0x8056940) at ../../../../../xlators/cluster/afr/src/afr.c:269 #9 0xb75740f8 in afr_flush_done (frame=0xb621f408, this=0x8056940) at ../../../../../xlators/cluster/afr/src/afr.c:1043 #10 0xb758d9ce in afr_unlock_common_cbk (frame=0xb621f408, cookie=0xb621def0, this=0x8056940, op_ret=0, op_errno=0) at ../../../../../xlators/cluster/afr/src/afr-transaction.c:398 #11 0xb75c8806 in client_finodelk_cbk (frame=0xb621def0, hdr=0xb620f400, hdrlen=32, iobuf=0x0) at ../../../../../xlators/protocol/client/src/client-protocol.c:5165 #12 0xb75ca7e3 in protocol_client_interpret (this=0x8051cf0, trans=0x808f270, hdr_p=0xb620f400 "", hdrlen=32, iobuf=0x0) at ../../../../../xlators/protocol/client/src/client-protocol.c:5969 #13 0xb75cb571 in protocol_client_pollin (this=0x8051cf0, trans=0x808f270) at ../../../../../xlators/protocol/client/src/client-protocol.c:6263 #14 0xb75cbbc0 in notify (this=0x8051cf0, event=2, data=0x808f270) at ../../../../../xlators/protocol/client/src/client-protocol.c:6382 #15 0xb7f97f82 in xlator_notify (xl=0x8051cf0, event=2, data=0x808f270) at ../../../libglusterfs/src/xlator.c:923 #16 0xb6d63663 in socket_event_poll_in (this=0x808f270) at ../../../../transport/socket/src/socket.c:713 #17 0xb6d63968 in socket_event_handler (fd=9, idx=0, data=0x808f270, poll_in=1, poll_out=0, poll_err=0) at ../../../../transport/socket/src/socket.c:813 #18 0xb7fb918c in event_dispatch_epoll_handler (event_pool=0x80511d8, events=0x8090e90, i=0) at ../../../libglusterfs/src/event.c:804 #19 0xb7fb935d in event_dispatch_epoll (event_pool=0x80511d8) at ../../../libglusterfs/src/event.c:867 #20 0xb7fb96b8 in event_dispatch (event_pool=0x80511d8) at ../../../libglusterfs/src/event.c:975 #21 0x0804dd2a in main (argc=7, argv=0xbffff864) at ../../../glusterfsd/src/glusterfsd.c:1379
Additional info: (gdb) p *fd $36 = {pid = 32037, flags = 66, refcount = 0, flush_unique = 104350, inode_list = {next = 0xb621f524, prev = 0xb621f524}, inode = 0xb621b388, lock = 1, _ctx = 0xeeeeeeee} (gdb) p *fd->inode $38 = {table = 0x8057258, lock = 1, nlookup = 1, generation = 397994504831633667, in_attic = 0, ref = 0, ino = 975752, st_mode = 33152, fd_list = {next = 0xb621b3b4, prev = 0xb621b3b4}, dentry_list = {next = 0xb621b3bc, prev = 0xb621b3bc}, hash = {next = 0xb621b3c4, prev = 0xb621b3c4}, list = {next = 0x8057298, prev = 0xb621d8b4}, _ctx = 0xb6214df0}
PATCH: http://patches.gluster.com/patch/2068 in master (protocol/server, mount/fuse: Handle returned inode in create properly)
Running dbench on an afr (2 server) configuration results in assertion failed: inode->ref Relevant portion of the log: The guilty inode is 975752, corresponding to fd=0xb621f510 [2009-10-29 17:49:51] T [fuse-bridge.c:1675:fuse_create_cbk] glusterfs-fuse: 104309: CREATE() /dbench-4.0/clients/client2/~dmtmp/WORD/~WRD0003.TMP => 0xb621f510 (ino=975752) [2009-10-29 17:49:51] T [fuse-bridge.c:1956:fuse_write] glusterfs-fuse: 104318: WRITE (0xb621f510, size=1, offset=15359) [2009-10-29 17:49:51] T [fuse-bridge.c:1164:fuse_err_cbk] glusterfs-fuse: 104279: FLUSH() ERR => 0 [2009-10-29 17:49:51] T [fuse-bridge.c:562:fuse_entry_cbk] glusterfs-fuse: 104315: LOOKUP() /dbench-4.0/clients/client0/~dmtmp/WORD/HANDOUT.DOC => -1 (No such file or directory) [2009-10-29 17:49:51] T [fuse-bridge.c:2120:fuse_opendir] glusterfs-fuse: 104319: OPENDIR /dbench-4.0/clients/client0/~dmtmp/WORD [2009-10-29 17:49:51] T [fuse-bridge.c:1164:fuse_err_cbk] glusterfs-fuse: 104286: FLUSH() ERR => 0 [2009-10-29 17:49:51] T [fuse-bridge.c:1223:fuse_unlink_cbk] glusterfs-fuse: 104287: UNLINK() /dbench-4.0/clients/client1/~dmtmp/WORD/HANDOUT.DOC => 0 [2009-10-29 17:49:51] T [fuse-bridge.c:1164:fuse_err_cbk] glusterfs-fuse: 104314: FLUSH() ERR => 0 [2009-10-29 17:49:51] T [fuse-bridge.c:1909:fuse_writev_cbk] glusterfs-fuse: 104313: WRITE => 1531/1531,47504/120756 [2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104317: FSTAT() /dbench-4.0/clients/client4/~dmtmp/WORD/~WRD0003.TMP => 975764 [2009-10-29 17:49:51] T [fuse-bridge.c:1164:fuse_err_cbk] glusterfs-fuse: 104316: FLUSH() ERR => 0 [2009-10-29 17:49:51] T [fuse-bridge.c:889:fuse_fd_cbk] glusterfs-fuse: 104319: OPENDIR() /dbench-4.0/clients/client0/~dmtmp/WORD => 0xb620ae18 [2009-10-29 17:49:51] T [fuse-bridge.c:847:fuse_getattr] glusterfs-fuse: 104321: GETATTR 3055646864 (/dbench-4.0/clients/client1/~dmtmp/WORD) [2009-10-29 17:49:51] T [fuse-bridge.c:1885:fuse_readv] glusterfs-fuse: 104322: READ (0xb621fa00, size=131072, offset=0) [2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104321: STAT() /dbench-4.0/clients/client1/~dmtmp/WORD => 969438 [2009-10-29 17:49:51] T [fuse-bridge.c:1051:fuse_setattr] glusterfs-fuse: 104323: SETATTR (3055672112)/dbench-4.0/clients/client4/~dmtmp/WORD/~WRD0003.TMP [2009-10-29 17:49:51] T [fuse-bridge.c:1909:fuse_writev_cbk] glusterfs-fuse: 104318: WRITE => 1/1,15359/0 [2009-10-29 17:49:51] T [fuse-bridge.c:2051:fuse_release] glusterfs-fuse: 104324: RELEASE 0x80b8a58 (FLUSH implied) [2009-10-29 17:49:51] T [fuse-bridge.c:856:fuse_getattr] glusterfs-fuse: 104325: FGETATTR 134981624 (/dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP/0x80b8a58) [2009-10-29 17:49:51] T [fuse-bridge.c:1842:fuse_readv_cbk] glusterfs-fuse: 104322: READ => 120756/131072,0/120756 [2009-10-29 17:49:51] T [fuse-bridge.c:2238:fuse_readdir] glusterfs-fuse: 104326: READDIR (0xb620ae18, size=4096, offset=0) [2009-10-29 17:49:51] T [fuse-bridge.c:635:fuse_lookup] glusterfs-fuse: 104327: LOOKUP /dbench-4.0/clients/client1/~dmtmp/WORD/HANDOUT.DOC [2009-10-29 17:49:51] D [inode.c:886:inode_path] fuse/inode: no dentry for non-root inode 975752 [2009-10-29 17:49:51] D [fuse-bridge.c:445:fuse_loc_fill] glusterfs-fuse: inode_path failed for 975752 [2009-10-29 17:49:51] T [fuse-bridge.c:856:fuse_getattr] glusterfs-fuse: 104328: FGETATTR 3055661960 ((null)/0xb621f510) [2009-10-29 17:49:51] T [fuse-bridge.c:1956:fuse_write] glusterfs-fuse: 104329: WRITE (0xb621fa00, size=32, offset=47432) [2009-10-29 17:49:51] T [fuse-bridge.c:934:fuse_setattr_cbk] glusterfs-fuse: 104323: SETATTR() /dbench-4.0/clients/client4/~dmtmp/WORD/~WRD0003.TMP => 975764 [2009-10-29 17:49:51] T [fuse-bridge.c:1051:fuse_setattr] glusterfs-fuse: 104330: SETATTR (3055672112)/dbench-4.0/clients/client4/~dmtmp/WORD/~WRD0003.TMP [2009-10-29 17:49:51] T [fuse-bridge.c:2183:fuse_readdir_cbk] glusterfs-fuse: 104326: READDIR => 7/4096,0 [2009-10-29 17:49:51] T [fuse-bridge.c:562:fuse_entry_cbk] glusterfs-fuse: 104327: LOOKUP() /dbench-4.0/clients/client1/~dmtmp/WORD/HANDOUT.DOC => -1 (No such file or directory) [2009-10-29 17:49:51] T [fuse-bridge.c:2238:fuse_readdir] glusterfs-fuse: 104331: READDIR (0xb620ae18, size=4096, offset=1856558593) [2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104325: FSTAT() /dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP => 975880 [2009-10-29 17:49:51] T [fuse-bridge.c:2120:fuse_opendir] glusterfs-fuse: 104332: OPENDIR /dbench-4.0/clients/client1/~dmtmp/WORD [2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104328: FSTAT() ERR => 975752 [2009-10-29 17:49:51] T [fuse-bridge.c:1164:fuse_err_cbk] glusterfs-fuse: 104324: FLUSH() ERR => 0 [2009-10-29 17:49:51] T [fuse-bridge.c:1815:fuse_open] glusterfs-fuse: 104333: OPEN /dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP [2009-10-29 17:49:51] T [fuse-bridge.c:2183:fuse_readdir_cbk] glusterfs-fuse: 104331: READDIR => 0/4096,1856558593 [2009-10-29 17:49:51] T [fuse-bridge.c:1956:fuse_write] glusterfs-fuse: 104334: WRITE (0xb621f510, size=15360, offset=0) [2009-10-29 17:49:51] T [fuse-bridge.c:934:fuse_setattr_cbk] glusterfs-fuse: 104330: SETATTR() /dbench-4.0/clients/client4/~dmtmp/WORD/~WRD0003.TMP => 975764 [2009-10-29 17:49:51] T [fuse-bridge.c:2256:fuse_releasedir] glusterfs-fuse: 104335: RELEASEDIR 0xb620ae18 [2009-10-29 17:49:51] T [fuse-bridge.c:847:fuse_getattr] glusterfs-fuse: 104336: GETATTR 3055622456 (/dbench-4.0/clients/client0/~dmtmp/WORD) [2009-10-29 17:49:51] T [fuse-bridge.c:889:fuse_fd_cbk] glusterfs-fuse: 104332: OPENDIR() /dbench-4.0/clients/client1/~dmtmp/WORD => 0xb620aac8 [2009-10-29 17:49:51] T [fuse-bridge.c:635:fuse_lookup] glusterfs-fuse: 104337: LOOKUP /dbench-4.0/clients/client4/~dmtmp/WORD/~WRL0004.TMP [2009-10-29 17:49:51] T [fuse-bridge.c:2238:fuse_readdir] glusterfs-fuse: 104338: READDIR (0xb620aac8, size=4096, offset=0) [2009-10-29 17:49:51] T [fuse-bridge.c:889:fuse_fd_cbk] glusterfs-fuse: 104333: OPEN() /dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP => 0xb621adc0 [2009-10-29 17:49:51] T [fuse-bridge.c:1909:fuse_writev_cbk] glusterfs-fuse: 104329: WRITE => 32/32,47432/120756 [2009-10-29 17:49:51] T [fuse-bridge.c:1885:fuse_readv] glusterfs-fuse: 104339: READ (0xb621adc0, size=4096, offset=0) [2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104336: STAT() /dbench-4.0/clients/client0/~dmtmp/WORD => 969518 [2009-10-29 17:49:51] T [fuse-bridge.c:1885:fuse_readv] glusterfs-fuse: 104340: READ (0xb621fa00, size=131072, offset=0) [2009-10-29 17:49:51] T [fuse-bridge.c:635:fuse_lookup] glusterfs-fuse: 104341: LOOKUP /dbench-4.0/clients/client0/~dmtmp/WORD/HANDOUT.DOC [2009-10-29 17:49:51] T [fuse-bridge.c:562:fuse_entry_cbk] glusterfs-fuse: 104337: LOOKUP() /dbench-4.0/clients/client4/~dmtmp/WORD/~WRL0004.TMP => -1 (No such file or directory) [2009-10-29 17:49:51] T [fuse-bridge.c:2120:fuse_opendir] glusterfs-fuse: 104342: OPENDIR /dbench-4.0/clients/client4/~dmtmp/WORD [2009-10-29 17:49:51] T [fuse-bridge.c:2183:fuse_readdir_cbk] glusterfs-fuse: 104338: READDIR => 6/4096,0 [2009-10-29 17:49:51] T [fuse-bridge.c:2238:fuse_readdir] glusterfs-fuse: 104343: READDIR (0xb620aac8, size=4096, offset=1856558593) [2009-10-29 17:49:51] T [fuse-bridge.c:1842:fuse_readv_cbk] glusterfs-fuse: 104340: READ => 120756/131072,0/120756 [2009-10-29 17:49:51] T [fuse-bridge.c:1956:fuse_write] glusterfs-fuse: 104344: WRITE (0xb621fa00, size=1568, offset=116096) [2009-10-29 17:49:51] T [fuse-bridge.c:562:fuse_entry_cbk] glusterfs-fuse: 104341: LOOKUP() /dbench-4.0/clients/client0/~dmtmp/WORD/HANDOUT.DOC => -1 (No such file or directory) [2009-10-29 17:49:51] T [fuse-bridge.c:1595:fuse_rename] glusterfs-fuse: 104345: RENAME `/dbench-4.0/clients/client0/~dmtmp/WORD/~WRD0000.TMP (975690)' -> `/dbench-4.0/clients/client0/~dmtmp/WORD/HANDOUT.DOC (0)' [2009-10-29 17:49:51] T [fuse-bridge.c:889:fuse_fd_cbk] glusterfs-fuse: 104342: OPENDIR() /dbench-4.0/clients/client4/~dmtmp/WORD => 0xb620ae18 [2009-10-29 17:49:51] T [fuse-bridge.c:1842:fuse_readv_cbk] glusterfs-fuse: 104339: READ => 4096/4096,0/15360 [2009-10-29 17:49:51] T [fuse-bridge.c:2238:fuse_readdir] glusterfs-fuse: 104346: READDIR (0xb620ae18, size=4096, offset=0) [2009-10-29 17:49:51] T [fuse-bridge.c:2183:fuse_readdir_cbk] glusterfs-fuse: 104343: READDIR => 0/4096,1856558593 [2009-10-29 17:49:51] T [fuse-bridge.c:1909:fuse_writev_cbk] glusterfs-fuse: 104334: WRITE => 15360/15360,0/15360 [2009-10-29 17:49:51] T [fuse-bridge.c:856:fuse_getattr] glusterfs-fuse: 104347: FGETATTR 134981624 (/dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP/0xb621adc0) [2009-10-29 17:49:51] T [fuse-bridge.c:2183:fuse_readdir_cbk] glusterfs-fuse: 104346: READDIR => 7/4096,0 [2009-10-29 17:49:51] T [fuse-bridge.c:2256:fuse_releasedir] glusterfs-fuse: 104348: RELEASEDIR 0xb620aac8 [2009-10-29 17:49:51] T [fuse-bridge.c:847:fuse_getattr] glusterfs-fuse: 104349: GETATTR 3055646864 (/dbench-4.0/clients/client1/~dmtmp/WORD) [2009-10-29 17:49:51] T [fuse-bridge.c:1993:fuse_flush] glusterfs-fuse: 104350: FLUSH 0xb621f510 [2009-10-29 17:49:51] T [fuse-bridge.c:2238:fuse_readdir] glusterfs-fuse: 104351: READDIR (0xb620ae18, size=4096, offset=1856558593) [2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104349: STAT() /dbench-4.0/clients/client1/~dmtmp/WORD => 969438 [2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104347: FSTAT() /dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP => 975880 [2009-10-29 17:49:51] T [fuse-bridge.c:2183:fuse_readdir_cbk] glusterfs-fuse: 104351: READDIR => 0/4096,1856558593 [2009-10-29 17:49:51] T [fuse-bridge.c:1909:fuse_writev_cbk] glusterfs-fuse: 104344: WRITE => 1568/1568,116096/120756 [2009-10-29 17:49:51] T [fuse-bridge.c:635:fuse_lookup] glusterfs-fuse: 104352: LOOKUP /dbench-4.0/clients/client1/~dmtmp/WORD/HANDOUT.DOC [2009-10-29 17:49:51] T [fuse-bridge.c:1164:fuse_err_cbk] glusterfs-fuse: 104350: FLUSH() ERR => 0 [2009-10-29 17:49:51] T [fuse-bridge.c:1051:fuse_setattr] glusterfs-fuse: 104353: SETATTR (134981624)/dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP [2009-10-29 17:49:51] T [fuse-bridge.c:1524:fuse_rename_cbk] glusterfs-fuse: 104345: /dbench-4.0/clients/client0/~dmtmp/WORD/~WRD0000.TMP -> /dbench-4.0/clients/client0/~dmtmp/WORD/HANDOUT.DOC => 0 (buf->st_ino=975690 , loc->ino=975690) [2009-10-29 17:49:51] T [fuse-bridge.c:2256:fuse_releasedir] glusterfs-fuse: 104354: RELEASEDIR 0xb620ae18 [2009-10-29 17:49:51] T [fuse-bridge.c:562:fuse_entry_cbk] glusterfs-fuse: 104352: LOOKUP() /dbench-4.0/clients/client1/~dmtmp/WORD/HANDOUT.DOC => -1 (No such file or directory) [2009-10-29 17:49:51] T [fuse-bridge.c:1993:fuse_flush] glusterfs-fuse: 104355: FLUSH 0x80a30b8 [2009-10-29 17:49:51] T [fuse-bridge.c:934:fuse_setattr_cbk] glusterfs-fuse: 104353: SETATTR() /dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP => 975880 [2009-10-29 17:49:51] T [fuse-bridge.c:1885:fuse_readv] glusterfs-fuse: 104356: READ (0xb621fa00, size=131072, offset=0) [2009-10-29 17:49:51] T [fuse-bridge.c:2051:fuse_release] glusterfs-fuse: 104357: RELEASE 0xb621f510 (FLUSH implied) [2009-10-29 17:49:51] D [inode.c:886:inode_path] fuse/inode: no dentry for non-root inode 975752 [2009-10-29 17:49:51] D [fuse-bridge.c:445:fuse_loc_fill] glusterfs-fuse: inode_path failed for 975752 [2009-10-29 17:49:51] T [fuse-bridge.c:856:fuse_getattr] glusterfs-fuse: 104358: FGETATTR 3055661960 ((null)/0xb621f510) [2009-10-29 17:49:51] T [fuse-bridge.c:847:fuse_getattr] glusterfs-fuse: 104359: GETATTR 3055622456 (/dbench-4.0/clients/client0/~dmtmp/WORD) [2009-10-29 17:49:51] T [fuse-bridge.c:2120:fuse_opendir] glusterfs-fuse: 104360: OPENDIR /dbench-4.0/clients/client1/~dmtmp/WORD [2009-10-29 17:49:51] T [fuse-bridge.c:1051:fuse_setattr] glusterfs-fuse: 104361: SETATTR (134981624)/dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP [2009-10-29 17:49:51] T [fuse-bridge.c:1842:fuse_readv_cbk] glusterfs-fuse: 104356: READ => 120756/131072,0/120756 [2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104359: STAT() /dbench-4.0/clients/client0/~dmtmp/WORD => 969518 [2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104358: FSTAT() ERR => 975752 [2009-10-29 17:49:51] T [fuse-bridge.c:889:fuse_fd_cbk] glusterfs-fuse: 104360: OPENDIR() /dbench-4.0/clients/client1/~dmtmp/WORD => 0xb6218ed8 [2009-10-29 17:49:51] T [fuse-bridge.c:847:fuse_getattr] glusterfs-fuse: 104362: GETATTR 134935672 (/dbench-4.0/clients/client0/~dmtmp/WORD/HANDOUT.DOC) [2009-10-29 17:49:51] T [fuse-bridge.c:1956:fuse_write] glusterfs-fuse: 104363: WRITE (0xb621fa00, size=7612, offset=87496) [2009-10-29 17:49:51] T [fuse-bridge.c:1164:fuse_err_cbk] glusterfs-fuse: 104355: FLUSH() ERR => 0 [2009-10-29 17:49:51] D [inode.c:886:inode_path] fuse/inode: no dentry for non-root inode 975752 [2009-10-29 17:49:51] T [fuse-bridge.c:1164:fuse_err_cbk] glusterfs-fuse: 104357: FLUSH() ERR => 0 [2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104362: STAT() /dbench-4.0/clients/client0/~dmtmp/WORD/HANDOUT.DOC => 975690 [2009-10-29 17:49:51] D [fuse-bridge.c:445:fuse_loc_fill] glusterfs-fuse: inode_path failed for 975752 [2009-10-29 17:49:51] W [fuse-bridge.c:1802:fuse_open] glusterfs-fuse: 104364: OPEN (null) (fuse_loc_fill() failed) [2009-10-29 17:49:51] T [fuse-bridge.c:2238:fuse_readdir] glusterfs-fuse: 104365: READDIR (0xb6218ed8, size=4096, offset=0) [2009-10-29 17:49:51] T [fuse-bridge.c:2051:fuse_release] glusterfs-fuse: 104366: RELEASE 0x80a30b8 (FLUSH implied) [2009-10-29 17:49:51] T [fuse-bridge.c:847:fuse_getattr] glusterfs-fuse: 104367: GETATTR 3055620528 (/dbench-4.0/clients/client4/~dmtmp/WORD) [2009-10-29 17:49:51] T [fuse-bridge.c:934:fuse_setattr_cbk] glusterfs-fuse: 104361: SETATTR() /dbench-4.0/clients/client3/~dmtmp/WORD/~WRD0003.TMP => 975880 [2009-10-29 17:49:51] T [fuse-bridge.c:1815:fuse_open] glusterfs-fuse: 104368: OPEN /dbench-4.0/clients/client0/~dmtmp/WORD/HANDOUT.DOC [2009-10-29 17:49:51] T [fuse-bridge.c:1993:fuse_flush] glusterfs-fuse: 104369: FLUSH 0xb6412c48 [2009-10-29 17:49:51] T [fuse-bridge.c:635:fuse_lookup] glusterfs-fuse: 104370: LOOKUP /dbench-4.0/clients/client3/~dmtmp/WORD/~WRL0004.TMP [2009-10-29 17:49:51] T [fuse-bridge.c:2183:fuse_readdir_cbk] glusterfs-fuse: 104365: READDIR => 6/4096,0 [2009-10-29 17:49:51] T [fuse-bridge.c:2238:fuse_readdir] glusterfs-fuse: 104371: READDIR (0xb6218ed8, size=4096, offset=1856558593) [2009-10-29 17:49:51] T [fuse-bridge.c:746:fuse_attr_cbk] glusterfs-fuse: 104367: STAT() /dbench-4.0/clients/client4/~dmtmp/WORD => 969362 [2009-10-29 17:49:51] T [fuse-bridge.c:856:fuse_getattr] glusterfs-fuse: 104372: FGETATTR 134934656 (/dbench-4.0/clients/client4/~dmtmp/WORD/HANDOUT.DOC/0x80a30b8) [2009-10-29 17:49:51] T [fuse-bridge.c:889:fuse_fd_cbk] glusterfs-fuse: 104368: OPEN() /dbench-4.0/clients/client0/~dmtmp/WORD/HANDOUT.DOC => 0x80a2208 [2009-10-29 17:49:51] T [fuse-bridge.c:1885:fuse_readv] glusterfs-fuse: 104373: READ (0x80a2208, size=4096, offset=0) [2009-10-29 17:49:51] T [fuse-bridge.c:2183:fuse_readdir_cbk] glusterfs-fuse: 104371: READDIR => 0/4096,1856558593 [2009-10-29 17:49:51] T [fuse-bridge.c:562:fuse_entry_cbk] glusterfs-fuse: 104370: LOOKUP() /dbench-4.0/clients/client3/~dmtmp/WORD/~WRL0004.TMP => -1 (No such file or directory) [2009-10-29 17:49:51] T [fuse-bridge.c:1164:fuse_err_cbk] glusterfs-fuse: 104366: FLUSH() ERR => 0 [2009-10-29 17:49:51] T [fuse-bridge.c:1842:fuse_readv_cbk] glusterfs-fuse: 104373: READ => 4096/4096,0/15360 glusterfs: ../../../libglusterfs/src/inode.c:352: __inode_unref: Assertion `inode->ref' failed.