Bug 762594 (GLUSTER-862)

Summary: Crash in dht_readv_cbk with error-gen
Product: [Community] GlusterFS Reporter: Anush Shetty <anush>
Component: distributeAssignee: Anand Avati <aavati>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: low Docs Contact:
Priority: low    
Version: mainlineCC: chrisw, gluster-bugs
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Anand Avati 2010-04-27 10:31:57 UTC
PATCH: http://patches.gluster.com/patch/3172 in master (cluster/dht: perform inode number overwrite in readv_cbk iatt with checks)

Comment 1 Anush Shetty 2010-04-27 12:36:50 UTC
Observed this crash while running  - for i in `seq 10`; do rm -rfv linux-2.6.34-rc5 & tar jxvf linux-2.6.34-rc5.tar.bz2; done on a Distribute over 4 subvolumes. One of the Distribute subvolumes was error-gen


(gdb) bt
#0  0x00007fd78cc30594 in dht_readv_cbk (frame=0x7fd78401b9d8, cookie=0x7fd78401b4f8, this=0x171c920, op_ret=-1, op_errno=9, vector=0x0, count=0, 
    stbuf=0x0, iobref=0x0) at dht-common.c:1753
#1  0x00007fd78ce52dca in error_gen_readv (frame=0x7fd78401b4f8, this=0x1719840, fd=0x172bc18, size=131072, offset=1048576) at error-gen.c:1070
#2  0x00007fd78cc30b28 in dht_readv (frame=0x7fd78401b9d8, this=0x171c920, fd=0x172bc18, size=131072, off=1048576) at dht-common.c:1790
#3  0x00007fd78ca05a56 in fuse_readv (this=0x171db58, finh=0x7fd78401b908, msg=0x7fd78401b930) at fuse-bridge.c:1888
#4  0x00007fd78ca0c1d2 in fuse_thread_proc (data=0x171db58) at fuse-bridge.c:3188
#5  0x00007fd78de02a04 in start_thread (arg=<value optimized out>) at pthread_create.c:300
#6  0x00007fd78db6c80d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Client log:

[2010-04-27 17:56:24] T [fuse-bridge.c:1039:fuse_setattr] glusterfs-fuse: 8833: SETATTR (140563609397336)/linux-2.6.34-rc5/Documentation/blockdev/nbd.txt
[2010-04-27 17:56:24] T [fuse-bridge.c:908:fuse_setattr_cbk] glusterfs-fuse: 8833: SETATTR() /linux-2.6.34-rc5/Documentation/blockdev/nbd.txt => 21300953
[2010-04-27 17:56:24] T [fuse-bridge.c:598:fuse_lookup] glusterfs-fuse: 8834: LOOKUP /linux-2.6.34-rc5/Documentation/blockdev/paride.txt
[2010-04-27 17:56:24] T [fuse-bridge.c:525:fuse_entry_cbk] glusterfs-fuse: 8834: LOOKUP() /linux-2.6.34-rc5/Documentation/blockdev/paride.txt => -1 (No such file or directory)
[2010-04-27 17:56:24] T [fuse-bridge.c:1769:fuse_create] glusterfs-fuse: 8835: CREATE /linux-2.6.34-rc5/Documentation/blockdev/paride.txt
[2010-04-27 17:56:24] T [dht-common.c:3048:dht_create] dist: creating /linux-2.6.34-rc5/Documentation/blockdev/paride.txt on client1
[2010-04-27 17:56:24] T [fuse-bridge.c:1661:fuse_create_cbk] glusterfs-fuse: 8835: CREATE() /linux-2.6.34-rc5/Documentation/blockdev/paride.txt => 0x7fd784003418 (ino=20381740)
[2010-04-27 17:56:24] T [fuse-bridge.c:2667:fuse_getxattr] glusterfs-fuse: 8836: GETXATTR /linux-2.6.34-rc5/Documentation/blockdev/paride.txt/140563609395528 (security.capability)
[2010-04-27 17:56:24] T [fuse-bridge.c:2527:fuse_xattr_cbk] glusterfs-fuse: 8836: GETXATTR() /linux-2.6.34-rc5/Documentation/blockdev/paride.txt => 0
[2010-04-27 17:56:24] T [fuse-bridge.c:1962:fuse_write] glusterfs-fuse: 8837: WRITE (0x7fd784003418, size=6144, offset=0)
[2010-04-27 17:56:24] T [fuse-bridge.c:1910:fuse_writev_cbk] glusterfs-fuse: 8837: WRITE => 6144/6144,0/0
[2010-04-27 17:56:24] T [fuse-bridge.c:525:fuse_entry_cbk] glusterfs-fuse: 8832: LOOKUP() /linux-2.6.34-rc5/drivers/net/igbvf/jazzsonic.c => -1 (No such file or directory)
[2010-04-27 17:56:24] T [fuse-bridge.c:2667:fuse_getxattr] glusterfs-fuse: 8838: GETXATTR /linux-2.6.34-rc5/Documentation/blockdev/paride.txt/140563609395528 (security.capability)
[2010-04-27 17:56:24] T [fuse-bridge.c:2527:fuse_xattr_cbk] glusterfs-fuse: 8838: GETXATTR() /linux-2.6.34-rc5/Documentation/blockdev/paride.txt => 0
[2010-04-27 17:56:24] T [fuse-bridge.c:1962:fuse_write] glusterfs-fuse: 8839: WRITE (0x7fd784003418, size=10240, offset=6144)
[2010-04-27 17:56:24] T [fuse-bridge.c:1910:fuse_writev_cbk] glusterfs-fuse: 8839: WRITE => 10240/10240,6144/6144
[2010-04-27 17:56:24] T [fuse-bridge.c:2667:fuse_getxattr] glusterfs-fuse: 8840: GETXATTR /linux-2.6.34-rc5/Documentation/blockdev/paride.txt/140563609395528 (security.capability)
[2010-04-27 17:56:24] T [fuse-bridge.c:2527:fuse_xattr_cbk] glusterfs-fuse: 8840: GETXATTR() /linux-2.6.34-rc5/Documentation/blockdev/paride.txt => 0
[2010-04-27 17:56:24] T [fuse-bridge.c:1962:fuse_write] glusterfs-fuse: 8841: WRITE (0x7fd784003418, size=682, offset=16384)
[2010-04-27 17:56:25] T [fuse-bridge.c:1886:fuse_readv] glusterfs-fuse: 8842: READ (0x172bc18, size=131072, offset=1048576)
[2010-04-27 17:56:25] E [error-gen.c:1069:error_gen_readv] eg: unwind(-1, Bad file descriptor)
pending frames:
frame : type(1) op(READ)
frame : type(1) op(WRITE)

patchset: git://git.sv.gnu.org/gluster.git
signal received: 11
time of crash: 2010-04-27 17:56:25
configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.0.0git
/lib/libc.so.6[0x7fd78dac0530]
/gluster/mainline/lib/glusterfs/3.0.0git/xlator/cluster/distribute.so(dht_readv_cbk+0x55)[0x7fd78cc30594]
/gluster/mainline/lib/glusterfs/3.0.0git/xlator/debug/error-gen.so(error_gen_readv+0x1ab)[0x7fd78ce52dca]
/gluster/mainline/lib/glusterfs/3.0.0git/xlator/cluster/distribute.so(dht_readv+0x444)[0x7fd78cc30b28]
/gluster/mainline/lib/glusterfs/3.0.0git/xlator/mount/fuse.so[0x7fd78ca05a56]
/gluster/mainline/lib/glusterfs/3.0.0git/xlator/mount/fuse.so[0x7fd78ca0c1d2]
/lib/libpthread.so.0[0x7fd78de02a04]
/lib/libc.so.6(clone+0x6d)[0x7fd78db6c80d]
---------