Hide Forgot
I've looked at the code and this is why it crashes, I think: In dht_readlink_cbk dht_local_t *local = NULL; local = frame->local; if (local) { sbuf->st_ino = local->st_ino; } else { op_ret = -1; op_errno = EINVAL; } It is trying to de-reference sbuf when sbuf is NULL because the operation itself failed. See the op_ret = -1 in the back trace. Fix is on the way.
The client crashed in readlink operation. This is the backtrace... #0 0x00002b8f00c3d12a in dht_readlink_cbk (frame=0x2aaab60fbc90, cookie=0x2aaab61c6280, this=<value optimized out>, op_ret=-1, op_errno=12, path=0x0, sbuf=0x0) at dht-common.c:1356 1356 dht-common.c: No such file or directory. in dht-common.c (gdb) bt #0 0x00002b8f00c3d12a in dht_readlink_cbk (frame=0x2aaab60fbc90, cookie=0x2aaab61c6280, this=<value optimized out>, op_ret=-1, op_errno=12, path=0x0, sbuf=0x0) at dht-common.c:1356 #1 0x00002b8f00a0b9a3 in afr_readlink (frame=0x2aaab61c6280, this=<value optimized out>, loc=<value optimized out>, size=4096) at afr-inode-read.c:543 #2 0x00002b8f00c418bd in dht_readlink (frame=0x2aaab60fbc90, this=<value optimized out>, loc=0x2aaab61c4858, size=4096) at dht-common.c:1400 #3 0x00002b8effd1cae6 in default_readlink (frame=<value optimized out>, this=0x18f3ab70, loc=0x2aaab61c4858, size=4096) at defaults.c:224 #4 0x00002b8effd1cae6 in default_readlink (frame=<value optimized out>, this=0x18f3b4c0, loc=0x2aaab61c4858, size=4096) at defaults.c:224 #5 0x00002b8effd1cae6 in default_readlink (frame=<value optimized out>, this=0x18f3bdf0, loc=0x2aaab61c4858, size=4096) at defaults.c:224 #6 0x00002b8effd1cae6 in default_readlink (frame=<value optimized out>, this=0x18f3c7f0, loc=0x2aaab61c4858, size=4096) at defaults.c:224 #7 0x00002b8f01682fcb in sp_readlink (frame=0x2aaab61f6fd0, this=0x18f3d1e0, loc=0x2aaab61c4858, size=4096) at stat-prefetch.c:2240 #8 0x00002b8f0188f0a2 in fuse_readlink (this=<value optimized out>, finh=<value optimized out>, msg=<value optimized out>) at fuse-bridge.c:1309 #9 0x00002b8f018935d6 in fuse_thread_proc (data=<value optimized out>) at fuse-bridge.c:3169 #10 0x0000003384c06307 in start_thread () from /lib64/libpthread.so.0 #11 0x00000033844d1ded in clone () from /lib64/libc.so.6 (gdb) l 1351 in dht-common.c (gdb) list *0x00002b8f00c3d12a 0x2b8f00c3d12a is in dht_readlink_cbk (dht-common.c:1356). 1351 in dht-common.c (gdb) p frame $1 = (call_frame_t *) 0x2aaab60fbc90 (gdb) p frame->local $2 = (void *) 0x2aaab61f5140 (gdb) The crash occured on client01 machine on availmedia. The log file can be found in /opt/availmedia/glusterfs/3.0.0/var/log/glusterfs/mount-availmedia-replicate.log
PATCH: http://patches.gluster.com/patch/2620 in master (distribute: Dont dereference sbuf on failed readlink_cbk)