Description of problem: pending frames: frame : type(1) op(READDIR) frame : type(0) op(0) patchset: git://git.gluster.com/glusterfs.git signal received: 11 time of crash: 2014-08-22 05:10:30 configuration details: backtrace 1 dlfcn 1 libpthread 1 xattr.h 1 st_atimespec.tv_nsec 1 package-string: glusterfs 3.7dev 0 libglusterfs.0.dylib 0x00000001061be824 _gf_msg_backtrace_nomem + 308^@ 1 libglusterfs.0.dylib 0x00000001061de475 gf_print_trace + 757^@ 2 glusterfs 0x000000010618c84d glusterfsd_print_trace + 29^@ 3 libsystem_c.dylib 0x00007fff8e25690a _sigtramp + 26^@ 4 ??? 0x000000010757c38c 0x0 + 4418159500^@ 5 client.so 0x0000000107298c2c client_readdir + 364^@ 6 libglusterfs.0.dylib 0x00000001061dc6af default_readdir + 175^@ 7 libglusterfs.0.dylib 0x00000001061dc6af default_readdir + 175^@ 8 meta.so 0x00000001073a91e2 meta_readdir + 98^@ 9 fuse.so 0x000000010700b49c fuse_readdir_resume + 1932^@ 10 fuse.so 0x0000000106ff98a0 fuse_resolve_done + 64^@ 11 fuse.so 0x0000000106ff96de fuse_resolve_all + 222^@ 12 fuse.so 0x0000000106ff983b fuse_resolve + 171^@ 13 fuse.so 0x0000000106ff96ae fuse_resolve_all + 174^@ 14 fuse.so 0x0000000106ff77e1 fuse_resolve_continue + 33^@ 15 fuse.so 0x0000000106ffa05d fuse_resolve_fd + 1965^@ 16 fuse.so 0x0000000106ff97ce fuse_resolve + 62^@ 17 fuse.so 0x0000000106ff9654 fuse_resolve_all + 84^@ 18 fuse.so 0x0000000106ff9774 fuse_resolve_and_resume + 52^@ 19 fuse.so 0x000000010701ca65 fuse_readdir + 373^@ 20 fuse.so 0x00000001070173f7 fuse_thread_proc + 2407^@ 21 libsystem_c.dylib 0x00007fff8e268772 _pthread_start + 327^@ 22 libsystem_c.dylib 0x00007fff8e2551a1 thread_start + 13^@ --------- bash-3.2# gdb -c /cores/core.22669 --exec /usr/local/sbin/glusterfs GNU gdb 6.3.50-20050815 (Apple version gdb-1824) (Wed Feb 6 22:51:23 UTC 2013) Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-apple-darwin".Reading symbols for shared libraries ...... done Reading symbols for shared libraries . done Reading symbols for shared libraries ............................................... done #0 0x00007fff90960f96 in poll () (gdb) info thr Ambiguous info command "thr": thread, threads. (gdb) info threads 6 0x00000001072c03e5 in client3_3_readdir (frame=0x7f8aa060521c, this=0x7f8aa1002020, data=0x10757c740) at client-rpc-fops.c:5702 5 0x00007fff909600fa in __psynch_cvwait () 4 0x00007fff909600fa in __psynch_cvwait () 3 0x00007fff9096057a in __sigwait () 2 0x00007fff90960386 in __semwait_signal () * 1 0x00007fff90960f96 in poll () Current language: auto; currently minimal (gdb) (gdb) #0 0x00000001072c03e5 in client3_3_readdir (frame=0x7f8aa060521c, this=0x7f8aa1002020, data=0x10757c740) at client-rpc-fops.c:5702 #1 0x0000000107298c2c in client_readdir (frame=0x7f8aa060521c, this=0x7f8aa1002020, fd=0x7f8aa040ce0c, size=1536, off=0, xdata=0x0) at client.c:1860 #2 0x00000001061dc6af in default_readdir (frame=0x7f8aa060521c, this=0x7f8aa0880620, fd=0x7f8aa040ce0c, size=1536, off=0, xdata=0x0) at defaults.c:2067 #3 0x00000001061dc6af in default_readdir (frame=0x7f8aa060521c, this=0x7f8aa0881220, fd=0x7f8aa040ce0c, size=1536, off=0, xdata=0x0) at defaults.c:2067 #4 0x00000001073a91e2 in meta_readdir (frame=0x7f8aa060521c, this=0x7f8aa0881220, fd=0x7f8aa040ce0c, size=1536, offset=0, xdata=0x0) at meta.c:116 #5 0x000000010700b49c in fuse_readdir_resume (state=0x7f8aa100b620) at fuse-bridge.c:2637 #6 0x0000000106ff98a0 in fuse_resolve_done (state=0x7f8aa100b620) at fuse-resolve.c:665 #7 0x0000000106ff96de in fuse_resolve_all (state=0x7f8aa100b620) at fuse-resolve.c:694 #8 0x0000000106ff983b in fuse_resolve (state=0x7f8aa100b620) at fuse-resolve.c:651 #9 0x0000000106ff96ae in fuse_resolve_all (state=0x7f8aa100b620) at fuse-resolve.c:690 #10 0x0000000106ff77e1 in fuse_resolve_continue (state=0x7f8aa100b620) at fuse-resolve.c:710 #11 0x0000000106ffa05d in fuse_resolve_fd (state=0x7f8aa100b620) at fuse-resolve.c:563 #12 0x0000000106ff97ce in fuse_resolve (state=0x7f8aa100b620) at fuse-resolve.c:640 #13 0x0000000106ff9654 in fuse_resolve_all (state=0x7f8aa100b620) at fuse-resolve.c:683 #14 0x0000000106ff9774 in fuse_resolve_and_resume (state=0x7f8aa100b620, fn=0x10700ad10 <fuse_readdir_resume>) at fuse-resolve.c:723 #15 0x000000010701ca65 in fuse_readdir (this=0x7f8aa084ec20, finh=0x7f8aa04102b0, msg=0x7f8aa04102d8) at fuse-bridge.c:2657 #16 0x00000001070173f7 in fuse_thread_proc (data=0x7f8aa084ec20) at fuse-bridge.c:4861 #17 0x00007fff8e268772 in _pthread_start () #18 0x00007fff8e2551a1 in thread_start () (gdb) fr 0 (gdb) l 5697 rsp_iobref = NULL; 5698 } 5699 5700 req.size = args->size; 5701 req.offset = args->offset; 5702 req.fd = remote_fd; 5703 5704 local->cmd = remote_fd; 5705 5706 memcpy (req.gfid, args->fd->inode->gfid, 16); (gdb) p local $1 = (clnt_local_t *) 0x0 (gdb) Local is NULL and it will fail. Version-Release number of selected component (if applicable): master How reproducible: Always ================= diff --git a/xlators/protocol/client/src/client-rpc-fops.c b/xlators/protocol/client/src/client-rpc-fops.c index 50ade5d..7bcc60e 100644 --- a/xlators/protocol/client/src/client-rpc-fops.c +++ b/xlators/protocol/client/src/client-rpc-fops.c @@ -5661,14 +5661,16 @@ client3_3_readdir (call_frame_t *frame, xlator_t *this, readdir_rsp_size = xdr_sizeof ((xdrproc_t) xdr_gfs3_readdir_rsp, &rsp) + args->size; + local = mem_get0 (this->local_pool); + if (!local) { + op_errno = ENOMEM; + goto unwind; + } + frame->local = local; + if ((readdir_rsp_size + GLUSTERFS_RPC_REPLY_SIZE + GLUSTERFS_RDMA_MAX_HEADER_SIZE) > (GLUSTERFS_RDMA_INLINE_THRESHOLD)) { - local = mem_get0 (this->local_pool); - if (!local) { - op_errno = ENOMEM; - goto unwind; - } - frame->local = local; + ============= Fixes this issue
(gdb) p readdir_rsp_size $1 = 1552 GLUSTERFS_RPC_REPLY_SIZE == 24 GLUSTERFS_RDMA_MAX_HEADER_SIZE == 228 # bc -l 1552 + 24 + 228 1804 GLUSTERFS_RDMA_INLINE_THRESHOLD == 2048 1804 > 2048 Is always false here on OSX, leads to crash later when local is in-fact NULL.
REVIEW: http://review.gluster.org/8511 (client: client3_3_readdir() - initialize ``local`` properly) posted (#1) for review on master by Harshavardhana (harsha)
REVIEW: http://review.gluster.org/8511 (client: client3_3_readdir() - initialize ``local`` properly) posted (#2) for review on master by Harshavardhana (harsha)
COMMIT: http://review.gluster.org/8511 committed in master by Raghavendra G (rgowdapp) ------ commit 60f12dfbc87818831a65ac80ad8ba2fe166a29e2 Author: Harshavardhana <harsha> Date: Thu Aug 21 22:22:37 2014 -0700 client: client3_3_readdir() - initialize ``local`` properly A crash is observed in the following scenario on OSX ~~~ (gdb) p readdir_rsp_size $1 = 1552 GLUSTERFS_RPC_REPLY_SIZE == 24 GLUSTERFS_RDMA_MAX_HEADER_SIZE == 228 ((1552 + 24 + 228)) == 1804 GLUSTERFS_RDMA_INLINE_THRESHOLD == 2048 if ((readdir_rsp_size + GLUSTERFS_RPC_REPLY_SIZE + GLUSTERFS_RDMA_MAX_HEADER_SIZE) > GLUSTERFS_RDMA_INLINE_THRESHOLD) ----> False ~~~ ``local`` is never initialized leads to NULL reference later. This patch makes sure that local is initialized, correctly as its done in client3_3_readdirp() call. Change-Id: I46931fc96900b7740ae71536c954bb9deda5c879 BUG: 1132796 Signed-off-by: Harshavardhana <harsha> Reviewed-on: http://review.gluster.org/8511 Reviewed-by: Niels de Vos <ndevos> Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Raghavendra G <rgowdapp> Tested-by: Raghavendra G <rgowdapp>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report. glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939 [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user