Description of problem: 'gluster volume status gfstest detail' makes glusterd crash with the following in syslog: Dec 13 18:38:34 hkvm usr-local-etc-glusterfs-glusterd.vol[26797]: pending frames: Dec 13 18:38:34 hkvm usr-local-etc-glusterfs-glusterd.vol[26797]: frame : type(0) op(0) Dec 13 18:38:34 hkvm usr-local-etc-glusterfs-glusterd.vol[26797]: patchset: git://git.gluster.com/glusterfs.git Dec 13 18:38:34 hkvm usr-local-etc-glusterfs-glusterd.vol[26797]: signal received: 11 Dec 13 18:38:34 hkvm usr-local-etc-glusterfs-glusterd.vol[26797]: time of crash: Dec 13 18:38:34 hkvm usr-local-etc-glusterfs-glusterd.vol[26797]: 2014-12-13 17:38:34 Dec 13 18:38:34 hkvm usr-local-etc-glusterfs-glusterd.vol[26797]: configuration details: Dec 13 18:38:34 hkvm usr-local-etc-glusterfs-glusterd.vol[26797]: argp 1 Dec 13 18:38:34 hkvm usr-local-etc-glusterfs-glusterd.vol[26797]: backtrace 1 Dec 13 18:38:34 hkvm usr-local-etc-glusterfs-glusterd.vol[26797]: dlfcn 1 Dec 13 18:38:34 hkvm usr-local-etc-glusterfs-glusterd.vol[26797]: libpthread 1 Dec 13 18:38:34 hkvm usr-local-etc-glusterfs-glusterd.vol[26797]: llistxattr 1 Dec 13 18:38:34 hkvm usr-local-etc-glusterfs-glusterd.vol[26797]: setfsid 1 Dec 13 18:38:34 hkvm usr-local-etc-glusterfs-glusterd.vol[26797]: spinlock 1 Dec 13 18:38:34 hkvm usr-local-etc-glusterfs-glusterd.vol[26797]: epoll.h 1 Dec 13 18:38:34 hkvm usr-local-etc-glusterfs-glusterd.vol[26797]: xattr.h 1 Dec 13 18:38:34 hkvm usr-local-etc-glusterfs-glusterd.vol[26797]: st_atim.tv_nsec 1 Dec 13 18:38:34 hkvm usr-local-etc-glusterfs-glusterd.vol[26797]: package-string: glusterfs 3.6.1 Dec 13 18:38:34 hkvm usr-local-etc-glusterfs-glusterd.vol[26797]: ———— Version-Release number of selected component (if applicable): glusterfs 3.6.1 compiled from source tarball. Ubuntu 12.04, 14.04 How reproducible: always Steps to Reproduce: 1. gluster volume status <volume> detail 2. 3. Actual results: Expected results: Additional info:
Do you have backtrace or core file generated from this crash ? if yes, can u share the same ?
Not sure if you got this via mail: hkvm:/tmp# gdb /usr/local/sbin/glusterd core GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04 Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". For bug reporting instructions, please see: <http://bugs.launchpad.net/gdb-linaro/>... Reading symbols from /usr/local/sbin/glusterd...done. [New LWP 20279] [New LWP 20276] [New LWP 20277] [New LWP 20275] [New LWP 20409] [New LWP 20278] warning: Can't read pathname for load map: Input/output error. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fffd4f96000 Core was generated by `glusterd'. Program terminated with signal 11, Segmentation fault. #0 0x00007f9c082dd56a in ?? () from /lib/x86_64-linux-gnu/libc.so.6 (gdb) where #0 0x00007f9c082dd56a in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f9bffd287ef in glusterd_add_inode_size_to_dict (count=0, dict=0x7f9c065fb538) at glusterd-utils.c:7297 #2 glusterd_add_brick_detail_to_dict (volinfo=<optimized out>, brickinfo=<optimized out>, dict=0x7f9c065fb538, count=0) at glusterd-utils.c:7583 #3 0x00007f9bffd126ba in glusterd_op_status_volume (rsp_dict=0x7f9c065fb538, dict=0x7f9c065fb5c4, op_errstr=<optimized out>) at glusterd-op-sm.c:2725 #4 glusterd_op_commit_perform (op=GD_OP_STATUS_VOLUME, dict=0x7f9c065fb5c4, op_errstr=<optimized out>, rsp_dict=0x7f9c065fb538) at glusterd-op-sm.c:4923 #5 0x00007f9bffd8a6ab in gd_commit_op_phase (peers=0x2014840, op=GD_OP_STATUS_VOLUME, op_ctx=0x7f9c065fb880, req_dict=0x7f9c065fb5c4, op_errstr=0x22238a8, npeers=0) at glusterd-syncop.c:1220 #6 0x00007f9bffd8b6b2 in gd_sync_task_begin (op_ctx=0x7f9c065fb880, req=0x200ef8c) at glusterd-syncop.c:1625 #7 0x00007f9bffd8ba8b in glusterd_op_begin_synctask (req=0x200ef8c, op=<optimized out>, dict=0x7f9c065fb880) at glusterd-syncop.c:1677 #8 0x00007f9bffcf24c4 in __glusterd_handle_status_volume (req=0x200ef8c) at glusterd-handler.c:3825 #9 0x00007f9bffd006df in glusterd_big_locked_handler (req=0x200ef8c, actor_fn=0x7f9bffcf2360 <__glusterd_handle_status_volume>) at glusterd-handler.c:80 #10 0x00007f9c08c17942 in synctask_wrap (old_task=<optimized out>) at syncop.c:335 #11 0x00007f9c081f3650 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #12 0x0000000000000000 in ?? () (gdb)
using zfs 0.6.3
Found the following: In xlators/mgmt/glusterd/src/glusterd-utils.c::glusterd_add_inode_size_to_dict() the file system, here zfs, is not found so libglusterfs/src/run.c::runner_start() is called with an uninitialized “runner struct” which is caught by GF_ASSERT (runner->argv[0]);
same issue, also using zfs 0.6.3. one addition, tried downgrading to gluster 3.5.3 and still getting same crash/error under 3.5.3, although the glusterfsd's start up and seem to be working correctly.
This was broken by commit b71d501392ae10de4424c325ff37afcf3bd83d32 (f20d0ef8ad7d2f65a9234fc11101830873a9f6ab in master) which did not account for unknown filesystems. Associated with bug 1081013
Bug 1254421 and 1081013 fix this in the glusterfs-3.6.x series. *** This bug has been marked as a duplicate of bug 1254421 ***