Hide Forgot
So it turned out that the generated volfile was wrong, which resulted in io-threads having a non-existing subvolume, which implied io-threads#init has failed. So far so goo.d The actual error is that the failure in init is not handled, the code continues as if nothing strange has had happened.
Check out gnotify-dbg branch of glusterfs-fusilli repo: http://git.gluster.com/?p=users/csaba/glusterfs-fusilli.git;a=shortlog;h=refs/heads/gnotifyd-dbg This branch adds two commits to master, one intoducing a new, completely skeletal xlator called gnotifyd, the other adding a gnotifyd volume to the server vol generated by glusterd. When starting the volume, the glusterfs server crashes with: [2010-09-13 10:22:03.731541] I [server-handshake.c:523:server_setvolume] pop-server: accepted client from 192.168.2.10:1023 pending frames: patchset: git://git.sv.gnu.org/gluster.git signal received: 11 time of crash: 2010-09-13 10:22:03 configuration details: argp 1 backtrace 1 dlfcn 1 fdatasync 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.1.0git /host/lib/libc.so.6(+0x325e0)[0x7f42d4ae45e0] /host/lib/libpthread.so.0(pthread_mutex_lock+0x4)[0x7f42d4e17124] /meta/progs/glusterfs-git/usr/lib/glusterfs/3.1.0git/xlator/performance/io-threads.so(iot_schedule+0x23)[0x7f42d1ac833b] /meta/progs/glusterfs-git/usr/lib/glusterfs/3.1.0git/xlator/performance/io-threads.so(iot_schedule_unordered+0x27)[0x7f42d1ac83a5] /meta/progs/glusterfs-git/usr/lib/glusterfs/3.1.0git/xlator/performance/io-threads.so(iot_lookup+0xcf)[0x7f42d1ac887d] /meta/progs/glusterfs-git/usr/lib/libglusterfs.so.0(default_lookup+0x23d)[0x7f42d5680d61] /meta/progs/glusterfs-git/usr/lib/glusterfs/3.1.0git/xlator/protocol/server.so(server_lookup_resume+0x270)[0x7f42d16b1a4a] /meta/progs/glusterfs-git/usr/lib/glusterfs/3.1.0git/xlator/protocol/server.so(server_resolve_done+0x65)[0x7f42d169f7c3] /meta/progs/glusterfs-git/usr/lib/glusterfs/3.1.0git/xlator/protocol/server.so(server_resolve_all+0xf4)[0x7f42d169f8be] /meta/progs/glusterfs-git/usr/lib/glusterfs/3.1.0git/xlator/protocol/server.so(server_resolve+0xda)[0x7f42d169f757] /meta/progs/glusterfs-git/usr/lib/glusterfs/3.1.0git/xlator/protocol/server.so(server_resolve_all+0xcb)[0x7f42d169f895] /meta/progs/glusterfs-git/usr/lib/glusterfs/3.1.0git/xlator/protocol/server.so(server_resolve_inode+0x93)[0x7f42d169f5b3] /meta/progs/glusterfs-git/usr/lib/glusterfs/3.1.0git/xlator/protocol/server.so(server_resolve+0x9b)[0x7f42d169f718] /meta/progs/glusterfs-git/usr/lib/glusterfs/3.1.0git/xlator/protocol/server.so(server_resolve_all+0x79)[0x7f42d169f843] /meta/progs/glusterfs-git/usr/lib/glusterfs/3.1.0git/xlator/protocol/server.so(resolve_and_resume+0x3f)[0x7f42d169f959] /meta/progs/glusterfs-git/usr/lib/glusterfs/3.1.0git/xlator/protocol/server.so(server_lookup+0x3b9)[0x7f42d16b71e9] /meta/progs/glusterfs-git/usr/lib/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x323)[0x7f42d545338c] /meta/progs/glusterfs-git/usr/lib/libgfrpc.so.0(rpcsvc_notify+0x176)[0x7f42d5453735] /meta/progs/glusterfs-git/usr/lib/libgfrpc.so.0(rpc_transport_notify+0xd2)[0x7f42d545970f] /meta/progs/glusterfs-git/usr/lib/glusterfs/3.1.0git/rpc-transport/socket.so(socket_event_poll_in+0x54)[0x7f42d31d39bb] /meta/progs/glusterfs-git/usr/lib/glusterfs/3.1.0git/rpc-transport/socket.so(socket_event_handler+0x106)[0x7f42d31d3d74] /meta/progs/glusterfs-git/usr/lib/libglusterfs.so.0(+0x423af)[0x7f42d56a53af] /meta/progs/glusterfs-git/usr/lib/libglusterfs.so.0(+0x425c2)[0x7f42d56a55c2] /meta/progs/glusterfs-git/usr/lib/libglusterfs.so.0(event_dispatch+0x85)[0x7f42d56a5939] /meta/progs/glusterfs-git/usr/sbin/glusterfs(main+0xfb)[0x406318] /host/lib/libc.so.6(__libc_start_main+0xfd)[0x7f42d4ad0c4d] /meta/progs/glusterfs-git/usr/sbin/glusterfs[0x4031e9] --------- Note that when I restarted the server manually from command line, the crash didn't occur.
PATCH: http://patches.gluster.com/patch/4738 in master (io-threads: return -1 in case of 'init()' failures)