Bug 763323 (GLUSTER-1591)

Summary: glusterfs server crashes with another translator inserted into volume chain
Product: [Community] GlusterFS Reporter: Csaba Henk <csaba>
Component: coreAssignee: Amar Tumballi <amarts>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: medium Docs Contact:
Priority: low    
Version: mainlineCC: gluster-bugs, vraman
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Csaba Henk 2010-09-13 06:58:06 UTC
So it turned out that the generated volfile was wrong, which resulted in io-threads having a non-existing subvolume, which implied io-threads#init has failed.

So far so goo.d The actual error is that the failure in init is not handled, the code continues as if nothing strange has had happened.

Comment 1 Csaba Henk 2010-09-13 08:26:20 UTC
Check out gnotify-dbg branch of glusterfs-fusilli repo:

http://git.gluster.com/?p=users/csaba/glusterfs-fusilli.git;a=shortlog;h=refs/heads/gnotifyd-dbg

This branch adds two commits to master, one intoducing a new, completely skeletal xlator called gnotifyd, the other adding a gnotifyd volume to the server vol generated by glusterd.

When starting the volume, the glusterfs server crashes with:

[2010-09-13 10:22:03.731541] I [server-handshake.c:523:server_setvolume] pop-server: accepted client from 192.168.2.10:1023
pending frames:

patchset: git://git.sv.gnu.org/gluster.git
signal received: 11
time of crash: 2010-09-13 10:22:03
configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.1.0git
/host/lib/libc.so.6(+0x325e0)[0x7f42d4ae45e0]
/host/lib/libpthread.so.0(pthread_mutex_lock+0x4)[0x7f42d4e17124]
/meta/progs/glusterfs-git/usr/lib/glusterfs/3.1.0git/xlator/performance/io-threads.so(iot_schedule+0x23)[0x7f42d1ac833b]
/meta/progs/glusterfs-git/usr/lib/glusterfs/3.1.0git/xlator/performance/io-threads.so(iot_schedule_unordered+0x27)[0x7f42d1ac83a5]
/meta/progs/glusterfs-git/usr/lib/glusterfs/3.1.0git/xlator/performance/io-threads.so(iot_lookup+0xcf)[0x7f42d1ac887d]
/meta/progs/glusterfs-git/usr/lib/libglusterfs.so.0(default_lookup+0x23d)[0x7f42d5680d61]
/meta/progs/glusterfs-git/usr/lib/glusterfs/3.1.0git/xlator/protocol/server.so(server_lookup_resume+0x270)[0x7f42d16b1a4a]
/meta/progs/glusterfs-git/usr/lib/glusterfs/3.1.0git/xlator/protocol/server.so(server_resolve_done+0x65)[0x7f42d169f7c3]
/meta/progs/glusterfs-git/usr/lib/glusterfs/3.1.0git/xlator/protocol/server.so(server_resolve_all+0xf4)[0x7f42d169f8be]
/meta/progs/glusterfs-git/usr/lib/glusterfs/3.1.0git/xlator/protocol/server.so(server_resolve+0xda)[0x7f42d169f757]
/meta/progs/glusterfs-git/usr/lib/glusterfs/3.1.0git/xlator/protocol/server.so(server_resolve_all+0xcb)[0x7f42d169f895]
/meta/progs/glusterfs-git/usr/lib/glusterfs/3.1.0git/xlator/protocol/server.so(server_resolve_inode+0x93)[0x7f42d169f5b3]
/meta/progs/glusterfs-git/usr/lib/glusterfs/3.1.0git/xlator/protocol/server.so(server_resolve+0x9b)[0x7f42d169f718]
/meta/progs/glusterfs-git/usr/lib/glusterfs/3.1.0git/xlator/protocol/server.so(server_resolve_all+0x79)[0x7f42d169f843]
/meta/progs/glusterfs-git/usr/lib/glusterfs/3.1.0git/xlator/protocol/server.so(resolve_and_resume+0x3f)[0x7f42d169f959]
/meta/progs/glusterfs-git/usr/lib/glusterfs/3.1.0git/xlator/protocol/server.so(server_lookup+0x3b9)[0x7f42d16b71e9]
/meta/progs/glusterfs-git/usr/lib/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x323)[0x7f42d545338c]
/meta/progs/glusterfs-git/usr/lib/libgfrpc.so.0(rpcsvc_notify+0x176)[0x7f42d5453735]
/meta/progs/glusterfs-git/usr/lib/libgfrpc.so.0(rpc_transport_notify+0xd2)[0x7f42d545970f]
/meta/progs/glusterfs-git/usr/lib/glusterfs/3.1.0git/rpc-transport/socket.so(socket_event_poll_in+0x54)[0x7f42d31d39bb]
/meta/progs/glusterfs-git/usr/lib/glusterfs/3.1.0git/rpc-transport/socket.so(socket_event_handler+0x106)[0x7f42d31d3d74]
/meta/progs/glusterfs-git/usr/lib/libglusterfs.so.0(+0x423af)[0x7f42d56a53af]
/meta/progs/glusterfs-git/usr/lib/libglusterfs.so.0(+0x425c2)[0x7f42d56a55c2]
/meta/progs/glusterfs-git/usr/lib/libglusterfs.so.0(event_dispatch+0x85)[0x7f42d56a5939]
/meta/progs/glusterfs-git/usr/sbin/glusterfs(main+0xfb)[0x406318]
/host/lib/libc.so.6(__libc_start_main+0xfd)[0x7f42d4ad0c4d]
/meta/progs/glusterfs-git/usr/sbin/glusterfs[0x4031e9]
---------

Note that when I restarted the server manually from command line, the crash didn't occur.

Comment 2 Vijay Bellur 2010-09-13 10:45:05 UTC
PATCH: http://patches.gluster.com/patch/4738 in master (io-threads: return -1 in case of 'init()' failures)