Hide Forgot
I accidentally (and stupidly) installed a 2.x version of the glusterfs client on a virtual machine and mounted the filesystem. This crash left the server filesystems in a split-brain condition (dht over afr) which could not be repaired in 3.0.2. (It was repaired, for the most part, after installing 3.0.4 though.) My expectation would be that the servers should disconnect the incompatible client and log an error. As long as you're in that section of code, "passed" is misspelled. The servers crashed with the following: [2010-06-23 21:41:20] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed [2010-06-23 21:41:20] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed [2010-06-23 21:41:20] N [server-protocol.c:6748:notify] server: 10.0.0.166:1005 disconnected [2010-06-23 21:41:20] N [server-protocol.c:6748:notify] server: 10.0.0.166:1011 disconnected [2010-06-23 21:41:22] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed [2010-06-23 21:41:22] N [server-protocol.c:6748:notify] server: 10.0.0.166:1022 disconnected [2010-06-23 21:41:22] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed [2010-06-23 21:41:22] N [server-protocol.c:6748:notify] server: 10.0.0.166:1016 disconnected [2010-06-23 21:41:22] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed [2010-06-23 21:41:22] N [server-protocol.c:6748:notify] server: 10.0.0.166:1010 disconnected [2010-06-23 21:41:23] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed [2010-06-23 21:41:23] N [server-protocol.c:6748:notify] server: 10.0.0.166:1004 disconnected [2010-06-23 21:41:30] N [server-protocol.c:6748:notify] server: 10.0.0.166:1019 disconnected [2010-06-23 21:41:30] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed [2010-06-23 21:41:30] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed [2010-06-23 21:41:30] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed [2010-06-23 21:41:30] N [server-protocol.c:6748:notify] server: 10.0.0.166:1012 disconnected [2010-06-23 21:41:30] N [server-protocol.c:6748:notify] server: 10.0.0.166:1006 disconnected [2010-06-23 21:41:30] N [server-protocol.c:6748:notify] server: 10.0.0.166:1001 disconnected [2010-06-23 21:41:30] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed [2010-06-23 21:41:30] N [server-protocol.c:6748:notify] server: 10.0.0.166:999 disconnected [2010-06-23 21:41:30] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed [2010-06-23 21:41:30] N [server-protocol.c:6748:notify] server: 10.0.0.166:998 disconnected [2010-06-23 21:41:33] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed [2010-06-23 21:41:33] N [server-protocol.c:6748:notify] server: 10.0.0.166:1019 disconnected [2010-06-23 21:41:37] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed [2010-06-23 21:41:37] N [server-protocol.c:6748:notify] server: 10.0.0.166:995 disconnected [2010-06-23 21:41:37] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed [2010-06-23 21:41:37] N [server-protocol.c:6748:notify] server: 10.0.0.166:993 disconnected [2010-06-23 21:41:37] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed [2010-06-23 21:41:37] N [server-protocol.c:6748:notify] server: 10.0.0.166:991 disconnected [2010-06-23 21:41:37] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed [2010-06-23 21:41:37] N [server-protocol.c:6748:notify] server: 10.0.0.166:989 disconnected [2010-06-23 21:41:37] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed [2010-06-23 21:41:37] N [server-protocol.c:6748:notify] server: 10.0.0.166:987 disconnected [2010-06-23 21:41:37] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed [2010-06-23 21:41:37] N [server-protocol.c:6748:notify] server: 10.0.0.166:985 disconnected [2010-06-23 21:41:37] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed [2010-06-23 21:41:37] N [server-protocol.c:6748:notify] server: 10.0.0.166:983 disconnected [2010-06-23 21:41:37] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed [2010-06-23 21:41:37] N [server-protocol.c:6748:notify] server: 10.0.0.166:981 disconnected [2010-06-23 21:41:38] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed [2010-06-23 21:41:38] N [server-protocol.c:6748:notify] server: 10.0.0.166:979 disconnected [2010-06-23 21:41:38] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed [2010-06-23 21:41:38] N [server-protocol.c:6748:notify] server: 10.0.0.166:977 disconnected [2010-06-23 21:41:38] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed [2010-06-23 21:41:38] N [server-protocol.c:6748:notify] server: 10.0.0.166:975 disconnected [2010-06-23 21:41:38] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed [2010-06-23 21:41:38] N [server-protocol.c:6748:notify] server: 10.0.0.166:973 disconnected pending frames: frame : type(2) op(SETVOLUME) frame : type(1) op(LOOKUP) patchset: v3.0.2 signal received: 11 time of crash: 2010-06-23 21:41:38 configuration details: argp 1 backtrace 1 dlfcn 1 fdatasync 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.0.2 [0x44d420] /usr/lib/libglusterfs.so.0(dict_unserialize+0x115)[0xf45fa5] /usr/lib/glusterfs/3.0.2/xlator/protocol/server.so(mop_setvolume+0x7d)[0xd5a4ed] /usr/lib/glusterfs/3.0.2/xlator/protocol/server.so(protocol_server_interpret+0xb0)[0xd559e0] /usr/lib/glusterfs/3.0.2/xlator/protocol/server.so(protocol_server_pollin+0x97)[0xd55c87] /usr/lib/glusterfs/3.0.2/xlator/protocol/server.so(notify+0xd1)[0xd55d61] /usr/lib/libglusterfs.so.0(xlator_notify+0x3f)[0xf4bfaf] /usr/lib/glusterfs/3.0.2/transport/socket.so(socket_event_poll_in+0x3d)[0x122cdd] /usr/lib/glusterfs/3.0.2/transport/socket.so(socket_event_handler+0xae)[0x123d1e] /usr/lib/libglusterfs.so.0[0xf66daa] /usr/lib/libglusterfs.so.0(event_dispatch+0x21)[0xf65c71] /usr/sbin/glusterfsd(main+0xcfc)[0x804bcfc] /lib/libc.so.6(__libc_start_main+0xdc)[0x59de9c] /usr/sbin/glusterfsd[0x8049c51] ---------
dict_unserialize does return a error (-1) when a undersized buffer is passed. And there is a disconnection error msg logged: [2010-06-23 21:41:20] N [server-protocol.c:6748:notify] server: 10.0.0.166:1005 disconnected Closing this bug, and will track it if found in newer versions. As for the typo in the log msg, I will fix it as part of another fix.
Fixed logmsg typo as part of fix in 269 Regards, Shishir
*** Bug 1158 has been marked as a duplicate of this bug. ***
PATCH: http://patches.gluster.com/patch/3994 in master (Fix for seg fault in dict_unserialize if undersized buffers are passed)
PATCH: http://patches.gluster.com/patch/3992 in release-3.0 (Fix for seg fault in dict_unserialize if undersized buffers are passed)