Bug 762763 (GLUSTER-1031)

Summary: dict_unserialize crash if undersized buffers passed
Product: [Community] GlusterFS Reporter: Joe Julian <joej>
Component: coreAssignee: shishir gowda <sgowda>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: high Docs Contact:
Priority: urgent    
Version: 3.0.2CC: amarts, chida, gluster-bugs, joe, nsathyan, rabhat
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Joe Julian 2010-06-25 17:50:23 UTC 
I accidentally (and stupidly) installed a 2.x version of the glusterfs client on a virtual machine and mounted the filesystem. This crash left the server filesystems in a split-brain condition (dht over afr) which could not be repaired in 3.0.2. (It was repaired, for the most part, after installing 3.0.4 though.)

My expectation would be that the servers should disconnect the incompatible client and log an error.

As long as you're in that section of code, "passed" is misspelled.

The servers crashed with the following:
[2010-06-23 21:41:20] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed
[2010-06-23 21:41:20] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed
[2010-06-23 21:41:20] N [server-protocol.c:6748:notify] server: 10.0.0.166:1005 disconnected
[2010-06-23 21:41:20] N [server-protocol.c:6748:notify] server: 10.0.0.166:1011 disconnected
[2010-06-23 21:41:22] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed
[2010-06-23 21:41:22] N [server-protocol.c:6748:notify] server: 10.0.0.166:1022 disconnected
[2010-06-23 21:41:22] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed
[2010-06-23 21:41:22] N [server-protocol.c:6748:notify] server: 10.0.0.166:1016 disconnected
[2010-06-23 21:41:22] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed
[2010-06-23 21:41:22] N [server-protocol.c:6748:notify] server: 10.0.0.166:1010 disconnected
[2010-06-23 21:41:23] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed
[2010-06-23 21:41:23] N [server-protocol.c:6748:notify] server: 10.0.0.166:1004 disconnected
[2010-06-23 21:41:30] N [server-protocol.c:6748:notify] server: 10.0.0.166:1019 disconnected
[2010-06-23 21:41:30] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed
[2010-06-23 21:41:30] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed
[2010-06-23 21:41:30] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed
[2010-06-23 21:41:30] N [server-protocol.c:6748:notify] server: 10.0.0.166:1012 disconnected
[2010-06-23 21:41:30] N [server-protocol.c:6748:notify] server: 10.0.0.166:1006 disconnected
[2010-06-23 21:41:30] N [server-protocol.c:6748:notify] server: 10.0.0.166:1001 disconnected
[2010-06-23 21:41:30] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed
[2010-06-23 21:41:30] N [server-protocol.c:6748:notify] server: 10.0.0.166:999 disconnected
[2010-06-23 21:41:30] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed
[2010-06-23 21:41:30] N [server-protocol.c:6748:notify] server: 10.0.0.166:998 disconnected
[2010-06-23 21:41:33] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed
[2010-06-23 21:41:33] N [server-protocol.c:6748:notify] server: 10.0.0.166:1019 disconnected
[2010-06-23 21:41:37] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed
[2010-06-23 21:41:37] N [server-protocol.c:6748:notify] server: 10.0.0.166:995 disconnected
[2010-06-23 21:41:37] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed
[2010-06-23 21:41:37] N [server-protocol.c:6748:notify] server: 10.0.0.166:993 disconnected
[2010-06-23 21:41:37] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed
[2010-06-23 21:41:37] N [server-protocol.c:6748:notify] server: 10.0.0.166:991 disconnected
[2010-06-23 21:41:37] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed
[2010-06-23 21:41:37] N [server-protocol.c:6748:notify] server: 10.0.0.166:989 disconnected
[2010-06-23 21:41:37] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed
[2010-06-23 21:41:37] N [server-protocol.c:6748:notify] server: 10.0.0.166:987 disconnected
[2010-06-23 21:41:37] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed
[2010-06-23 21:41:37] N [server-protocol.c:6748:notify] server: 10.0.0.166:985 disconnected
[2010-06-23 21:41:37] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed
[2010-06-23 21:41:37] N [server-protocol.c:6748:notify] server: 10.0.0.166:983 disconnected
[2010-06-23 21:41:37] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed
[2010-06-23 21:41:37] N [server-protocol.c:6748:notify] server: 10.0.0.166:981 disconnected
[2010-06-23 21:41:38] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed
[2010-06-23 21:41:38] N [server-protocol.c:6748:notify] server: 10.0.0.166:979 disconnected
[2010-06-23 21:41:38] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed
[2010-06-23 21:41:38] N [server-protocol.c:6748:notify] server: 10.0.0.166:977 disconnected
[2010-06-23 21:41:38] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed
[2010-06-23 21:41:38] N [server-protocol.c:6748:notify] server: 10.0.0.166:975 disconnected
[2010-06-23 21:41:38] E [dict.c:2437:dict_unserialize] dict: undersized buffer passsed
[2010-06-23 21:41:38] N [server-protocol.c:6748:notify] server: 10.0.0.166:973 disconnected
pending frames:
frame : type(2) op(SETVOLUME)
frame : type(1) op(LOOKUP)

patchset: v3.0.2
signal received: 11
time of crash: 2010-06-23 21:41:38
configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.0.2
[0x44d420]
/usr/lib/libglusterfs.so.0(dict_unserialize+0x115)[0xf45fa5]
/usr/lib/glusterfs/3.0.2/xlator/protocol/server.so(mop_setvolume+0x7d)[0xd5a4ed]
/usr/lib/glusterfs/3.0.2/xlator/protocol/server.so(protocol_server_interpret+0xb0)[0xd559e0]
/usr/lib/glusterfs/3.0.2/xlator/protocol/server.so(protocol_server_pollin+0x97)[0xd55c87]
/usr/lib/glusterfs/3.0.2/xlator/protocol/server.so(notify+0xd1)[0xd55d61]
/usr/lib/libglusterfs.so.0(xlator_notify+0x3f)[0xf4bfaf]
/usr/lib/glusterfs/3.0.2/transport/socket.so(socket_event_poll_in+0x3d)[0x122cdd]
/usr/lib/glusterfs/3.0.2/transport/socket.so(socket_event_handler+0xae)[0x123d1e]
/usr/lib/libglusterfs.so.0[0xf66daa]
/usr/lib/libglusterfs.so.0(event_dispatch+0x21)[0xf65c71]
/usr/sbin/glusterfsd(main+0xcfc)[0x804bcfc]
/lib/libc.so.6(__libc_start_main+0xdc)[0x59de9c]
/usr/sbin/glusterfsd[0x8049c51]
---------
Comment 1 shishir gowda 2010-07-19 07:18:23 UTC 
dict_unserialize does return a error (-1) when a undersized buffer is passed.

And there is a disconnection error msg logged:
[2010-06-23 21:41:20] N [server-protocol.c:6748:notify] server: 10.0.0.166:1005
disconnected

Closing this bug, and will track it if found in newer versions.

As for the typo in the log msg, I will fix it as part of another fix.
Comment 2 shishir gowda 2010-07-19 08:48:53 UTC 
Fixed logmsg typo as part of fix in 269

Regards,
Shishir
Comment 3 Joe Julian 2010-08-01 15:44:39 UTC 
*** Bug 1158 has been marked as a duplicate of this bug. ***
Comment 4 Anand Avati 2010-08-06 07:37:27 UTC 
PATCH: http://patches.gluster.com/patch/3994 in master (Fix for seg fault in dict_unserialize if undersized buffers are passed)
Comment 5 Anand Avati 2010-08-06 07:37:54 UTC 
PATCH: http://patches.gluster.com/patch/3992 in release-3.0 (Fix for seg fault in dict_unserialize if undersized buffers are passed)