Description of problem: On a two node gluster with 4 distributed volumes after upgrade of glusterfs server to version 3.7.1 from 3.7.0 on one of the two nodes glusterd crashes with a buffer overflow error. This happens on node0 when it tries to communicate with node1. The nodes are exactly the same. Same hardware, same OS, same packages installed. Version-Release number of selected component (if applicable): 3.7.1 How reproducible: Always on our environment Steps to Reproduce: 1. On node0: [root@gnode00 ~]# systemctl start glusterd [root@gnode00 ~]# systemctl status glusterd.service glusterd.service - GlusterFS, a clustered file-system server Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled) Active: active (running) since Thu 2015-06-04 10:38:48 EEST; 7min ago Process: 12501 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid (code=exited, status=0/SUCCESS) Main PID: 12502 (glusterd) CGroup: /system.slice/glusterd.service └─12502 /usr/sbin/glusterd -p /var/run/glusterd.pid Jun 04 10:38:48 gnode00 systemd[1]: Starting GlusterFS, a clustered file-system server... Jun 04 10:38:48 gnode00 systemd[1]: Started GlusterFS, a clustered file-system server. 2. On node1: [root@gnode01 ~]# systemctl start glusterd [root@gnode01 ~]# systemctl status glusterd glusterd.service - GlusterFS, a clustered file-system server Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled) Active: active (running) since Thu 2015-06-04 10:54:55 EEST; 6s ago Process: 8179 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid (code=exited, status=0/SUCCESS) Main PID: 8180 (glusterd) CGroup: /system.slice/glusterd.service ├─8180 /usr/sbin/glusterd -p /var/run/glusterd.pid ├─8202 /usr/sbin/glusterfsd -s gnode01 --volfile-id mmphoto.gnode01.BRICKS-lv02a-mmphoto -p /var/lib/glusterd/vols/mmphoto/run/gnode01... ├─8209 /usr/sbin/glusterfsd -s gnode01 --volfile-id mmphoto.gnode01.BRICKS-lv02b-mmphoto -p /var/lib/glusterd/vols/mmphoto/run/gnode01... ├─8215 /usr/sbin/glusterfsd -s gnode01 --volfile-id phdms.gnode01.BRICKS-lv00-phdms -p /var/lib/glusterd/vols/phdms/run/gnode01-BRICKS... ├─8220 /usr/sbin/glusterfsd -s gnode01 --volfile-id restore.gnode01.BRICKS-lv03a-restore -p /var/lib/glusterd/vols/restore/run/gnode01... └─8225 /usr/sbin/glusterfsd -s gnode01 --volfile-id thesis.gnode01.BRICKS-lv01-thesis -p /var/lib/glusterd/vols/thesis/run/gnode01-BRI... Jun 04 10:54:55 gnode01 systemd[1]: Starting GlusterFS, a clustered file-system server... Jun 04 10:54:55 gnode01 systemd[1]: Started GlusterFS, a clustered file-system server. 3. On node0: [root@gnode00 ~]# systemctl status glusterd.service glusterd.service - GlusterFS, a clustered file-system server Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled) Active: failed (Result: signal) since Thu 2015-06-04 10:54:57 EEST; 1min 51s ago Process: 12501 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid (code=exited, status=0/SUCCESS) Main PID: 12502 (code=killed, signal=ABRT) Jun 04 10:54:57 gnode00 etc-glusterfs-glusterd.vol[12502]: llistxattr 1 Jun 04 10:54:57 gnode00 etc-glusterfs-glusterd.vol[12502]: setfsid 1 Jun 04 10:54:57 gnode00 etc-glusterfs-glusterd.vol[12502]: spinlock 1 Jun 04 10:54:57 gnode00 etc-glusterfs-glusterd.vol[12502]: epoll.h 1 Jun 04 10:54:57 gnode00 etc-glusterfs-glusterd.vol[12502]: xattr.h 1 Jun 04 10:54:57 gnode00 etc-glusterfs-glusterd.vol[12502]: st_atim.tv_nsec 1 Jun 04 10:54:57 gnode00 etc-glusterfs-glusterd.vol[12502]: package-string: glusterfs 3.7.1 Jun 04 10:54:57 gnode00 etc-glusterfs-glusterd.vol[12502]: --------- Jun 04 10:54:57 gnode00 systemd[1]: glusterd.service: main process exited, code=killed, status=6/ABRT Jun 04 10:54:57 gnode00 systemd[1]: Unit glusterd.service entered failed state. The log files has the following entry: [2015-06-04 07:24:07.098158] D [rpc-transport.c:287:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.7.1/rpc-transport/socket.so [2015-06-04 07:24:07.098247] D [socket.c:3816:socket_init] 0-management: disabling nodelay [2015-06-04 07:24:07.098273] D [socket.c:3867:socket_init] 0-management: Configued transport.tcp-user-timeout=0 [2015-06-04 07:24:07.098294] D [socket.c:3938:socket_init] 0-management: SSL support on the I/O path is NOT enabled [2015-06-04 07:24:07.098313] D [socket.c:3941:socket_init] 0-management: SSL support for glusterd is NOT enabled [2015-06-04 07:24:07.098329] D [socket.c:3958:socket_init] 0-management: using system polling thread [2015-06-04 07:24:07.098382] D [glusterd-handler.c:3153:glusterd_rpc_create] 0-management: returning 0 [2015-06-04 07:24:07.098408] D [glusterd-utils.c:1493:glusterd_brick_connect] 0-: Returning 0 [2015-06-04 07:24:07.098428] D [glusterd-utils.c:4387:glusterd_brick_start] 0-management: returning 0 [2015-06-04 07:24:07.098453] D [glusterd-utils.c:4387:glusterd_brick_start] 0-management: returning 0 *** buffer overflow detected ***: glusterd terminated ======= Backtrace: ========= /lib64/libc.so.6(__fortify_fail+0x37)[0x7f0493709a57] /lib64/libc.so.6(+0x10bc10)[0x7f0493707c10] /lib64/libc.so.6(+0x10b32b)[0x7f049370732b] /lib64/libc.so.6(__snprintf_chk+0x78)[0x7f0493707248] /usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(glusterd_volume_defrag_restart+0x191)[0x7f0489ad2931] /usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(glusterd_restart_rebalance+0x82)[0x7f0489ad8aa2] /usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(glusterd_spawn_daemons+0x4f)[0x7f0489ad8b1f] /lib64/libglusterfs.so.0(synctask_wrap+0x12)[0x7f0494c7a482] /lib64/libc.so.6(+0x470f0)[0x7f04936430f0] ======= Memory map: ======== 7f0478000000-7f0478021000 rw-p 00000000 00:00 0 7f0478021000-7f047c000000 ---p 00000000 00:00 0 7f047c000000-7f047c023000 rw-p 00000000 00:00 0 7f047c023000-7f0480000000 ---p 00000000 00:00 0 7f0480000000-7f0480021000 rw-p 00000000 00:00 0 7f0480021000-7f0484000000 ---p 00000000 00:00 0 7f0484b35000-7f0484d36000 rw-p 00000000 00:00 0 [stack:12401] 7f0484d36000-7f0484d37000 ---p 00000000 00:00 0 7f0484d37000-7f0485537000 rw-p 00000000 00:00 0 [stack:12405] 7f0485537000-7f0485538000 ---p 00000000 00:00 0 7f0485538000-7f0485d38000 rw-p 00000000 00:00 0 [stack:12404] 7f0485d38000-7f0485d3d000 r-xp 00000000 08:04 3016 /usr/lib64/libnss_dns-2.17.so 7f0485d3d000-7f0485f3c000 ---p 00005000 08:04 3016 /usr/lib64/libnss_dns-2.17.so 7f0485f3c000-7f0485f3d000 r--p 00004000 08:04 3016 /usr/lib64/libnss_dns-2.17.so 7f0485f3d000-7f0485f3e000 rw-p 00005000 08:04 3016 /usr/lib64/libnss_dns-2.17.so 7f0485f3e000-7f0485f49000 r-xp 00000000 08:04 3018 /usr/lib64/libnss_files-2.17.so 7f0485f49000-7f0486148000 ---p 0000b000 08:04 3018 /usr/lib64/libnss_files-2.17.so 7f0486148000-7f0486149000 r--p 0000a000 08:04 3018 /usr/lib64/libnss_files-2.17.so 7f0486149000-7f048614a000 rw-p 0000b000 08:04 3018 /usr/lib64/libnss_files-2.17.so 7f048614a000-7f04862f5000 rw-p 00000000 00:00 0 7f04862f5000-7f048630b000 r-xp 00000000 08:04 3028 /usr/lib64/libresolv-2.17.so 7f048630b000-7f048650b000 ---p 00016000 08:04 3028 /usr/lib64/libresolv-2.17.so 7f048650b000-7f048650c000 r--p 00016000 08:04 3028 /usr/lib64/libresolv-2.17.so 7f048650c000-7f048650d000 rw-p 00017000 08:04 3028 /usr/lib64/libresolv-2.17.so 7f048650d000-7f048650f000 rw-p 00000000 00:00 0 7f048650f000-7f0486512000 r-xp 00000000 08:04 3374 /usr/lib64/libkeyutils.so.1.5 7f0486512000-7f0486711000 ---p 00003000 08:04 3374 /usr/lib64/libkeyutils.so.1.5 7f0486711000-7f0486712000 r--p 00002000 08:04 3374 /usr/lib64/libkeyutils.so.1.5 7f0486712000-7f0486713000 rw-p 00003000 08:04 3374 /usr/lib64/libkeyutils.so.1.5 7f0486713000-7f0486720000 r-xp 00000000 08:04 35693 /usr/lib64/libkrb5support.so.0.1 7f0486720000-7f0486920000 ---p 0000d000 08:04 35693 /usr/lib64/libkrb5support.so.0.1 7f0486920000-7f0486921000 r--p 0000d000 08:04 35693 /usr/lib64/libkrb5support.so.0.1 7f0486921000-7f0486922000 rw-p 0000e000 08:04 35693 /usr/lib64/libkrb5support.so.0.1 7f0486922000-7f0486951000 r-xp 00000000 08:04 35681 /usr/lib64/libk5crypto.so.3.1 7f0486951000-7f0486b50000 ---p 0002f000 08:04 35681 /usr/lib64/libk5crypto.so.3.1 7f0486b50000-7f0486b52000 r--p 0002e000 08:04 35681 /usr/lib64/libk5crypto.so.3.1 7f0486b52000-7f0486b53000 rw-p 00030000 08:04 35681 /usr/lib64/libk5crypto.so.3.1 7f0486b53000-7f0486b54000 rw-p 00000000 00:00 0 7f0486b54000-7f0486b57000 r-xp 00000000 08:04 3157 /usr/lib64/libcom_err.so.2.1 7f0486b57000-7f0486d56000 ---p 00003000 08:04 3157 /usr/lib64/libcom_err.so.2.1 7f0486d56000-7f0486d57000 r--p 00002000 08:04 3157 /usr/lib64/libcom_err.so.2.1 7f0486d57000-7f0486d58000 rw-p 00003000 08:04 3157 /usr/lib64/libcom_err.so.2.1 7f0486d58000-7f0486e2b000 r-xp 00000000 08:04 35691 /usr/lib64/libkrb5.so.3.3 7f0486e2b000-7f048702b000 ---p 000d3000 08:04 35691 /usr/lib64/libkrb5.so.3.3 7f048702b000-7f0487038000 r--p 000d3000 08:04 35691 /usr/lib64/libkrb5.so.3.3 7f0487038000-7f048703b000 rw-p 000e0000 08:04 35691 /usr/lib64/libkrb5.so.3.3 7f048703b000-7f0487084000 r-xp 00000000 08:04 4350 /usr/lib64/libgssapi_krb5.so.2.2 7f0487084000-7f0487284000 ---p 00049000 08:04 4350 /usr/lib64/libgssapi_krb5.so.2.2 7f0487284000-7f0487285000 r--p 00049000 08:04 4350 /usr/lib64/libgssapi_krb5.so.2.2 7f0487285000-7f0487287000 rw-p 0004a000 08:04 4350 /usr/lib64/libgssapi_krb5.so.2.2 7f0487287000-7f04872e9000 r-xp 00000000 08:04 35705 /usr/lib64/libssl.so.1.0.1e 7f04872e9000-7f04874e9000 ---p 00062000 08:04 35705 /usr/lib64/libssl.so.1.0.1e 7f04874e9000-7f04874ed000 r--p 00062000 08:04 35705 /usr/lib64/libssl.so.1.0.1e 7f04874ed000-7f04874f4000 rw-p 00066000 08:04 35705 /usr/lib64/libssl.so.1.0.1e 7f04874f4000-7f0487505000 r-xp 00000000 08:04 135881318 /usr/lib64/glusterfs/3.7.1/rpc-transport/socket.so 7f0487505000-7f0487704000 ---p 00011000 08:04 135881318 /usr/lib64/glusterfs/3.7.1/rpc-transport/socket.so 7f0487704000-7f0487705000 r--p 00010000 08:04 135881318 /usr/lib64/glusterfs/3.7.1/rpc-transport/socket.so 7f0487705000-7f0487709000 rw-p 00011000 08:04 135881318 /usr/lib64/glusterfs/3.7.1/rpc-transport/socket.so 7f0487709000-7f0487736000 rw-p 00000000 00:00 0 7f0487736000-7f0487795000 r-xp 00000000 08:04 3080 /usr/lib64/libpcre.so.1.2.0 7f0487795000-7f0487995000 ---p 0005f000 08:04 3080 /usr/lib64/libpcre.so.1.2.0 7f0487995000-7f0487996000 r--p 0005f000 08:04 3080 /usr/lib64/libpcre.so.1.2.0 7f0487996000-7f0487997000 rw-p 00060000 08:04 3080 /usr/lib64/libpcre.so.1.2.0 7f0487997000-7f04879db000 r-xp 00000000 08:04 3054 /usr/lib64/libsepol.so.1 7f04879db000-7f0487bda000 ---p 00044000 08:04 3054 /usr/lib64/libsepol.so.1 7f0487bda000-7f0487bdb000 r--p 00043000 08:04 3054 /usr/lib64/libsepol.so.1 7f0487bdb000-7f0487bdc000 rw-p 00044000 08:04 3054 /usr/lib64/libsepol.so.1 7f0487bdc000-7f0487bf1000 r-xp 00000000 08:04 209 /usr/lib64/libgcc_s-4.8.3-20140911.so.1 7f0487bf1000-7f0487df0000 ---p 00015000 08:04 209 /usr/lib64/libgcc_s-4.8.3-20140911.so.1 7f0487df0000-7f0487df1000 r--p 00014000 08:04 209 /usr/lib64/libgcc_s-4.8.3-20140911.so.1 7f0487df1000-7f0487df2000 rw-p 00015000 08:04 209 /usr/lib64/libgcc_s-4.8.3-20140911.so.1 7f0487df2000-7f0487e13000 r-xp 00000000 08:04 3091 /usr/lib64/libselinux.so.1 7f0487e13000-7f0488013000 ---p 00021000 08:04 3091 /usr/lib64/libselinux.so.1 7f0488013000-7f0488014000 r--p 00021000 08:04 3091 /usr/lib64/libselinux.so.1 7f0488014000-7f0488015000 rw-p 00022000 08:04 3091 /usr/lib64/libselinux.so.1 7f0488015000-7f0488017000 rw-p 00000000 00:00 0 7f0488017000-7f048801c000 r-xp 00000000 08:04 1647807 /usr/lib64/libdevmapper-event.so.1.02 7f048801c000-7f048821c000 ---p 00005000 08:04 1647807 /usr/lib64/libdevmapper-event.so.1.02 7f048821c000-7f048821d000 r--p 00005000 08:04 1647807 /usr/lib64/libdevmapper-event.so.1.02 7f048821d000-7f048821e000 rw-p 00006000 08:04 1647807 /usr/lib64/libdevmapper-event.so.1.02 7f048821e000-7f0488263000 r-xp 00000000 08:04 206103 /usr/lib64/libdevmapper.so.1.02 7f0488263000-7f0488462000 ---p 00045000 08:04 206103 /usr/lib64/libdevmapper.so.1.02 7f0488462000-7f0488463000 r--p 00044000 08:04 206103 /usr/lib64/libdevmapper.so.1.02 7f0488463000-7f0488466000 rw-p 00045000 08:04 206103 /usr/lib64/libdevmapper.so.1.02 7f0488466000-7f0488467000 rw-p 00000000 00:00 0 7f0488467000-7f048849f000 r-xp 00000000 08:04 131775 /usr/lib64/libblkid.so.1.1.0 7f048849f000-7f048869e000 ---p 00038000 08:04 131775 /usr/lib64/libblkid.so.1.1.0 7f048869e000-7f04886a1000 r--p 00037000 08:04 131775 /usr/lib64/libblkid.so.1.1.0 7f04886a1000-7f04886a2000 rw-p 0003a000 08:04 131775 /usr/lib64/libblkid.so.1.1.0 7f04886a2000-7f04886a3000 rw-p 00000000 00:00 0 7f04886a3000-7f04886b4000 r-xp 00000000 08:04 35842 /usr/lib64/libudev.so.1.4.0 7f04886b4000-7f04888b3000 ---p 00011000 08:04 35842 /usr/lib64/libudev.so.1.4.0 7f04888b3000-7f04888b4000 r--p 00010000 08:04 35842 /usr/lib64/libudev.so.1.4.0 7f04888b4000-7f04888b5000 rw-p 00011000 08:04 35842 /usr/lib64/libudev.so.1.4.0 7f04888b5000-7f04888b7000 r-xp 00000000 08:04 35806 /usr/lib64/liburcu-common.so.1.0.0 7f04888b7000-7f0488ab6000 ---p 00002000 08:04 35806 /usr/lib64/liburcu-common.so.1.0.0 7f0488ab6000-7f0488ab7000 r--p 00001000 08:04 35806 /usr/lib64/liburcu-common.so.1.0.0 7f0488ab7000-7f0488ab8000 rw-p 00002000 08:04 35806 /usr/lib64/liburcu-common.so.1.0.0 7f0488ab8000-7f0488adc000 r-xp 00000000 08:04 3047 /usr/lib64/liblzma.so.5.0.99 7f0488adc000-7f0488cdb000 ---p 00024000 08:04 3047 /usr/lib64/liblzma.so.5.0.99 7f0488cdb000-7f0488cdc000 r--p 00023000 08:04 3047 /usr/lib64/liblzma.so.5.0.99 7f0488cdc000-7f0488cdd000 rw-p 00024000 08:04 3047 /usr/lib64/liblzma.so.5.0.99 7f0488cdd000-7f0488dba000 r-xp 00000000 08:04 275744 /usr/lib64/liblvm2app.so.2.2 7f0488dba000-7f0488fb9000 ---p 000dd000 08:04 275744 /usr/lib64/liblvm2app.so.2.2 7f0488fb9000-7f0488fba000 r--p 000dc000 08:04 275744 /usr/lib64/liblvm2app.so.2.2 7f0488fba000-7f0488fc0000 rw-p 000dd000 08:04 275744 /usr/lib64/liblvm2app.so.2.2 7f0488fc0000-7f0488fdc000 rw-p 00000000 00:00 0 7f0488fdc000-7f0488fe2000 r-xp 00000000 08:04 249077 /usr/lib64/liburcu-cds.so.1.0.0 7f0488fe2000-7f04891e1000 ---p 00006000 08:04 249077 /usr/lib64/liburcu-cds.so.1.0.0 7f04891e1000-7f04891e2000 r--p 00005000 08:04 249077 /usr/lib64/liburcu-cds.so.1.0.0 7f04891e2000-7f04891e3000 rw-p 00006000 08:04 249077 /usr/lib64/liburcu-cds.so.1.0.0 7f04891e3000-7f04891e9000 r-xp 00000000 08:04 249075 /usr/lib64/liburcu-bp.so.1.0.0 7f04891e9000-7f04893e8000 ---p 00006000 08:04 249075 /usr/lib64/liburcu-bp.so.1.0.0 7f04893e8000-7f04893e9000 r--p 00005000 08:04 249075 /usr/lib64/liburcu-bp.so.1.0.0 7f04893e9000-7f04893ea000 rw-p 00006000 08:04 249075 /usr/lib64/liburcu-bp.so.1.0.0 7f04893ea000-7f04894eb000 r-xp 00000000 08:04 3008 /usr/lib64/libm-2.17.so 7f04894eb000-7f04896ea000 ---p 00101000 08:04 3008 /usr/lib64/libm-2.17.so 7f04896ea000-7f04896eb000 r--p 00100000 08:04 3008 /usr/lib64/libm-2.17.so 7f04896eb000-7f04896ec000 rw-p 00101000 08:04 3008 /usr/lib64/libm-2.17.so 7f04896ec000-7f048984a000 r-xp 00000000 08:04 1434624 /usr/lib64/libxml2.so.2.9.1 7f048984a000-7f0489a49000 ---p 0015e000 08:04 1434624 /usr/lib64/libxml2.so.2.9.1 7f0489a49000-7f0489a51000 r--p 0015d000 08:04 1434624 /usr/lib64/libxml2.so.2.9.1 7f0489a51000-7f0489a53000 rw-p 00165000 08:04 1434624 /usr/lib64/libxml2.so.2.9.1 7f0489a53000-7f0489a55000 rw-p 00000000 00:00 0 7f0489a5e000-7f0489bb5000 r-xp 00000000 08:04 135881320 /usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so 7f0489bb5000-7f0489db5000 ---p 00157000 08:04 135881320 /usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so 7f0489db5000-7f0489db6000 r--p 00157000 08:04 135881320 /usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so 7f0489db6000-7f0489dc7000 rw-p 00158000 08:04 135881320 /usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so 7f0489dc7000-7f0489dc8000 rw-p 00000000 00:00 0 7f0489dc8000-7f0489dc9000 ---p 00000000 00:00 0 7f0489dc9000-7f048a5c9000 rw-p 00000000 00:00 0 [stack:12402] 7f048a5c9000-7f048a5ca000 ---p 00000000 00:00 0 7f048a5ca000-7f048adca000 rw-p 00000000 00:00 0 7f048adca000-7f048adcb000 ---p 00000000 00:00 0 7f048adcb000-7f048b5cb000 rw-p 00000000 00:00 0 [stack:12400] 7f048b5cb000-7f048b5cc000 ---p 00000000 00:00 0 7f048b5cc000-7f048bdcc000 rw-p 00000000 00:00 0 [stack:12399] 7f048bdcc000-7f04922f3000 r--p 00000000 08:04 201329209 /usr/lib/locale/locale-archive 7f04922f3000-7f04935fc000 rw-p 00000000 00:00 0 7f04935fc000-7f04937b2000 r-xp 00000000 08:04 3000 /usr/lib64/libc-2.17.so 7f04937b2000-7f04939b2000 ---p 001b6000 08:04 3000 /usr/lib64/libc-2.17.so 7f04939b2000-7f04939b6000 r--p 001b6000 08:04 3000 /usr/lib64/libc-2.17.so 7f04939b6000-7f04939b8000 rw-p 001ba000 08:04 3000 /usr/lib64/libc-2.17.so 7f04939b8000-7f04939bd000 rw-p 00000000 00:00 0 7f04939bd000-7f0493b7a000 r-xp 00000000 08:04 4346 /usr/lib64/libcrypto.so.1.0.1e 7f0493b7a000-7f0493d7a000 ---p 001bd000 08:04 4346 /usr/lib64/libcrypto.so.1.0.1e 7f0493d7a000-7f0493d94000 r--p 001bd000 08:04 4346 /usr/lib64/libcrypto.so.1.0.1e 7f0493d94000-7f0493da0000 rw-p 001d7000 08:04 4346 /usr/lib64/libcrypto.so.1.0.1e 7f0493da0000-7f0493da4000 rw-p 00000000 00:00 0 7f0493da4000-7f0493dba000 r-xp 00000000 08:04 3026 /usr/lib64/libpthread-2.17.so 7f0493dba000-7f0493fba000 ---p 00016000 08:04 3026 /usr/lib64/libpthread-2.17.so 7f0493fba000-7f0493fbb000 r--p 00016000 08:04 3026 /usr/lib64/libpthread-2.17.so 7f0493fbb000-7f0493fbc000 rw-p 00017000 08:04 3026 /usr/lib64/libpthread-2.17.so 7f0493fbc000-7f0493fc0000 rw-p 00000000 00:00 0 7f0493fc0000-7f0493fc3000 r-xp 00000000 08:04 3006 /usr/lib64/libdl-2.17.so 7f0493fc3000-7f04941c2000 ---p 00003000 08:04 3006 /usr/lib64/libdl-2.17.so 7f04941c2000-7f04941c3000 r--p 00002000 08:04 3006 /usr/lib64/libdl-2.17.so 7f04941c3000-7f04941c4000 rw-p 00003000 08:04 3006 /usr/lib64/libdl-2.17.so 7f04941c4000-7f04941cb000 r-xp 00000000 08:04 3030 /usr/lib64/librt-2.17.so 7f04941cb000-7f04943ca000 ---p 00007000 08:04 3030 /usr/lib64/librt-2.17.so 7f04943ca000-7f04943cb000 r--p 00006000 08:04 3030 /usr/lib64/librt-2.17.so 7f04943cb000-7f04943cc000 rw-p 00007000 08:04 3030 /usr/lib64/librt-2.17.so 7f04943cc000-7f04943d0000 r-xp 00000000 08:04 131774 /usr/lib64/libuuid.so.1.3.0 7f04943d0000-7f04945cf000 ---p 00004000 08:04 131774 /usr/lib64/libuuid.so.1.3.0 7f04945cf000-7f04945d0000 r--p 00003000 08:04 131774 /usr/lib64/libuuid.so.1.3.0 7f04945d0000-7f04945d1000 rw-p 00004000 08:04 131774 /usr/lib64/libuuid.so.1.3.0 7f04945d1000-7f04945e6000 r-xp 00000000 08:04 3094 /usr/lib64/libz.so.1.2.7 7f04945e6000-7f04947e5000 ---p 00015000 08:04 3094 /usr/lib64/libz.so.1.2.7 7f04947e5000-7f04947e6000 r--p 00014000 08:04 3094 /usr/lib64/libz.so.1.2.7 7f04947e6000-7f04947e7000 rw-p 00015000 08:04 3094 /usr/lib64/libz.so.1.2.7 7f04947e7000-7f04947fe000 r-xp 00000000 08:04 35840 /usr/lib64/libgfxdr.so.0.0.1 7f04947fe000-7f04949fd000 ---p 00017000 08:04 35840 /usr/lib64/libgfxdr.so.0.0.1 7f04949fd000-7f04949fe000 r--p 00016000 08:04 35840 /usr/lib64/libgfxdr.so.0.0.1 7f04949fe000-7f04949ff000 rw-p 00017000 08:04 35840 /usr/lib64/libgfxdr.so.0.0.1 7f04949ff000-7f0494a19000 r-xp 00000000 08:04 127607 /usr/lib64/libgfrpc.so.0.0.1 7f0494a19000-7f0494c18000 ---p 0001a000 08:04 127607 /usr/lib64/libgfrpc.so.0.0.1 7f0494c18000-7f0494c19000 r--p 00019000 08:04 127607 /usr/lib64/libgfrpc.so.0.0.1 7f0494c19000-7f0494c1a000 rw-p 0001a000 08:04 127607 /usr/lib64/libgfrpc.so.0.0.1 7f0494c1a000-7f0494cd5000 r-xp 00000000 08:04 39467 /usr/lib64/libglusterfs.so.0.0.1 7f0494cd5000-7f0494ed5000 ---p 000bb000 08:04 39467 /usr/lib64/libglusterfs.so.0.0.1 7f0494ed5000-7f0494ed7000 r--p 000bb000 08:04 39467 /usr/lib64/libglusterfs.so.0.0.1 7f0494ed7000-7f0494ed8000 rw-p 000bd000 08:04 39467 /usr/lib64/libglusterfs.so.0.0.1 7f0494ed8000-7f0494edb000 rw-p 00000000 00:00 0 7f0494edb000-7f0494efc000 r-xp 00000000 08:04 2993 /usr/lib64/ld-2.17.so 7f0494efd000-7f04950f2000 rw-p 00000000 00:00 0 7f04950f2000-7f04950f9000 r--s 00000000 08:04 67111173 /usr/lib64/gconv/gconv-modules.cache 7f04950f9000-7f04950fc000 rw-p 00000000 00:00 0 7f04950fc000-7f04950fd000 r--p 00021000 08:04 2993 /usr/lib64/ld-2.17.so 7f04950fd000-7f04950fe000 rw-p 00022000 08:04 2993 /usr/lib64/ld-2.17.so 7f04950fe000-7f04950ff000 rw-p 00000000 00:00 0 7f04950ff000-7f0495113000 r-xp 00000000 08:04 35418 /usr/sbin/glusterfsd 7f0495313000-7f0495314000 r--p 00014000 08:04 35418 /usr/sbin/glusterfsd 7f0495314000-7f0495316000 rw-p 00015000 08:04 35418 /usr/sbin/glusterfsd 7f0496c8d000-7f0496da3000 rw-p 00000000 00:00 0 [heap] 7fff0b610000-7fff0b631000 rw-p 00000000 00:00 0 [stack] 7fff0b714000-7fff0b716000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] [2015-06-04 07:24:07.101320] D [logging.c:1765:gf_log_flush_extra_msgs] 0-logging-infra: Log buffer size reduced. About to flush 5 extra log messages The message "D [MSGID: 0] [glusterd-svc-mgmt.c:116:glusterd_svc_init_common] 0-management: Returning 0" repeated 4 times between [2015-06-04 07:24:04.537856] and [2015-06-04 07:24:04.539461] [2015-06-04 07:24:07.101384] D [logging.c:1768:gf_log_flush_extra_msgs] 0-logging-infra: Just flushed 5 extra log messages pending frames: frame : type(0) op(0) patchset: git://git.gluster.com/glusterfs.git signal received: 6 time of crash: 2015-06-04 07:24:07 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.7.1 /lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xb2)[0x7f0494c3fd92] /lib64/libglusterfs.so.0(gf_print_trace+0x32d)[0x7f0494c5a9ed] /lib64/libc.so.6(+0x35650)[0x7f0493631650] /lib64/libc.so.6(gsignal+0x37)[0x7f04936315d7] /lib64/libc.so.6(abort+0x148)[0x7f0493632cc8] /lib64/libc.so.6(+0x75e07)[0x7f0493671e07] /lib64/libc.so.6(__fortify_fail+0x37)[0x7f0493709a57] /lib64/libc.so.6(+0x10bc10)[0x7f0493707c10] /lib64/libc.so.6(+0x10b32b)[0x7f049370732b] /lib64/libc.so.6(__snprintf_chk+0x78)[0x7f0493707248] /usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(glusterd_volume_defrag_restart+0x191)[0x7f0489ad2931] /usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(glusterd_restart_rebalance+0x82)[0x7f0489ad8aa2] /usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(glusterd_spawn_daemons+0x4f)[0x7f0489ad8b1f] /lib64/libglusterfs.so.0(synctask_wrap+0x12)[0x7f0494c7a482] /lib64/libc.so.6(+0x470f0)[0x7f04936430f0] --------- Aborted (core dumped)
Can you please attach the coredump if it is available?
Created attachment 1034608 [details] Core dump of glusterd This is the core dump file created when running on the console of node0 (while node1 is up'n'running) glusterd -LDEBUG --debug
REVIEW: http://review.gluster.org/11090 (glusterd: Buffer overflow causing crash for glusterd) posted (#4) for review on master by Atin Mukherjee (amukherj)
REVIEW: http://review.gluster.org/11090 (glusterd: Buffer overflow causing crash for glusterd) posted (#6) for review on master by Atin Mukherjee (amukherj)
COMMIT: http://review.gluster.org/11090 committed in master by Vijay Bellur (vbellur) ------ commit 1ea9c9a3d59d8c3ddf1e27e2939c136a4ed3c6e8 Author: Susant Palai <spalai> Date: Thu Jun 4 22:37:11 2015 +0530 glusterd: Buffer overflow causing crash for glusterd Problem: In GLUSTERD_GET_DEFRAG_PROCESS we are using PATH_MAX (4096) as the max size of the input for target path, but we have allocated NAME_MAX (255) size of buffer for the target. Now this crash is not seen with source install, but is seen with RPMS. The reason is _foritfy_fail. This check happens when _FORTIFY_SOURCE flag is enabled. This option tries to figure out possible overflow scenarios like the bug here and does crash the process. Change-Id: I26261be85936d2e94a526fdcaa8d3249f8af11c3 BUG: 1228093 Signed-off-by: Susant Palai <spalai> Reviewed-on: http://review.gluster.org/11090 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: N Balachandran <nbalacha> Reviewed-by: Raghavendra G <rgowdapp>
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report. glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/ [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user