Bug 1228093 - Glusterd crash
Summary: Glusterd crash
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: glusterd
Version: mainline
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Susant Kumar Palai
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks: 1228541
TreeView+ depends on / blocked
 
Reported: 2015-06-04 08:01 UTC by Alexandros Soumplis
Modified: 2016-06-16 13:08 UTC (History)
3 users (show)

Fixed In Version: glusterfs-3.8rc2
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1228541 (view as bug list)
Environment:
Last Closed: 2016-06-16 13:08:04 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:


Attachments (Terms of Use)
Core dump of glusterd (653.18 KB, application/x-gzip)
2015-06-04 08:35 UTC, Alexandros Soumplis
no flags Details

Description Alexandros Soumplis 2015-06-04 08:01:28 UTC
Description of problem:

On a two node gluster with 4 distributed volumes after upgrade of glusterfs server to version 3.7.1 from 3.7.0 on one of the two nodes glusterd crashes with a buffer overflow error. This happens on node0 when it tries to communicate with node1. 

The nodes are exactly the same. Same hardware, same OS, same packages installed.

Version-Release number of selected component (if applicable): 3.7.1


How reproducible:
Always on our environment

Steps to Reproduce:
1. On node0:

[root@gnode00 ~]# systemctl start glusterd
[root@gnode00 ~]# systemctl status glusterd.service 
glusterd.service - GlusterFS, a clustered file-system server
   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled)
   Active: active (running) since Thu 2015-06-04 10:38:48 EEST; 7min ago
  Process: 12501 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid (code=exited, status=0/SUCCESS)
 Main PID: 12502 (glusterd)
   CGroup: /system.slice/glusterd.service
           └─12502 /usr/sbin/glusterd -p /var/run/glusterd.pid

Jun 04 10:38:48 gnode00 systemd[1]: Starting GlusterFS, a clustered file-system server...
Jun 04 10:38:48 gnode00 systemd[1]: Started GlusterFS, a clustered file-system server.

2. On node1:
[root@gnode01 ~]# systemctl start glusterd
[root@gnode01 ~]# systemctl status glusterd
glusterd.service - GlusterFS, a clustered file-system server
   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled)
   Active: active (running) since Thu 2015-06-04 10:54:55 EEST; 6s ago
  Process: 8179 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid (code=exited, status=0/SUCCESS)
 Main PID: 8180 (glusterd)
   CGroup: /system.slice/glusterd.service
           ├─8180 /usr/sbin/glusterd -p /var/run/glusterd.pid
           ├─8202 /usr/sbin/glusterfsd -s gnode01 --volfile-id mmphoto.gnode01.BRICKS-lv02a-mmphoto -p /var/lib/glusterd/vols/mmphoto/run/gnode01...
           ├─8209 /usr/sbin/glusterfsd -s gnode01 --volfile-id mmphoto.gnode01.BRICKS-lv02b-mmphoto -p /var/lib/glusterd/vols/mmphoto/run/gnode01...
           ├─8215 /usr/sbin/glusterfsd -s gnode01 --volfile-id phdms.gnode01.BRICKS-lv00-phdms -p /var/lib/glusterd/vols/phdms/run/gnode01-BRICKS...
           ├─8220 /usr/sbin/glusterfsd -s gnode01 --volfile-id restore.gnode01.BRICKS-lv03a-restore -p /var/lib/glusterd/vols/restore/run/gnode01...
           └─8225 /usr/sbin/glusterfsd -s gnode01 --volfile-id thesis.gnode01.BRICKS-lv01-thesis -p /var/lib/glusterd/vols/thesis/run/gnode01-BRI...

Jun 04 10:54:55 gnode01 systemd[1]: Starting GlusterFS, a clustered file-system server...
Jun 04 10:54:55 gnode01 systemd[1]: Started GlusterFS, a clustered file-system server.


3. On node0:

[root@gnode00 ~]# systemctl status glusterd.service 
glusterd.service - GlusterFS, a clustered file-system server
   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled)
   Active: failed (Result: signal) since Thu 2015-06-04 10:54:57 EEST; 1min 51s ago
  Process: 12501 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid (code=exited, status=0/SUCCESS)
 Main PID: 12502 (code=killed, signal=ABRT)

Jun 04 10:54:57 gnode00 etc-glusterfs-glusterd.vol[12502]: llistxattr 1
Jun 04 10:54:57 gnode00 etc-glusterfs-glusterd.vol[12502]: setfsid 1
Jun 04 10:54:57 gnode00 etc-glusterfs-glusterd.vol[12502]: spinlock 1
Jun 04 10:54:57 gnode00 etc-glusterfs-glusterd.vol[12502]: epoll.h 1
Jun 04 10:54:57 gnode00 etc-glusterfs-glusterd.vol[12502]: xattr.h 1
Jun 04 10:54:57 gnode00 etc-glusterfs-glusterd.vol[12502]: st_atim.tv_nsec 1
Jun 04 10:54:57 gnode00 etc-glusterfs-glusterd.vol[12502]: package-string: glusterfs 3.7.1
Jun 04 10:54:57 gnode00 etc-glusterfs-glusterd.vol[12502]: ---------
Jun 04 10:54:57 gnode00 systemd[1]: glusterd.service: main process exited, code=killed, status=6/ABRT
Jun 04 10:54:57 gnode00 systemd[1]: Unit glusterd.service entered failed state.


The log files has the following entry:
[2015-06-04 07:24:07.098158] D [rpc-transport.c:287:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.7.1/rpc-transport/socket.so
[2015-06-04 07:24:07.098247] D [socket.c:3816:socket_init] 0-management: disabling nodelay
[2015-06-04 07:24:07.098273] D [socket.c:3867:socket_init] 0-management: Configued transport.tcp-user-timeout=0
[2015-06-04 07:24:07.098294] D [socket.c:3938:socket_init] 0-management: SSL support on the I/O path is NOT enabled
[2015-06-04 07:24:07.098313] D [socket.c:3941:socket_init] 0-management: SSL support for glusterd is NOT enabled
[2015-06-04 07:24:07.098329] D [socket.c:3958:socket_init] 0-management: using system polling thread
[2015-06-04 07:24:07.098382] D [glusterd-handler.c:3153:glusterd_rpc_create] 0-management: returning 0
[2015-06-04 07:24:07.098408] D [glusterd-utils.c:1493:glusterd_brick_connect] 0-: Returning 0
[2015-06-04 07:24:07.098428] D [glusterd-utils.c:4387:glusterd_brick_start] 0-management: returning 0 
[2015-06-04 07:24:07.098453] D [glusterd-utils.c:4387:glusterd_brick_start] 0-management: returning 0 
*** buffer overflow detected ***: glusterd terminated
======= Backtrace: =========
/lib64/libc.so.6(__fortify_fail+0x37)[0x7f0493709a57]
/lib64/libc.so.6(+0x10bc10)[0x7f0493707c10]
/lib64/libc.so.6(+0x10b32b)[0x7f049370732b]
/lib64/libc.so.6(__snprintf_chk+0x78)[0x7f0493707248]
/usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(glusterd_volume_defrag_restart+0x191)[0x7f0489ad2931]
/usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(glusterd_restart_rebalance+0x82)[0x7f0489ad8aa2]
/usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(glusterd_spawn_daemons+0x4f)[0x7f0489ad8b1f]
/lib64/libglusterfs.so.0(synctask_wrap+0x12)[0x7f0494c7a482]
/lib64/libc.so.6(+0x470f0)[0x7f04936430f0]
======= Memory map: ========
7f0478000000-7f0478021000 rw-p 00000000 00:00 0 
7f0478021000-7f047c000000 ---p 00000000 00:00 0 
7f047c000000-7f047c023000 rw-p 00000000 00:00 0 
7f047c023000-7f0480000000 ---p 00000000 00:00 0 
7f0480000000-7f0480021000 rw-p 00000000 00:00 0 
7f0480021000-7f0484000000 ---p 00000000 00:00 0 
7f0484b35000-7f0484d36000 rw-p 00000000 00:00 0                          [stack:12401]
7f0484d36000-7f0484d37000 ---p 00000000 00:00 0 
7f0484d37000-7f0485537000 rw-p 00000000 00:00 0                          [stack:12405]
7f0485537000-7f0485538000 ---p 00000000 00:00 0 
7f0485538000-7f0485d38000 rw-p 00000000 00:00 0                          [stack:12404]
7f0485d38000-7f0485d3d000 r-xp 00000000 08:04 3016                       /usr/lib64/libnss_dns-2.17.so
7f0485d3d000-7f0485f3c000 ---p 00005000 08:04 3016                       /usr/lib64/libnss_dns-2.17.so
7f0485f3c000-7f0485f3d000 r--p 00004000 08:04 3016                       /usr/lib64/libnss_dns-2.17.so
7f0485f3d000-7f0485f3e000 rw-p 00005000 08:04 3016                       /usr/lib64/libnss_dns-2.17.so
7f0485f3e000-7f0485f49000 r-xp 00000000 08:04 3018                       /usr/lib64/libnss_files-2.17.so
7f0485f49000-7f0486148000 ---p 0000b000 08:04 3018                       /usr/lib64/libnss_files-2.17.so
7f0486148000-7f0486149000 r--p 0000a000 08:04 3018                       /usr/lib64/libnss_files-2.17.so
7f0486149000-7f048614a000 rw-p 0000b000 08:04 3018                       /usr/lib64/libnss_files-2.17.so
7f048614a000-7f04862f5000 rw-p 00000000 00:00 0 
7f04862f5000-7f048630b000 r-xp 00000000 08:04 3028                       /usr/lib64/libresolv-2.17.so
7f048630b000-7f048650b000 ---p 00016000 08:04 3028                       /usr/lib64/libresolv-2.17.so
7f048650b000-7f048650c000 r--p 00016000 08:04 3028                       /usr/lib64/libresolv-2.17.so
7f048650c000-7f048650d000 rw-p 00017000 08:04 3028                       /usr/lib64/libresolv-2.17.so
7f048650d000-7f048650f000 rw-p 00000000 00:00 0 
7f048650f000-7f0486512000 r-xp 00000000 08:04 3374                       /usr/lib64/libkeyutils.so.1.5
7f0486512000-7f0486711000 ---p 00003000 08:04 3374                       /usr/lib64/libkeyutils.so.1.5
7f0486711000-7f0486712000 r--p 00002000 08:04 3374                       /usr/lib64/libkeyutils.so.1.5
7f0486712000-7f0486713000 rw-p 00003000 08:04 3374                       /usr/lib64/libkeyutils.so.1.5
7f0486713000-7f0486720000 r-xp 00000000 08:04 35693                      /usr/lib64/libkrb5support.so.0.1
7f0486720000-7f0486920000 ---p 0000d000 08:04 35693                      /usr/lib64/libkrb5support.so.0.1
7f0486920000-7f0486921000 r--p 0000d000 08:04 35693                      /usr/lib64/libkrb5support.so.0.1
7f0486921000-7f0486922000 rw-p 0000e000 08:04 35693                      /usr/lib64/libkrb5support.so.0.1
7f0486922000-7f0486951000 r-xp 00000000 08:04 35681                      /usr/lib64/libk5crypto.so.3.1
7f0486951000-7f0486b50000 ---p 0002f000 08:04 35681                      /usr/lib64/libk5crypto.so.3.1
7f0486b50000-7f0486b52000 r--p 0002e000 08:04 35681                      /usr/lib64/libk5crypto.so.3.1
7f0486b52000-7f0486b53000 rw-p 00030000 08:04 35681                      /usr/lib64/libk5crypto.so.3.1
7f0486b53000-7f0486b54000 rw-p 00000000 00:00 0 
7f0486b54000-7f0486b57000 r-xp 00000000 08:04 3157                       /usr/lib64/libcom_err.so.2.1
7f0486b57000-7f0486d56000 ---p 00003000 08:04 3157                       /usr/lib64/libcom_err.so.2.1
7f0486d56000-7f0486d57000 r--p 00002000 08:04 3157                       /usr/lib64/libcom_err.so.2.1
7f0486d57000-7f0486d58000 rw-p 00003000 08:04 3157                       /usr/lib64/libcom_err.so.2.1
7f0486d58000-7f0486e2b000 r-xp 00000000 08:04 35691                      /usr/lib64/libkrb5.so.3.3
7f0486e2b000-7f048702b000 ---p 000d3000 08:04 35691                      /usr/lib64/libkrb5.so.3.3
7f048702b000-7f0487038000 r--p 000d3000 08:04 35691                      /usr/lib64/libkrb5.so.3.3
7f0487038000-7f048703b000 rw-p 000e0000 08:04 35691                      /usr/lib64/libkrb5.so.3.3
7f048703b000-7f0487084000 r-xp 00000000 08:04 4350                       /usr/lib64/libgssapi_krb5.so.2.2
7f0487084000-7f0487284000 ---p 00049000 08:04 4350                       /usr/lib64/libgssapi_krb5.so.2.2
7f0487284000-7f0487285000 r--p 00049000 08:04 4350                       /usr/lib64/libgssapi_krb5.so.2.2
7f0487285000-7f0487287000 rw-p 0004a000 08:04 4350                       /usr/lib64/libgssapi_krb5.so.2.2
7f0487287000-7f04872e9000 r-xp 00000000 08:04 35705                      /usr/lib64/libssl.so.1.0.1e
7f04872e9000-7f04874e9000 ---p 00062000 08:04 35705                      /usr/lib64/libssl.so.1.0.1e
7f04874e9000-7f04874ed000 r--p 00062000 08:04 35705                      /usr/lib64/libssl.so.1.0.1e
7f04874ed000-7f04874f4000 rw-p 00066000 08:04 35705                      /usr/lib64/libssl.so.1.0.1e
7f04874f4000-7f0487505000 r-xp 00000000 08:04 135881318                  /usr/lib64/glusterfs/3.7.1/rpc-transport/socket.so
7f0487505000-7f0487704000 ---p 00011000 08:04 135881318                  /usr/lib64/glusterfs/3.7.1/rpc-transport/socket.so
7f0487704000-7f0487705000 r--p 00010000 08:04 135881318                  /usr/lib64/glusterfs/3.7.1/rpc-transport/socket.so
7f0487705000-7f0487709000 rw-p 00011000 08:04 135881318                  /usr/lib64/glusterfs/3.7.1/rpc-transport/socket.so
7f0487709000-7f0487736000 rw-p 00000000 00:00 0 
7f0487736000-7f0487795000 r-xp 00000000 08:04 3080                       /usr/lib64/libpcre.so.1.2.0
7f0487795000-7f0487995000 ---p 0005f000 08:04 3080                       /usr/lib64/libpcre.so.1.2.0
7f0487995000-7f0487996000 r--p 0005f000 08:04 3080                       /usr/lib64/libpcre.so.1.2.0
7f0487996000-7f0487997000 rw-p 00060000 08:04 3080                       /usr/lib64/libpcre.so.1.2.0
7f0487997000-7f04879db000 r-xp 00000000 08:04 3054                       /usr/lib64/libsepol.so.1
7f04879db000-7f0487bda000 ---p 00044000 08:04 3054                       /usr/lib64/libsepol.so.1
7f0487bda000-7f0487bdb000 r--p 00043000 08:04 3054                       /usr/lib64/libsepol.so.1
7f0487bdb000-7f0487bdc000 rw-p 00044000 08:04 3054                       /usr/lib64/libsepol.so.1
7f0487bdc000-7f0487bf1000 r-xp 00000000 08:04 209                        /usr/lib64/libgcc_s-4.8.3-20140911.so.1
7f0487bf1000-7f0487df0000 ---p 00015000 08:04 209                        /usr/lib64/libgcc_s-4.8.3-20140911.so.1
7f0487df0000-7f0487df1000 r--p 00014000 08:04 209                        /usr/lib64/libgcc_s-4.8.3-20140911.so.1
7f0487df1000-7f0487df2000 rw-p 00015000 08:04 209                        /usr/lib64/libgcc_s-4.8.3-20140911.so.1
7f0487df2000-7f0487e13000 r-xp 00000000 08:04 3091                       /usr/lib64/libselinux.so.1
7f0487e13000-7f0488013000 ---p 00021000 08:04 3091                       /usr/lib64/libselinux.so.1
7f0488013000-7f0488014000 r--p 00021000 08:04 3091                       /usr/lib64/libselinux.so.1
7f0488014000-7f0488015000 rw-p 00022000 08:04 3091                       /usr/lib64/libselinux.so.1
7f0488015000-7f0488017000 rw-p 00000000 00:00 0 
7f0488017000-7f048801c000 r-xp 00000000 08:04 1647807                    /usr/lib64/libdevmapper-event.so.1.02
7f048801c000-7f048821c000 ---p 00005000 08:04 1647807                    /usr/lib64/libdevmapper-event.so.1.02
7f048821c000-7f048821d000 r--p 00005000 08:04 1647807                    /usr/lib64/libdevmapper-event.so.1.02
7f048821d000-7f048821e000 rw-p 00006000 08:04 1647807                    /usr/lib64/libdevmapper-event.so.1.02
7f048821e000-7f0488263000 r-xp 00000000 08:04 206103                     /usr/lib64/libdevmapper.so.1.02
7f0488263000-7f0488462000 ---p 00045000 08:04 206103                     /usr/lib64/libdevmapper.so.1.02
7f0488462000-7f0488463000 r--p 00044000 08:04 206103                     /usr/lib64/libdevmapper.so.1.02
7f0488463000-7f0488466000 rw-p 00045000 08:04 206103                     /usr/lib64/libdevmapper.so.1.02
7f0488466000-7f0488467000 rw-p 00000000 00:00 0 
7f0488467000-7f048849f000 r-xp 00000000 08:04 131775                     /usr/lib64/libblkid.so.1.1.0
7f048849f000-7f048869e000 ---p 00038000 08:04 131775                     /usr/lib64/libblkid.so.1.1.0
7f048869e000-7f04886a1000 r--p 00037000 08:04 131775                     /usr/lib64/libblkid.so.1.1.0
7f04886a1000-7f04886a2000 rw-p 0003a000 08:04 131775                     /usr/lib64/libblkid.so.1.1.0
7f04886a2000-7f04886a3000 rw-p 00000000 00:00 0 
7f04886a3000-7f04886b4000 r-xp 00000000 08:04 35842                      /usr/lib64/libudev.so.1.4.0
7f04886b4000-7f04888b3000 ---p 00011000 08:04 35842                      /usr/lib64/libudev.so.1.4.0
7f04888b3000-7f04888b4000 r--p 00010000 08:04 35842                      /usr/lib64/libudev.so.1.4.0
7f04888b4000-7f04888b5000 rw-p 00011000 08:04 35842                      /usr/lib64/libudev.so.1.4.0
7f04888b5000-7f04888b7000 r-xp 00000000 08:04 35806                      /usr/lib64/liburcu-common.so.1.0.0
7f04888b7000-7f0488ab6000 ---p 00002000 08:04 35806                      /usr/lib64/liburcu-common.so.1.0.0
7f0488ab6000-7f0488ab7000 r--p 00001000 08:04 35806                      /usr/lib64/liburcu-common.so.1.0.0
7f0488ab7000-7f0488ab8000 rw-p 00002000 08:04 35806                      /usr/lib64/liburcu-common.so.1.0.0
7f0488ab8000-7f0488adc000 r-xp 00000000 08:04 3047                       /usr/lib64/liblzma.so.5.0.99
7f0488adc000-7f0488cdb000 ---p 00024000 08:04 3047                       /usr/lib64/liblzma.so.5.0.99
7f0488cdb000-7f0488cdc000 r--p 00023000 08:04 3047                       /usr/lib64/liblzma.so.5.0.99
7f0488cdc000-7f0488cdd000 rw-p 00024000 08:04 3047                       /usr/lib64/liblzma.so.5.0.99
7f0488cdd000-7f0488dba000 r-xp 00000000 08:04 275744                     /usr/lib64/liblvm2app.so.2.2
7f0488dba000-7f0488fb9000 ---p 000dd000 08:04 275744                     /usr/lib64/liblvm2app.so.2.2
7f0488fb9000-7f0488fba000 r--p 000dc000 08:04 275744                     /usr/lib64/liblvm2app.so.2.2
7f0488fba000-7f0488fc0000 rw-p 000dd000 08:04 275744                     /usr/lib64/liblvm2app.so.2.2
7f0488fc0000-7f0488fdc000 rw-p 00000000 00:00 0 
7f0488fdc000-7f0488fe2000 r-xp 00000000 08:04 249077                     /usr/lib64/liburcu-cds.so.1.0.0
7f0488fe2000-7f04891e1000 ---p 00006000 08:04 249077                     /usr/lib64/liburcu-cds.so.1.0.0
7f04891e1000-7f04891e2000 r--p 00005000 08:04 249077                     /usr/lib64/liburcu-cds.so.1.0.0
7f04891e2000-7f04891e3000 rw-p 00006000 08:04 249077                     /usr/lib64/liburcu-cds.so.1.0.0
7f04891e3000-7f04891e9000 r-xp 00000000 08:04 249075                     /usr/lib64/liburcu-bp.so.1.0.0
7f04891e9000-7f04893e8000 ---p 00006000 08:04 249075                     /usr/lib64/liburcu-bp.so.1.0.0
7f04893e8000-7f04893e9000 r--p 00005000 08:04 249075                     /usr/lib64/liburcu-bp.so.1.0.0
7f04893e9000-7f04893ea000 rw-p 00006000 08:04 249075                     /usr/lib64/liburcu-bp.so.1.0.0
7f04893ea000-7f04894eb000 r-xp 00000000 08:04 3008                       /usr/lib64/libm-2.17.so
7f04894eb000-7f04896ea000 ---p 00101000 08:04 3008                       /usr/lib64/libm-2.17.so
7f04896ea000-7f04896eb000 r--p 00100000 08:04 3008                       /usr/lib64/libm-2.17.so
7f04896eb000-7f04896ec000 rw-p 00101000 08:04 3008                       /usr/lib64/libm-2.17.so
7f04896ec000-7f048984a000 r-xp 00000000 08:04 1434624                    /usr/lib64/libxml2.so.2.9.1
7f048984a000-7f0489a49000 ---p 0015e000 08:04 1434624                    /usr/lib64/libxml2.so.2.9.1
7f0489a49000-7f0489a51000 r--p 0015d000 08:04 1434624                    /usr/lib64/libxml2.so.2.9.1
7f0489a51000-7f0489a53000 rw-p 00165000 08:04 1434624                    /usr/lib64/libxml2.so.2.9.1
7f0489a53000-7f0489a55000 rw-p 00000000 00:00 0 
7f0489a5e000-7f0489bb5000 r-xp 00000000 08:04 135881320                  /usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so
7f0489bb5000-7f0489db5000 ---p 00157000 08:04 135881320                  /usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so
7f0489db5000-7f0489db6000 r--p 00157000 08:04 135881320                  /usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so
7f0489db6000-7f0489dc7000 rw-p 00158000 08:04 135881320                  /usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so
7f0489dc7000-7f0489dc8000 rw-p 00000000 00:00 0 
7f0489dc8000-7f0489dc9000 ---p 00000000 00:00 0 
7f0489dc9000-7f048a5c9000 rw-p 00000000 00:00 0                          [stack:12402]
7f048a5c9000-7f048a5ca000 ---p 00000000 00:00 0 
7f048a5ca000-7f048adca000 rw-p 00000000 00:00 0 
7f048adca000-7f048adcb000 ---p 00000000 00:00 0 
7f048adcb000-7f048b5cb000 rw-p 00000000 00:00 0                          [stack:12400]
7f048b5cb000-7f048b5cc000 ---p 00000000 00:00 0 
7f048b5cc000-7f048bdcc000 rw-p 00000000 00:00 0                          [stack:12399]
7f048bdcc000-7f04922f3000 r--p 00000000 08:04 201329209                  /usr/lib/locale/locale-archive
7f04922f3000-7f04935fc000 rw-p 00000000 00:00 0 
7f04935fc000-7f04937b2000 r-xp 00000000 08:04 3000                       /usr/lib64/libc-2.17.so
7f04937b2000-7f04939b2000 ---p 001b6000 08:04 3000                       /usr/lib64/libc-2.17.so
7f04939b2000-7f04939b6000 r--p 001b6000 08:04 3000                       /usr/lib64/libc-2.17.so
7f04939b6000-7f04939b8000 rw-p 001ba000 08:04 3000                       /usr/lib64/libc-2.17.so
7f04939b8000-7f04939bd000 rw-p 00000000 00:00 0 
7f04939bd000-7f0493b7a000 r-xp 00000000 08:04 4346                       /usr/lib64/libcrypto.so.1.0.1e
7f0493b7a000-7f0493d7a000 ---p 001bd000 08:04 4346                       /usr/lib64/libcrypto.so.1.0.1e
7f0493d7a000-7f0493d94000 r--p 001bd000 08:04 4346                       /usr/lib64/libcrypto.so.1.0.1e
7f0493d94000-7f0493da0000 rw-p 001d7000 08:04 4346                       /usr/lib64/libcrypto.so.1.0.1e
7f0493da0000-7f0493da4000 rw-p 00000000 00:00 0 
7f0493da4000-7f0493dba000 r-xp 00000000 08:04 3026                       /usr/lib64/libpthread-2.17.so
7f0493dba000-7f0493fba000 ---p 00016000 08:04 3026                       /usr/lib64/libpthread-2.17.so
7f0493fba000-7f0493fbb000 r--p 00016000 08:04 3026                       /usr/lib64/libpthread-2.17.so
7f0493fbb000-7f0493fbc000 rw-p 00017000 08:04 3026                       /usr/lib64/libpthread-2.17.so
7f0493fbc000-7f0493fc0000 rw-p 00000000 00:00 0 
7f0493fc0000-7f0493fc3000 r-xp 00000000 08:04 3006                       /usr/lib64/libdl-2.17.so
7f0493fc3000-7f04941c2000 ---p 00003000 08:04 3006                       /usr/lib64/libdl-2.17.so
7f04941c2000-7f04941c3000 r--p 00002000 08:04 3006                       /usr/lib64/libdl-2.17.so
7f04941c3000-7f04941c4000 rw-p 00003000 08:04 3006                       /usr/lib64/libdl-2.17.so
7f04941c4000-7f04941cb000 r-xp 00000000 08:04 3030                       /usr/lib64/librt-2.17.so
7f04941cb000-7f04943ca000 ---p 00007000 08:04 3030                       /usr/lib64/librt-2.17.so
7f04943ca000-7f04943cb000 r--p 00006000 08:04 3030                       /usr/lib64/librt-2.17.so
7f04943cb000-7f04943cc000 rw-p 00007000 08:04 3030                       /usr/lib64/librt-2.17.so
7f04943cc000-7f04943d0000 r-xp 00000000 08:04 131774                     /usr/lib64/libuuid.so.1.3.0
7f04943d0000-7f04945cf000 ---p 00004000 08:04 131774                     /usr/lib64/libuuid.so.1.3.0
7f04945cf000-7f04945d0000 r--p 00003000 08:04 131774                     /usr/lib64/libuuid.so.1.3.0
7f04945d0000-7f04945d1000 rw-p 00004000 08:04 131774                     /usr/lib64/libuuid.so.1.3.0
7f04945d1000-7f04945e6000 r-xp 00000000 08:04 3094                       /usr/lib64/libz.so.1.2.7
7f04945e6000-7f04947e5000 ---p 00015000 08:04 3094                       /usr/lib64/libz.so.1.2.7
7f04947e5000-7f04947e6000 r--p 00014000 08:04 3094                       /usr/lib64/libz.so.1.2.7
7f04947e6000-7f04947e7000 rw-p 00015000 08:04 3094                       /usr/lib64/libz.so.1.2.7
7f04947e7000-7f04947fe000 r-xp 00000000 08:04 35840                      /usr/lib64/libgfxdr.so.0.0.1
7f04947fe000-7f04949fd000 ---p 00017000 08:04 35840                      /usr/lib64/libgfxdr.so.0.0.1
7f04949fd000-7f04949fe000 r--p 00016000 08:04 35840                      /usr/lib64/libgfxdr.so.0.0.1
7f04949fe000-7f04949ff000 rw-p 00017000 08:04 35840                      /usr/lib64/libgfxdr.so.0.0.1
7f04949ff000-7f0494a19000 r-xp 00000000 08:04 127607                     /usr/lib64/libgfrpc.so.0.0.1
7f0494a19000-7f0494c18000 ---p 0001a000 08:04 127607                     /usr/lib64/libgfrpc.so.0.0.1
7f0494c18000-7f0494c19000 r--p 00019000 08:04 127607                     /usr/lib64/libgfrpc.so.0.0.1
7f0494c19000-7f0494c1a000 rw-p 0001a000 08:04 127607                     /usr/lib64/libgfrpc.so.0.0.1
7f0494c1a000-7f0494cd5000 r-xp 00000000 08:04 39467                      /usr/lib64/libglusterfs.so.0.0.1
7f0494cd5000-7f0494ed5000 ---p 000bb000 08:04 39467                      /usr/lib64/libglusterfs.so.0.0.1
7f0494ed5000-7f0494ed7000 r--p 000bb000 08:04 39467                      /usr/lib64/libglusterfs.so.0.0.1
7f0494ed7000-7f0494ed8000 rw-p 000bd000 08:04 39467                      /usr/lib64/libglusterfs.so.0.0.1
7f0494ed8000-7f0494edb000 rw-p 00000000 00:00 0 
7f0494edb000-7f0494efc000 r-xp 00000000 08:04 2993                       /usr/lib64/ld-2.17.so
7f0494efd000-7f04950f2000 rw-p 00000000 00:00 0 
7f04950f2000-7f04950f9000 r--s 00000000 08:04 67111173                   /usr/lib64/gconv/gconv-modules.cache
7f04950f9000-7f04950fc000 rw-p 00000000 00:00 0 
7f04950fc000-7f04950fd000 r--p 00021000 08:04 2993                       /usr/lib64/ld-2.17.so
7f04950fd000-7f04950fe000 rw-p 00022000 08:04 2993                       /usr/lib64/ld-2.17.so
7f04950fe000-7f04950ff000 rw-p 00000000 00:00 0 
7f04950ff000-7f0495113000 r-xp 00000000 08:04 35418                      /usr/sbin/glusterfsd
7f0495313000-7f0495314000 r--p 00014000 08:04 35418                      /usr/sbin/glusterfsd
7f0495314000-7f0495316000 rw-p 00015000 08:04 35418                      /usr/sbin/glusterfsd
7f0496c8d000-7f0496da3000 rw-p 00000000 00:00 0                          [heap]
7fff0b610000-7fff0b631000 rw-p 00000000 00:00 0                          [stack]
7fff0b714000-7fff0b716000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
[2015-06-04 07:24:07.101320] D [logging.c:1765:gf_log_flush_extra_msgs] 0-logging-infra: Log buffer size reduced. About to flush 5 extra log messages
The message "D [MSGID: 0] [glusterd-svc-mgmt.c:116:glusterd_svc_init_common] 0-management: Returning 0" repeated 4 times between [2015-06-04 07:24:04.537856] and [2015-06-04 07:24:04.539461]
[2015-06-04 07:24:07.101384] D [logging.c:1768:gf_log_flush_extra_msgs] 0-logging-infra: Just flushed 5 extra log messages
pending frames:
frame : type(0) op(0)
patchset: git://git.gluster.com/glusterfs.git
signal received: 6
time of crash: 
2015-06-04 07:24:07
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.7.1
/lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xb2)[0x7f0494c3fd92]
/lib64/libglusterfs.so.0(gf_print_trace+0x32d)[0x7f0494c5a9ed]
/lib64/libc.so.6(+0x35650)[0x7f0493631650]
/lib64/libc.so.6(gsignal+0x37)[0x7f04936315d7]
/lib64/libc.so.6(abort+0x148)[0x7f0493632cc8]
/lib64/libc.so.6(+0x75e07)[0x7f0493671e07]
/lib64/libc.so.6(__fortify_fail+0x37)[0x7f0493709a57]
/lib64/libc.so.6(+0x10bc10)[0x7f0493707c10]
/lib64/libc.so.6(+0x10b32b)[0x7f049370732b]
/lib64/libc.so.6(__snprintf_chk+0x78)[0x7f0493707248]
/usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(glusterd_volume_defrag_restart+0x191)[0x7f0489ad2931]
/usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(glusterd_restart_rebalance+0x82)[0x7f0489ad8aa2]
/usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(glusterd_spawn_daemons+0x4f)[0x7f0489ad8b1f]
/lib64/libglusterfs.so.0(synctask_wrap+0x12)[0x7f0494c7a482]
/lib64/libc.so.6(+0x470f0)[0x7f04936430f0]
---------
Aborted (core dumped)

Comment 1 Nithya Balachandran 2015-06-04 08:18:23 UTC
Can you please attach the coredump if it is available?

Comment 2 Alexandros Soumplis 2015-06-04 08:35:00 UTC
Created attachment 1034608 [details]
Core dump of glusterd

This is the core dump file created when running on the console of node0 (while node1 is up'n'running)
glusterd -LDEBUG --debug

Comment 3 Anand Avati 2015-06-05 04:04:53 UTC
REVIEW: http://review.gluster.org/11090 (glusterd: Buffer overflow causing crash for glusterd) posted (#4) for review on master by Atin Mukherjee (amukherj@redhat.com)

Comment 4 Anand Avati 2015-06-09 08:34:27 UTC
REVIEW: http://review.gluster.org/11090 (glusterd: Buffer overflow causing crash for glusterd) posted (#6) for review on master by Atin Mukherjee (amukherj@redhat.com)

Comment 5 Anand Avati 2015-06-13 08:13:59 UTC
COMMIT: http://review.gluster.org/11090 committed in master by Vijay Bellur (vbellur@redhat.com) 
------
commit 1ea9c9a3d59d8c3ddf1e27e2939c136a4ed3c6e8
Author: Susant Palai <spalai@redhat.com>
Date:   Thu Jun 4 22:37:11 2015 +0530

    glusterd: Buffer overflow causing crash for glusterd
    
    Problem: In GLUSTERD_GET_DEFRAG_PROCESS we are using PATH_MAX (4096)
    as the max size of the input for target path, but we have allocated
    NAME_MAX (255) size of buffer for the target.
    
    Now this crash is not seen with source install, but is seen with RPMS.
    The reason is _foritfy_fail. This check happens when _FORTIFY_SOURCE
    flag is enabled. This option tries to figure out possible
    overflow scenarios like the bug here and does crash the process.
    
    Change-Id: I26261be85936d2e94a526fdcaa8d3249f8af11c3
    BUG: 1228093
    Signed-off-by: Susant Palai <spalai@redhat.com>
    Reviewed-on: http://review.gluster.org/11090
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: N Balachandran <nbalacha@redhat.com>
    Reviewed-by: Raghavendra G <rgowdapp@redhat.com>

Comment 6 Mike McCune 2016-03-28 23:36:13 UTC
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune@redhat.com with any questions

Comment 7 Niels de Vos 2016-06-16 13:08:04 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report.

glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user


Note You need to log in before you can comment on or make changes to this bug.