Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 837684

Summary: Most commands cause glusterd 3.2.7 to crash on Debian Wheezy
Product: [Community] GlusterFS Reporter: Louis Zuckerman <glusterbugs>
Component: cliAssignee: Kaushal <kaushal>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: medium Docs Contact:
Priority: medium    
Version: 3.2.7CC: amarts, glusterbugs, gluster-bugs, kkeithle, pmatthaei
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Other   
Whiteboard:
Fixed In Version: glusterfs-3.4.0 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-07-24 17:30:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Louis Zuckerman 2012-07-04 18:28:07 UTC
Description of problem:

Glusterd dies during handling of most gluster CLI commands (vol create, start, stop, delete, at least)

Version-Release number of selected component (if applicable):

Glusterd v. 3.2.7

How reproducible:

Easy to reproduce & consistent.

Steps to Reproduce:
1. Build glusterfs on Debian Sid/Wheezy or install from Debian repository
2. Try to create a volume (gluster volume create foo thishost:/var/tmp/foo0)
3. The volume gets created but glusterd dies in the process
  
Actual results:

"Connection failed. Please check if gluster daemon is operational."

Starting glusterd up again you can run 'gluster volume info' and see the volume was created.  You can then start the volume, but glusterd will crash once more.  Same for stopping & deleting the volume: glusterd crashes after each operation, but operations appear to have worked after restarting glusterd & running 'gluster volume info' (which does not crash glusterd.)

Expected results:

Normal "volume created" message reminding me to start the volume.

Additional info:

Original bug report in Debian bug tracker:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=679767

Crash dump in the glusterd log file from my (successful) attempt to reproduce:

patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash: 2012-07-04 14:11:41
configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.2.7
/lib/x86_64-linux-gnu/libc.so.6(+0x324f0)[0x7faf8f3744f0]
/lib/x86_64-linux-gnu/libc.so.6(+0x7fbf1)[0x7faf8f3c1bf1]
/lib/x86_64-linux-gnu/libc.so.6(xdr_string+0x37)[0x7faf8f444357]
/usr/lib/libgfxdr.so.0(xdr_gf1_cli_create_vol_rsp+0x51)[0x7faf8faf6061]
/usr/lib/libgfrpc.so.0(xdr_serialize_generic+0x41)[0x7faf8fd0ff11]
/usr/lib/glusterfs/3.2.7/xlator/mgmt/glusterd.so(glusterd_serialize_reply+0x56)[0x7faf8daf1906]
/usr/lib/glusterfs/3.2.7/xlator/mgmt/glusterd.so(glusterd_submit_reply+0x65)[0x7faf8daf1a25]
/usr/lib/glusterfs/3.2.7/xlator/mgmt/glusterd.so(glusterd_op_send_cli_response+0x8f)[0x7faf8db0027f]
/usr/lib/glusterfs/3.2.7/xlator/mgmt/glusterd.so(glusterd_op_txn_complete+0xcc)[0x7faf8daef84c]
/usr/lib/glusterfs/3.2.7/xlator/mgmt/glusterd.so(+0x4bb39)[0x7faf8daefb39]
/usr/lib/glusterfs/3.2.7/xlator/mgmt/glusterd.so(glusterd_op_sm+0xe2)[0x7faf8dae89d2]
/usr/lib/glusterfs/3.2.7/xlator/mgmt/glusterd.so(glusterd_handle_create_volume+0x290)[0x7faf8dad0f60]
/usr/lib/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x27f)[0x7faf8fd071ff]
/usr/lib/libgfrpc.so.0(rpcsvc_notify+0x7b)[0x7faf8fd0778b]
/usr/lib/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7faf8fd0af13]
/usr/lib/glusterfs/3.2.7/rpc-transport/socket.so(socket_event_poll_in+0x34)[0x7faf8d89b224]
/usr/lib/glusterfs/3.2.7/rpc-transport/socket.so(socket_event_handler+0xb5)[0x7faf8d89b565]
/usr/lib/libglusterfs.so.0(+0x404c8)[0x7faf8ff564c8]
/usr/sbin/glusterd(main+0x29b)[0x7faf903a3a6b]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd)[0x7faf8f360ead]
/usr/sbin/glusterd(+0x4cc9)[0x7faf903a3cc9]

Comment 1 Amar Tumballi 2012-07-11 04:59:36 UTC
Looks like an issue with gcc optimization flags. Working on it to get more clarity.

As a work around, giving 'CFLAGS=-O0' fixed the issue.

Comment 2 Patrick Matthäi 2012-07-13 13:56:03 UTC
Thanks but to have glusterfs in our release we have to use the optimizing flags :(

Comment 3 Vijay Bellur 2012-07-20 23:13:19 UTC
CHANGE: http://review.gluster.com/3707 (Most commands cause glusterd 3.2.7 to crash on Debian Wheezy) merged in release-3.2 by Anand Avati (avati)

Comment 4 Patrick Matthäi 2012-07-24 07:38:02 UTC
Looks good, Louis and myself tested it.

Patch already uploaded to Debian, thanks!

Comment 5 Kaleb KEITHLEY 2012-08-28 12:27:11 UTC
Despite having tested previously on Fedora 17 (w/ gcc-4.7.0) and found no problem, yesterday while running the 'official' Fedora build of 3.2.7 I ran into the same problem.

glusterfs-3.2.7-2 contains the fix.

Comment 6 Patrick Matthäi 2012-08-28 12:28:37 UTC
3.2.7-2 from Debian also has got the patch and it works :)