This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 837684 - Most commands cause glusterd 3.2.7 to crash on Debian Wheezy
Most commands cause glusterd 3.2.7 to crash on Debian Wheezy
Status: CLOSED CURRENTRELEASE
Product: GlusterFS
Classification: Community
Component: cli (Show other bugs)
3.2.7
Unspecified Other
medium Severity medium
: ---
: ---
Assigned To: Kaushal
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-07-04 14:28 EDT by Louis Zuckerman
Modified: 2013-07-24 13:30 EDT (History)
5 users (show)

See Also:
Fixed In Version: glusterfs-3.4.0
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-07-24 13:30:10 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Debian BTS 679767 None None None 2012-07-04 14:28:07 EDT

  None (edit)
Description Louis Zuckerman 2012-07-04 14:28:07 EDT
Description of problem:

Glusterd dies during handling of most gluster CLI commands (vol create, start, stop, delete, at least)

Version-Release number of selected component (if applicable):

Glusterd v. 3.2.7

How reproducible:

Easy to reproduce & consistent.

Steps to Reproduce:
1. Build glusterfs on Debian Sid/Wheezy or install from Debian repository
2. Try to create a volume (gluster volume create foo thishost:/var/tmp/foo0)
3. The volume gets created but glusterd dies in the process
  
Actual results:

"Connection failed. Please check if gluster daemon is operational."

Starting glusterd up again you can run 'gluster volume info' and see the volume was created.  You can then start the volume, but glusterd will crash once more.  Same for stopping & deleting the volume: glusterd crashes after each operation, but operations appear to have worked after restarting glusterd & running 'gluster volume info' (which does not crash glusterd.)

Expected results:

Normal "volume created" message reminding me to start the volume.

Additional info:

Original bug report in Debian bug tracker:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=679767

Crash dump in the glusterd log file from my (successful) attempt to reproduce:

patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash: 2012-07-04 14:11:41
configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.2.7
/lib/x86_64-linux-gnu/libc.so.6(+0x324f0)[0x7faf8f3744f0]
/lib/x86_64-linux-gnu/libc.so.6(+0x7fbf1)[0x7faf8f3c1bf1]
/lib/x86_64-linux-gnu/libc.so.6(xdr_string+0x37)[0x7faf8f444357]
/usr/lib/libgfxdr.so.0(xdr_gf1_cli_create_vol_rsp+0x51)[0x7faf8faf6061]
/usr/lib/libgfrpc.so.0(xdr_serialize_generic+0x41)[0x7faf8fd0ff11]
/usr/lib/glusterfs/3.2.7/xlator/mgmt/glusterd.so(glusterd_serialize_reply+0x56)[0x7faf8daf1906]
/usr/lib/glusterfs/3.2.7/xlator/mgmt/glusterd.so(glusterd_submit_reply+0x65)[0x7faf8daf1a25]
/usr/lib/glusterfs/3.2.7/xlator/mgmt/glusterd.so(glusterd_op_send_cli_response+0x8f)[0x7faf8db0027f]
/usr/lib/glusterfs/3.2.7/xlator/mgmt/glusterd.so(glusterd_op_txn_complete+0xcc)[0x7faf8daef84c]
/usr/lib/glusterfs/3.2.7/xlator/mgmt/glusterd.so(+0x4bb39)[0x7faf8daefb39]
/usr/lib/glusterfs/3.2.7/xlator/mgmt/glusterd.so(glusterd_op_sm+0xe2)[0x7faf8dae89d2]
/usr/lib/glusterfs/3.2.7/xlator/mgmt/glusterd.so(glusterd_handle_create_volume+0x290)[0x7faf8dad0f60]
/usr/lib/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x27f)[0x7faf8fd071ff]
/usr/lib/libgfrpc.so.0(rpcsvc_notify+0x7b)[0x7faf8fd0778b]
/usr/lib/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7faf8fd0af13]
/usr/lib/glusterfs/3.2.7/rpc-transport/socket.so(socket_event_poll_in+0x34)[0x7faf8d89b224]
/usr/lib/glusterfs/3.2.7/rpc-transport/socket.so(socket_event_handler+0xb5)[0x7faf8d89b565]
/usr/lib/libglusterfs.so.0(+0x404c8)[0x7faf8ff564c8]
/usr/sbin/glusterd(main+0x29b)[0x7faf903a3a6b]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd)[0x7faf8f360ead]
/usr/sbin/glusterd(+0x4cc9)[0x7faf903a3cc9]
Comment 1 Amar Tumballi 2012-07-11 00:59:36 EDT
Looks like an issue with gcc optimization flags. Working on it to get more clarity.

As a work around, giving 'CFLAGS=-O0' fixed the issue.
Comment 2 Patrick Matthäi 2012-07-13 09:56:03 EDT
Thanks but to have glusterfs in our release we have to use the optimizing flags :(
Comment 3 Vijay Bellur 2012-07-20 19:13:19 EDT
CHANGE: http://review.gluster.com/3707 (Most commands cause glusterd 3.2.7 to crash on Debian Wheezy) merged in release-3.2 by Anand Avati (avati@redhat.com)
Comment 4 Patrick Matthäi 2012-07-24 03:38:02 EDT
Looks good, Louis and myself tested it.

Patch already uploaded to Debian, thanks!
Comment 5 Kaleb KEITHLEY 2012-08-28 08:27:11 EDT
Despite having tested previously on Fedora 17 (w/ gcc-4.7.0) and found no problem, yesterday while running the 'official' Fedora build of 3.2.7 I ran into the same problem.

glusterfs-3.2.7-2 contains the fix.
Comment 6 Patrick Matthäi 2012-08-28 08:28:37 EDT
3.2.7-2 from Debian also has got the patch and it works :)

Note You need to log in before you can comment on or make changes to this bug.