Bug 1038990

Summary: [SNAPSHOT] : glusterd crash while creating snapshot
Product: Red Hat Gluster Storage Reporter: senaik
Component: snapshotAssignee: Raghavendra Bhat <rabhat>
Status: CLOSED ERRATA QA Contact: senaik
Severity: urgent Docs Contact:
Priority: urgent    
Version: rhgs-3.0CC: asengupt, nsathyan, rhs-bugs, ssamanta, storage-qa-internal
Target Milestone: ---   
Target Release: RHGS 3.0.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: SNAPSHOT
Fixed In Version: glusterfs-3.4.1.snap.feb05.2014git-1 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-09-22 19:30:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description senaik 2013-12-06 10:09:43 UTC 
Description of problem:
=======================
glusterd crashed while creating snapshot 

Version-Release number of selected component (if applicable):
============================================================
glusterfs 3.4.0.snap.dec03.2013git


How reproducible:


Steps to Reproduce:
==================
1.Create a distributed volume with 4 bricks and start it 

2.On the second node ,mount the volume and create 2 files 

3.On the third node , take a snapshot of the volume . Command failed with below error :
[root@snapshot-03 glusterfs]# gluster snapshot create vol6 -n snap6
snapshot create: failed: Commit failed on localhost. Please check log file for details.
Snapshot command failed

Please refer bug https://bugzilla.redhat.com/show_bug.cgi?id=1038973

4.Tried to take a snaphot again : 

[root@snapshot-03 bricks]# gluster snapshot create vol6 -n snap6
snapshot create: failed: snap with name snap6 already exists
Snapshot command failed

[root@snapshot-03 bricks]# gluster snapshot create vol6 -n SNAP6
Connection failed. Please check if gluster daemon is operational.
Snapshot command failed



Actual results:
===============
glusterd crashed while taking taking snapshot 


Expected results:


Additional info:
================

pending frames:
frame : type(0) op(0)

patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash: 2013-12-06 02:52:33configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.4.0.snap.dec03.2013git
/lib64/libc.so.6[0x3f794329a0]
/usr/lib64/libglusterfs.so.0(gf_store_mkstemp+0x38)[0x7fe8aec40ee8]
/usr/lib64/glusterfs/3.4.0.snap.dec03.2013git/xlator/mgmt/glusterd.so(glusterd_store_perform_snap_volume_store+0x162)[0x7fe8aae06782]
/usr/lib64/glusterfs/3.4.0.snap.dec03.2013git/xlator/mgmt/glusterd.so(glusterd_store_snap_volume+0x5f)[0x7fe8aae0683f]
/usr/lib64/glusterfs/3.4.0.snap.dec03.2013git/xlator/mgmt/glusterd.so(glusterd_store_perform_snap_store+0x1b3)[0x7fe8aae06b13]
/usr/lib64/glusterfs/3.4.0.snap.dec03.2013git/xlator/mgmt/glusterd.so(glusterd_do_snap+0x369)[0x7fe8aae44c19]
/usr/lib64/glusterfs/3.4.0.snap.dec03.2013git/xlator/mgmt/glusterd.so(glusterd_snapshot_create_commit+0x515)[0x7fe8aae465a5]
/usr/lib64/glusterfs/3.4.0.snap.dec03.2013git/xlator/mgmt/glusterd.so(glusterd_snapshot+0x10e)[0x7fe8aae4692e]
/usr/lib64/glusterfs/3.4.0.snap.dec03.2013git/xlator/mgmt/glusterd.so(gd_mgmt_v3_commit_fn+0xae)[0x7fe8aae4a58e]
/usr/lib64/glusterfs/3.4.0.snap.dec03.2013git/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_commit+0xa7)[0x7fe8aae4b0e7]
/usr/lib64/glusterfs/3.4.0.snap.dec03.2013git/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_initiate_snap_phases+0x6cc)[0x7fe8aae4d70c]
/usr/lib64/glusterfs/3.4.0.snap.dec03.2013git/xlator/mgmt/glusterd.so(glusterd_handle_snapshot_fn+0x4ee)[0x7fe8aae46f4e]
/usr/lib64/glusterfs/3.4.0.snap.dec03.2013git/xlator/mgmt/glusterd.so(glusterd_big_locked_handler+0x3f)[0x7fe8aadc0d4f]
/usr/lib64/libglusterfs.so.0(synctask_wrap+0x12)[0x7fe8aec44dd2]
/lib64/libc.so.6[0x3f79443bf0]
Comment 1 senaik 2013-12-06 10:16:32 UTC 
sosreport :

http://rhsqe-repo.lab.eng.blr.redhat.com/bugs_necessary_info/snapshots/1038990/
Comment 3 senaik 2013-12-20 09:31:17 UTC 
faced similar glusterd crash while trying to create multiple snaps and CG s : 


[root@snapshot-01 brick7]# gluster snapshot create volume2 -n snap17
snapshot create: snap17: snap created successfully
[root@snapshot-01 brick7]# gluster snapshot create volume1 -n snap18
snapshot create: snap18: snap created successfully
[root@snapshot-01 brick7]# gluster snapshot create volume1 -n SNAP18
snapshot create: SNAP18: snap created successfully

[root@snapshot-01 brick7]# gluster snapshot create volume2 volume1 -n CG-11
snapshot create: failed: Commit failed on localhost. Please check log file for details.
Snapshot command failed
[root@snapshot-01 brick7]# gluster snapshot create volume2 volume1 -n CG11
Connection failed. Please check if gluster daemon is operational.
Snapshot command failed
Comment 4 Avra Sengupta 2013-12-26 08:31:55 UTC 
Fixed with http://review.gluster.org/#/c/6379/
Comment 5 senaik 2014-02-07 13:06:21 UTC 
Version : glusterfs 3.4.1.0.snap.feb05.2014git

Repeated the same steps as mentioned in the Description , no crash reported. 

Marking the bug as 'verified'
Comment 7 Nagaprasad Sathyanarayana 2014-04-21 06:17:49 UTC 
Marking snapshot BZs to RHS 3.0.
Comment 8 Nagaprasad Sathyanarayana 2014-05-19 10:56:37 UTC 
Setting flags required to add BZs to RHS 3.0 Errata
Comment 11 errata-xmlrpc 2014-09-22 19:30:03 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2014-1278.html