Bug 1094720

Summary:	[SNAPSHOT]: snapshot creation fails but df -h shows the snapshot brick as mounted
Product:	[Community] GlusterFS	Reporter:	rjoseph
Component:	core	Assignee:	Vijaikumar Mallikarjuna <vmallika>
Status:	CLOSED DUPLICATE	QA Contact:
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	pre-release	CC:	gluster-bugs, ndevos, rhinduja, smohan, vmallika
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:	SNAPSHOT
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	1094714	Environment:
Last Closed:	2014-07-13 10:53:16 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1094714
Bug Blocks:

Comment 1 rjoseph 2014-05-06 11:06:59 UTC

Following is the extract from the logs where failure occurred.

[2014-05-06 18:01:18.752857] D [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_snapshot_create_commit+0x2b1) [0x7f2627dad3e1] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_do_snap_vol+0x892) [0x7f2627da94e2] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_take_lvm_snapshot+0x10d) [0x7f2627da891d]))) 0-: running lvcreate help: /sbin/lvcreate --help
[2014-05-06 18:01:18.761000] D [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_snapshot_create_commit+0x2b1) [0x7f2627dad3e1] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_do_snap_vol+0x892) [0x7f2627da94e2] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_take_lvm_snapshot+0x1f5) [0x7f2627da8a05]))) 0-: taking snapshot of the brick snapshot09.lab.eng.blr.redhat.com:/brick2/b2: /sbin/lvcreate -s /dev/mapper/VolGroup0-thin_vol2 --setactivationskip n --name a98f98898ade4433b2272b430cf7779f
[2014-05-06 18:01:19.328292] D [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_do_snap_vol+0x892) [0x7f2627da94e2] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_take_lvm_snapshot+0x218) [0x7f2627da8a28] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_build_snap_device_path+0x12c) [0x7f2627da85bc]))) 0-management: Get volume group for device /dev/mapper/VolGroup0-thin_vol2: /sbin/lvs --noheadings -o vg_name /dev/mapper/VolGroup0-thin_vol2
[2014-05-06 18:01:19.373457] D [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_do_snap_vol+0x8b4) [0x7f2627da9504] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_snap_brick_create+0x24b) [0x7f2627d9f6fb] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_mount_lvm_snapshot+0xf5) [0x7f2627d352c5]))) 0-management: mount -o nouuid /dev/VolGroup0/a98f98898ade4433b2272b430cf7779f /var/run/gluster/snaps/a98f98898ade4433b2272b430cf7779f/brick1: mount -o nouuid /dev/VolGroup0/a98f98898ade4433b2272b430cf7779f /var/run/gluster/snaps/a98f98898ade4433b2272b430cf7779f/brick1
[2014-05-06 18:01:20.346320] D [glusterd-utils.c:11605:glusterd_mount_lvm_snapshot] 0-management: mounting the snapshot logical device /dev/VolGroup0/a98f98898ade4433b2272b430cf7779f successful
[2014-05-06 18:01:20.346492] D [glusterd-utils.c:892:glusterd_brickinfo_new] 0-management: Returning 0
[2014-05-06 18:01:20.346597] D [glusterd-utils.c:892:glusterd_brickinfo_new] 0-management: Returning 0
[2014-05-06 18:01:20.346890] D [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_snapshot_create_commit+0x2b1) [0x7f2627dad3e1] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_do_snap_vol+0x892) [0x7f2627da94e2] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_take_lvm_snapshot+0x10d) [0x7f2627da891d]))) 0-: running lvcreate help: /sbin/lvcreate --help
[2014-05-06 18:01:20.354907] D [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_snapshot_create_commit+0x2b1) [0x7f2627dad3e1] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_do_snap_vol+0x892) [0x7f2627da94e2] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_take_lvm_snapshot+0x1f5) [0x7f2627da8a05]))) 0-: taking snapshot of the brick snapshot09.lab.eng.blr.redhat.com:/brick3/b3: /sbin/lvcreate -s /dev/mapper/VolGroup0-thin_vol3 --setactivationskip n --name a98f98898ade4433b2272b430cf7779f
[2014-05-06 18:01:20.409856] D [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_do_snap_vol+0x892) [0x7f2627da94e2] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_take_lvm_snapshot+0x218) [0x7f2627da8a28] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_build_snap_device_path+0x12c) [0x7f2627da85bc]))) 0-management: Get volume group for device /dev/mapper/VolGroup0-thin_vol3: /sbin/lvs --noheadings -o vg_name /dev/mapper/VolGroup0-thin_vol3
[2014-05-06 18:01:20.451011] D [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_do_snap_vol+0x8b4) [0x7f2627da9504] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_snap_brick_create+0x24b) [0x7f2627d9f6fb] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_mount_lvm_snapshot+0xf5) [0x7f2627d352c5]))) 0-management: mount -o nouuid /dev/VolGroup0/a98f98898ade4433b2272b430cf7779f /var/run/gluster/snaps/a98f98898ade4433b2272b430cf7779f/brick3: mount -o nouuid /dev/VolGroup0/a98f98898ade4433b2272b430cf7779f /var/run/gluster/snaps/a98f98898ade4433b2272b430cf7779f/brick3
[2014-05-06 18:01:20.475025] D [glusterd-utils.c:11605:glusterd_mount_lvm_snapshot] 0-management: mounting the snapshot logical device /dev/VolGroup0/a98f98898ade4433b2272b430cf7779f successful
[2014-05-06 18:01:20.475133] W [glusterd-snapshot.c:3369:glusterd_snap_brick_create] 0-management: stat of the brick /var/run/gluster/snaps/a98f98898ade4433b2272b430cf7779f/brick3/b3(brick mount: /var/run/gluster/snaps/a98f98898ade4433b2272b430cf7779f/brick3) failed (No such file or directory)

In this test case multiple LVMs are present in the same Volume Group. If bricks are formed from these LVMs snapshot creation will fail. (Fix for this is already under review). 
We are using runner apis to create lvm snapshot, but the runner_start call does not give error in case the command fails. 
To fix this we should be using runner_run which gives the correct error or check error in runner_end.

Comment 2 Vijaikumar Mallikarjuna 2014-05-06 11:43:45 UTC

Patch http://review.gluster.org/#/c/7581/ will fix this issue.

Comment 3 Niels de Vos 2014-07-13 10:53:16 UTC


*** This bug has been marked as a duplicate of bug 1091934 ***