Bug 1427039

Summary:	Create glusterfs snapshot error
Product:	[Community] GlusterFS	Reporter:	shibin <1220642745>
Component:	snapshot	Assignee:	Avra Sengupta <asengupt>
Status:	CLOSED NOTABUG	QA Contact:
Severity:	high	Docs Contact:
Priority:	medium
Version:	3.8	CC:	amukherj, bugs
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-03-06 14:21:56 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description shibin 2017-02-27 07:48:57 UTC

Create glusterfs snapshot error in ebs。The log is:


[2017-02-27 07:48:12.300293] E [MSGID: 106058] [glusterd-snapshot.c:5088:glusterd_update_fs_label] 0-management: Failed to change filesystem label of /run/gluster/snaps/b9bb548cfbed4b609a9d9e3de377456e/brick1/shibin01 brick to 6e40ede875a146e1
[2017-02-27 07:48:12.300341] E [MSGID: 106058] [glusterd-snapshot.c:5148:glusterd_take_brick_snapshot] 0-management: Failed to update file-system label for /run/gluster/snaps/b9bb548cfbed4b609a9d9e3de377456e/brick1/shibin01 brick
[2017-02-27 07:48:12.303181] E [MSGID: 106098] [glusterd-snapshot-utils.c:2709:glusterd_mount_lvm_snapshot] 0-management: mounting the snapshot logical device /dev/vg/b9bb548cfbed4b609a9d9e3de377456e_0 failed (error: File exists)
[2017-02-27 07:48:12.303208] E [MSGID: 106059] [glusterd-snapshot.c:4788:glusterd_snap_brick_create] 0-management: Failed to mount lvm snapshot.
[2017-02-27 07:48:12.303220] W [MSGID: 106055] [glusterd-snapshot.c:4819:glusterd_snap_brick_create] 0-management: unmounting the snap brick mount /run/gluster/snaps/b9bb548cfbed4b609a9d9e3de377456e/brick1
[2017-02-27 07:48:12.305901] E [MSGID: 106095] [glusterd-snapshot-utils.c:3364:glusterd_umount] 0-management: umounting /run/gluster/snaps/b9bb548cfbed4b609a9d9e3de377456e/brick1 failed (File exists) [File exists]
[2017-02-27 07:48:12.305932] E [MSGID: 106050] [glusterd-snapshot.c:5162:glusterd_take_brick_snapshot] 0-management: not able to create the brick for the snap df, volume b9bb548cfbed4b609a9d9e3de377456e
[2017-02-27 07:48:12.306230] E [MSGID: 106030] [glusterd-snapshot.c:6484:glusterd_take_brick_snapshot_task] 0-management: Failed to take backend snapshot for brick 172.31.28.255:/run/gluster/snaps/b9bb548cfbed4b609a9d9e3de377456e/brick1/shibin01 volume(b9bb548cfbed4b609a9d9e3de377456e)
[2017-02-27 07:48:12.306987] E [MSGID: 106030] [glusterd-snapshot.c:6626:glusterd_schedule_brick_snapshot] 0-management: Failed to create snapshot
[2017-02-27 07:48:12.307026] E [MSGID: 106030] [glusterd-snapshot.c:6952:glusterd_snapshot_create_commit] 0-management: Failed to take backend snapshot df
[2017-02-27 07:48:12.308255] E [MSGID: 106030] [glusterd-snapshot.c:8297:glusterd_snapshot] 0-management: Failed to create snapshot
[2017-02-27 07:48:12.308277] W [MSGID: 106123] [glusterd-mgmt.c:272:gd_mgmt_v3_commit_fn] 0-management: Snapshot Commit Failed
[2017-02-27 07:48:12.308295] E [MSGID: 106123] [glusterd-mgmt.c:1427:glusterd_mgmt_v3_commit] 0-management: Commit failed for operation Snapshot on local node
[2017-02-27 07:48:12.308306] E [MSGID: 106123] [glusterd-mgmt.c:2304:glusterd_mgmt_v3_initiate_snap_phases] 0-management: Commit Op Failed
[2017-02-27 07:48:12.383510] E [MSGID: 106062] [glusterd-snapshot.c:6359:glusterd_do_snap_cleanup] 0-glusterd: Unable to get volume name
[2017-02-27 07:48:12.383885] W [MSGID: 106040] [glusterd-snapshot.c:8188:glusterd_snapshot_create_postvalidate] 0-management: cleanup operation failed
[2017-02-27 07:48:12.384160] W [MSGID: 106030] [glusterd-snapshot.c:9050:glusterd_snapshot_postvalidate] 0-management: Snapshot create post-validation failed
[2017-02-27 07:48:12.384409] W [MSGID: 106121] [glusterd-mgmt.c:351:gd_mgmt_v3_post_validate_fn] 0-management: postvalidate operation failed
[2017-02-27 07:48:12.384692] E [MSGID: 106121] [glusterd-mgmt.c:1660:glusterd_mgmt_v3_post_validate] 0-management: Post Validation failed for operation Snapshot on local node
[2017-02-27 07:48:12.384934] E [MSGID: 106122] [glusterd-mgmt.c:2363:glusterd_mgmt_v3_initiate_snap_phases] 0-management: Post Validation Failed

Comment 1 shibin 2017-02-27 07:52:38 UTC

/run/gluster/snaps/b9bb548cfbed4b609a9d9e3de377456e/brick1/shibin01 is not exist

Comment 2 shibin 2017-03-06 14:14:29 UTC

Create glusterfs snapshot error in docker container.  

The log:
[2017-02-27 07:48:12.300293] E [MSGID: 106058] [glusterd-snapshot.c:5088:glusterd_update_fs_label] 0-management: Failed to change filesystem label of /run/gluster/snaps/b9bb548cfbed4b609a9d9e3de377456e/brick1/shibin01 brick to 6e40ede875a146e1
[2017-02-27 07:48:12.300341] E [MSGID: 106058] [glusterd-snapshot.c:5148:glusterd_take_brick_snapshot] 0-management: Failed to update file-system label for /run/gluster/snaps/b9bb548cfbed4b609a9d9e3de377456e/brick1/shibin01 brick
[2017-02-27 07:48:12.303181] E [MSGID: 106098] [glusterd-snapshot-utils.c:2709:glusterd_mount_lvm_snapshot] 0-management: mounting the snapshot logical device /dev/vg/b9bb548cfbed4b609a9d9e3de377456e_0 failed (error: File exists)
[2017-02-27 07:48:12.303208] E [MSGID: 106059] [glusterd-snapshot.c:4788:glusterd_snap_brick_create] 0-management: Failed to mount lvm snapshot.


That is because lvmetad which caches lvm metadata and with
docker the cache doesn't seem to update properly.
Therefore it takes some time before the snapshot device appear in
docker container. Therefore after snapshot creation changing label and
mount fails because it fails to see the snapshot lvm.


modify the /etc/lvm/lvm.conf in container
udev_sync = 0
udev_rules = 0
use_lvmetad = 0


the glusterfs snapshot create sucessfully

Comment 3 shibin 2017-03-06 14:16:23 UTC

Thanks Rajesh Joseph help me to solve the problem.