Bug 1427039 - Create glusterfs snapshot error
Summary: Create glusterfs snapshot error
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: GlusterFS
Classification: Community
Component: snapshot
Version: 3.8
Hardware: x86_64
OS: Linux
medium
high
Target Milestone: ---
Assignee: Avra Sengupta
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-02-27 07:48 UTC by shibin
Modified: 2017-03-06 14:21 UTC (History)
2 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2017-03-06 14:21:56 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:

Attachments (Terms of Use)

Description shibin 2017-02-27 07:48:57 UTC 
Create glusterfs snapshot error in ebs。The log is:


[2017-02-27 07:48:12.300293] E [MSGID: 106058] [glusterd-snapshot.c:5088:glusterd_update_fs_label] 0-management: Failed to change filesystem label of /run/gluster/snaps/b9bb548cfbed4b609a9d9e3de377456e/brick1/shibin01 brick to 6e40ede875a146e1
[2017-02-27 07:48:12.300341] E [MSGID: 106058] [glusterd-snapshot.c:5148:glusterd_take_brick_snapshot] 0-management: Failed to update file-system label for /run/gluster/snaps/b9bb548cfbed4b609a9d9e3de377456e/brick1/shibin01 brick
[2017-02-27 07:48:12.303181] E [MSGID: 106098] [glusterd-snapshot-utils.c:2709:glusterd_mount_lvm_snapshot] 0-management: mounting the snapshot logical device /dev/vg/b9bb548cfbed4b609a9d9e3de377456e_0 failed (error: File exists)
[2017-02-27 07:48:12.303208] E [MSGID: 106059] [glusterd-snapshot.c:4788:glusterd_snap_brick_create] 0-management: Failed to mount lvm snapshot.
[2017-02-27 07:48:12.303220] W [MSGID: 106055] [glusterd-snapshot.c:4819:glusterd_snap_brick_create] 0-management: unmounting the snap brick mount /run/gluster/snaps/b9bb548cfbed4b609a9d9e3de377456e/brick1
[2017-02-27 07:48:12.305901] E [MSGID: 106095] [glusterd-snapshot-utils.c:3364:glusterd_umount] 0-management: umounting /run/gluster/snaps/b9bb548cfbed4b609a9d9e3de377456e/brick1 failed (File exists) [File exists]
[2017-02-27 07:48:12.305932] E [MSGID: 106050] [glusterd-snapshot.c:5162:glusterd_take_brick_snapshot] 0-management: not able to create the brick for the snap df, volume b9bb548cfbed4b609a9d9e3de377456e
[2017-02-27 07:48:12.306230] E [MSGID: 106030] [glusterd-snapshot.c:6484:glusterd_take_brick_snapshot_task] 0-management: Failed to take backend snapshot for brick 172.31.28.255:/run/gluster/snaps/b9bb548cfbed4b609a9d9e3de377456e/brick1/shibin01 volume(b9bb548cfbed4b609a9d9e3de377456e)
[2017-02-27 07:48:12.306987] E [MSGID: 106030] [glusterd-snapshot.c:6626:glusterd_schedule_brick_snapshot] 0-management: Failed to create snapshot
[2017-02-27 07:48:12.307026] E [MSGID: 106030] [glusterd-snapshot.c:6952:glusterd_snapshot_create_commit] 0-management: Failed to take backend snapshot df
[2017-02-27 07:48:12.308255] E [MSGID: 106030] [glusterd-snapshot.c:8297:glusterd_snapshot] 0-management: Failed to create snapshot
[2017-02-27 07:48:12.308277] W [MSGID: 106123] [glusterd-mgmt.c:272:gd_mgmt_v3_commit_fn] 0-management: Snapshot Commit Failed
[2017-02-27 07:48:12.308295] E [MSGID: 106123] [glusterd-mgmt.c:1427:glusterd_mgmt_v3_commit] 0-management: Commit failed for operation Snapshot on local node
[2017-02-27 07:48:12.308306] E [MSGID: 106123] [glusterd-mgmt.c:2304:glusterd_mgmt_v3_initiate_snap_phases] 0-management: Commit Op Failed
[2017-02-27 07:48:12.383510] E [MSGID: 106062] [glusterd-snapshot.c:6359:glusterd_do_snap_cleanup] 0-glusterd: Unable to get volume name
[2017-02-27 07:48:12.383885] W [MSGID: 106040] [glusterd-snapshot.c:8188:glusterd_snapshot_create_postvalidate] 0-management: cleanup operation failed
[2017-02-27 07:48:12.384160] W [MSGID: 106030] [glusterd-snapshot.c:9050:glusterd_snapshot_postvalidate] 0-management: Snapshot create post-validation failed
[2017-02-27 07:48:12.384409] W [MSGID: 106121] [glusterd-mgmt.c:351:gd_mgmt_v3_post_validate_fn] 0-management: postvalidate operation failed
[2017-02-27 07:48:12.384692] E [MSGID: 106121] [glusterd-mgmt.c:1660:glusterd_mgmt_v3_post_validate] 0-management: Post Validation failed for operation Snapshot on local node
[2017-02-27 07:48:12.384934] E [MSGID: 106122] [glusterd-mgmt.c:2363:glusterd_mgmt_v3_initiate_snap_phases] 0-management: Post Validation Failed
Comment 1 shibin 2017-02-27 07:52:38 UTC 
/run/gluster/snaps/b9bb548cfbed4b609a9d9e3de377456e/brick1/shibin01 is not exist
Comment 2 shibin 2017-03-06 14:14:29 UTC 
Create glusterfs snapshot error in docker container.  

The log:
[2017-02-27 07:48:12.300293] E [MSGID: 106058] [glusterd-snapshot.c:5088:glusterd_update_fs_label] 0-management: Failed to change filesystem label of /run/gluster/snaps/b9bb548cfbed4b609a9d9e3de377456e/brick1/shibin01 brick to 6e40ede875a146e1
[2017-02-27 07:48:12.300341] E [MSGID: 106058] [glusterd-snapshot.c:5148:glusterd_take_brick_snapshot] 0-management: Failed to update file-system label for /run/gluster/snaps/b9bb548cfbed4b609a9d9e3de377456e/brick1/shibin01 brick
[2017-02-27 07:48:12.303181] E [MSGID: 106098] [glusterd-snapshot-utils.c:2709:glusterd_mount_lvm_snapshot] 0-management: mounting the snapshot logical device /dev/vg/b9bb548cfbed4b609a9d9e3de377456e_0 failed (error: File exists)
[2017-02-27 07:48:12.303208] E [MSGID: 106059] [glusterd-snapshot.c:4788:glusterd_snap_brick_create] 0-management: Failed to mount lvm snapshot.


That is because lvmetad which caches lvm metadata and with
docker the cache doesn't seem to update properly.
Therefore it takes some time before the snapshot device appear in
docker container. Therefore after snapshot creation changing label and
mount fails because it fails to see the snapshot lvm.


modify the /etc/lvm/lvm.conf in container
udev_sync = 0
udev_rules = 0
use_lvmetad = 0


the glusterfs snapshot create sucessfully
Comment 3 shibin 2017-03-06 14:16:23 UTC 
Thanks Rajesh Joseph help me to solve the problem.
Note You need to log in before you can comment on or make changes to this bug.