Bug 1591634

Summary: Snapshots fail when underlying bricks are btrfs
Product: [Community] GlusterFS Reporter: Evili del Rio <evili.del.rio>
Component: snapshotAssignee: bugs <bugs>
Status: CLOSED DUPLICATE QA Contact:
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.1CC: bugs, rkavunga
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-10-16 05:52:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Relevant part of local node glusterd.log
none
Proposed patch
none
Proposed patch none

Description Evili del Rio 2018-06-15 07:27:11 UTC 
Created attachment 1451792 [details]
Relevant part of local node glusterd.log

Description of problem:

I have a disperse volume [1 x (4 + 2) = 6] with 3 servers 2 bricks each. Each   brick is a thin LV with a BTRFS filesystem on it. When I try to make a gluster snapshot of the volume:

gluster snapshot create web_data-24 web_data no-timestamp description Backup_snapshot

it fails with the following error:

snapshot create: failed: Commit failed on localhost. Please check log file for details.
Snapshot command failed

Then the local node has strange mounts on the volume bricks:

 /dev/mapper/vg_brickN-SOMEUUID_1

The other nodes on the cluster are unaffected.

If I make the same setup (servers, volume, mount points) but with xfs bricks everything works as it should.

Version-Release number of selected component (if applicable):

4.0.2

How reproducible: Whenever there are btrfs bricks in a volume:

Steps to Reproduce:

1. Setup a gluster cluster.
2. For each node create 2 vg with 2 thin pools, and 2 thin volumes as stated in 
https://docs.gluster.org/en/latest/Administrator%20Guide/Managing%20Snapshots/
3. Mount all bricks
4. Create the volume:
gluster volume create web_data  redundancy 2 gluster0.storage:/srv/glusterfs/web/brick0/brick gluster0.storage:/srv/glusterfs/web/brick3/brick gluster1.storage:/srv/glusterfs/web/brick1/brick gluster1.storage:/srv/glusterfs/web/brick4/brick  gluster2.storage:/srv/glusterfs/web/brick2/brick gluster2.storage:/srv/glusterfs/web/brick5/brick force
5. Try to make a snapshot:
gluster snapshot create web_data-24 web_data no-timestamp description Backup_snapshot

Actual results: The snapshot fails on local node:
snapshot create: failed: Commit failed on localhost. Please check log file for details.

And there are strange mounts on system like:

/dev/mapper/vg_web5-95d900edd0564a8296ea63bd6886ba73_1 on /srv/glusterfs/web/brick5 type btrfs (rw,noatime,nodiratime,seclabel,nobarrier,compress=lzo,space_cache,autodefrag,commit=60,subvolid=5,subvol=/)

instead of the original one:

/dev/mapper/vg_web5-brick on /srv/glusterfs/web/brick5 type btrfs (rw,noatime,nodiratime,seclabel,nobarrier,compress=lzo,space_cache,autodefrag,commit=60,subvolid=5,subvol=/)

Expected results:
Creation of the gluster snapshot.

Additional info:
Comment 1 Shyamsundar 2018-06-20 18:29:20 UTC 
This bug reported is against a version of Gluster that is no longer maintained
(or has been EOL'd). See https://www.gluster.org/release-schedule/ for the
versions currently maintained.

As a result this bug is being closed.

If the bug persists on a maintained version of gluster or against the mainline
gluster repository, request that it be reopened and the Version field be marked
appropriately.
Comment 2 Evili del Rio 2018-06-21 07:55:47 UTC 
That's a bit unkind. EOL for 4.0 was on 2018-06-20 and today, one day after there are still not a working RPM repository for Gluster 4.1 on CentOS 7 or RedHat EL 7.

You should have waited, at least, to have a working repositroy for 4.1.
Comment 3 Evili del Rio 2018-08-09 06:59:16 UTC 
Hi,

Here it is. The bug is present en the 4.1 version.

The logs report the following:


[2018-08-09 06:49:39.680183] W [MSGID: 106055] [glusterd-snapshot.c:5115:glusterd_update_fs_label] 0-management: Changing file-system label of btrfs file-system is not supported as of now [Operation not supported]
[2018-08-09 06:49:39.680251] E [MSGID: 106057] [glusterd-snapshot.c:5192:glusterd_take_brick_snapshot] 0-management: Failed to update file-system label for /run/gluster/snaps/c59f672b1b614451bbcbecc5bef82fcb/brick5/brick brick
[2018-08-09 06:49:39.680273] E [MSGID: 106029] [glusterd-snapshot.c:6542:glusterd_take_brick_snapshot_task] 0-management: Failed to take backend snapshot for brick gluster2.storage:/run/gluster/snaps/c59f672b1b614451bbcbecc5bef82fcb/brick5/brick volume(c59f672b1b614451bbcbecc5bef82fcb)
[2018-08-09 06:49:39.680637] E [MSGID: 106057] [glusterd-snapshot.c:5192:glusterd_take_brick_snapshot] 0-management: Failed to update file-system label for /run/gluster/snaps/c59f672b1b614451bbcbecc5bef82fcb/brick6/brick brick
[2018-08-09 06:49:39.680698] E [MSGID: 106029] [glusterd-snapshot.c:6542:glusterd_take_brick_snapshot_task] 0-management: Failed to take backend snapshot for brick gluster2.storage:/run/gluster/snaps/c59f672b1b614451bbcbecc5bef82fcb/brick6/brick volume(c59f672b1b614451bbcbecc5bef82fcb)
[2018-08-09 06:49:39.697489] E [MSGID: 106029] [glusterd-snapshot.c:6686:glusterd_schedule_brick_snapshot] 0-management: Failed to create snapshot
[2018-08-09 06:49:39.697524] E [MSGID: 106029] [glusterd-snapshot.c:7002:glusterd_snapshot_create_commit] 0-management: Failed to take backend snapshot web_data_20180809
[2018-08-09 06:49:39.697785] W [MSGID: 106072] [glusterd-snapshot.c:2779:glusterd_do_lvm_snapshot_remove] 0-management: Getting the root of the brick for volume c59f672b1b614451bbcbecc5bef82fcb (snap web_data_20180809) failed. Removing lv (/dev/vg_web2/c59f672b1b614451bbcbecc5bef82fcb_0).
[2018-08-09 06:49:39.798500] W [MSGID: 106072] [glusterd-snapshot.c:2779:glusterd_do_lvm_snapshot_remove] 0-management: Getting the root of the brick for volume c59f672b1b614451bbcbecc5bef82fcb (snap web_data_20180809) failed. Removing lv (/dev/vg_web5/c59f672b1b614451bbcbecc5bef82fcb_1).
[2018-08-09 06:49:39.929974] E [MSGID: 106029] [glusterd-snapshot.c:8449:glusterd_snapshot] 0-management: Failed to create snapshot
[2018-08-09 06:49:40.110278] E [MSGID: 106061] [glusterd-snapshot.c:6413:glusterd_do_snap_cleanup] 0-glusterd: Unable to get volume name
[2018-08-09 06:49:40.110352] W [MSGID: 106039] [glusterd-snapshot.c:8263:glusterd_snapshot_create_postvalidate] 0-management: cleanup operation failed
[2018-08-09 06:49:40.110404] W [MSGID: 106029] [glusterd-snapshot.c:9246:glusterd_snapshot_postvalidate] 0-management: Snapshot create post-validation failed
[2018-08-09 06:49:39.680625] W [MSGID: 106055] [glusterd-snapshot.c:5115:glusterd_update_fs_label] 0-management: Changing file-system label of btrfs file-system is not supported as of now [Operation not supported]
Comment 4 Evili del Rio 2018-08-09 09:08:59 UTC 
Created attachment 1474612 [details]
Proposed patch

This is a proposed patch to support the relabeling issue for btrfs. It uses the btrfs command:

btrfs filesystem label /path/mounted/brick newlabel
Comment 5 Evili del Rio 2018-08-09 09:13:44 UTC 
Created attachment 1474613 [details]
Proposed patch
Comment 6 Mohammed Rafi KC 2018-10-16 05:52:33 UTC 

*** This bug has been marked as a duplicate of bug 1426749 ***