Bug 1403672

Summary: Snapshot: After snapshot restore failure , snapshot goes into inconsistent state
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Anil Shah <ashah>
Component: snapshotAssignee: Avra Sengupta <asengupt>
Status: CLOSED ERRATA QA Contact: Anil Shah <ashah>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: rhgs-3.2CC: amukherj, rcyriac, rhinduja, rhs-bugs, storage-qa-internal
Target Milestone: ---   
Target Release: RHGS 3.2.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: glusterfs-3.8.4-10 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1404118 (view as bug list) Environment:
Last Closed: 2017-03-23 05:55:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1351528, 1404118, 1405909    

Description Anil Shah 2016-12-12 07:06:35 UTC 
Description of problem:

With reference to bug #1403169, After snapshot restore failure, snapshot goes into inconsistent state.
Can't activate this snapshot, coz activate command shows snapshot is already activated and snapshot status command shows some bricks process are down. 


Version-Release number of selected component (if applicable):

glusterfs-3.8.4-7.el7rhgs.x86_64


How reproducible:

100%

Steps to Reproduce:
1. Create 2*2 distribute replicate volume
2. Enable cluster.enable-shared-storage 
3. enable nfs ganesha
4. create snapshot 
5. disable nfs ganesha
6. bring gluster-shared-storage volume
7. Restore snapshot  , this command will fail
8. Check snapshot status, or trying taking clone of snapshot

Actual results:

clone command fails 
snapshot status command shows some bricks process are down and activate command says snapshot is already activated

Expected results:

clone command should not fail


Additional info:



[2016-12-12 06:51:25.513917] E [MSGID: 106122] [glusterd-snapshot.c:2389:glusterd_snapshot_clone_prevalidate] 0-management: Failed to pre validate
[2016-12-12 06:51:25.513948] E [MSGID: 106443] [glusterd-snapshot.c:2405:glusterd_snapshot_clone_prevalidate] 0-management: One or more bricks are not running. Please run snapshot status command to see brick status.
Please start the stopped brick and then issue snapshot clone command 
[2016-12-12 06:51:25.513960] W [MSGID: 106443] [glusterd-snapshot.c:8636:glusterd_snapshot_prevalidate] 0-management: Snapshot clone pre-validation failed
[2016-12-12 06:51:25.513969] W [MSGID: 106122] [glusterd-mgmt.c:167:gd_mgmt_v3_pre_validate_fn] 0-management: Snapshot Prevalidate Failed
[2016-12-12 06:51:25.513978] E [MSGID: 106122] [glusterd-mgmt.c:916:glusterd_mgmt_v3_pre_validate] 0-management: Pre Validation failed for operation Snapshot on local node
[2016-12-12 06:51:25.513987] E [MSGID: 106122] [glusterd-mgmt.c:2272:glusterd_mgmt_v3_initiate_snap_phases] 0-management: Pre Validation Failed
[2016-12-12 06:51:25.514003] E [MSGID: 106027] [glusterd-snapshot.c:8113:glusterd_snapshot_clone_postvalidate] 0-management: unable to find clone clone1 volinfo
[2016-12-12 06:51:25.514012] W [MSGID: 106444] [glusterd-snapshot.c:9136:glusterd_snapshot_postvalidate] 0-management: Snapshot create post-validation failed
[2016-12-12 06:51:25.514019] W [MSGID: 106121] [glusterd-mgmt.c:373:gd_mgmt_v3_post_validate_fn] 0-management: postvalidate operation failed
[2016-12-12 06:51:25.514027] E [MSGID: 106121] [glusterd-mgmt.c:1689:glusterd_mgmt_v3_post_validate] 0-management: Post Validation failed for operation Snapshot on local node
[2016-12-12 06:51:25.514035] E [MSGID: 106122] [glusterd-mgmt.c:2392:glusterd_mgmt_v3_initiate_snap_phases] 0-management: Post Validation Failed

===========================================================

[2016-12-12 07:02:29.274196] E [MSGID: 106116] [glusterd-mgmt.c:135:gd_mgmt_v3_collate_errors] 0-management: Pre Validation failed on 10.70.36.46. Snapshot snap1 is already activated.
[2016-12-12 07:02:29.274267] E [MSGID: 106116] [glusterd-mgmt.c:135:gd_mgmt_v3_collate_errors] 0-management: Pre Validation failed on 10.70.36.71. Snapshot snap1 is already activated.
[2016-12-12 07:02:29.274294] E [MSGID: 106116] [glusterd-mgmt.c:135:gd_mgmt_v3_collate_errors] 0-management: Pre Validation failed on 10.70.44.7. Snapshot snap1 is already activated.
[2016-12-12 07:02:29.274328] E [MSGID: 106122] [glusterd-mgmt.c:979:glusterd_mgmt_v3_pre_validate] 0-management: Pre Validation failed on peers
[2016-12-12 07:02:29.274390] E [MSGID: 106122] [glusterd-mgmt.c:2272:glusterd_mgmt_v3_initiate_snap_phases] 0-management: Pre Validation Failed
===========================================================


[root@rhs-client46 glusterfs]# gluster snapshot status snap1

Snap Name : snap1
Snap UUID : 09114c3e-9ac3-42d7-b8a2-d1c65e0782b8

	Brick Path        :   10.70.36.70:/run/gluster/snaps/d57afcb0ccd74e9cada4953a70831515/brick1/b1
	Volume Group      :   RHS_vg1
	Brick Running     :   No
	Brick PID         :   N/A
	Data Percentage   :   0.55
	LV Size           :   199.00g


	Brick Path        :   10.70.36.71:/run/gluster/snaps/d57afcb0ccd74e9cada4953a70831515/brick2/b2
	Volume Group      :   RHS_vg1
	Brick Running     :   Yes
	Brick PID         :   11850
	Data Percentage   :   0.57
	LV Size           :   199.00g


	Brick Path        :   10.70.36.46:/run/gluster/snaps/d57afcb0ccd74e9cada4953a70831515/brick3/b3
	Volume Group      :   RHS_vg1
	Brick Running     :   Yes
	Brick PID         :   28314
	Data Percentage   :   0.11
	LV Size           :   1.80t


	Brick Path        :   10.70.44.7:/run/gluster/snaps/d57afcb0ccd74e9cada4953a70831515/brick4/b4
	Volume Group      :   RHS_vg1
	Brick Running     :   Yes
	Brick PID         :   24756
	Data Percentage   :   0.16
	LV Size           :   926.85g


	Brick Path        :   10.70.36.70:/run/gluster/snaps/d57afcb0ccd74e9cada4953a70831515/brick5/b5
	Volume Group      :   RHS_vg2
	Brick Running     :   No
	Brick PID         :   N/A
	Data Percentage   :   0.55
	LV Size           :   199.00g


	Brick Path        :   10.70.36.71:/run/gluster/snaps/d57afcb0ccd74e9cada4953a70831515/brick6/b6
	Volume Group      :   RHS_vg2
	Brick Running     :   Yes
	Brick PID         :   11870
	Data Percentage   :   0.57
	LV Size           :   199.00g
Comment 4 Avra Sengupta 2016-12-13 07:25:09 UTC 
Master Url : http://review.gluster.org/#/c/16116/
Comment 5 Avra Sengupta 2016-12-19 08:46:52 UTC 
Master Url : http://review.gluster.org/#/c/16116/
Release 3.9 Url: http://review.gluster.org/#/c/16195/
RHGS 3.2.0 Url: https://code.engineering.redhat.com/gerrit/#/c/93275/
Comment 9 errata-xmlrpc 2017-03-23 05:55:40 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2017-0486.html