Bug 1403672 - Snapshot: After snapshot restore failure , snapshot goes into inconsistent state
Summary: Snapshot: After snapshot restore failure , snapshot goes into inconsistent state
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: snapshot
Version: rhgs-3.2
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
: RHGS 3.2.0
Assignee: Avra Sengupta
QA Contact: Anil Shah
URL:
Whiteboard:
Depends On:
Blocks: 1351528 1404118 1405909
TreeView+ depends on / blocked
 
Reported: 2016-12-12 07:06 UTC by Anil Shah
Modified: 2017-03-23 05:55 UTC (History)
5 users (show)

Fixed In Version: glusterfs-3.8.4-10
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1404118 (view as bug list)
Environment:
Last Closed: 2017-03-23 05:55:40 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:0486 normal SHIPPED_LIVE Moderate: Red Hat Gluster Storage 3.2.0 security, bug fix, and enhancement update 2017-03-23 09:18:45 UTC

Description Anil Shah 2016-12-12 07:06:35 UTC
Description of problem:

With reference to bug #1403169, After snapshot restore failure, snapshot goes into inconsistent state.
Can't activate this snapshot, coz activate command shows snapshot is already activated and snapshot status command shows some bricks process are down. 


Version-Release number of selected component (if applicable):

glusterfs-3.8.4-7.el7rhgs.x86_64


How reproducible:

100%

Steps to Reproduce:
1. Create 2*2 distribute replicate volume
2. Enable cluster.enable-shared-storage 
3. enable nfs ganesha
4. create snapshot 
5. disable nfs ganesha
6. bring gluster-shared-storage volume
7. Restore snapshot  , this command will fail
8. Check snapshot status, or trying taking clone of snapshot

Actual results:

clone command fails 
snapshot status command shows some bricks process are down and activate command says snapshot is already activated

Expected results:

clone command should not fail


Additional info:



[2016-12-12 06:51:25.513917] E [MSGID: 106122] [glusterd-snapshot.c:2389:glusterd_snapshot_clone_prevalidate] 0-management: Failed to pre validate
[2016-12-12 06:51:25.513948] E [MSGID: 106443] [glusterd-snapshot.c:2405:glusterd_snapshot_clone_prevalidate] 0-management: One or more bricks are not running. Please run snapshot status command to see brick status.
Please start the stopped brick and then issue snapshot clone command 
[2016-12-12 06:51:25.513960] W [MSGID: 106443] [glusterd-snapshot.c:8636:glusterd_snapshot_prevalidate] 0-management: Snapshot clone pre-validation failed
[2016-12-12 06:51:25.513969] W [MSGID: 106122] [glusterd-mgmt.c:167:gd_mgmt_v3_pre_validate_fn] 0-management: Snapshot Prevalidate Failed
[2016-12-12 06:51:25.513978] E [MSGID: 106122] [glusterd-mgmt.c:916:glusterd_mgmt_v3_pre_validate] 0-management: Pre Validation failed for operation Snapshot on local node
[2016-12-12 06:51:25.513987] E [MSGID: 106122] [glusterd-mgmt.c:2272:glusterd_mgmt_v3_initiate_snap_phases] 0-management: Pre Validation Failed
[2016-12-12 06:51:25.514003] E [MSGID: 106027] [glusterd-snapshot.c:8113:glusterd_snapshot_clone_postvalidate] 0-management: unable to find clone clone1 volinfo
[2016-12-12 06:51:25.514012] W [MSGID: 106444] [glusterd-snapshot.c:9136:glusterd_snapshot_postvalidate] 0-management: Snapshot create post-validation failed
[2016-12-12 06:51:25.514019] W [MSGID: 106121] [glusterd-mgmt.c:373:gd_mgmt_v3_post_validate_fn] 0-management: postvalidate operation failed
[2016-12-12 06:51:25.514027] E [MSGID: 106121] [glusterd-mgmt.c:1689:glusterd_mgmt_v3_post_validate] 0-management: Post Validation failed for operation Snapshot on local node
[2016-12-12 06:51:25.514035] E [MSGID: 106122] [glusterd-mgmt.c:2392:glusterd_mgmt_v3_initiate_snap_phases] 0-management: Post Validation Failed

===========================================================

[2016-12-12 07:02:29.274196] E [MSGID: 106116] [glusterd-mgmt.c:135:gd_mgmt_v3_collate_errors] 0-management: Pre Validation failed on 10.70.36.46. Snapshot snap1 is already activated.
[2016-12-12 07:02:29.274267] E [MSGID: 106116] [glusterd-mgmt.c:135:gd_mgmt_v3_collate_errors] 0-management: Pre Validation failed on 10.70.36.71. Snapshot snap1 is already activated.
[2016-12-12 07:02:29.274294] E [MSGID: 106116] [glusterd-mgmt.c:135:gd_mgmt_v3_collate_errors] 0-management: Pre Validation failed on 10.70.44.7. Snapshot snap1 is already activated.
[2016-12-12 07:02:29.274328] E [MSGID: 106122] [glusterd-mgmt.c:979:glusterd_mgmt_v3_pre_validate] 0-management: Pre Validation failed on peers
[2016-12-12 07:02:29.274390] E [MSGID: 106122] [glusterd-mgmt.c:2272:glusterd_mgmt_v3_initiate_snap_phases] 0-management: Pre Validation Failed
===========================================================


[root@rhs-client46 glusterfs]# gluster snapshot status snap1

Snap Name : snap1
Snap UUID : 09114c3e-9ac3-42d7-b8a2-d1c65e0782b8

	Brick Path        :   10.70.36.70:/run/gluster/snaps/d57afcb0ccd74e9cada4953a70831515/brick1/b1
	Volume Group      :   RHS_vg1
	Brick Running     :   No
	Brick PID         :   N/A
	Data Percentage   :   0.55
	LV Size           :   199.00g


	Brick Path        :   10.70.36.71:/run/gluster/snaps/d57afcb0ccd74e9cada4953a70831515/brick2/b2
	Volume Group      :   RHS_vg1
	Brick Running     :   Yes
	Brick PID         :   11850
	Data Percentage   :   0.57
	LV Size           :   199.00g


	Brick Path        :   10.70.36.46:/run/gluster/snaps/d57afcb0ccd74e9cada4953a70831515/brick3/b3
	Volume Group      :   RHS_vg1
	Brick Running     :   Yes
	Brick PID         :   28314
	Data Percentage   :   0.11
	LV Size           :   1.80t


	Brick Path        :   10.70.44.7:/run/gluster/snaps/d57afcb0ccd74e9cada4953a70831515/brick4/b4
	Volume Group      :   RHS_vg1
	Brick Running     :   Yes
	Brick PID         :   24756
	Data Percentage   :   0.16
	LV Size           :   926.85g


	Brick Path        :   10.70.36.70:/run/gluster/snaps/d57afcb0ccd74e9cada4953a70831515/brick5/b5
	Volume Group      :   RHS_vg2
	Brick Running     :   No
	Brick PID         :   N/A
	Data Percentage   :   0.55
	LV Size           :   199.00g


	Brick Path        :   10.70.36.71:/run/gluster/snaps/d57afcb0ccd74e9cada4953a70831515/brick6/b6
	Volume Group      :   RHS_vg2
	Brick Running     :   Yes
	Brick PID         :   11870
	Data Percentage   :   0.57
	LV Size           :   199.00g

Comment 4 Avra Sengupta 2016-12-13 07:25:09 UTC
Master Url : http://review.gluster.org/#/c/16116/

Comment 9 errata-xmlrpc 2017-03-23 05:55:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2017-0486.html


Note You need to log in before you can comment on or make changes to this bug.