Description of problem: With reference to bug #1403169, After snapshot restore failure, snapshot goes into inconsistent state. Can't activate this snapshot, coz activate command shows snapshot is already activated and snapshot status command shows some bricks process are down. Version-Release number of selected component (if applicable): glusterfs-3.8.4-7.el7rhgs.x86_64 How reproducible: 100% Steps to Reproduce: 1. Create 2*2 distribute replicate volume 2. Enable cluster.enable-shared-storage 3. enable nfs ganesha 4. create snapshot 5. disable nfs ganesha 6. bring gluster-shared-storage volume 7. Restore snapshot , this command will fail 8. Check snapshot status, or trying taking clone of snapshot Actual results: clone command fails snapshot status command shows some bricks process are down and activate command says snapshot is already activated Expected results: clone command should not fail Additional info: [2016-12-12 06:51:25.513917] E [MSGID: 106122] [glusterd-snapshot.c:2389:glusterd_snapshot_clone_prevalidate] 0-management: Failed to pre validate [2016-12-12 06:51:25.513948] E [MSGID: 106443] [glusterd-snapshot.c:2405:glusterd_snapshot_clone_prevalidate] 0-management: One or more bricks are not running. Please run snapshot status command to see brick status. Please start the stopped brick and then issue snapshot clone command [2016-12-12 06:51:25.513960] W [MSGID: 106443] [glusterd-snapshot.c:8636:glusterd_snapshot_prevalidate] 0-management: Snapshot clone pre-validation failed [2016-12-12 06:51:25.513969] W [MSGID: 106122] [glusterd-mgmt.c:167:gd_mgmt_v3_pre_validate_fn] 0-management: Snapshot Prevalidate Failed [2016-12-12 06:51:25.513978] E [MSGID: 106122] [glusterd-mgmt.c:916:glusterd_mgmt_v3_pre_validate] 0-management: Pre Validation failed for operation Snapshot on local node [2016-12-12 06:51:25.513987] E [MSGID: 106122] [glusterd-mgmt.c:2272:glusterd_mgmt_v3_initiate_snap_phases] 0-management: Pre Validation Failed [2016-12-12 06:51:25.514003] E [MSGID: 106027] [glusterd-snapshot.c:8113:glusterd_snapshot_clone_postvalidate] 0-management: unable to find clone clone1 volinfo [2016-12-12 06:51:25.514012] W [MSGID: 106444] [glusterd-snapshot.c:9136:glusterd_snapshot_postvalidate] 0-management: Snapshot create post-validation failed [2016-12-12 06:51:25.514019] W [MSGID: 106121] [glusterd-mgmt.c:373:gd_mgmt_v3_post_validate_fn] 0-management: postvalidate operation failed [2016-12-12 06:51:25.514027] E [MSGID: 106121] [glusterd-mgmt.c:1689:glusterd_mgmt_v3_post_validate] 0-management: Post Validation failed for operation Snapshot on local node [2016-12-12 06:51:25.514035] E [MSGID: 106122] [glusterd-mgmt.c:2392:glusterd_mgmt_v3_initiate_snap_phases] 0-management: Post Validation Failed =========================================================== [2016-12-12 07:02:29.274196] E [MSGID: 106116] [glusterd-mgmt.c:135:gd_mgmt_v3_collate_errors] 0-management: Pre Validation failed on 10.70.36.46. Snapshot snap1 is already activated. [2016-12-12 07:02:29.274267] E [MSGID: 106116] [glusterd-mgmt.c:135:gd_mgmt_v3_collate_errors] 0-management: Pre Validation failed on 10.70.36.71. Snapshot snap1 is already activated. [2016-12-12 07:02:29.274294] E [MSGID: 106116] [glusterd-mgmt.c:135:gd_mgmt_v3_collate_errors] 0-management: Pre Validation failed on 10.70.44.7. Snapshot snap1 is already activated. [2016-12-12 07:02:29.274328] E [MSGID: 106122] [glusterd-mgmt.c:979:glusterd_mgmt_v3_pre_validate] 0-management: Pre Validation failed on peers [2016-12-12 07:02:29.274390] E [MSGID: 106122] [glusterd-mgmt.c:2272:glusterd_mgmt_v3_initiate_snap_phases] 0-management: Pre Validation Failed =========================================================== [root@rhs-client46 glusterfs]# gluster snapshot status snap1 Snap Name : snap1 Snap UUID : 09114c3e-9ac3-42d7-b8a2-d1c65e0782b8 Brick Path : 10.70.36.70:/run/gluster/snaps/d57afcb0ccd74e9cada4953a70831515/brick1/b1 Volume Group : RHS_vg1 Brick Running : No Brick PID : N/A Data Percentage : 0.55 LV Size : 199.00g Brick Path : 10.70.36.71:/run/gluster/snaps/d57afcb0ccd74e9cada4953a70831515/brick2/b2 Volume Group : RHS_vg1 Brick Running : Yes Brick PID : 11850 Data Percentage : 0.57 LV Size : 199.00g Brick Path : 10.70.36.46:/run/gluster/snaps/d57afcb0ccd74e9cada4953a70831515/brick3/b3 Volume Group : RHS_vg1 Brick Running : Yes Brick PID : 28314 Data Percentage : 0.11 LV Size : 1.80t Brick Path : 10.70.44.7:/run/gluster/snaps/d57afcb0ccd74e9cada4953a70831515/brick4/b4 Volume Group : RHS_vg1 Brick Running : Yes Brick PID : 24756 Data Percentage : 0.16 LV Size : 926.85g Brick Path : 10.70.36.70:/run/gluster/snaps/d57afcb0ccd74e9cada4953a70831515/brick5/b5 Volume Group : RHS_vg2 Brick Running : No Brick PID : N/A Data Percentage : 0.55 LV Size : 199.00g Brick Path : 10.70.36.71:/run/gluster/snaps/d57afcb0ccd74e9cada4953a70831515/brick6/b6 Volume Group : RHS_vg2 Brick Running : Yes Brick PID : 11870 Data Percentage : 0.57 LV Size : 199.00g
Master Url : http://review.gluster.org/#/c/16116/
Master Url : http://review.gluster.org/#/c/16116/ Release 3.9 Url: http://review.gluster.org/#/c/16195/ RHGS 3.2.0 Url: https://code.engineering.redhat.com/gerrit/#/c/93275/
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2017-0486.html