Bug 1111479
| Summary: | [SNAPSHOT]: Attaching a new node to the cluster while snapshot delete was in progress, deleted snapshots successfuly but gluster snapshot list shows some of the the snaps is still present | |||
|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | senaik | |
| Component: | snapshot | Assignee: | rjoseph | |
| Status: | CLOSED DEFERRED | QA Contact: | ||
| Severity: | medium | Docs Contact: | ||
| Priority: | medium | |||
| Version: | rhgs-3.0 | CC: | asengupt, asriram, josferna, nsathyan, rhs-bugs, rjoseph, storage-qa-internal, vagarwal | |
| Target Milestone: | --- | |||
| Target Release: | --- | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | SNAPSHOT | |||
| Fixed In Version: | Doc Type: | Known Issue | ||
| Doc Text: |
Attaching and detaching new node to the cluster while snapshot is in progress leads to inconsistent snapshot. Workaround: Do not attach or detach new node to the cluster operation while snapshot is in progress.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1115806 (view as bug list) | Environment: | ||
| Last Closed: | 2016-01-29 13:42:01 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1087818, 1115806 | |||
The issue is same as bug 1114403. The issue is glusterd quorum failure in snap phases, Glusterd Log from snapshot-15 host during deletion of snap_1_6 : [2014-06-19 09:27:45.937337] W [glusterd-utils.c:13069:glusterd_snap_quorum_check] 0-management: glusterds are not in quorum [2014-06-19 09:27:45.937396] W [glusterd-mgmt.c:1846:glusterd_mgmt_v3_initiate_snap_phases] 0-management: quorum check failed The fix is submitted upstream, in the review process Fix submitted upstream: REVIEW: http://review.gluster.org/8200 (glusterd/snapshot: fixing glusterd quorum during snap operation) posted (#1) for review on master by Joseph Fernandes (josferna) I have tested the fix with this scenario and it fixed this issue. Please review and sign-off edited doc text. Not targetting for 3.1 based on commen 5, removing from the list for 3.1 This Bug is not fixed with the submitted patch and it requires design changes in glusterd. Hence moving this back to New. Current Gluster architecture does not support implementation of this feature. Therefore this feature request is deferred till Gluterd 2.0. |
Description of problem: ======================= Attaching a new node to the cluster while snapshot delete is in progress, deleted snapshots successfuly but gluster snapshot list shows some of the successfully deleted snapshosts are still present Version-Release number of selected component (if applicable): ============================================================= glusterfs 3.6.0.19 built on Jun 18 2014 How reproducible: ================= 1/1 Steps to Reproduce: ==================== -4 node cluster (snapshot13-16) -created few snapshots with IO going on -deleted snapshots in loop -while delete in is in progress, probed another node (10.70.44.54) for i in {1..20} ; do time gluster --mode=script snapshot delete snap_1_$i ; done snapshot delete: snap_1_1: snap removed successfully real 0m2.057s user 0m0.206s sys 0m0.045s snapshot delete: snap_1_2: snap removed successfully real 0m1.719s user 0m0.197s sys 0m0.057s snapshot delete: snap_1_3: snap removed successfully real 0m1.807s user 0m0.217s sys 0m0.045s snapshot delete: snap_1_4: snap removed successfully----------------> removed successfully real 0m1.739s user 0m0.220s sys 0m0.033s snapshot delete: snap_1_5: snap removed successfully----------------> removed successfully real 0m1.839s user 0m0.203s sys 0m0.049s snapshot delete: failed: snap snap_1_6 might not be in an usable state.--> probe was in progress Snapshot command failed real 0m0.522s user 0m0.203s sys 0m0.047s snapshot delete: snap_1_7: snap removed successfully real 0m4.233s user 0m0.218s sys 0m0.038s snapshot delete: snap_1_8: snap removed successfully real 0m2.918s user 0m0.214s sys 0m0.040s . . . snapshot delete: snap_1_20: snap removed successfully real 0m2.043s user 0m0.211s sys 0m0.040s gluster snapshot list still shows snap_1_4 and snap_1_5 [root@snapshot16 ~]# gluster snapshot list snap_1_4 snap_1_5 snap_1_6 snap_1_21 snap_1_22 snap_1_23 gluster snapshot status snap_1_5 Snapshot Status : failed: Commit failed on localhost. Please check log file for details. Snapshot command failed gluster snapshot status snap_1_4 Snapshot Status : failed: Commit failed on localhost. Please check log file for details. Snapshot command failed gluster snapshot info snap_1_5 Snapshot : snap_1_5 Snap UUID : 39d5dfbe-a54e-4637-8151-fbc98118ce4d Created : 2014-06-19 14:51:42 Snap Volumes: Snap Volume Name : 02a8fa61e9674aa1babac102011bb544 Origin Volume name : vol0 Snaps taken for vol0 : 6 Snaps available for vol0 : 250 Status : Started gluster snapshot info snap_1_4 Snapshot : snap_1_4 Snap UUID : 2df58e99-9547-46bf-896f-b07316a1c27b Created : 2014-06-19 14:51:26 Snap Volumes: Snap Volume Name : 62b973d1b0ad408dae74468d923a2112 Origin Volume name : vol0 Snaps taken for vol0 : 6 Snaps available for vol0 : 250 Status : Started df -h does not list it , but gluster snapshot list and info shows the snapshot is still present. Actual results: ============== snap_1_4 and snap_1_5 were deleted successfully, but gluster snapshot list shows the snapshot is still present Expected results: ================ gluster snapshot list should not list the deleted snapshots. Additional info: