Bug 1091926 - [SNAPSHOT] : glusterd crash on 2 nodes while snapshot was in progress when IO was in progress on the client
Summary: [SNAPSHOT] : glusterd crash on 2 nodes while snapshot was in progress when IO...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: glusterd
Version: 3.5.0
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
Assignee: Vijaikumar Mallikarjuna
QA Contact:
URL:
Whiteboard: SNAPSHOT
: 1103685 (view as bug list)
Depends On: 1054705 1088355 1091934
Blocks: 1097217 1103685
TreeView+ depends on / blocked
 
Reported: 2014-04-28 10:42 UTC by Vijaikumar Mallikarjuna
Modified: 2016-05-11 22:47 UTC (History)
10 users (show)

Fixed In Version: glusterfs-3.6.0beta1
Doc Type: Bug Fix
Doc Text:
Clone Of: 1088355
Environment:
Last Closed: 2014-11-11 08:31:08 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Comment 1 Anand Avati 2014-04-28 10:45:27 UTC
REVIEW: http://review.gluster.org/7579 (glusterd/snapshot: Don't release big_lock before completing snapshot creation) posted (#1) for review on master by Vijaikumar Mallikarjuna (vmallika)

Comment 2 Anand Avati 2014-04-30 08:28:16 UTC
REVIEW: http://review.gluster.org/7579 (glusterd/snapshot: Don't release big_lock before completing snapshot creation) posted (#2) for review on master by Vijaikumar Mallikarjuna (vmallika)

Comment 3 Anand Avati 2014-05-02 06:16:02 UTC
REVIEW: http://review.gluster.org/7579 (glusterd/snapshot: Don't release big_lock before completing snapshot creation) posted (#3) for review on master by Vijaikumar Mallikarjuna (vmallika)

Comment 4 Anand Avati 2014-05-05 06:12:03 UTC
REVIEW: http://review.gluster.org/7579 (glusterd/snapshot: Don't release big_lock before completing snapshot creation) posted (#4) for review on master by Vijaikumar Mallikarjuna (vmallika)

Comment 5 Anand Avati 2014-05-06 09:31:18 UTC
REVIEW: http://review.gluster.org/7579 (glusterd/snapshot: Don't release big_lock before completing snapshot creation) posted (#5) for review on master by Vijaikumar Mallikarjuna (vmallika)

Comment 6 Anand Avati 2014-05-07 10:14:14 UTC
REVIEW: http://review.gluster.org/7579 (glusterd/snapshot: Don't release big_lock before completing snapshot creation) posted (#6) for review on master by Rajesh Joseph (rjoseph)

Comment 7 Anand Avati 2014-05-08 07:29:51 UTC
REVIEW: http://review.gluster.org/7579 (glusterd/snapshot: Don't release big_lock before completing snapshot creation) posted (#7) for review on master by Rajesh Joseph (rjoseph)

Comment 8 Anand Avati 2014-05-08 08:09:12 UTC
REVIEW: http://review.gluster.org/7579 (glusterd/snapshot: Don't release big_lock before completing snapshot creation) posted (#8) for review on master by Vijaikumar Mallikarjuna (vmallika)

Comment 9 Anand Avati 2014-05-08 12:06:02 UTC
REVIEW: http://review.gluster.org/7579 (glusterd/snapshot: Don't release big_lock before completing snapshot creation) posted (#9) for review on master by Vijaikumar Mallikarjuna (vmallika)

Comment 10 Anand Avati 2014-05-09 05:25:29 UTC
COMMIT: http://review.gluster.org/7579 committed in master by Krishnan Parthasarathi (kparthas) 
------
commit 26043d1f8ed6892aca47abcb75cedfdc51e86a0a
Author: Vijaikumar M <vmallika>
Date:   Tue May 6 14:56:23 2014 +0530

    glusterd/snapshot: Don't release big_lock before completing snapshot creation
    
    Releasing the big-lock can cause problem like deadlock or memory
    corruption.
    
    Same happened with bug 1091926 where glusterd on node-2 entered a commit
    phase and released a big-lock.
    
    The originator node received timeout for the commit phase and
    triggered a post-validate cleanup to the node-2.
    
    Now node-2 continued to work with the object that are alreday cleaned-up
    and resulted in a crash.
    
    Solution is to not to release big-lock in the commit phase of snapshot
    creation.
    
    Change-Id: I571194fdb0b0ecc91bd13f2a9fc92fe4338d14dc
    BUG: 1091926
    Signed-off-by: Vijaikumar M <vmallika>
    Reviewed-on: http://review.gluster.org/7579
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Rajesh Joseph <rjoseph>
    Reviewed-by: Krishnan Parthasarathi <kparthas>
    Tested-by: Krishnan Parthasarathi <kparthas>

Comment 11 Niels de Vos 2014-07-13 11:17:52 UTC
*** Bug 1103685 has been marked as a duplicate of this bug. ***

Comment 12 Niels de Vos 2014-09-22 12:39:07 UTC
A beta release for GlusterFS 3.6.0 has been released. Please verify if the release solves this bug report for you. In case the glusterfs-3.6.0beta1 release does not have a resolution for this issue, leave a comment in this bug and move the status to ASSIGNED. If this release fixes the problem for you, leave a note and change the status to VERIFIED.

Packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update (possibly an "updates-testing" repository) infrastructure for your distribution.

[1] http://supercolony.gluster.org/pipermail/gluster-users/2014-September/018836.html
[2] http://supercolony.gluster.org/pipermail/gluster-users/

Comment 13 Niels de Vos 2014-11-11 08:31:08 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.6.1, please reopen this bug report.

glusterfs-3.6.1 has been announced [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://supercolony.gluster.org/pipermail/gluster-users/2014-November/019410.html
[2] http://supercolony.gluster.org/mailman/listinfo/gluster-users


Note You need to log in before you can comment on or make changes to this bug.