Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1360174

Summary: IO error seen with Rolling or non-disruptive upgrade of an distribute-disperse(EC) volume from 3.7.5 to 3.7.9
Product: [Community] GlusterFS Reporter: Ashish Pandey <aspandey>
Component: disperseAssignee: Ashish Pandey <aspandey>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 3.8.1CC: aspandey, bugs, byarlaga, nchilaka, pkarampu
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.8.2 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1347686 Environment:
Last Closed: 2016-08-12 09:47:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1347251, 1347686    
Bug Blocks: 1360152    

Comment 1 Vijay Bellur 2016-07-26 07:36:09 UTC 
REVIEW: http://review.gluster.org/15013 (cluster/ec: Handle absence of keys in some callback dict) posted (#1) for review on release-3.8 by Ashish Pandey (aspandey)
Comment 2 Vijay Bellur 2016-07-27 05:51:23 UTC 
COMMIT: http://review.gluster.org/15013 committed in release-3.8 by Pranith Kumar Karampuri (pkarampu) 
------
commit 197bbc9d150554b333504cc9616aaa19b937a8a6
Author: Ashish Pandey <aspandey>
Date:   Fri Jun 17 17:52:56 2016 +0530

    cluster/ec: Handle absence of keys in some callback dict
    
    Problem: This issue arises when we do a rolling update
    from 3.7.5 to 3.7.9.
    For 4+2 volume running 3.7.5, if we update 2 nodes
    and after heal completion  kill 2 older nodes, this
    problem can be seen. After update and killing of
    bricks, 2 nodes will return inodelk count key in dict
    while other 2 nodes will not have inodelk count in dict.
    This is also true for get-link-count.
    During dictionary match , ec_dict_compare, this will
    lead to mismatch of answers and the file operation
    on mount point will fail with IO error.
    
    Solution:
    Don't match inode, entry and link count keys while
    comparing two dictionaries. However, while combining the
    data in ec_dict_combine, go through all the dictionaries
    and select the maximum values received in different dicts
    for these keys.
    
    master -
    http://review.gluster.org/#/c/14761/
    
    Change-Id: I33546e3619fe8f909286ee48fb0df2009cd3d22f
    BUG: 1360174
    Signed-off-by: Ashish Pandey <aspandey>
    Reviewed-on: http://review.gluster.org/14761
    Reviewed-by: Xavier Hernandez <xhernandez>
    Smoke: Gluster Build System <jenkins.org>
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>
    CentOS-regression: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    Signed-off-by: Ashish Pandey <aspandey>
    Reviewed-on: http://review.gluster.org/15013
Comment 3 Niels de Vos 2016-08-12 09:47:50 UTC 
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.2, please open a new bug report.

glusterfs-3.8.2 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://www.gluster.org/pipermail/announce/2016-August/000058.html
[2] https://www.gluster.org/pipermail/gluster-users/