Bug 1293223 - Disperse: Disperse volume (cold vol) crashes while writing files on tier volume
Disperse: Disperse volume (cold vol) crashes while writing files on tier volume
Status: CLOSED CURRENTRELEASE
Product: GlusterFS
Classification: Community
Component: disperse (Show other bugs)
mainline
Unspecified Unspecified
unspecified Severity high
: ---
: ---
Assigned To: Ashish Pandey
:
: 1311368 (view as bug list)
Depends On:
Blocks: 1293224 1293228
  Show dependency treegraph
 
Reported: 2015-12-21 01:35 EST by Ashish Pandey
Modified: 2016-06-16 09:51 EDT (History)
3 users (show)

See Also:
Fixed In Version: glusterfs-3.8rc2
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1293224 1293228 (view as bug list)
Environment:
Last Closed: 2016-06-16 09:51:54 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Ashish Pandey 2015-12-21 01:35:49 EST
Description of problem:

Disperse volume crashes while trying to write multiple files using multiple threads on fuse mounted tier volume.

Version-Release number of selected component (if applicable):
[root@apandey glusterfs]# glusterfs --version
glusterfs 3.8dev built on Dec 21 2015 10:49:16
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/>
GlusterFS comes with ABSOLUTELY NO WARRANTY.


How reproducible:
100%

Steps to Reproduce:
1. Create a tier volume with 2 X (4+2) disperse volume and 6 X (2) replica volume.
2. Mount it through fuse.
3. start writing various files with multiple threads on mount point.
crefi --multi -n 10 -b 10 -d 10 --max=1024k --min=5k --random -T 5 -t text -I 5 --fop=create /mnt/gfs

Actual results:
After some time (1 min to 30 min) CRASH happens in disperse volume.


Expected results:
No Crash should be there and all the read, write and modify operation should be successful. 

Additional info:
[root@apandey glusterfs]# gluster v info
 
Volume Name: vol
Type: Tier
Volume ID: a9007561-0c50-463c-b37d-59f3992f339e
Status: Started
Number of Bricks: 24
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distributed-Replicate
Number of Bricks: 6 x 2 = 12
Brick1: apandey:/brick/gluster/r12
Brick2: apandey:/brick/gluster/r11
Brick3: apandey:/brick/gluster/r10
Brick4: apandey:/brick/gluster/r9
Brick5: apandey:/brick/gluster/r8
Brick6: apandey:/brick/gluster/r7
Brick7: apandey:/brick/gluster/r6
Brick8: apandey:/brick/gluster/r5
Brick9: apandey:/brick/gluster/r4
Brick10: apandey:/brick/gluster/r3
Brick11: apandey:/brick/gluster/r2
Brick12: apandey:/brick/gluster/r1
Cold Tier:
Cold Tier Type : Distributed-Disperse
Number of Bricks: 2 x (4 + 2) = 12
Brick13: apandey:/brick/gluster/v1
Brick14: apandey:/brick/gluster/v2
Brick15: apandey:/brick/gluster/v3
Brick16: apandey:/brick/gluster/v4
Brick17: apandey:/brick/gluster/v5
Brick18: apandey:/brick/gluster/v6
Brick19: apandey:/brick/gluster/v7
Brick20: apandey:/brick/gluster/v8
Brick21: apandey:/brick/gluster/v9
Brick22: apandey:/brick/gluster/v10
Brick23: apandey:/brick/gluster/v11
Brick24: apandey:/brick/gluster/v12
Options Reconfigured:
cluster.tier-demote-frequency: 60
cluster.tier-promote-frequency: 60
cluster.write-freq-threshold: 1
cluster.read-freq-threshold: 1
features.record-counters: on
cluster.watermark-hi: 5
cluster.watermark-low: 1
cluster.tier-mode: cache
features.ctr-enabled: on
diagnostics.client-log-level: WARNING
performance.readdir-ahead: on
Comment 1 Vijay Bellur 2015-12-21 06:05:21 EST
REVIEW: http://review.gluster.org/13039 (cluster/ec: Get size and config for invalid inode) posted (#1) for review on master by Ashish Pandey (aspandey@redhat.com)
Comment 2 Vijay Bellur 2015-12-21 06:06:58 EST
REVIEW: http://review.gluster.org/13039 (cluster/ec: Get size and config for invalid inode) posted (#2) for review on master by Ashish Pandey (aspandey@redhat.com)
Comment 3 Vijay Bellur 2015-12-22 05:57:28 EST
REVIEW: http://review.gluster.org/13039 (cluster/ec: Get size and config for invalid inode) posted (#3) for review on master by Ashish Pandey (aspandey@redhat.com)
Comment 4 Vijay Bellur 2016-01-13 03:31:00 EST
REVIEW: http://review.gluster.org/13039 (cluster/ec: Get size and config for invalid inode) posted (#4) for review on master by Xavier Hernandez (xhernandez@datalab.es)
Comment 5 Vijay Bellur 2016-01-13 06:57:03 EST
COMMIT: http://review.gluster.org/13039 committed in master by Pranith Kumar Karampuri (pkarampu@redhat.com) 
------
commit 3882408103973eac6983c2efdd5af8b1d51f272c
Author: Ashish Pandey <aspandey@redhat.com>
Date:   Mon Dec 21 16:04:20 2015 +0530

    cluster/ec: Get size and config for invalid inode
    
    Problem:
    After creating an inode and before linking it
    to inode table, if there is a request to setattr
    for that file, it fails and leads to crash.
    Before linking inode to inode table ia_type is IA_INVAL
    which will casue have_size and have_config as zero.
    
    Solution:
    Check and get size and config if an inode is invalid
    
    Change-Id: I0c0e564940b1b9f351369a76ab14f6b4aa81f23b
    BUG: 1293223
    Signed-off-by: Ashish Pandey <aspandey@redhat.com>
    Reviewed-on: http://review.gluster.org/13039
    Reviewed-by: Xavier Hernandez <xhernandez@datalab.es>
    Tested-by: NetBSD Build System <jenkins@build.gluster.org>
Comment 6 Vijay Bellur 2016-01-14 03:41:38 EST
REVIEW: http://review.gluster.org/13238 (cluster/ec: Handle non-existent config xattr for non regular files) posted (#1) for review on master by Xavier Hernandez (xhernandez@datalab.es)
Comment 7 Vijay Bellur 2016-01-14 03:51:51 EST
REVIEW: http://review.gluster.org/13238 (cluster/ec: Handle non-existent config xattr for non regular files) posted (#2) for review on master by Xavier Hernandez (xhernandez@datalab.es)
Comment 8 Vijay Bellur 2016-01-15 05:01:16 EST
REVIEW: http://review.gluster.org/13238 (cluster/ec: Handle non-existent config xattr for non regular files) posted (#3) for review on master by Xavier Hernandez (xhernandez@datalab.es)
Comment 9 Vijay Bellur 2016-02-05 03:29:14 EST
REVIEW: http://review.gluster.org/13238 (cluster/ec: Handle non-existent config xattr for non regular files) posted (#4) for review on master by Xavier Hernandez (xhernandez@datalab.es)
Comment 10 Vijay Bellur 2016-02-05 10:02:04 EST
COMMIT: http://review.gluster.org/13238 committed in master by Jeff Darcy (jdarcy@redhat.com) 
------
commit 0aba36ab99fbf537ba2e5ceddfc950528506011f
Author: Xavier Hernandez <xhernandez@datalab.es>
Date:   Thu Jan 14 09:36:33 2016 +0100

    cluster/ec: Handle non-existent config xattr for non regular files
    
    Since we now try to get the 'trusted.ec.config' xattr for inodes of
    type IA_INVAL (these inodes will be set to some valid type later),
    if that inode corresponds to a non regular file, the xattr won't
    exist and we will handle this as an error when it's not.
    
    This patch solves the problem by only considering errors for inodes
    that are already known to be regular files.
    
    Change-Id: Id72f314e209459236d75cf087fc51e09943756b4
    BUG: 1293223
    Signed-off-by: Xavier Hernandez <xhernandez@datalab.es>
    Reviewed-on: http://review.gluster.org/13238
    Smoke: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Ashish Pandey <aspandey@redhat.com>
    NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
Comment 11 Vijay Bellur 2016-02-15 05:03:28 EST
REVIEW: http://review.gluster.org/13446 (cluster/ec: Fix invalid config check for directories) posted (#1) for review on master by Xavier Hernandez (xhernandez@datalab.es)
Comment 12 Vijay Bellur 2016-03-01 02:56:45 EST
COMMIT: http://review.gluster.org/13446 committed in master by Jeff Darcy (jdarcy@redhat.com) 
------
commit 9afe93bee897c8ead507a8dadb0e0fc32ed8abf0
Author: Xavier Hernandez <xhernandez@datalab.es>
Date:   Mon Feb 15 10:59:29 2016 +0100

    cluster/ec: Fix invalid config check for directories
    
    The trusted.ec.config xattr is not defined for directories. However
    sometimes it could be requested because the inode type of a directory
    can temporarily be IA_INVAL.
    
    Requesting such xattr using the xattrop fop when it doesn't exist,
    returns a config value full of 0's, which is invalid and caused some
    fops to fail.
    
    This patch filters out this case by ignoring config xattr == 0.
    
    Change-Id: Ied51c35b313ea8c3eeae27812f9bae61d3808e92
    BUG: 1293223
    Signed-off-by: Xavier Hernandez <xhernandez@datalab.es>
    Reviewed-on: http://review.gluster.org/13446
    Smoke: Gluster Build System <jenkins@build.gluster.com>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Ashish Pandey <aspandey@redhat.com>
    NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
Comment 13 Manikandan 2016-03-01 07:25:03 EST
*** Bug 1311368 has been marked as a duplicate of this bug. ***
Comment 14 Mike McCune 2016-03-28 18:17:27 EDT
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune@redhat.com with any questions
Comment 15 Niels de Vos 2016-06-16 09:51:54 EDT
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report.

glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Note You need to log in before you can comment on or make changes to this bug.