Description of problem: Disperse volume crashes while trying to write multiple files using multiple threads on fuse mounted tier volume. Version-Release number of selected component (if applicable): [root@apandey glusterfs]# glusterfs --version glusterfs 3.8dev built on Dec 21 2015 10:49:16 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/> GlusterFS comes with ABSOLUTELY NO WARRANTY. How reproducible: 100% Steps to Reproduce: 1. Create a tier volume with 2 X (4+2) disperse volume and 6 X (2) replica volume. 2. Mount it through fuse. 3. start writing various files with multiple threads on mount point. crefi --multi -n 10 -b 10 -d 10 --max=1024k --min=5k --random -T 5 -t text -I 5 --fop=create /mnt/gfs Actual results: After some time (1 min to 30 min) CRASH happens in disperse volume. Expected results: No Crash should be there and all the read, write and modify operation should be successful. Additional info: [root@apandey glusterfs]# gluster v info Volume Name: vol Type: Tier Volume ID: a9007561-0c50-463c-b37d-59f3992f339e Status: Started Number of Bricks: 24 Transport-type: tcp Hot Tier : Hot Tier Type : Distributed-Replicate Number of Bricks: 6 x 2 = 12 Brick1: apandey:/brick/gluster/r12 Brick2: apandey:/brick/gluster/r11 Brick3: apandey:/brick/gluster/r10 Brick4: apandey:/brick/gluster/r9 Brick5: apandey:/brick/gluster/r8 Brick6: apandey:/brick/gluster/r7 Brick7: apandey:/brick/gluster/r6 Brick8: apandey:/brick/gluster/r5 Brick9: apandey:/brick/gluster/r4 Brick10: apandey:/brick/gluster/r3 Brick11: apandey:/brick/gluster/r2 Brick12: apandey:/brick/gluster/r1 Cold Tier: Cold Tier Type : Distributed-Disperse Number of Bricks: 2 x (4 + 2) = 12 Brick13: apandey:/brick/gluster/v1 Brick14: apandey:/brick/gluster/v2 Brick15: apandey:/brick/gluster/v3 Brick16: apandey:/brick/gluster/v4 Brick17: apandey:/brick/gluster/v5 Brick18: apandey:/brick/gluster/v6 Brick19: apandey:/brick/gluster/v7 Brick20: apandey:/brick/gluster/v8 Brick21: apandey:/brick/gluster/v9 Brick22: apandey:/brick/gluster/v10 Brick23: apandey:/brick/gluster/v11 Brick24: apandey:/brick/gluster/v12 Options Reconfigured: cluster.tier-demote-frequency: 60 cluster.tier-promote-frequency: 60 cluster.write-freq-threshold: 1 cluster.read-freq-threshold: 1 features.record-counters: on cluster.watermark-hi: 5 cluster.watermark-low: 1 cluster.tier-mode: cache features.ctr-enabled: on diagnostics.client-log-level: WARNING performance.readdir-ahead: on
REVIEW: http://review.gluster.org/13039 (cluster/ec: Get size and config for invalid inode) posted (#1) for review on master by Ashish Pandey (aspandey)
REVIEW: http://review.gluster.org/13039 (cluster/ec: Get size and config for invalid inode) posted (#2) for review on master by Ashish Pandey (aspandey)
REVIEW: http://review.gluster.org/13039 (cluster/ec: Get size and config for invalid inode) posted (#3) for review on master by Ashish Pandey (aspandey)
REVIEW: http://review.gluster.org/13039 (cluster/ec: Get size and config for invalid inode) posted (#4) for review on master by Xavier Hernandez (xhernandez)
COMMIT: http://review.gluster.org/13039 committed in master by Pranith Kumar Karampuri (pkarampu) ------ commit 3882408103973eac6983c2efdd5af8b1d51f272c Author: Ashish Pandey <aspandey> Date: Mon Dec 21 16:04:20 2015 +0530 cluster/ec: Get size and config for invalid inode Problem: After creating an inode and before linking it to inode table, if there is a request to setattr for that file, it fails and leads to crash. Before linking inode to inode table ia_type is IA_INVAL which will casue have_size and have_config as zero. Solution: Check and get size and config if an inode is invalid Change-Id: I0c0e564940b1b9f351369a76ab14f6b4aa81f23b BUG: 1293223 Signed-off-by: Ashish Pandey <aspandey> Reviewed-on: http://review.gluster.org/13039 Reviewed-by: Xavier Hernandez <xhernandez> Tested-by: NetBSD Build System <jenkins.org>
REVIEW: http://review.gluster.org/13238 (cluster/ec: Handle non-existent config xattr for non regular files) posted (#1) for review on master by Xavier Hernandez (xhernandez)
REVIEW: http://review.gluster.org/13238 (cluster/ec: Handle non-existent config xattr for non regular files) posted (#2) for review on master by Xavier Hernandez (xhernandez)
REVIEW: http://review.gluster.org/13238 (cluster/ec: Handle non-existent config xattr for non regular files) posted (#3) for review on master by Xavier Hernandez (xhernandez)
REVIEW: http://review.gluster.org/13238 (cluster/ec: Handle non-existent config xattr for non regular files) posted (#4) for review on master by Xavier Hernandez (xhernandez)
COMMIT: http://review.gluster.org/13238 committed in master by Jeff Darcy (jdarcy) ------ commit 0aba36ab99fbf537ba2e5ceddfc950528506011f Author: Xavier Hernandez <xhernandez> Date: Thu Jan 14 09:36:33 2016 +0100 cluster/ec: Handle non-existent config xattr for non regular files Since we now try to get the 'trusted.ec.config' xattr for inodes of type IA_INVAL (these inodes will be set to some valid type later), if that inode corresponds to a non regular file, the xattr won't exist and we will handle this as an error when it's not. This patch solves the problem by only considering errors for inodes that are already known to be regular files. Change-Id: Id72f314e209459236d75cf087fc51e09943756b4 BUG: 1293223 Signed-off-by: Xavier Hernandez <xhernandez> Reviewed-on: http://review.gluster.org/13238 Smoke: Gluster Build System <jenkins.com> Reviewed-by: Ashish Pandey <aspandey> NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.com> Reviewed-by: Jeff Darcy <jdarcy>
REVIEW: http://review.gluster.org/13446 (cluster/ec: Fix invalid config check for directories) posted (#1) for review on master by Xavier Hernandez (xhernandez)
COMMIT: http://review.gluster.org/13446 committed in master by Jeff Darcy (jdarcy) ------ commit 9afe93bee897c8ead507a8dadb0e0fc32ed8abf0 Author: Xavier Hernandez <xhernandez> Date: Mon Feb 15 10:59:29 2016 +0100 cluster/ec: Fix invalid config check for directories The trusted.ec.config xattr is not defined for directories. However sometimes it could be requested because the inode type of a directory can temporarily be IA_INVAL. Requesting such xattr using the xattrop fop when it doesn't exist, returns a config value full of 0's, which is invalid and caused some fops to fail. This patch filters out this case by ignoring config xattr == 0. Change-Id: Ied51c35b313ea8c3eeae27812f9bae61d3808e92 BUG: 1293223 Signed-off-by: Xavier Hernandez <xhernandez> Reviewed-on: http://review.gluster.org/13446 Smoke: Gluster Build System <jenkins.com> CentOS-regression: Gluster Build System <jenkins.com> Reviewed-by: Ashish Pandey <aspandey> NetBSD-regression: NetBSD Build System <jenkins.org> Reviewed-by: Jeff Darcy <jdarcy>
*** Bug 1311368 has been marked as a duplicate of this bug. ***
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report. glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/ [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user