Bug 1310171 - Incorrect file size on mount if stat is served from the arbiter brick.
Incorrect file size on mount if stat is served from the arbiter brick.
Status: CLOSED CURRENTRELEASE
Product: GlusterFS
Classification: Community
Component: arbiter (Show other bugs)
mainline
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Ravishankar N
:
Depends On:
Blocks: 1313921
  Show dependency treegraph
 
Reported: 2016-02-19 11:25 EST by Ravishankar N
Modified: 2016-06-16 09:57 EDT (History)
2 users (show)

See Also:
Fixed In Version: glusterfs-3.8rc2
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1313921 (view as bug list)
Environment:
Last Closed: 2016-06-16 09:57:48 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Ravishankar N 2016-02-19 11:25:06 EST
Description of problem:
glusterfs user Mat Clayton told me on IRC that sometimes `ls -l of /mount/filename` on an arbiter volume reported zero bytes despite all bricks being up and the actual data file on the non-arbiter bricks being 120 Meg.

Version-Release number of selected component (if applicable):
v3.7.8

How reproducible:
Sometimes the value is right and other times it is zero.
Comment 1 Ravishankar N 2016-02-19 11:32:47 EST
RCA:I was able to re-create the issue intermittently. Basically if afr_stat() gets called on the file whose gfid hashes to the arbiter brick, afr_read_subvol_select_by_policy() picks up the arbiter brick to wind the call and hence we end up serving zero as the iatt size.
Comment 2 Vijay Bellur 2016-02-29 00:33:38 EST
REVIEW: http://review.gluster.org/13539 (afr: do not set arbiter as a readable subvol in inode context) posted (#1) for review on master by Ravishankar N (ravishankar@redhat.com)
Comment 3 Vijay Bellur 2016-03-01 11:59:09 EST
REVIEW: http://review.gluster.org/13539 (afr: do not set arbiter as a readable subvol in inode context) posted (#2) for review on master by Ravishankar N (ravishankar@redhat.com)
Comment 4 Vijay Bellur 2016-03-01 23:07:00 EST
REVIEW: http://review.gluster.org/13539 (afr: do not set arbiter as a readable subvol in inode context) posted (#3) for review on master by Ravishankar N (ravishankar@redhat.com)
Comment 5 Vijay Bellur 2016-03-03 07:58:18 EST
REVIEW: http://review.gluster.org/13539 (afr: do not set arbiter as a readable subvol in inode context) posted (#4) for review on master by Ravishankar N (ravishankar@redhat.com)
Comment 6 Vijay Bellur 2016-03-03 12:03:27 EST
REVIEW: http://review.gluster.org/13539 (afr: do not set arbiter as a readable subvol in inode context) posted (#5) for review on master by Ravishankar N (ravishankar@redhat.com)
Comment 7 Vijay Bellur 2016-03-04 05:16:30 EST
REVIEW: http://review.gluster.org/13539 (afr: do not set arbiter as a readable subvol in inode context) posted (#6) for review on master by Ravishankar N (ravishankar@redhat.com)
Comment 8 Vijay Bellur 2016-03-04 05:19:41 EST
REVIEW: http://review.gluster.org/13539 (afr: do not set arbiter as a readable subvol in inode context) posted (#7) for review on master by Ravishankar N (ravishankar@redhat.com)
Comment 9 Vijay Bellur 2016-03-04 06:30:44 EST
REVIEW: http://review.gluster.org/13539 (afr: do not set arbiter as a readable subvol in inode context) posted (#8) for review on master by Ravishankar N (ravishankar@redhat.com)
Comment 10 Vijay Bellur 2016-03-04 12:32:25 EST
COMMIT: http://review.gluster.org/13539 committed in master by Jeff Darcy (jdarcy@redhat.com) 
------
commit 8ab87137c855c9a0551a9100c146e4ca76dbbda2
Author: Ravishankar N <ravishankar@redhat.com>
Date:   Mon Feb 29 05:16:50 2016 +0000

    afr: do not set arbiter as a readable subvol in inode context
    
    Problem:
    If afr_lookup_done() or afr_read_subvol_select_by_policy() chooses the
    arbiter brick to serve the stat() data, file size will be reported as
    zero from the mount, despite other data bricks being available. This can
    break programs like tar which use the stat info to decide how much to read.
    
    Fix:
    In the inode-context, mark arbiter as a non-readable subvol for both
    data and metadata.
    
    It it to be noted that by making this fix, we are *not* going to serve
    metadata FOPS anymore from the arbiter brick despite the brick storing
    the metadata. It makes sense to do this because the ever increasing
    over-loaded FOPs (getxattr returning stat data etc.) and compound FOPS
    in gluster will otherwise make it difficult to add checks in code to
    handle corner cases.
    
    Change-Id: Ic60b25d77fd05e0897481b7fcb3716d4f2101001
    BUG: 1310171
    Signed-off-by: Ravishankar N <ravishankar@redhat.com>
    Reported-by: Mat Clayton <mat@mixcloud.com>
    Reviewed-on: http://review.gluster.org/13539
    Reviewed-by: Anuradha Talur <atalur@redhat.com>
    Reviewed-by: Krutika Dhananjay <kdhananj@redhat.com>
    Smoke: Gluster Build System <jenkins@build.gluster.com>
    NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
Comment 11 Vijay Bellur 2016-03-05 11:12:44 EST
REVIEW: http://review.gluster.org/13611 (tests: Use fource_umount instead of umount) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu@redhat.com)
Comment 12 Vijay Bellur 2016-03-09 03:38:25 EST
REVIEW: http://review.gluster.org/13611 (tests: Use force_umount instead of umount) posted (#2) for review on master by Pranith Kumar Karampuri (pkarampu@redhat.com)
Comment 13 Vijay Bellur 2016-03-09 03:38:57 EST
REVIEW: http://review.gluster.org/13611 (tests: Use force_umount instead of umount) posted (#3) for review on master by Pranith Kumar Karampuri (pkarampu@redhat.com)
Comment 14 Vijay Bellur 2016-03-09 03:39:14 EST
REVIEW: http://review.gluster.org/13611 (tests: Use force_umount instead of umount) posted (#4) for review on master by Pranith Kumar Karampuri (pkarampu@redhat.com)
Comment 15 Vijay Bellur 2016-03-11 03:37:50 EST
COMMIT: http://review.gluster.org/13611 committed in master by Pranith Kumar Karampuri (pkarampu@redhat.com) 
------
commit 805a9c7a40e22f811019f4a854636c30c164525b
Author: Pranith Kumar K <pkarampu@redhat.com>
Date:   Sat Mar 5 21:41:33 2016 +0530

    tests: Use force_umount instead of umount
    
    umount leads to spurious failures with "mount is busy" kind of errors
    at the time of umount. Use 'force_umount' instead.
    
    BUG: 1310171
    Change-Id: I5a5579288f002de14effc00b793143fef86eb828
    Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
    Reviewed-on: http://review.gluster.org/13611
    NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    Smoke: Gluster Build System <jenkins@build.gluster.com>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Ravishankar N <ravishankar@redhat.com>
    Reviewed-by: Krutika Dhananjay <kdhananj@redhat.com>
Comment 16 Niels de Vos 2016-06-16 09:57:48 EDT
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report.

glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Note You need to log in before you can comment on or make changes to this bug.