Bug 1297213

Summary: Stale stat information for corrupted objects (replicated volume)
Product: [Community] GlusterFS Reporter: Venky Shankar <vshankar>
Component: bitrotAssignee: Venky Shankar <vshankar>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: medium Docs Contact: bugs <bugs>
Priority: medium    
Version: 3.7.7CC: bugs, hgowtham, khiremat, ndevos
Target Milestone: ---Keywords: Patch, Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.7.7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1296399 Environment:
Last Closed: 2016-04-19 07:52:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1296399, 1297300    
Bug Blocks: 1279240    

Description Venky Shankar 2016-01-10 16:07:04 UTC
+++ This bug was initially created as a clone of Bug #1296399 +++

Description of problem:
Stale stat information returned when an object is corrupted on a replicated volume.

Version-Release number of selected component (if applicable):
mainline

How reproducible:
always

Steps to Reproduce:
1. Create a replicated volume, enable bitrot, mount, perform I/O
2. Wait for files to get signed
3. Corrupt an object (modify form brick directly) from one of the replica (preferably first replica)
4. Wait till scrubber marks the object as corrupted
5. Perform I/O on the file again
6. stat to check object size

Actual results:
Stale (+ incorrect) stat information (size, mtime, etc..) reported

Expected results:
Correct stat information (updated size, mtim, etc..) should be returned

Additional info:

--- Additional comment from Vijay Bellur on 2016-01-07 02:22:37 EST ---

REVIEW: http://review.gluster.org/13120 (features/bitrot: add check for corrupted object in f{stat}) posted (#3) for review on master by Venky Shankar (vshankar)

--- Additional comment from Vijay Bellur on 2016-01-10 07:43:33 EST ---

COMMIT: http://review.gluster.org/13120 committed in master by Venky Shankar (vshankar) 
------
commit d5d6918ce7dc9f54496da435af546611dfbe7d5c
Author: Venky Shankar <vshankar>
Date:   Wed Dec 30 14:56:12 2015 +0530

    features/bitrot: add check for corrupted object in f{stat}
    
    Check for corrupted objects is done bt bitrot stub component
    for data operations and such fops are denied processing by
    returning EIO. These checks were not done for operations such
    as get/set extended attribute, stat and the likes - IOW, stub
    only blocked pure data operations.
    
    However, its necessary to have these checks for certain other
    fops, most importantly stat (and fstat). This is due to the
    fact that clients could possibly get stale stat information
    (such as size, {a,c,m}time) resulting in incorrect operation
    of the application that rely on these fields. Note that, the
    data that replication would take care of fetching good (and
    correct) data, but the staleness of stat information could
    lead to data inconsistencies (e.g., rebalance, tier).
    
    Change-Id: I5a22780373b182a13f8d2c4ca6b7d9aa0ffbfca3
    BUG: 1296399
    Signed-off-by: Venky Shankar <vshankar>
    Reviewed-on: http://review.gluster.org/13120
    Reviewed-by: Kotresh HR <khiremat>
    Reviewed-by: mohammed rafi  kc <rkavunga>
    Reviewed-by: Raghavendra Bhat <raghavendra>
    Tested-by: NetBSD Build System <jenkins.org>
    Tested-by: Gluster Build System <jenkins.com>

Comment 1 Vijay Bellur 2016-01-22 06:00:36 UTC
REVIEW: http://review.gluster.org/13276 (features/bitrot: add check for corrupted object in f{stat}) posted (#1) for review on release-3.7 by Venky Shankar (vshankar)

Comment 2 Vijay Bellur 2016-01-26 02:19:57 UTC
REVIEW: http://review.gluster.org/13276 (features/bitrot: add check for corrupted object in f{stat}) posted (#2) for review on release-3.7 by Pranith Kumar Karampuri (pkarampu)

Comment 3 Vijay Bellur 2016-01-27 02:45:07 UTC
COMMIT: http://review.gluster.org/13276 committed in release-3.7 by Pranith Kumar Karampuri (pkarampu) 
------
commit 0bfb02e3b56abf411882f245655921a91df885ef
Author: Venky Shankar <vshankar>
Date:   Wed Dec 30 14:56:12 2015 +0530

    features/bitrot: add check for corrupted object in f{stat}
    
             Backport of http://review.gluster.org/13120
    
    Check for corrupted objects is done bt bitrot stub component
    for data operations and such fops are denied processing by
    returning EIO. These checks were not done for operations such
    as get/set extended attribute, stat and the likes - IOW, stub
    only blocked pure data operations.
    
    However, its necessary to have these checks for certain other
    fops, most importantly stat (and fstat). This is due to the
    fact that clients could possibly get stale stat information
    (such as size, {a,c,m}time) resulting in incorrect operation
    of the application that rely on these fields. Note that, the
    data that replication would take care of fetching good (and
    correct) data, but the staleness of stat information could
    lead to data inconsistencies (e.g., rebalance, tier).
    
    Change-Id: I5a22780373b182a13f8d2c4ca6b7d9aa0ffbfca3
    BUG: 1297213
    Signed-off-by: Venky Shankar <vshankar>
    Reviewed-on: http://review.gluster.org/13276
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>
    Tested-by: Pranith Kumar Karampuri <pkarampu>
    Smoke: Gluster Build System <jenkins.com>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.com>

Comment 4 Kaushal 2016-04-19 07:52:37 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.7, please open a new bug report.

glusterfs-3.7.7 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] https://www.gluster.org/pipermail/gluster-users/2016-February/025292.html
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user