Bug 1302948 - tar complains: <fileName>: file changed as we read it
Summary: tar complains: <fileName>: file changed as we read it
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: distribute
Version: mainline
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
Assignee: Sakshi
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks: 1312721 1335285
TreeView+ depends on / blocked
 
Reported: 2016-01-29 06:10 UTC by Sakshi
Modified: 2018-01-10 08:07 UTC (History)
10 users (show)

Fixed In Version: glusterfs-3.8rc2
Doc Type: Bug Fix
Doc Text:
Clone Of: 1298724
: 1312721 1335285 (view as bug list)
Environment:
Last Closed: 2016-06-16 13:56:06 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Description Sakshi 2016-01-29 06:10:48 UTC
+++ This bug was initially created as a clone of Bug #1298724 +++

Description of problem:
Encounter 'file changed as we read it' problem, with quota disabled:

# pwd
# tar czvf linux-4.4.tar.gz.test linux-4.4
linux-4.4/
linux-4.4/Kbuild
tar: linux-4.4/Kbuild: file changed as we read it
linux-4.4/block/
linux-4.4/block/blk-lib.c
tar: linux-4.4/block/blk-lib.c: file changed as we read it
linux-4.4/block/blk-settings.c
tar: linux-4.4/block/blk-settings.c: file changed as we read it
linux-4.4/block/scsi_ioctl.c
linux-4.4/block/blk-mq.h
linux-4.4/block/noop-iosched.c
linux-4.4/block/Kconfig.iosched
tar: linux-4.4/block/Kconfig.iosched: file changed as we read it
linux-4.4/block/blk-throttle.c
linux-4.4/block/ioprio.c
linux-4.4/block/t10-pi.c
linux-4.4/block/Kconfig
linux-4.4/block/bio.c
linux-4.4/block/blk-exec.c
linux-4.4/block/blk-ioc.c
linux-4.4/block/blk-iopoll.c
linux-4.4/block/blk-map.c
linux-4.4/block/blk-timeout.c
linux-4.4/block/compat_ioctl.c
linux-4.4/block/deadline-iosched.c
linux-4.4/block/elevator.c
linux-4.4/block/partition-generic.c
linux-4.4/block/Makefile
linux-4.4/block/blk-cgroup.c
linux-4.4/block/blk-core.c
tar: linux-4.4/block/blk-core.c: file changed as we read it
linux-4.4/block/blk-mq-cpu.c
tar: linux-4.4/block/blk-mq-cpu.c: file changed as we read it
linux-4.4/block/bsg.c
tar: linux-4.4/block/bsg.c: file changed as we read it
linux-4.4/block/cfq-iosched.c
tar: linux-4.4/block/cfq-iosched.c: file changed as we read it
linux-4.4/block/cmdline-parser.c
tar: linux-4.4/block/cmdline-parser.c: file changed as we read it
linux-4.4/block/genhd.c
linux-4.4/block/bio-integrity.c
linux-4.4/block/blk-merge.c
linux-4.4/block/blk-softirq.c
linux-4.4/block/blk-tag.c
linux-4.4/block/bounce.c
linux-4.4/block/blk-mq-cpumap.c
tar: linux-4.4/block/blk-mq-cpumap.c: file changed as we read it
linux-4.4/block/blk-mq-sysfs.c
tar: linux-4.4/block/blk-mq-sysfs.c: file changed as we read it
linux-4.4/block/blk-mq-tag.h
tar: linux-4.4/block/blk-mq-tag.h: file changed as we read it
linux-4.4/block/blk-mq.c
linux-4.4/block/blk-sysfs.c
linux-4.4/block/blk.h
linux-4.4/block/partitions/
linux-4.4/block/partitions/amiga.c
linux-4.4/block/partitions/efi.h
linux-4.4/block/partitions/karma.h
linux-4.4/block/partitions/ldm.c
tar: linux-4.4/block/partitions/ldm.c: file changed as we read it
linux-4.4/block/partitions/msdos.c
tar: linux-4.4/block/partitions/msdos.c: file changed as we read it
linux-4.4/block/partitions/osf.h
linux-4.4/block/partitions/atari.c
linux-4.4/block/partitions/cmdline.c
linux-4.4/block/partitions/mac.h
linux-4.4/block/partitions/sgi.h
linux-4.4/block/partitions/sun.h
linux-4.4/block/partitions/Kconfig
linux-4.4/block/partitions/ldm.h
tar: linux-4.4/block/partitions/ldm.h: file changed as we read it
linux-4.4/block/partitions/msdos.h
linux-4.4/block/partitions/sysv68.c
linux-4.4/block/partitions/sysv68.h
linux-4.4/block/partitions/Makefile
linux-4.4/block/partitions/acorn.h
linux-4.4/block/partitions/sgi.c
linux-4.4/block/partitions/atari.h
linux-4.4/block/partitions/efi.c
tar: linux-4.4/block/partitions/efi.c: file changed as we read it
linux-4.4/block/partitions/ibm.c
linux-4.4/block/partitions/aix.h
linux-4.4/block/partitions/check.h
linux-4.4/block/partitions/karma.c
linux-4.4/block/partitions/mac.c
linux-4.4/block/partitions/osf.c
linux-4.4/block/partitions/ultrix.h
linux-4.4/block/partitions/acorn.c
linux-4.4/block/partitions/amiga.h
linux-4.4/block/partitions/check.c
linux-4.4/block/partitions/sun.c
tar: linux-4.4/block/partitions/sun.c: file changed as we read it
linux-4.4/block/partitions/aix.c
linux-4.4/block/partitions/cmdline.h
linux-4.4/block/partitions/ibm.h
linux-4.4/block/partitions/ultrix.c
tar: linux-4.4/block/partitions: file changed as we read it
linux-4.4/block/blk-flush.c
linux-4.4/block/blk-integrity.c
linux-4.4/block/blk-mq-tag.c
linux-4.4/block/bsg-lib.c
linux-4.4/block/ioctl.c
tar: linux-4.4/block: file changed as we read it
linux-4.4/certs/
linux-4.4/certs/system_keyring.c
linux-4.4/certs/system_certificates.S
linux-4.4/certs/.gitignore
linux-4.4/certs/Kconfig
tar: linux-4.4/certs/Kconfig: file changed as we read it
linux-4.4/certs/Makefile
tar: linux-4.4/certs: file changed as we read it
linux-4.4/fs/
linux-4.4/fs/affs/
linux-4.4/fs/affs/inode.c
linux-4.4/fs/affs/symlink.c
tar: linux-4.4/fs/affs/symlink.c: file changed as we read it
linux-4.4/fs/affs/affs.h
tar: linux-4.4/fs/affs/affs.h: file changed as we read it
linux-4.4/fs/affs/amigaffs.c
linux-4.4/fs/affs/dir.c
tar: linux-4.4/fs/affs/dir.c: file changed as we read it
linux-4.4/fs/affs/super.c
linux-4.4/fs/affs/Changes
linux-4.4/fs/affs/Kconfig
linux-4.4/fs/affs/file.c
linux-4.4/fs/affs/Makefile
tar: linux-4.4/fs/affs/Makefile: file changed as we read it
linux-4.4/fs/affs/bitmap.c
linux-4.4/fs/affs/namei.c
tar: linux-4.4/fs/affs: file changed as we read it
linux-4.4/fs/coda/
linux-4.4/fs/coda/dir.c
linux-4.4/fs/coda/upcall.c
linux-4.4/fs/coda/Kconfig
linux-4.4/fs/coda/file.c
linux-4.4/fs/coda/Makefile
linux-4.4/fs/coda/coda_linux.h
linux-4.4/fs/coda/pioctl.c
linux-4.4/fs/coda/coda_cache.h
tar: linux-4.4/fs/coda/coda_cache.h: file changed as we read it
linux-4.4/fs/coda/coda_fs_i.h
linux-4.4/fs/coda/inode.c
linux-4.4/fs/coda/psdev.c
linux-4.4/fs/coda/symlink.c
linux-4.4/fs/coda/sysctl.c
linux-4.4/fs/coda/cache.c
linux-4.4/fs/coda/coda_int.h
linux-4.4/fs/coda/coda_linux.c
linux-4.4/fs/coda/cnode.c
tar: linux-4.4/fs/coda: file changed as we read it
linux-4.4/fs/dcookies.c
tar: linux-4.4/fs/dcookies.c: file changed as we read it
linux-4.4/fs/exec.c
tar: linux-4.4/fs/exec.c: file changed as we read it
linux-4.4/fs/fs-writeback.c
linux-4.4/fs/hfs/
linux-4.4/fs/hfs/Makefile
linux-4.4/fs/hfs/string.c
tar: linux-4.4/fs/hfs/string.c: file changed as we read it
linux-4.4/fs/hfs/bitmap.c
linux-4.4/fs/hfs/bnode.c
linux-4.4/fs/hfs/btree.h
linux-4.4/fs/hfs/hfs.h
linux-4.4/fs/hfs/hfs_fs.h
linux-4.4/fs/hfs/attr.c
tar: linux-4.4/fs/hfs/attr.c: file changed as we read it
linux-4.4/fs/hfs/btree.c
linux-4.4/fs/hfs/inode.c
linux-4.4/fs/hfs/mdb.c
linux-4.4/fs/hfs/trans.c
tar: linux-4.4/fs/hfs/trans.c: file changed as we read it
linux-4.4/fs/hfs/part_tbl.c
linux-4.4/fs/hfs/sysdep.c
linux-4.4/fs/hfs/brec.c
tar: linux-4.4/fs/hfs/brec.c: file changed as we read it
linux-4.4/fs/hfs/catalog.c
linux-4.4/fs/hfs/dir.c
linux-4.4/fs/hfs/super.c
tar: linux-4.4/fs/hfs/super.c: file changed as we read it
linux-4.4/fs/hfs/Kconfig
linux-4.4/fs/hfs/bfind.c
linux-4.4/fs/hfs/extent.c
tar: linux-4.4/fs/hfs: file changed as we read it
linux-4.4/fs/ioctl.c
linux-4.4/fs/jffs2/
linux-4.4/fs/jffs2/README.Locking
linux-4.4/fs/jffs2/acl.h
tar: linux-4.4/fs/jffs2/acl.h: file changed as we read it
linux-4.4/fs/jffs2/compr.h
linux-4.4/fs/jffs2/compr_rubin.c
tar: linux-4.4/fs/jffs2/compr_rubin.c: file changed as we read it
linux-4.4/fs/jffs2/gc.c
linux-4.4/fs/jffs2/jffs2_fs_sb.h
tar: linux-4.4/fs/jffs2/jffs2_fs_sb.h: file changed as we read it
linux-4.4/fs/jffs2/nodelist.c
linux-4.4/fs/jffs2/acl.c
linux-4.4/fs/jffs2/debug.h
linux-4.4/fs/jffs2/os-linux.h
linux-4.4/fs/jffs2/readinode.c
linux-4.4/fs/jffs2/scan.c
linux-4.4/fs/jffs2/xattr_trusted.c
linux-4.4/fs/jffs2/LICENCE
tar: linux-4.4/fs/jffs2/LICENCE: file changed as we read it
linux-4.4/fs/jffs2/build.c
tar: linux-4.4/fs/jffs2/build.c: file changed as we read it
linux-4.4/fs/jffs2/dir.c
tar: linux-4.4/fs/jffs2/dir.c: file changed as we read it
linux-4.4/fs/jffs2/super.c
linux-4.4/fs/jffs2/xattr.c
linux-4.4/fs/jffs2/Kconfig
linux-4.4/fs/jffs2/TODO
linux-4.4/fs/jffs2/file.c
linux-4.4/fs/jffs2/fs.c
linux-4.4/fs/jffs2/nodemgmt.c
linux-4.4/fs/jffs2/Makefile
linux-4.4/fs/jffs2/background.c
linux-4.4/fs/jffs2/compr.c
linux-4.4/fs/jffs2/compr_lzo.c
linux-4.4/fs/jffs2/jffs2_fs_i.h
linux-4.4/fs/jffs2/xattr_user.c
tar: linux-4.4/fs/jffs2/xattr_user.c: file changed as we read it
linux-4.4/fs/jffs2/summary.h
linux-4.4/fs/jffs2/wbuf.c
linux-4.4/fs/jffs2/erase.c
linux-4.4/fs/jffs2/malloc.c
tar: linux-4.4/fs/jffs2/malloc.c: file changed as we read it
linux-4.4/fs/jffs2/nodelist.h
tar: linux-4.4/fs/jffs2/nodelist.h: file changed as we read it
linux-4.4/fs/jffs2/read.c
tar: linux-4.4/fs/jffs2/read.c: file changed as we read it
linux-4.4/fs/jffs2/summary.c
tar: linux-4.4/fs/jffs2/summary.c: file changed as we read it
linux-4.4/fs/jffs2/symlink.c
tar: linux-4.4/fs/jffs2/symlink.c: file changed as we read it
linux-4.4/fs/jffs2/writev.c
linux-4.4/fs/jffs2/compr_rtime.c
linux-4.4/fs/jffs2/compr_zlib.c
linux-4.4/fs/jffs2/debug.c
linux-4.4/fs/jffs2/ioctl.c
linux-4.4/fs/jffs2/security.c
linux-4.4/fs/jffs2/write.c
linux-4.4/fs/jffs2/xattr.h
tar: linux-4.4/fs/jffs2: file changed as we read it
linux-4.4/fs/ncpfs/
linux-4.4/fs/ncpfs/dir.c
linux-4.4/fs/ncpfs/ncplib_kernel.h
tar: linux-4.4/fs/ncpfs/ncplib_kernel.h: file changed as we read it
linux-4.4/fs/ncpfs/ncpsign_kernel.c
linux-4.4/fs/ncpfs/Kconfig
linux-4.4/fs/ncpfs/file.c
linux-4.4/fs/ncpfs/getopt.h
linux-4.4/fs/ncpfs/ncplib_kernel.c
linux-4.4/fs/ncpfs/Makefile
tar: linux-4.4/fs/ncpfs/Makefile: file changed as we read it
linux-4.4/fs/ncpfs/ncp_fs_sb.h
linux-4.4/fs/ncpfs/sock.c
linux-4.4/fs/ncpfs/inode.c
tar: linux-4.4/fs/ncpfs/inode.c: file changed as we read it
linux-4.4/fs/ncpfs/ncp_fs_i.h
tar: linux-4.4/fs/ncpfs/ncp_fs_i.h: file changed as we read it
linux-4.4/fs/ncpfs/symlink.c
linux-4.4/fs/ncpfs/ioctl.c
linux-4.4/fs/ncpfs/getopt.c
linux-4.4/fs/ncpfs/ncp_fs.h
tar: linux-4.4/fs/ncpfs/ncp_fs.h: file changed as we read it
linux-4.4/fs/ncpfs/mmap.c
linux-4.4/fs/ncpfs/ncpsign_kernel.h
tar: linux-4.4/fs/ncpfs: file changed as we read it
linux-4.4/fs/notify/
linux-4.4/fs/notify/Makefile
linux-4.4/fs/notify/notification.c
linux-4.4/fs/notify/dnotify/
linux-4.4/fs/notify/dnotify/Makefile
tar: linux-4.4/fs/notify/dnotify/Makefile: file changed as we read it
linux-4.4/fs/notify/dnotify/Kconfig
linux-4.4/fs/notify/dnotify/dnotify.c
tar: linux-4.4/fs/notify/dnotify: file changed as we read it
linux-4.4/fs/notify/fanotify/
linux-4.4/fs/notify/fanotify/Kconfig
tar: linux-4.4/fs/notify/fanotify/Kconfig: file changed as we read it
linux-4.4/fs/notify/fanotify/Makefile
linux-4.4/fs/notify/fanotify/fanotify_user.c
linux-4.4/fs/notify/fanotify/fanotify.c
tar: linux-4.4/fs/notify/fanotify/fanotify.c: file changed as we read it
linux-4.4/fs/notify/fanotify/fanotify.h
tar: linux-4.4/fs/notify/fanotify: file changed as we read it
linux-4.4/fs/notify/fdinfo.c
linux-4.4/fs/notify/fdinfo.h
linux-4.4/fs/notify/group.c
linux-4.4/fs/notify/fsnotify.c
linux-4.4/fs/notify/fsnotify.h
linux-4.4/fs/notify/mark.c
linux-4.4/fs/notify/inode_mark.c
linux-4.4/fs/notify/inotify/
linux-4.4/fs/notify/inotify/Kconfig
linux-4.4/fs/notify/inotify/Makefile
linux-4.4/fs/notify/inotify/inotify_user.c
linux-4.4/fs/notify/inotify/inotify.h
linux-4.4/fs/notify/inotify/inotify_fsnotify.c
tar: linux-4.4/fs/notify/inotify: file changed as we read it
linux-4.4/fs/notify/vfsmount_mark.c
tar: linux-4.4/fs/notify/vfsmount_mark.c: file changed as we read it


Version-Release number of selected component (if applicable):
glusterfs-3.7.1-11.el7rhgs.x86_64

How reproducible:
Always

Steps to Reproduce:
1.Run 'tar czvf linux-4.4.tar.gz.test linux-4.4'
2.
3.

Actual results:
tar: linux-4.4/fs/notify/inotify: file changed as we read it

Expected results:
tar should not complain

Additional info:
Same symptoms as the below bug which was fixed in glusterfs-3.7.1-6


Bug 1223757 - tar on a glusterfs mount displays "file changed as we read it" even though the file was not changed

--- Additional comment from Pranith Kumar K on 2016-01-18 20:13:40 EST ---

Anuradha is looking at this bug.


--- Additional comment from Anuradha on 2016-01-20 20:15:57 EST ---

Hi,                                                                            
                                                                               
I was able to recreate the issue yesterday.                                    
On debugging (with Pranith and Krutika), cause for "file changed as we read it" 
for directories was found. Further debugging needs to be done to understand why 
the message is displayed for files. I am going to apply a fix in my setup for  
the cause found yesterday and see the results.                                 
                                                                               
Will keep you posted.                                                          
                                                                               
Thanks,                                                                        
Anuradha.

--- Additional comment from Anuradha on 2016-01-21 09:20:01 EST ---

Hi,

After testing with the fix, as mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1298724#c5, I have not seen the issue on FUSE mount yet. Will test more.


--- Additional comment from Anuradha on 2016-01-22 11:33:39 EST ---

Hi,

As mentioned in the previous comment, I wasn't able to recreate the issue      
on a FUSE mount with the fix applied. But, the issue appeared on NFS mount.    On debugging, the following RCA was found (thanks to Krutika Dhananjay for finding the root cause):    

When a directory is created in a distribute or a distribute-replicate volume,
after creating the directory, dht sets its layout using setxattr call.
So the ctime is updated 2 times for the directory (say, t1 is the first ctime after mkdir, and t2 is the latest one after setxattr). The ctime to be returned to the application should be t2. But, dht doesn't update ctime from its perspective, causing it to return t1 to the upper layers, thus to the application.

In case of NFS mount in our test setup, it appears that NFS caches this metadata and provides it to tar during tar's first query. After the archiving is done, tar makes another query for comparison. At this point, glusterfs is queried during which dht returns the latest value that it finds (that is t2). This leads to mismatch in ctime seen by tar application.

The mismatch in these ctimes causes file changed as we read it warning.

Moving the component to dht as the root cause of the issue lies in dht.

Thanks,
Anuradha.

Comment 1 Vijay Bellur 2016-02-04 10:55:50 UTC
REVIEW: http://review.gluster.org/13352 (dht: mkdir must unwind with latest ctime) posted (#1) for review on master by Sakshi Bansal

Comment 2 Vijay Bellur 2016-02-24 07:53:55 UTC
REVIEW: http://review.gluster.org/13352 (dht: mkdir must unwind with latest ctime) posted (#2) for review on master by Sakshi Bansal

Comment 3 Vijay Bellur 2016-02-26 12:00:32 UTC
COMMIT: http://review.gluster.org/13352 committed in master by Raghavendra G (rgowdapp) 
------
commit dbf0ca53a5704c7f94257fc4223b41865458301a
Author: Sakshi Bansal <sabansal>
Date:   Thu Feb 4 16:02:59 2016 +0530

    dht: mkdir must unwind with latest ctime
    
    Currently fops like mkdir used the the ctime it gets
    after creating the directory entry. But setting layout
    also updates the ctime of a directory. Hence DHT must
    get the ctime after the setxattr call and unwind with
    the latest ctime to avoid mismatch in time seen by
    applications like tar.
    
    Change-Id: Iecbbe3aac5244af5da9788b48ccf299ca56b4bae
    BUG: 1302948
    Signed-off-by: Sakshi Bansal <sabansal>
    Reviewed-on: http://review.gluster.org/13352
    Smoke: Gluster Build System <jenkins.com>
    CentOS-regression: Gluster Build System <jenkins.com>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    Reviewed-by: Raghavendra G <rgowdapp>

Comment 4 Vijay Bellur 2016-03-19 04:47:58 UTC
REVIEW: http://review.gluster.org/13781 (dht: lock around dht_iatt_merge after setting layout) posted (#1) for review on master by Sakshi Bansal

Comment 5 Vijay Bellur 2016-03-19 05:23:48 UTC
REVIEW: http://review.gluster.org/13782 (NFS: do not fill attr information when returning      directory entries in readdirp) posted (#1) for review on master by Sakshi Bansal

Comment 6 Vijay Bellur 2016-03-19 05:48:11 UTC
REVIEW: http://review.gluster.org/13781 (dht: update attr information in refresh layout to avoid      stale timestamp) posted (#2) for review on master by Sakshi Bansal

Comment 7 Vijay Bellur 2016-03-19 07:48:18 UTC
REVIEW: http://review.gluster.org/13770 (dht: report constant directory size) posted (#3) for review on master by N Balachandran (nbalacha)

Comment 8 Vijay Bellur 2016-03-19 07:53:34 UTC
REVIEW: http://review.gluster.org/13782 (NFS: do not fill attr information when returning      directory entries in readdirp) posted (#2) for review on master by Sakshi Bansal

Comment 9 Vijay Bellur 2016-03-19 08:07:28 UTC
REVIEW: http://review.gluster.org/13770 (dht: report constant directory size) posted (#4) for review on master by N Balachandran (nbalacha)

Comment 10 Vijay Bellur 2016-03-19 09:34:31 UTC
REVIEW: http://review.gluster.org/13770 (dht: report constant directory size) posted (#5) for review on master by N Balachandran (nbalacha)

Comment 11 Vijay Bellur 2016-03-20 21:37:49 UTC
COMMIT: http://review.gluster.org/13770 committed in master by Vijay Bellur (vbellur) 
------
commit 942d127fee13653daba49195692c5e8465f1fbff
Author: Jeff Darcy <jdarcy>
Date:   Thu Mar 17 13:33:34 2016 -0400

    dht: report constant directory size
    
    Directory size is meaningless.  Every filesystem has its own
    unpredictable way of increasing or decreasing it, based on internal data
    structures and even transient conditions.  Some filesystems (e.g. ext4)
    never decrease it at all.  Others (e.g. btrfs) don't even report it.
    Very few programs look at it, and those that do are broken.
    Unfortunately, one such program is GNU tar, which will complain when it
    sees different values because at different times we got the value from
    different DHT subvolumes.  To avoid such problems, just report a
    constant value.
    
    Change-Id: Id64ce917c75b5f7ff50cb55b6e997f3b3556e7e3
    BUG: 1302948
    Original-author: Shyam <srangana>
    Signed-off-by: Jeff Darcy <jdarcy>
    Signed-off-by: N Balachandran <nbalacha>
    Reviewed-on: http://review.gluster.org/13770
    Smoke: Gluster Build System <jenkins.com>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.com>
    Reviewed-by: Shyamsundar Ranganathan <srangana>
    Reviewed-by: Vijay Bellur <vbellur>

Comment 12 Vijay Bellur 2016-03-24 05:50:44 UTC
COMMIT: http://review.gluster.org/13781 committed in master by Raghavendra G (rgowdapp) 
------
commit f14e800cd9bfe34822e1530f71f233bfe4fb04c0
Author: Sakshi Bansal <sabansal>
Date:   Sat Mar 19 10:15:24 2016 +0530

    dht: update attr information in refresh layout to avoid
         stale timestamp
    
    Consider the scenario where an mkdir has just created the
    directory but has not healed it yet. A parallel lookup on
    this entry will find anomalies and trigger a selfheal which
    will sample the ctime of the directory after the mkdir phase.
    Meanwhile the mkdir has completed setting the layout and
    updated the ctime. The selfheal then sees the layout to be
    healed and returns with the ctime it got after the mkdir phase
    which has now become stale. However if the lookup happens
    to unwind before the mkdir then the inode associated with
    lookup will get linked in the inode table which has the
    stale ctime. To avoid this selfheal must do an iatt_merge in
    refresh layout to get the latest timestamp irrespective of
    whether it needs to heal the layout or not.
    
    Change-Id: I3634c3978bcc1710705f44b48f3876601682d33e
    BUG: 1302948
    Signed-off-by: Sakshi Bansal <sabansal>
    Reviewed-on: http://review.gluster.org/13781
    Smoke: Gluster Build System <jenkins.com>
    Reviewed-by: Raghavendra G <rgowdapp>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.com>

Comment 13 Vijay Bellur 2016-03-25 02:57:44 UTC
REVIEW: http://review.gluster.org/13827 (cluster/afr: Fix read-child selection in entry create fop) posted (#3) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 14 Vijay Bellur 2016-03-25 03:37:15 UTC
REVIEW: http://review.gluster.org/13827 (cluster/afr: Fix read-child selection in entry create fop) posted (#4) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 15 Vijay Bellur 2016-03-25 06:43:35 UTC
REVIEW: http://review.gluster.org/13826 (debug/trace: Print {acm}times as integers) posted (#2) for review on master by Krutika Dhananjay (kdhananj)

Comment 16 Vijay Bellur 2016-03-26 12:41:56 UTC
REVIEW: http://review.gluster.org/13785 (cluster/afr: Don't let NFS cache stat after writes) posted (#4) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 17 Vijay Bellur 2016-03-26 12:44:54 UTC
REVIEW: http://review.gluster.org/13785 (cluster/afr: Don't let NFS cache stat after writes) posted (#5) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 18 Vijay Bellur 2016-03-28 08:03:19 UTC
REVIEW: http://review.gluster.org/13826 (debug/trace: Print {acm}times as integers) posted (#4) for review on master by Krutika Dhananjay (kdhananj)

Comment 19 Vijay Bellur 2016-03-28 12:13:47 UTC
COMMIT: http://review.gluster.org/13827 committed in master by Pranith Kumar Karampuri (pkarampu) 
------
commit 2d0c2a9e15a581950a966841866f1f18a8888440
Author: Pranith Kumar K <pkarampu>
Date:   Fri Mar 25 06:04:23 2016 +0530

    cluster/afr: Fix read-child selection in entry create fop
    
    When an entry is being created the inode is yet to be linked
    so args must be filled with gfid and ia_type for it to give
    consistent iatt.
    
    Also handle Dht sending fops on inode not yet linked.
    
    BUG: 1302948
    Change-Id: I6969cacb437cad02f66716f3bf8ec004ffe7c691
    Signed-off-by: Pranith Kumar K <pkarampu>
    Reviewed-on: http://review.gluster.org/13827
    Reviewed-by: Krutika Dhananjay <kdhananj>
    Smoke: Gluster Build System <jenkins.com>
    Reviewed-by: Anuradha Talur <atalur>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.com>

Comment 20 Vijay Bellur 2016-03-28 13:16:38 UTC
REVIEW: http://review.gluster.org/13785 (cluster/afr: Don't let NFS cache stat after writes) posted (#6) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 21 Mike McCune 2016-03-28 23:31:34 UTC
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions

Comment 22 Vijay Bellur 2016-03-29 10:39:47 UTC
REVIEW: http://review.gluster.org/13785 (cluster/afr: Don't let NFS cache stat after writes) posted (#7) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 23 Vijay Bellur 2016-03-29 12:50:05 UTC
COMMIT: http://review.gluster.org/13826 committed in master by Jeff Darcy (jdarcy) 
------
commit 516db811a3bc1b3e778c383fc18aa5eb56342f77
Author: Krutika Dhananjay <kdhananj>
Date:   Thu Mar 24 20:43:05 2016 +0530

    debug/trace: Print {acm}times as integers
    
    This helps in debugging issues with mtime, ctime etc.
    
    Change-Id: I4a0335f17c6b40e6c627658e91f43989c3efca22
    BUG: 1302948
    Signed-off-by: Krutika Dhananjay <kdhananj>
    Reviewed-on: http://review.gluster.org/13826
    Smoke: Gluster Build System <jenkins.com>
    Reviewed-by: Niels de Vos <ndevos>
    CentOS-regression: Gluster Build System <jenkins.com>
    NetBSD-regression: NetBSD Build System <jenkins.org>

Comment 24 Vijay Bellur 2016-04-04 10:08:55 UTC
REVIEW: http://review.gluster.org/13782 (NFS: new option nfs.dirplus-stat added) posted (#3) for review on master by Sakshi Bansal

Comment 25 Vijay Bellur 2016-04-04 13:21:55 UTC
REVIEW: http://review.gluster.org/13782 (NFS: new option nfs.rdirplus added) posted (#4) for review on master by Sakshi Bansal

Comment 26 Vijay Bellur 2016-04-05 06:26:45 UTC
REVIEW: http://review.gluster.org/13782 (NFS: new option nfs.rdirplus added) posted (#5) for review on master by Sakshi Bansal

Comment 27 Vijay Bellur 2016-04-05 10:22:20 UTC
REVIEW: http://review.gluster.org/13782 (NFS: new option nfs.rdirplus added) posted (#6) for review on master by Sakshi Bansal

Comment 28 Vijay Bellur 2016-04-06 09:39:10 UTC
REVIEW: http://review.gluster.org/13782 (NFS: new option nfs.rdirplus added) posted (#7) for review on master by Sakshi Bansal

Comment 29 Vijay Bellur 2016-04-07 08:40:07 UTC
COMMIT: http://review.gluster.org/13782 committed in master by Niels de Vos (ndevos) 
------
commit 50b93f72f9a3f5bfd1d610801aecb06823adaa63
Author: Sakshi Bansal <sabansal>
Date:   Thu Mar 31 15:02:03 2016 +0530

    NFS: new option nfs.rdirplus added
    
    When this option is 'disabled', NFS falls back to standard readdir instead
    of readdirp
    
    Change-Id: Icaaf4da6533bee56160d4a81e42bb60f7d341945
    BUG: 1302948
    Signed-off-by: Sakshi Bansal <sabansal>
    Reviewed-on: http://review.gluster.org/13782
    Smoke: Gluster Build System <jenkins.com>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.com>
    Reviewed-by: Niels de Vos <ndevos>

Comment 30 Vijay Bellur 2016-04-17 01:54:54 UTC
REVIEW: http://review.gluster.org/13785 (cluster/afr: Don't let NFS cache stat after writes) posted (#8) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 31 Vijay Bellur 2016-04-18 01:32:34 UTC
REVIEW: http://review.gluster.org/13785 (cluster/afr: Don't let NFS cache stat after writes) posted (#9) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 32 Vijay Bellur 2016-04-18 01:59:33 UTC
REVIEW: http://review.gluster.org/13785 (cluster/afr: Don't let NFS cache stat after writes) posted (#10) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 33 Vijay Bellur 2016-05-04 04:21:41 UTC
REVIEW: http://review.gluster.org/13785 (cluster/afr: Don't let NFS cache stat after writes) posted (#11) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 34 Vijay Bellur 2016-05-04 04:47:20 UTC
REVIEW: http://review.gluster.org/13785 (cluster/afr: Don't let NFS cache stat after writes) posted (#12) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 35 Vijay Bellur 2016-05-05 01:42:17 UTC
COMMIT: http://review.gluster.org/13785 committed in master by Pranith Kumar Karampuri (pkarampu) 
------
commit 4c4624c9bad2edf27128cb122c64f15d7d63bbc8
Author: Pranith Kumar K <pkarampu>
Date:   Sat Mar 19 11:40:26 2016 +0530

    cluster/afr: Don't let NFS cache stat after writes
    
    Problem:
    Afr does post-ops after write but the stat buffer it unwinds is at the
    time of write, so if nfs client caches this, it will see different
    ctime when it does stat on it after post-op is done. From NFS client's
    perspective it thinks the file is changed. Tar which depends on this
    to be correct keeps giving 'file changed as we read it' warning.
    If Afr instead has to choose to unwind after post-op, eager-lock,
    delayed-post-op will have to be disabled which will lead to bad
    performance for all write usecases.
    
    Fix:
    Don't let client cache stat after write.
    
    Change-Id: Ic6062acc6e5cdd97a9c83c56bd529ec83cee8a23
    BUG: 1302948
    Signed-off-by: Pranith Kumar K <pkarampu>
    Signed-off-by: Anuradha Talur <atalur>
    Reviewed-on: http://review.gluster.org/13785
    Smoke: Gluster Build System <jenkins.com>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.com>
    Reviewed-by: Niels de Vos <ndevos>

Comment 36 Sakshi 2016-05-09 05:37:54 UTC
Are there any more patches sent for this issue. If not please move the status to MODIFIED

Comment 37 Niels de Vos 2016-06-16 13:56:06 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report.

glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 38 George 2018-01-10 08:07:11 UTC
I suppose a new issue maybe introduced by this fix, the new issue I has report in 
https://bugzilla.redhat.com/show_bug.cgi?id=1531457


Note You need to log in before you can comment on or make changes to this bug.