Bug 1312721 - tar complains: <fileName>: file changed as we read it
tar complains: <fileName>: file changed as we read it
Status: CLOSED CURRENTRELEASE
Product: GlusterFS
Classification: Community
Component: distribute (Show other bugs)
3.7.9
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Sakshi
: Reopened
Depends On: 1302948 1335285
Blocks: 1314508
  Show dependency treegraph
 
Reported: 2016-02-28 22:49 EST by Sakshi
Modified: 2016-06-28 08:13 EDT (History)
11 users (show)

See Also:
Fixed In Version: glusterfs-3.7.12
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1302948
: 1314508 (view as bug list)
Environment:
Last Closed: 2016-06-28 08:13:22 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Sakshi 2016-02-28 22:49:25 EST
+++ This bug was initially created as a clone of Bug #1302948 +++

+++ This bug was initially created as a clone of Bug #1298724 +++

Description of problem:
Encounter 'file changed as we read it' problem, with quota disabled:

# pwd
# tar czvf linux-4.4.tar.gz.test linux-4.4
linux-4.4/
linux-4.4/Kbuild
tar: linux-4.4/Kbuild: file changed as we read it
linux-4.4/block/
linux-4.4/block/blk-lib.c
tar: linux-4.4/block/blk-lib.c: file changed as we read it
linux-4.4/block/blk-settings.c
tar: linux-4.4/block/blk-settings.c: file changed as we read it
linux-4.4/block/scsi_ioctl.c
linux-4.4/block/blk-mq.h
linux-4.4/block/noop-iosched.c
linux-4.4/block/Kconfig.iosched
tar: linux-4.4/block/Kconfig.iosched: file changed as we read it
linux-4.4/block/blk-throttle.c
linux-4.4/block/ioprio.c
linux-4.4/block/t10-pi.c
linux-4.4/block/Kconfig
linux-4.4/block/bio.c
linux-4.4/block/blk-exec.c
linux-4.4/block/blk-ioc.c
linux-4.4/block/blk-iopoll.c
linux-4.4/block/blk-map.c
linux-4.4/block/blk-timeout.c
linux-4.4/block/compat_ioctl.c
linux-4.4/block/deadline-iosched.c
linux-4.4/block/elevator.c
linux-4.4/block/partition-generic.c
linux-4.4/block/Makefile
linux-4.4/block/blk-cgroup.c
linux-4.4/block/blk-core.c
tar: linux-4.4/block/blk-core.c: file changed as we read it
linux-4.4/block/blk-mq-cpu.c
tar: linux-4.4/block/blk-mq-cpu.c: file changed as we read it
linux-4.4/block/bsg.c
tar: linux-4.4/block/bsg.c: file changed as we read it
linux-4.4/block/cfq-iosched.c
tar: linux-4.4/block/cfq-iosched.c: file changed as we read it
linux-4.4/block/cmdline-parser.c
tar: linux-4.4/block/cmdline-parser.c: file changed as we read it
linux-4.4/block/genhd.c
linux-4.4/block/bio-integrity.c
linux-4.4/block/blk-merge.c
linux-4.4/block/blk-softirq.c
linux-4.4/block/blk-tag.c
linux-4.4/block/bounce.c
linux-4.4/block/blk-mq-cpumap.c
tar: linux-4.4/block/blk-mq-cpumap.c: file changed as we read it
linux-4.4/block/blk-mq-sysfs.c
tar: linux-4.4/block/blk-mq-sysfs.c: file changed as we read it
linux-4.4/block/blk-mq-tag.h
tar: linux-4.4/block/blk-mq-tag.h: file changed as we read it
linux-4.4/block/blk-mq.c
linux-4.4/block/blk-sysfs.c
linux-4.4/block/blk.h
linux-4.4/block/partitions/
linux-4.4/block/partitions/amiga.c
linux-4.4/block/partitions/efi.h
linux-4.4/block/partitions/karma.h
linux-4.4/block/partitions/ldm.c
tar: linux-4.4/block/partitions/ldm.c: file changed as we read it
linux-4.4/block/partitions/msdos.c
tar: linux-4.4/block/partitions/msdos.c: file changed as we read it
linux-4.4/block/partitions/osf.h
linux-4.4/block/partitions/atari.c
linux-4.4/block/partitions/cmdline.c
linux-4.4/block/partitions/mac.h
linux-4.4/block/partitions/sgi.h
linux-4.4/block/partitions/sun.h
linux-4.4/block/partitions/Kconfig
linux-4.4/block/partitions/ldm.h
tar: linux-4.4/block/partitions/ldm.h: file changed as we read it
linux-4.4/block/partitions/msdos.h
linux-4.4/block/partitions/sysv68.c
linux-4.4/block/partitions/sysv68.h
linux-4.4/block/partitions/Makefile
linux-4.4/block/partitions/acorn.h
linux-4.4/block/partitions/sgi.c
linux-4.4/block/partitions/atari.h
linux-4.4/block/partitions/efi.c
tar: linux-4.4/block/partitions/efi.c: file changed as we read it
linux-4.4/block/partitions/ibm.c
linux-4.4/block/partitions/aix.h
linux-4.4/block/partitions/check.h
linux-4.4/block/partitions/karma.c
linux-4.4/block/partitions/mac.c
linux-4.4/block/partitions/osf.c
linux-4.4/block/partitions/ultrix.h
linux-4.4/block/partitions/acorn.c
linux-4.4/block/partitions/amiga.h
linux-4.4/block/partitions/check.c
linux-4.4/block/partitions/sun.c
tar: linux-4.4/block/partitions/sun.c: file changed as we read it
linux-4.4/block/partitions/aix.c
linux-4.4/block/partitions/cmdline.h
linux-4.4/block/partitions/ibm.h
linux-4.4/block/partitions/ultrix.c
tar: linux-4.4/block/partitions: file changed as we read it
linux-4.4/block/blk-flush.c
linux-4.4/block/blk-integrity.c
linux-4.4/block/blk-mq-tag.c
linux-4.4/block/bsg-lib.c
linux-4.4/block/ioctl.c
tar: linux-4.4/block: file changed as we read it
linux-4.4/certs/
linux-4.4/certs/system_keyring.c
linux-4.4/certs/system_certificates.S
linux-4.4/certs/.gitignore
linux-4.4/certs/Kconfig
tar: linux-4.4/certs/Kconfig: file changed as we read it
linux-4.4/certs/Makefile
tar: linux-4.4/certs: file changed as we read it
linux-4.4/fs/
linux-4.4/fs/affs/
linux-4.4/fs/affs/inode.c
linux-4.4/fs/affs/symlink.c
tar: linux-4.4/fs/affs/symlink.c: file changed as we read it
linux-4.4/fs/affs/affs.h
tar: linux-4.4/fs/affs/affs.h: file changed as we read it
linux-4.4/fs/affs/amigaffs.c
linux-4.4/fs/affs/dir.c
tar: linux-4.4/fs/affs/dir.c: file changed as we read it
linux-4.4/fs/affs/super.c
linux-4.4/fs/affs/Changes
linux-4.4/fs/affs/Kconfig
linux-4.4/fs/affs/file.c
linux-4.4/fs/affs/Makefile
tar: linux-4.4/fs/affs/Makefile: file changed as we read it
linux-4.4/fs/affs/bitmap.c
linux-4.4/fs/affs/namei.c
tar: linux-4.4/fs/affs: file changed as we read it
linux-4.4/fs/coda/
linux-4.4/fs/coda/dir.c
linux-4.4/fs/coda/upcall.c
linux-4.4/fs/coda/Kconfig
linux-4.4/fs/coda/file.c
linux-4.4/fs/coda/Makefile
linux-4.4/fs/coda/coda_linux.h
linux-4.4/fs/coda/pioctl.c
linux-4.4/fs/coda/coda_cache.h
tar: linux-4.4/fs/coda/coda_cache.h: file changed as we read it
linux-4.4/fs/coda/coda_fs_i.h
linux-4.4/fs/coda/inode.c
linux-4.4/fs/coda/psdev.c
linux-4.4/fs/coda/symlink.c
linux-4.4/fs/coda/sysctl.c
linux-4.4/fs/coda/cache.c
linux-4.4/fs/coda/coda_int.h
linux-4.4/fs/coda/coda_linux.c
linux-4.4/fs/coda/cnode.c
tar: linux-4.4/fs/coda: file changed as we read it
linux-4.4/fs/dcookies.c
tar: linux-4.4/fs/dcookies.c: file changed as we read it
linux-4.4/fs/exec.c
tar: linux-4.4/fs/exec.c: file changed as we read it
linux-4.4/fs/fs-writeback.c
linux-4.4/fs/hfs/
linux-4.4/fs/hfs/Makefile
linux-4.4/fs/hfs/string.c
tar: linux-4.4/fs/hfs/string.c: file changed as we read it
linux-4.4/fs/hfs/bitmap.c
linux-4.4/fs/hfs/bnode.c
linux-4.4/fs/hfs/btree.h
linux-4.4/fs/hfs/hfs.h
linux-4.4/fs/hfs/hfs_fs.h
linux-4.4/fs/hfs/attr.c
tar: linux-4.4/fs/hfs/attr.c: file changed as we read it
linux-4.4/fs/hfs/btree.c
linux-4.4/fs/hfs/inode.c
linux-4.4/fs/hfs/mdb.c
linux-4.4/fs/hfs/trans.c
tar: linux-4.4/fs/hfs/trans.c: file changed as we read it
linux-4.4/fs/hfs/part_tbl.c
linux-4.4/fs/hfs/sysdep.c
linux-4.4/fs/hfs/brec.c
tar: linux-4.4/fs/hfs/brec.c: file changed as we read it
linux-4.4/fs/hfs/catalog.c
linux-4.4/fs/hfs/dir.c
linux-4.4/fs/hfs/super.c
tar: linux-4.4/fs/hfs/super.c: file changed as we read it
linux-4.4/fs/hfs/Kconfig
linux-4.4/fs/hfs/bfind.c
linux-4.4/fs/hfs/extent.c
tar: linux-4.4/fs/hfs: file changed as we read it
linux-4.4/fs/ioctl.c
linux-4.4/fs/jffs2/
linux-4.4/fs/jffs2/README.Locking
linux-4.4/fs/jffs2/acl.h
tar: linux-4.4/fs/jffs2/acl.h: file changed as we read it
linux-4.4/fs/jffs2/compr.h
linux-4.4/fs/jffs2/compr_rubin.c
tar: linux-4.4/fs/jffs2/compr_rubin.c: file changed as we read it
linux-4.4/fs/jffs2/gc.c
linux-4.4/fs/jffs2/jffs2_fs_sb.h
tar: linux-4.4/fs/jffs2/jffs2_fs_sb.h: file changed as we read it
linux-4.4/fs/jffs2/nodelist.c
linux-4.4/fs/jffs2/acl.c
linux-4.4/fs/jffs2/debug.h
linux-4.4/fs/jffs2/os-linux.h
linux-4.4/fs/jffs2/readinode.c
linux-4.4/fs/jffs2/scan.c
linux-4.4/fs/jffs2/xattr_trusted.c
linux-4.4/fs/jffs2/LICENCE
tar: linux-4.4/fs/jffs2/LICENCE: file changed as we read it
linux-4.4/fs/jffs2/build.c
tar: linux-4.4/fs/jffs2/build.c: file changed as we read it
linux-4.4/fs/jffs2/dir.c
tar: linux-4.4/fs/jffs2/dir.c: file changed as we read it
linux-4.4/fs/jffs2/super.c
linux-4.4/fs/jffs2/xattr.c
linux-4.4/fs/jffs2/Kconfig
linux-4.4/fs/jffs2/TODO
linux-4.4/fs/jffs2/file.c
linux-4.4/fs/jffs2/fs.c
linux-4.4/fs/jffs2/nodemgmt.c
linux-4.4/fs/jffs2/Makefile
linux-4.4/fs/jffs2/background.c
linux-4.4/fs/jffs2/compr.c
linux-4.4/fs/jffs2/compr_lzo.c
linux-4.4/fs/jffs2/jffs2_fs_i.h
linux-4.4/fs/jffs2/xattr_user.c
tar: linux-4.4/fs/jffs2/xattr_user.c: file changed as we read it
linux-4.4/fs/jffs2/summary.h
linux-4.4/fs/jffs2/wbuf.c
linux-4.4/fs/jffs2/erase.c
linux-4.4/fs/jffs2/malloc.c
tar: linux-4.4/fs/jffs2/malloc.c: file changed as we read it
linux-4.4/fs/jffs2/nodelist.h
tar: linux-4.4/fs/jffs2/nodelist.h: file changed as we read it
linux-4.4/fs/jffs2/read.c
tar: linux-4.4/fs/jffs2/read.c: file changed as we read it
linux-4.4/fs/jffs2/summary.c
tar: linux-4.4/fs/jffs2/summary.c: file changed as we read it
linux-4.4/fs/jffs2/symlink.c
tar: linux-4.4/fs/jffs2/symlink.c: file changed as we read it
linux-4.4/fs/jffs2/writev.c
linux-4.4/fs/jffs2/compr_rtime.c
linux-4.4/fs/jffs2/compr_zlib.c
linux-4.4/fs/jffs2/debug.c
linux-4.4/fs/jffs2/ioctl.c
linux-4.4/fs/jffs2/security.c
linux-4.4/fs/jffs2/write.c
linux-4.4/fs/jffs2/xattr.h
tar: linux-4.4/fs/jffs2: file changed as we read it
linux-4.4/fs/ncpfs/
linux-4.4/fs/ncpfs/dir.c
linux-4.4/fs/ncpfs/ncplib_kernel.h
tar: linux-4.4/fs/ncpfs/ncplib_kernel.h: file changed as we read it
linux-4.4/fs/ncpfs/ncpsign_kernel.c
linux-4.4/fs/ncpfs/Kconfig
linux-4.4/fs/ncpfs/file.c
linux-4.4/fs/ncpfs/getopt.h
linux-4.4/fs/ncpfs/ncplib_kernel.c
linux-4.4/fs/ncpfs/Makefile
tar: linux-4.4/fs/ncpfs/Makefile: file changed as we read it
linux-4.4/fs/ncpfs/ncp_fs_sb.h
linux-4.4/fs/ncpfs/sock.c
linux-4.4/fs/ncpfs/inode.c
tar: linux-4.4/fs/ncpfs/inode.c: file changed as we read it
linux-4.4/fs/ncpfs/ncp_fs_i.h
tar: linux-4.4/fs/ncpfs/ncp_fs_i.h: file changed as we read it
linux-4.4/fs/ncpfs/symlink.c
linux-4.4/fs/ncpfs/ioctl.c
linux-4.4/fs/ncpfs/getopt.c
linux-4.4/fs/ncpfs/ncp_fs.h
tar: linux-4.4/fs/ncpfs/ncp_fs.h: file changed as we read it
linux-4.4/fs/ncpfs/mmap.c
linux-4.4/fs/ncpfs/ncpsign_kernel.h
tar: linux-4.4/fs/ncpfs: file changed as we read it
linux-4.4/fs/notify/
linux-4.4/fs/notify/Makefile
linux-4.4/fs/notify/notification.c
linux-4.4/fs/notify/dnotify/
linux-4.4/fs/notify/dnotify/Makefile
tar: linux-4.4/fs/notify/dnotify/Makefile: file changed as we read it
linux-4.4/fs/notify/dnotify/Kconfig
linux-4.4/fs/notify/dnotify/dnotify.c
tar: linux-4.4/fs/notify/dnotify: file changed as we read it
linux-4.4/fs/notify/fanotify/
linux-4.4/fs/notify/fanotify/Kconfig
tar: linux-4.4/fs/notify/fanotify/Kconfig: file changed as we read it
linux-4.4/fs/notify/fanotify/Makefile
linux-4.4/fs/notify/fanotify/fanotify_user.c
linux-4.4/fs/notify/fanotify/fanotify.c
tar: linux-4.4/fs/notify/fanotify/fanotify.c: file changed as we read it
linux-4.4/fs/notify/fanotify/fanotify.h
tar: linux-4.4/fs/notify/fanotify: file changed as we read it
linux-4.4/fs/notify/fdinfo.c
linux-4.4/fs/notify/fdinfo.h
linux-4.4/fs/notify/group.c
linux-4.4/fs/notify/fsnotify.c
linux-4.4/fs/notify/fsnotify.h
linux-4.4/fs/notify/mark.c
linux-4.4/fs/notify/inode_mark.c
linux-4.4/fs/notify/inotify/
linux-4.4/fs/notify/inotify/Kconfig
linux-4.4/fs/notify/inotify/Makefile
linux-4.4/fs/notify/inotify/inotify_user.c
linux-4.4/fs/notify/inotify/inotify.h
linux-4.4/fs/notify/inotify/inotify_fsnotify.c
tar: linux-4.4/fs/notify/inotify: file changed as we read it
linux-4.4/fs/notify/vfsmount_mark.c
tar: linux-4.4/fs/notify/vfsmount_mark.c: file changed as we read it


Version-Release number of selected component (if applicable):
glusterfs-3.7.1-11.el7rhgs.x86_64

How reproducible:
Always

Steps to Reproduce:
1.Run 'tar czvf linux-4.4.tar.gz.test linux-4.4'
2.
3.

Actual results:
tar: linux-4.4/fs/notify/inotify: file changed as we read it

Expected results:
tar should not complain

Additional info:
Same symptoms as the below bug which was fixed in glusterfs-3.7.1-6


Bug 1223757 - tar on a glusterfs mount displays "file changed as we read it" even though the file was not changed

--- Additional comment from Pranith Kumar K on 2016-01-18 20:13:40 EST ---

Anuradha is looking at this bug.


--- Additional comment from Anuradha on 2016-01-20 20:15:57 EST ---

Hi,                                                                            
                                                                               
I was able to recreate the issue yesterday.                                    
On debugging (with Pranith and Krutika), cause for "file changed as we read it" 
for directories was found. Further debugging needs to be done to understand why 
the message is displayed for files. I am going to apply a fix in my setup for  
the cause found yesterday and see the results.                                 
                                                                               
Will keep you posted.                                                          
                                                                               
Thanks,                                                                        
Anuradha.

--- Additional comment from Anuradha on 2016-01-21 09:20:01 EST ---

Hi,

After testing with the fix, as mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1298724#c5, I have not seen the issue on FUSE mount yet. Will test more.


--- Additional comment from Anuradha on 2016-01-22 11:33:39 EST ---

Hi,

As mentioned in the previous comment, I wasn't able to recreate the issue      
on a FUSE mount with the fix applied. But, the issue appeared on NFS mount.    On debugging, the following RCA was found (thanks to Krutika Dhananjay for finding the root cause):    

When a directory is created in a distribute or a distribute-replicate volume,
after creating the directory, dht sets its layout using setxattr call.
So the ctime is updated 2 times for the directory (say, t1 is the first ctime after mkdir, and t2 is the latest one after setxattr). The ctime to be returned to the application should be t2. But, dht doesn't update ctime from its perspective, causing it to return t1 to the upper layers, thus to the application.

In case of NFS mount in our test setup, it appears that NFS caches this metadata and provides it to tar during tar's first query. After the archiving is done, tar makes another query for comparison. At this point, glusterfs is queried during which dht returns the latest value that it finds (that is t2). This leads to mismatch in ctime seen by tar application.

The mismatch in these ctimes causes file changed as we read it warning.

Moving the component to dht as the root cause of the issue lies in dht.

Thanks,
Anuradha.

--- Additional comment from Vijay Bellur on 2016-02-04 05:55:50 EST ---

REVIEW: http://review.gluster.org/13352 (dht: mkdir must unwind with latest ctime) posted (#1) for review on master by Sakshi Bansal

--- Additional comment from Vijay Bellur on 2016-02-24 02:53:55 EST ---

REVIEW: http://review.gluster.org/13352 (dht: mkdir must unwind with latest ctime) posted (#2) for review on master by Sakshi Bansal

--- Additional comment from Vijay Bellur on 2016-02-26 07:00:32 EST ---

COMMIT: http://review.gluster.org/13352 committed in master by Raghavendra G (rgowdapp@redhat.com) 
------
commit dbf0ca53a5704c7f94257fc4223b41865458301a
Author: Sakshi Bansal <sabansal@redhat.com>
Date:   Thu Feb 4 16:02:59 2016 +0530

    dht: mkdir must unwind with latest ctime
    
    Currently fops like mkdir used the the ctime it gets
    after creating the directory entry. But setting layout
    also updates the ctime of a directory. Hence DHT must
    get the ctime after the setxattr call and unwind with
    the latest ctime to avoid mismatch in time seen by
    applications like tar.
    
    Change-Id: Iecbbe3aac5244af5da9788b48ccf299ca56b4bae
    BUG: 1302948
    Signed-off-by: Sakshi Bansal <sabansal@redhat.com>
    Reviewed-on: http://review.gluster.org/13352
    Smoke: Gluster Build System <jenkins@build.gluster.com>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
    NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    Reviewed-by: Raghavendra G <rgowdapp@redhat.com>
Comment 1 Vijay Bellur 2016-02-28 22:54:51 EST
REVIEW: http://review.gluster.org/13536 (dht: mkdir must unwind with latest ctime) posted (#1) for review on release-3.7 by Sakshi Bansal
Comment 2 Vijay Bellur 2016-02-29 23:54:20 EST
COMMIT: http://review.gluster.org/13536 committed in release-3.7 by Raghavendra G (rgowdapp@redhat.com) 
------
commit 080af56a02aeec20899ecbc7202a8d293378c1f9
Author: Sakshi Bansal <sabansal@redhat.com>
Date:   Thu Feb 4 16:02:59 2016 +0530

    dht: mkdir must unwind with latest ctime
    
    Currently fops like mkdir used the the ctime it gets
    after creating the directory entry. But setting layout
    also updates the ctime of a directory. Hence DHT must
    get the ctime after the setxattr call and unwind with
    the latest ctime to avoid mismatch in time seen by
    applications like tar.
    
    Backport of http://review.gluster.org/#/c/13352/
    
    > Change-Id: Iecbbe3aac5244af5da9788b48ccf299ca56b4bae
    > BUG: 1302948
    > Signed-off-by: Sakshi Bansal <sabansal@redhat.com>
    > Reviewed-on: http://review.gluster.org/13352
    > Smoke: Gluster Build System <jenkins@build.gluster.com>
    > CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
    > NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    > Reviewed-by: Raghavendra G <rgowdapp@redhat.com>
    
    BUG: 1312721
    Change-Id: Ie09342d3705b40ce98b2935f05ad4402f74ba069
    Signed-off-by: Sakshi Bansal <sabansal@redhat.com>
    Reviewed-on: http://review.gluster.org/13536
    Smoke: Gluster Build System <jenkins@build.gluster.com>
    NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Raghavendra G <rgowdapp@redhat.com>
Comment 3 Niels de Vos 2016-03-27 06:32:27 EDT
Bug 1302948 is still in POST, this change for release-3.7 can not be in MODIFIED yet, not all backports are available.
Comment 4 Vijay Bellur 2016-03-28 03:09:12 EDT
REVIEW: http://review.gluster.org/13830 (dht: update attr information in refresh layout to avoid      stale timestamp) posted (#1) for review on release-3.7 by Sakshi Bansal
Comment 5 Vijay Bellur 2016-03-28 03:13:15 EDT
REVIEW: http://review.gluster.org/13831 (dht: report constant directory size) posted (#1) for review on release-3.7 by Sakshi Bansal
Comment 6 Vijay Bellur 2016-03-28 06:56:37 EDT
COMMIT: http://review.gluster.org/13830 committed in release-3.7 by Raghavendra G (rgowdapp@redhat.com) 
------
commit 76645128ed54d2cb43cc6a047c0bcc3ee43e9aa8
Author: Sakshi Bansal <sabansal@redhat.com>
Date:   Sat Mar 19 10:15:24 2016 +0530

    dht: update attr information in refresh layout to avoid
         stale timestamp
    
    Consider the scenario where an mkdir has just created the
    directory but has not healed it yet. A parallel lookup on
    this entry will find anomalies and trigger a selfheal which
    will sample the ctime of the directory after the mkdir phase.
    Meanwhile the mkdir has completed setting the layout and
    updated the ctime. The selfheal then sees the layout to be
    healed and returns with the ctime it got after the mkdir phase
    which has now become stale. However if the lookup happens
    to unwind before the mkdir then the inode associated with
    lookup will get linked in the inode table which has the
    stale ctime. To avoid this selfheal must do an iatt_merge in
    refresh layout to get the latest timestamp irrespective of
    whether it needs to heal the layout or not.
    
    Backport of http://review.gluster.org/#/c/13781/
    
    > Change-Id: I3634c3978bcc1710705f44b48f3876601682d33e
    > BUG: 1302948
    > Signed-off-by: Sakshi Bansal <sabansal@redhat.com>
    > Reviewed-on: http://review.gluster.org/13781
    > Smoke: Gluster Build System <jenkins@build.gluster.com>
    > Reviewed-by: Raghavendra G <rgowdapp@redhat.com>
    > NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    > CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
    
    Change-Id: I3634c3978bcc1710705f44b48f3876601682d33e
    BUG: 1312721
    Signed-off-by: Sakshi Bansal <sabansal@redhat.com>
    Reviewed-on: http://review.gluster.org/13830
    Smoke: Gluster Build System <jenkins@build.gluster.com>
    NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Raghavendra G <rgowdapp@redhat.com>
Comment 7 Vijay Bellur 2016-03-28 06:56:57 EDT
COMMIT: http://review.gluster.org/13831 committed in release-3.7 by Raghavendra G (rgowdapp@redhat.com) 
------
commit cd32751867252219bd66fa300418579080eef924
Author: Jeff Darcy <jdarcy@redhat.com>
Date:   Thu Mar 17 13:33:34 2016 -0400

    dht: report constant directory size
    
    Directory size is meaningless.  Every filesystem has its own
    unpredictable way of increasing or decreasing it, based on internal data
    structures and even transient conditions.  Some filesystems (e.g. ext4)
    never decrease it at all.  Others (e.g. btrfs) don't even report it.
    Very few programs look at it, and those that do are broken.
    Unfortunately, one such program is GNU tar, which will complain when it
    sees different values because at different times we got the value from
    different DHT subvolumes.  To avoid such problems, just report a
    constant value.
    
    Backport of http://review.gluster.org/#/c/13770/
    
    > Change-Id: Id64ce917c75b5f7ff50cb55b6e997f3b3556e7e3
    > BUG: 1302948
    > Original-author: Shyam <srangana@redhat.com>
    > Signed-off-by: Jeff Darcy <jdarcy@redhat.com>
    > Signed-off-by: N Balachandran <nbalacha@redhat.com>
    > Reviewed-on: http://review.gluster.org/13770
    > Smoke: Gluster Build System <jenkins@build.gluster.com>
    > NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    > CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
    > Reviewed-by: Shyamsundar Ranganathan <srangana@redhat.com>
    > Reviewed-by: Vijay Bellur <vbellur@redhat.com>
    
    Change-Id: Id64ce917c75b5f7ff50cb55b6e997f3b3556e7e3
    BUG: 1312721
    Original-author: Shyam <srangana@redhat.com>
    Signed-off-by: Jeff Darcy <jdarcy@redhat.com>
    Signed-off-by: N Balachandran <nbalacha@redhat.com>
    Reviewed-on: http://review.gluster.org/13831
    Smoke: Gluster Build System <jenkins@build.gluster.com>
    NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Raghavendra G <rgowdapp@redhat.com>
Comment 8 Vijay Bellur 2016-03-28 09:28:15 EDT
REVIEW: http://review.gluster.org/13836 (cluster/afr: Fix read-child selection in entry create fop) posted (#1) for review on release-3.7 by Pranith Kumar Karampuri (pkarampu@redhat.com)
Comment 9 Vijay Bellur 2016-03-29 04:07:48 EDT
COMMIT: http://review.gluster.org/13836 committed in release-3.7 by Pranith Kumar Karampuri (pkarampu@redhat.com) 
------
commit b487738c06ed537ab7fac61bd64db395602a6fed
Author: Pranith Kumar K <pkarampu@redhat.com>
Date:   Fri Mar 25 06:04:23 2016 +0530

    cluster/afr: Fix read-child selection in entry create fop
    
    When an entry is being created the inode is yet to be linked
    so args must be filled with gfid and ia_type for it to give
    consistent iatt.
    
    Also handle Dht sending fops on inode not yet linked.
    
     >BUG: 1302948
     >Change-Id: I6969cacb437cad02f66716f3bf8ec004ffe7c691
     >Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
     >Reviewed-on: http://review.gluster.org/13827
     >Reviewed-by: Krutika Dhananjay <kdhananj@redhat.com>
     >Smoke: Gluster Build System <jenkins@build.gluster.com>
     >Reviewed-by: Anuradha Talur <atalur@redhat.com>
     >NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
     >CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
     >(cherry picked from commit 2d0c2a9e15a581950a966841866f1f18a8888440)
    
    Change-Id: I4be73580368b17af71486693682af66d97d6f70d
    BUG: 1312721
    Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
    Reviewed-on: http://review.gluster.org/13836
    Smoke: Gluster Build System <jenkins@build.gluster.com>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
    NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    Reviewed-by: Krutika Dhananjay <kdhananj@redhat.com>
Comment 10 Vijay Bellur 2016-03-29 08:33:38 EDT
REVIEW: http://review.gluster.org/13844 (debug/trace: Print {acm}times as integers) posted (#1) for review on release-3.7 by Krutika Dhananjay (kdhananj@redhat.com)
Comment 11 Vijay Bellur 2016-03-30 04:40:19 EDT
COMMIT: http://review.gluster.org/13844 committed in release-3.7 by Pranith Kumar Karampuri (pkarampu@redhat.com) 
------
commit 324471574dd1f4278b347486187c711b722a9424
Author: Krutika Dhananjay <kdhananj@redhat.com>
Date:   Thu Mar 24 20:43:05 2016 +0530

    debug/trace: Print {acm}times as integers
    
            Backport of: http://review.gluster.org/#/c/13826/
    
    This helps in debugging issues with mtime, ctime etc.
    
    Change-Id: Ib98a0845d17351dd86ba90c3837b6f4bc78a94a1
    BUG: 1312721
    Signed-off-by: Krutika Dhananjay <kdhananj@redhat.com>
    Reviewed-on: http://review.gluster.org/13844
    Smoke: Gluster Build System <jenkins@build.gluster.com>
    NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    Reviewed-by: Niels de Vos <ndevos@redhat.com>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
Comment 12 Vijay Bellur 2016-04-06 01:43:37 EDT
REVIEW: http://review.gluster.org/13916 (NFS: new option nfs.rdirplus added) posted (#1) for review on release-3.7 by Sakshi Bansal
Comment 13 Vijay Bellur 2016-04-06 05:44:11 EDT
REVIEW: http://review.gluster.org/13916 (NFS: new option nfs.rdirplus added) posted (#2) for review on release-3.7 by Sakshi Bansal
Comment 14 Vijay Bellur 2016-04-17 21:35:07 EDT
REVIEW: http://review.gluster.org/14015 (cluster/afr: Don't let NFS cache stat after writes) posted (#1) for review on release-3.7 by Pranith Kumar Karampuri (pkarampu@redhat.com)
Comment 15 Vijay Bellur 2016-04-18 13:04:02 EDT
COMMIT: http://review.gluster.org/13916 committed in release-3.7 by Kaleb KEITHLEY (kkeithle@redhat.com) 
------
commit a47beb0dee2ab40b695fffcc46c650cc8ec34dc1
Author: Sakshi Bansal <sabansal@redhat.com>
Date:   Thu Mar 31 15:02:03 2016 +0530

    NFS: new option nfs.rdirplus added
    
    When this option is 'disabled', NFS falls back to standard readdir instead
    of readdirp
    
    Backport of http://review.gluster.org/#/c/13782/
    
    > Change-Id: Icaaf4da6533bee56160d4a81e42bb60f7d341945
    > BUG: 1302948
    > Signed-off-by: Sakshi Bansal <sabansal@redhat.com>
    
    Change-Id: Icaaf4da6533bee56160d4a81e42bb60f7d341945
    BUG: 1312721
    Signed-off-by: Sakshi Bansal <sabansal@redhat.com>
    Reviewed-on: http://review.gluster.org/13916
    Smoke: Gluster Build System <jenkins@build.gluster.com>
    NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Niels de Vos <ndevos@redhat.com>
Comment 16 Kaushal 2016-04-19 02:58:43 EDT
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.10, please open a new bug report.

glusterfs-3.7.10 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] https://www.gluster.org/pipermail/gluster-users/2016-April/026164.html
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user
Comment 17 Kaushal 2016-04-19 03:22:29 EDT
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.9, please open a new bug report.

glusterfs-3.7.9 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] https://www.gluster.org/pipermail/gluster-users/2016-March/025922.html
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user
Comment 18 Vijay Bellur 2016-05-12 18:55:04 EDT
REVIEW: http://review.gluster.org/14314 (cluster/afr: Don't let NFS cache stat after writes) posted (#1) for review on release-3.8 by Pranith Kumar Karampuri (pkarampu@redhat.com)
Comment 19 Pranith Kumar K 2016-05-12 18:59:30 EDT
Afr change is yet to land in 3.7.x
Comment 20 Vijay Bellur 2016-05-20 04:48:21 EDT
REVIEW: http://review.gluster.org/14454 (cluster/afr: Don't let NFS cache stat after writes) posted (#1) for review on release-3.7 by Pranith Kumar Karampuri (pkarampu@redhat.com)
Comment 21 Vijay Bellur 2016-06-02 04:42:37 EDT
REVIEW: http://review.gluster.org/14454 (cluster/afr: Don't let NFS cache stat after writes) posted (#2) for review on release-3.7 by Pranith Kumar Karampuri (pkarampu@redhat.com)
Comment 22 Vijay Bellur 2016-06-03 03:17:28 EDT
REVIEW: http://review.gluster.org/14454 (cluster/afr: Don't let NFS cache stat after writes.) posted (#3) for review on release-3.7 by Pranith Kumar Karampuri (pkarampu@redhat.com)
Comment 23 Vijay Bellur 2016-06-03 09:23:51 EDT
COMMIT: http://review.gluster.org/14454 committed in release-3.7 by Pranith Kumar Karampuri (pkarampu@redhat.com) 
------
commit 0c2fb5d80e248f8b49e0ed13de0165c30429a072
Author: Pranith Kumar K <pkarampu@redhat.com>
Date:   Sat Mar 19 11:40:26 2016 +0530

    cluster/afr: Don't let NFS cache stat after writes.
    
    Problem:
    Afr does post-ops after write but the stat buffer it unwinds is at the
    time of write, so if nfs client caches this, it will see different
    ctime when it does stat on it after post-op is done. From NFS client's
    perspective it thinks the file is changed. Tar which depends on this
    to be correct keeps giving 'file changed as we read it' warning.
    If Afr instead has to choose to unwind after post-op, eager-lock,
    delayed-post-op will have to be disabled which will lead to bad
    performance for all write usecases.
    
    Fix:
    Don't let client cache stat after write.
    
     >Change-Id: Ic6062acc6e5cdd97a9c83c56bd529ec83cee8a23
     >BUG: 1302948
     >Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
     >Signed-off-by: Anuradha Talur <atalur@redhat.com>
     >Reviewed-on: http://review.gluster.org/13785
     >Smoke: Gluster Build System <jenkins@build.gluster.com>
     >NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
     >CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
     >Reviewed-by: Niels de Vos <ndevos@redhat.com>
    
    BUG: 1312721
    Change-Id: I42a5d524bcf2a2034fe48ee8454812ca26a98c37
    Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
    Reviewed-on: http://review.gluster.org/14454
    NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
    Smoke: Gluster Build System <jenkins@build.gluster.com>
Comment 24 Kaushal 2016-06-28 08:13:22 EDT
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.12, please open a new bug report.

glusterfs-3.7.12 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] https://www.gluster.org/pipermail/gluster-devel/2016-June/049918.html
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Note You need to log in before you can comment on or make changes to this bug.