Created attachment 957921 [details]
strace of tar command demonstrating problem

Description of problem:
openat() syscall fails against a gluster mount.

Version-Release number of selected component (if applicable):
Client:
glusterfs-3.5.2-1.fc20.x86_64
kernel-3.16.6-200.fc20.x86_64
Server:
glusterfs-server-3.5.2-1.fc20.x86_64
kernel-3.16.3-200.fc20.x86_64


How reproducible:
Typical behavior, BUT some files fail to opened by openat() at one time and then can be opened later.  Cause unknown.

Steps to Reproduce:
1. Mount a glusterfs filesystem
2. Attempt to create a tarball from a directory on the glusterfs mount
3. Note that some/many files will be report error by 'tar' as 
 "file removed before we read it"
Actual results:
(from client)
[root@mycena Desktop]# tar -cf - /home/common/WORK/linux/src-mainline/Documentation/mic | cat > /dev/null
tar: Removing leading `/' from member names
tar: /home/common/WORK/linux/src-mainline/Documentation/mic/mpssd/.gitignore: File removed before we read it
tar: /home/common/WORK/linux/src-mainline/Documentation/mic/mpssd/Makefile: File removed before we read it
tar: /home/common/WORK/linux/src-mainline/Documentation/mic/mpssd/micctrl: File removed before we read it
tar: /home/common/WORK/linux/src-mainline/Documentation/mic/mpssd/mpss: File removed before we read it
tar: /home/common/WORK/linux/src-mainline/Documentation/mic/mpssd/mpssd.c: File removed before we read it
tar: /home/common/WORK/linux/src-mainline/Documentation/mic/mpssd/mpssd.h: File removed before we read it
tar: /home/common/WORK/linux/src-mainline/Documentation/mic/mpssd/sysfs.c: File removed before we read it


Expected results:
(from server)
[root@hypoxylon glusterfs]# tar -cf - /home/common/WORK/linux/src-mainline/Documentation/mic | cat > /dev/null
tar: Removing leading `/' from member names

Additional info:

==
Filesystems involved in server:
[root@hypoxylon glusterfs]# mount | grep /home
/dev/sda5 on /home type ext4 (rw,noatime,data=writeback)
/dev/sda3 on /home/common/WORK/Play type btrfs (rw,noatime,compress=lzo,space_cache)
/dev/sda3 on /home/common/WORK/Consulting type btrfs (rw,noatime,compress=lzo,space_cache)
/dev/sda3 on /home/common/WORK/linux type btrfs (rw,noatime,compress=lzo,space_cache)

/home/common is the 'share'

** Note that accessing files on the ext4 or btrfs portion of the share results in the same problem

==
Mount on client:
[root@mycena Desktop]# mount | grep /home
/dev/sda4 on /home type btrfs (rw,noatime,seclabel,compress=lzo,ssd,discard,space_cache)
hypoxylon:/sohocommon on /home/common type fuse.glusterfs (rw,nosuid,nodev,noatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)

==

stat of an example file
(server)
[root@hypoxylon glusterfs]# stat /home/common/WORK/linux/src-mainline/Documentation/mic/mpssd/sysfs.c
  File: ‘/home/common/WORK/linux/src-mainline/Documentation/mic/mpssd/sysfs.c’
  Size: 2240      	Blocks: 8          IO Block: 4096   regular file
Device: 31h/49d	Inode: 405348      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2014-06-04 09:31:01.439770922 -0400
Modify: 2014-06-04 09:31:01.439770922 -0400
Change: 2014-08-04 23:56:15.671400207 -0400
 Birth: -

(client)
[root@mycena Desktop]# stat /home/common/WORK/linux/src-mainline/Documentation/mic/mpssd/sysfs.c
  File: ‘/home/common/WORK/linux/src-mainline/Documentation/mic/mpssd/sysfs.c’
  Size: 2240      	Blocks: 5          IO Block: 131072 regular file
Device: 31h/49d	Inode: 11437353666924416731  Links: 0
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:fusefs_t:s0
Access: 2014-06-04 09:31:01.439770922 -0400
Modify: 2014-06-04 09:31:01.439770922 -0400
Change: 2014-08-04 23:56:15.671400207 -0400
 Birth: -

==
strace -f of the tar command above is attached, however relevant bits appear to be:
[pid  7703] openat(AT_FDCWD, "/home/common/WORK/linux/src-mainline/Documentation/mic", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW|O_CLOEXEC) = 3
[pid  7703] fstat(3, {st_mode=S_IFDIR|0755, st_size=42, ...}) = 0
[pid  7703] fstat(3, {st_mode=S_IFDIR|0755, st_size=42, ...}) = 0
[pid  7703] fcntl(3, F_GETFL)           = 0x28800 (flags O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_NOFOLLOW)
[pid  7703] fcntl(3, F_SETFD, FD_CLOEXEC) = 0
...              = 0
[pid  7703] newfstatat(3, "mpssd", {st_mode=S_IFDIR|0755, st_size=100, ...}, AT_SYMLINK_NOFOLLOW) = 0
[pid  7703] openat(3, "mpssd", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW|O_CLOEXEC) = 4
[pid  7703] fstat(4, {st_mode=S_IFDIR|0755, st_size=100, ...}) = 0
[pid  7703] fstat(4, {st_mode=S_IFDIR|0755, st_size=100, ...}) = 0
[pid  7703] fcntl(4, F_GETFL)           = 0x28800 (flags O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_NOFOLLOW)
[pid  7703] fcntl(4, F_SETFD, FD_CLOEXEC) = 0
[pid  7703] mmap(NULL, 135168, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fa5bda20000
[pid  7703] getdents(4, /* 9 entries */, 131072) = 264
[pid  7703] getdents(4, /* 0 entries */, 131072) = 0
[pid  7703] newfstatat(4, ".gitignore", {st_mode=S_IFREG|0644, st_size=6, ...}, AT_SYMLINK_NOFOLLOW) = 0
[pid  7703] openat(4, ".gitignore", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid  7703] write(2, "tar: ", 5tar: )        = 5
[pid  7703] write(2, "/home/common/WORK/linux/src-main"..., 103/home/common/WORK/linux/src-mainline/Documentation/mic/mpssd/.gitignore: File removed before we read it) = 103