Bug 762984 (GLUSTER-1252) - smbtorture on glusterfs renders the client mountpoint return (Stale NFS File Handle)
Summary: smbtorture on glusterfs renders the client mountpoint return (Stale NFS File ...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: GLUSTER-1252
Product: GlusterFS
Classification: Community
Component: core
Version: 3.0.5
Hardware: All
OS: Linux
urgent
high
Target Milestone: ---
Assignee: Pavan Vilas Sondur
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2010-07-29 20:15 UTC by Harshavardhana
Modified: 2015-12-01 16:45 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:

Attachments (Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

Description Harshavardhana 2010-07-29 17:28:14 UTC 
Trace logs
---

[2010-07-29 16:19:39] T [fuse-bridge.c:598:fuse_lookup] glusterfs-fuse: 575837: LOOKUP /smbtorture/clients/client20/~dmtmp/PARADOX/ENROLL.DB
[2010-07-29 16:19:39] T [fuse-bridge.c:525:fuse_entry_cbk] glusterfs-fuse: 575837: LOOKUP() /smbtorture/clients/client20/~dmtmp/PARADOX/ENROLL.DB => -1 (No such file or directory)
[2010-07-29 16:19:39] T [fuse-bridge.c:598:fuse_lookup] glusterfs-fuse: 575838: LOOKUP /smbtorture/clients/client20/~dmtmp/PARADOX/ENROLL.DB
[2010-07-29 16:19:39] T [fuse-bridge.c:525:fuse_entry_cbk] glusterfs-fuse: 575838: LOOKUP() /smbtorture/clients/client20/~dmtmp/PARADOX/ENROLL.DB => -1 (No such file or directory)
[2010-07-29 16:19:39] T [fuse-bridge.c:2134:fuse_opendir] glusterfs-fuse: 575839: OPENDIR /smbtorture/clients/client20/~dmtmp/PARADOX
[2010-07-29 16:19:39] T [fuse-bridge.c:849:fuse_fd_cbk] glusterfs-fuse: 575839: OPENDIR() /smbtorture/clients/client20/~dmtmp/PARADOX => 0x65a740
[2010-07-29 16:19:39] T [fuse-bridge.c:2254:fuse_readdir] glusterfs-fuse: 575840: READDIR (0x65a740, size=4096, offset=0)
[2010-07-29 16:19:39] T [fuse-bridge.c:2200:fuse_readdir_cbk] glusterfs-fuse: 575840: READDIR => 6/4096,0
[2010-07-29 16:19:39] T [fuse-bridge.c:2254:fuse_readdir] glusterfs-fuse: 575841: READDIR (0x65a740, size=4096, offset=1519598358)
[2010-07-29 16:19:39] T [fuse-bridge.c:2200:fuse_readdir_cbk] glusterfs-fuse: 575841: READDIR => 6/4096,1519598358
[2010-07-29 16:19:39] T [fuse-bridge.c:2254:fuse_readdir] glusterfs-fuse: 575842: READDIR (0x65a740, size=4096, offset=1487453869)
[2010-07-29 16:19:39] T [fuse-bridge.c:2200:fuse_readdir_cbk] glusterfs-fuse: 575842: READDIR => 2/4096,1487453869
[2010-07-29 16:19:39] T [fuse-bridge.c:2254:fuse_readdir] glusterfs-fuse: 575843: READDIR (0x65a740, size=4096, offset=1173103769)
[2010-07-29 16:19:39] T [fuse-bridge.c:2200:fuse_readdir_cbk] glusterfs-fuse: 575843: READDIR => 0/4096,1173103769
[2010-07-29 16:19:39] T [fuse-bridge.c:2272:fuse_releasedir] glusterfs-fuse: 575844: RELEASEDIR 0x65a740
[2010-07-29 16:19:39] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
[2010-07-29 16:19:39] D [dht-common.c:274:dht_revalidate_cbk] distribute: mismatching layouts for /
[2010-07-29 16:19:39] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
[2010-07-29 16:19:39] D [dht-common.c:274:dht_revalidate_cbk] distribute: mismatching layouts for /
[2010-07-29 16:19:39] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
[2010-07-29 16:19:39] D [dht-common.c:274:dht_revalidate_cbk] distribute: mismatching layouts for /
[2010-07-29 16:19:39] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
[2010-07-29 16:19:39] D [dht-common.c:274:dht_revalidate_cbk] distribute: mismatching layouts for /
[2010-07-29 16:19:39] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
[2010-07-29 16:19:39] D [dht-common.c:274:dht_revalidate_cbk] distribute: mismatching layouts for /
[2010-07-29 16:19:39] T [fuse-bridge.c:598:fuse_lookup] glusterfs-fuse: 575851: LOOKUP /smbtorture/clients/*
[2010-07-29 16:19:39] T [fuse-bridge.c:525:fuse_entry_cbk] glusterfs-fuse: 575851: LOOKUP() /smbtorture/clients/* => -1 (No such file or directory)
[2010-07-29 16:19:39] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
[2010-07-29 16:19:39] D [dht-common.c:274:dht_revalidate_cbk] distribute: mismatching layouts for /
[2010-07-29 16:19:39] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
[2010-07-29 16:19:39] D [dht-common.c:274:dht_revalidate_cbk] distribute: mismatching layouts for /
[2010-07-29 16:19:39] T [fuse-bridge.c:598:fuse_lookup] glusterfs-fuse: 575854: LOOKUP /smbtorture/clients/*
[2010-07-29 16:19:39] T [fuse-bridge.c:525:fuse_entry_cbk] glusterfs-fuse: 575854: LOOKUP() /smbtorture/clients/* => -1 (No such file or directory)
[2010-07-29 16:19:39] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
[2010-07-29 16:19:39] D [dht-common.c:274:dht_revalidate_cbk] distribute: mismatching layouts for /
[2010-07-29 16:19:39] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
[2010-07-29 16:19:39] D [dht-common.c:274:dht_revalidate_cbk] distribute: mismatching layouts for /
[2010-07-29 16:19:39] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
[2010-07-29 16:19:39] D [dht-common.c:274:dht_revalidate_cbk] distribute: mismatching layouts for /
[2010-07-29 16:19:39] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
[2010-07-29 16:19:39] D [dht-common.c:274:dht_revalidate_cbk] distribute: mismatching layouts for /
[2010-07-29 16:19:42] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
[2010-07-29 16:19:42] D [dht-common.c:274:dht_revalidate_cbk] distribute: mismatching layouts for /
[2010-07-29 16:19:42] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
[2010-07-29 16:19:42] D [dht-common.c:274:dht_revalidate_cbk] distribute: mismatching layouts for /
[2010-07-29 16:19:42] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
[2010-07-29 16:19:42] D [dht-common.c:274:dht_revalidate_cbk] distribute: mismatching layouts for /
[2010-07-29 16:19:42] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
[2010-07-29 16:19:42] D [dht-common.c:274:dht_revalidate_cbk] distribute: mismatching layouts for /
[2010-07-29 16:19:43] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
[2010-07-29 16:19:43] D [dht-common.c:274:dht_revalidate_cbk] distribute: mismatching layouts for /
[2010-07-29 16:19:43] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
[2010-07-29 16:19:43] D [dht-common.c:274:dht_revalidate_cbk] distribute: mismatching layouts for /
[2010-07-29 16:19:48] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
[2010-07-29 16:19:48] D [dht-common.c:274:dht_revalidate_cbk] distribute: mismatching layouts for /
[2010-07-29 16:19:48] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
[2010-07-29 16:19:48] D [dht-common.c:274:dht_revalidate_cbk] distribute: mismatching layouts for /
[2010-07-29 16:19:48] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
[2010-07-29 16:19:48] D [dht-common.c:274:dht_revalidate_cbk] distribute: mismatching layouts for /
[2010-07-29 16:19:48] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
[2010-07-29 16:19:48] D [dht-common.c:274:dht_revalidate_cbk] distribute: mismatching layouts for /
[2010-07-29 16:19:48] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
[2010-07-29 16:19:48] D [dht-common.c:274:dht_revalidate_cbk] distribute: mismatching layouts for /
[2010-07-29 16:19:48] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
[2010-07-29 16:19:48] D [dht-common.c:274:dht_revalidate_cbk] distribute: mismatching layouts for /
[2010-07-29 16:19:55] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
[2010-07-29 16:19:55] D [dht-common.c:274:dht_revalidate_cbk] distribute: mismatching layouts for /
[2010-07-29 16:19:55] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
[2010-07-29 16:19:55] D [dht-common.c:274:dht_revalidate_cbk] distribute: mismatching layouts for /
[2010-07-29 16:19:58] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
[2010-07-29 16:19:58] D [dht-common.c:274:dht_revalidate_cbk] distribute: mismatching layouts for /
[2010-07-29 16:19:58] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
[2010-07-29 16:19:58] D [dht-common.c:274:dht_revalidate_cbk] distribute: mismatching layouts for /
[2010-07-29 16:20:13] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
[2010-07-29 16:20:13] D [dht-common.c:274:dht_revalidate_cbk] distribute: mismatching layouts for /
[2010-07-29 16:20:13] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
[2010-07-29 16:20:13] D [dht-common.c:274:dht_revalidate_cbk] distribute: mismatching layouts for /
[2010-07-29 16:20:21] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
[2010-07-29 16:20:21] D [dht-common.c:274:dht_revalidate_cbk] distribute: mismatching layouts for /
[2010-07-29 16:20:22] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
[2010-07-29 16:20:22] D [dht-common.c:274:dht_revalidate_cbk] distribute: mismatching layouts for /
[2010-07-29 16:20:28] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
[2010-07-29 16:20:28] D [dht-common.c:274:dht_revalidate_cbk] distribute: mismatching layouts for /
[2010-07-29 16:20:28] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
[2010-07-29 16:20:28] D [dht-common.c:274:dht_revalidate_cbk] distribute: mismatching layouts for /
[2010-07-29 16:20:43] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
[2010-07-29 16:20:43] D [dht-common.c:274:dht_revalidate_cbk] distribute: mismatching layouts for /
[2010-07-29 16:20:43] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
[2010-07-29 16:20:43] D [dht-common.c:274:dht_revalidate_cbk] distribute: mismatching layouts for /
[2010-07-29 16:20:43] W [fuse-bridge.c:727:fuse_attr_cbk] glusterfs-fuse: 575882: LOOKUP() / => -1 (Stale NFS file handle)
[2010-07-29 16:20:58] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
[2010-07-29 16:20:58] D [dht-common.c:274:dht_revalidate_cbk] distribute: mismatching layouts for /
[2010-07-29 16:20:58] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
[2010-07-29 16:20:58] D [dht-common.c:274:dht_revalidate_cbk] distribute: mismatching layouts for /
[2010-07-29 16:21:13] D [dht-layout.c:649:dht_layout_dir_mismatch] distribute: / - disk layout missing
Comment 1 Harshavardhana 2010-07-29 20:15:19 UTC 
Servers logs
---------------[
2010-07-29 15:56:48] E [posix.c:3787:do_xattrop] posix1: fsetxattr failed on fd=51 while doing xattrop: key=trus
ted.afr.be01g-1 (No such file or directory)
[2010-07-29 15:57:01] E [posix.c:3787:do_xattrop] posix1: fsetxattr failed on fd=69 while doing xattrop: key=trus
ted.afr.be01g-1 (No such file or directory)
[2010-07-29 15:57:37] E [posix.c:3787:do_xattrop] posix1: fsetxattr failed on fd=49 while doing xattrop: key=trus
ted.afr.be01g-1 (No such file or directory)
[2010-07-29 15:57:39] E [posix.c:3787:do_xattrop] posix1: fsetxattr failed on fd=83 while doing xattrop: key=trus
ted.afr.be01g-1 (No such file or directory)
[2010-07-29 15:58:19] E [posix.c:2698:posix_writev] posix2: post-operation fstat failed on fd=819c7e8: No such fi
le or directory
[2010-07-29 15:58:19] E [posix.c:2603:posix_writev] posix2: pre-operation fstat failed on fd=819c7e8: No such fil
e or directory
[2010-07-29 15:58:19] E [posix.c:2603:posix_writev] posix2: pre-operation fstat failed on fd=819c7e8: No such fil
e or directory
[2010-07-29 15:58:19] E [posix.c:2603:posix_writev] posix2: pre-operation fstat failed on fd=819c7e8: No such fil
e or directory
[2010-07-29 15:58:19] E [posix.c:2603:posix_writev] posix2: pre-operation fstat failed on fd=819c7e8: No such fil
e or directory
[2010-07-29 15:58:19] E [posix.c:2603:posix_writev] posix2: pre-operation fstat failed on fd=819c7e8: No such fil
e or directory
[2010-07-29 15:58:19] E [posix.c:2603:posix_writev] posix2: pre-operation fstat failed on fd=819c7e8: No such fil
e or directory
[2010-07-29 15:58:19] E [posix.c:2603:posix_writev] posix2: pre-operation fstat failed on fd=819c7e8: No such fil
e or directory
[2010-07-29 15:58:19] E [posix.c:2603:posix_writev] posix2: pre-operation fstat failed on fd=819c7e8: No such fil
e or directory
[2010-07-29 15:58:19] E [posix.c:2603:posix_writev] posix2: pre-operation fstat failed on fd=819c7e8: No such fil
e or directory
[2010-07-29 15:58:19] E [posix.c:2603:posix_writev] posix2: pre-operation fstat failed on fd=819c7e8: No such fil
e or directory
[2010-07-29 15:58:19] E [posix.c:2603:posix_writev] posix2: pre-operation fstat failed on fd=819c7e8: No such fil
e or directory
[2010-07-29 15:58:19] E [posix.c:654:posix_setattr] posix2: setattr (lstat) on /store/y/smbtorture/clients/client
2/~dmtmp/SEED/LARGE.FIL failed: No such file or directory
[2010-07-29 15:58:52] E [posix.c:3787:do_xattrop] posix2: fsetxattr failed on fd=58 while doing xattrop: key=trus
ted.afr.be03g-2 (No such file or directory)
-bash-3.00# less /opt/glusterfs/var/log/glusterfs/opt-glusterfs-etc-glusterfs-glusterfsd.vol.log
-bash-3.00# less /opt/glusterfs/var/log/glusterfs/opt-glusterfs-etc-glusterfs-glusterfsd.vol.log 
WARNING: terminal is not fully functional
ted.afr.be01g-1 (No such file or directory)rfs-etc-glusterfs-glusterfsd.vol.log  (press RETURN)
[2010-07-29 15:56:48] E [posix.c:3787:do_xattrop] posix1: fsetxattr failed on fd=51 while doing xattrop: key=trus
ted.afr.be01g-1 (No such file or directory)
[2010-07-29 15:57:01] E [posix.c:3787:do_xattrop] posix1: fsetxattr failed on fd=69 while doing xattrop: key=trus
ted.afr.be01g-1 (No such file or directory)
[2010-07-29 15:57:37] E [posix.c:3787:do_xattrop] posix1: fsetxattr failed on fd=49 while doing xattrop: key=trus
ted.afr.be01g-1 (No such file or directory)
[2010-07-29 15:57:39] E [posix.c:3787:do_xattrop] posix1: fsetxattr failed on fd=83 while doing xattrop: key=trus
ted.afr.be01g-1 (No such file or directory)
[2010-07-29 15:58:19] E [posix.c:2698:posix_writev] posix2: post-operation fstat failed on fd=819c7e8: No such fi
le or directory
[2010-07-29 15:58:19] E [posix.c:2603:posix_writev] posix2: pre-operation fstat failed on fd=819c7e8: No such fil
e or directory
[2010-07-29 15:58:19] E [posix.c:2603:posix_writev] posix2: pre-operation fstat failed on fd=819c7e8: No such fil
e or directory
[2010-07-29 15:58:19] E [posix.c:2603:posix_writev] posix2: pre-operation fstat failed on fd=819c7e8: No such fil
e or directory
[2010-07-29 15:58:19] E [posix.c:2603:posix_writev] posix2: pre-operation fstat failed on fd=819c7e8: No such fil
e or directory
[2010-07-29 15:58:19] E [posix.c:2603:posix_writev] posix2: pre-operation fstat failed on fd=819c7e8: No such fil
e or directory
[2010-07-29 15:58:19] E [posix.c:2603:posix_writev] posix2: pre-operation fstat failed on fd=819c7e8: No such fil
e or directory
[2010-07-29 15:58:19] E [posix.c:2603:posix_writev] posix2: pre-operation fstat failed on fd=819c7e8: No such fil
e or directory
[2010-07-29 15:58:19] E [posix.c:2603:posix_writev] posix2: pre-operation fstat failed on fd=819c7e8: No such fil
e or directory
[2010-07-29 15:58:19] E [posix.c:2603:posix_writev] posix2: pre-operation fstat failed on fd=819c7e8: No such fil
e or directory
[2010-07-29 15:58:19] E [posix.c:2603:posix_writev] posix2: pre-operation fstat failed on fd=819c7e8: No such fil
e or directory
[2010-07-29 15:58:19] E [posix.c:2603:posix_writev] posix2: pre-operation fstat failed on fd=819c7e8: No such fil
e or directory
[2010-07-29 15:58:19] E [posix.c:654:posix_setattr] posix2: setattr (lstat) on /store/y/smbtorture/clients/client
2/~dmtmp/SEED/LARGE.FIL failed: No such file or directory
[2010-07-29 15:58:52] E [posix.c:3787:do_xattrop] posix2: fsetxattr failed on fd=58 while doing xattrop: key=trus
ted.afr.be03g-2 (No such file or directory)

Client logs
------------
[2010-07-29 15:47:13] N [client-protocol.c:6288:client_setvolume_cbk] be02g-2: Connected to 172.16.41.21:10001, attached to remote volume 'brick2'.
[2010-07-29 15:47:13] N [client-protocol.c:6288:client_setvolume_cbk] be03g-2: Connected to 172.16.41.22:10001, attached to remote volume 'brick2'.
[2010-07-29 15:47:13] N [afr.c:2648:notify] mirror-2: Subvolume 'be03g-2' came back up; going online.
[2010-07-29 15:47:13] N [client-protocol.c:6288:client_setvolume_cbk] be01g-2: Connected to 172.16.41.20:10001, attached to remote volume 'brick2'.
[2010-07-29 15:47:13] N [client-protocol.c:6288:client_setvolume_cbk] be03g-2: Connected to 172.16.41.22:10001, attached to remote volume 'brick2'.
[2010-07-29 15:47:13] N [client-protocol.c:6288:client_setvolume_cbk] be01g-2: Connected to 172.16.41.20:10001, attached to remote volume 'brick2'.
[2010-07-29 15:54:11] N [latency.c:189:gf_latency_toggle] [core]: Latency measurement turned on
[2010-07-29 15:58:52] W [fuse-bridge.c:727:fuse_attr_cbk] glusterfs-fuse: 546290: LOOKUP() / => -1 (Stale NFS file handle)
[2010-07-29 15:59:16] W [fuse-bridge.c:727:fuse_attr_cbk] glusterfs-fuse: 670181: LOOKUP() / => -1 (Stale NFS file handle)
[2010-07-29 16:03:09] W [fuse-bridge.c:727:fuse_attr_cbk] glusterfs-fuse: 917813: LOOKUP() / => -1 (Stale NFS file handle)
[2010-07-29 16:03:09] W [fuse-bridge.c:727:fuse_attr_cbk] glusterfs-fuse: 918139: LOOKUP() / => -1 (Stale NFS file handle)
[2010-07-29 16:03:23] W [fuse-bridge.c:727:fuse_attr_cbk] glusterfs-fuse: 918225: LOOKUP() / => -1 (Stale NFS file handle)
[2010-07-29 16:07:49] W [fuse-bridge.c:727:fuse_attr_cbk] glusterfs-fuse: 920076: LOOKUP() / => -1 (Stale NFS file handle)
[2010-07-29 16:12:51] W [fuse-bridge.c:727:fuse_attr_cbk] glusterfs-fuse: 920118: LOOKUP() / => -1 (Stale NFS file handle)
Comment 2 Harshavardhana 2010-08-03 14:25:40 UTC 
Can't reproduce the issue with --attribute-timeout and --entry-timeout set to "0"
Comment 3 Harshavardhana 2010-08-03 21:12:10 UTC 
Additional info - Tried 3.0.4 can't reproduce the same issue.
Comment 4 Vijay Bellur 2010-10-27 08:49:30 UTC 
Not observable in 3.0.6 / 3.1.0
Note You need to log in before you can comment on or make changes to this bug.