Hide Forgot
distributed replicate setup. (2 servers => 2 bricks on each server). 1 fuse client and 1 nfs client. running test as a normal user by chowning a directory on the mount point to the user. On fuse client dbench 100 and on nfs client clone a glusterfs git repository, autogen, configure and make. On the fuse client after dbench is complete stating glusterfs.git directory says Invalid argument. This is what the client log file in debug mode says. [2011-06-16 07:14:17.727953] D [client3_1-fops.c:1937:client3_1_lookup_cbk] 2-mirror-client-2: gfid changed for /dir/glusterfs.git [2011-06-16 07:14:17.728045] D [client3_1-fops.c:1937:client3_1_lookup_cbk] 2-mirror-client-3: gfid changed for /dir/glusterfs.git [2011-06-16 07:14:17.728069] I [dht-common.c:240:dht_revalidate_cbk] 2-mirror-dht: subvolume mirror-replicate-1 for /dir/glusterfs.git returned -1 (Invalid argument) Attaching gdb to the mount point and having break point to client3_1_lookup_cbk reveals that the gfid of the inode stored in the frame->local.loc and the gfid in the inode attribute for the inode obtained from the server are different. Turning off stat-prefetch seemed to correct the problem and EINVAL was not observed upo stating the glusterfs.git directory. But rm -rf on the mount point (fuse) of the glusterfs.git again gave EINVAL. rm -rf worked fine on the nfs mount point. Running the same test with stat-prefetch turned off from the beginning did not reproduce the issue.
PATCH: http://patches.gluster.com/patch/7588 in master (fuse: consider a lookup as revalidate even if the inode is present in new graph.)
PATCH: http://patches.gluster.com/patch/7576 in release-3.1 (fuse: consider a lookup as revalidate even if the inode is present in new graph.)
PATCH: http://patches.gluster.com/patch/7589 in release-3.2 (fuse: consider a lookup as revalidate even if the inode is present in new graph.)
A simple test to reproduce the bug would be: 1. create a directory on mount and cd into it - /mnt/gluster1/dir 2. have another mount and create a file in that directory /mnt/gluster2/dir# touch file 3. do ls on the first mount /mnt/gluster1/dir# ls file 4. create a new graph in both clients by turning off a translator #gluster volume ptop performance.write-behind off 5. on client2, remove and recreate file /mnt/gluster2/dir# rm -f file && touch file 6. on client1, ls on the file should produce junk output or some error /mnt/gluster1/dir# ls file
Works with 3.2.2qa1
PATCH: http://patches.gluster.com/patch/7786 in release-3.2 (mnt/fuse: generate uuids in fuse_lookup, not in fuse_lookup_resume.)
PATCH: http://patches.gluster.com/patch/7769 in master (mnt/fuse: generate uuids in fuse_lookup, not in fuse_lookup_resume.)
PATCH: http://patches.gluster.com/patch/7770 in release-3.1 (mnt/fuse: generate uuids in fuse_lookup, not in fuse_lookup_resume.)
Tried to reproduce with Du's comment on reproduce the bug. Tested against 0c01d96a065deb992a8c3a0b24c74a70ab99ea86 on 3.1 branch