Description of problem: ======================= Ambiguous behavior seen while renaming a symlink and having a constant lookup on the second mount point (the volume was mounted on) , MKDIR of the original symlink was not seen on the second mount point. The directory is seen on the second mount point only after about 20-30 min Version-Release number of selected component (if applicable): ============================================================= [root@dhcp42-18 scripts]# rpm -qa | grep gluster glusterfs-rdma-3.12.2-13.el7rhgs.x86_64 glusterfs-server-3.12.2-13.el7rhgs.x86_64 gluster-nagios-common-0.2.4-1.el7rhgs.noarch glusterfs-client-xlators-3.12.2-13.el7rhgs.x86_64 glusterfs-cli-3.12.2-13.el7rhgs.x86_64 python2-gluster-3.12.2-13.el7rhgs.x86_64 libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.6.x86_64 glusterfs-3.12.2-13.el7rhgs.x86_64 glusterfs-api-3.12.2-13.el7rhgs.x86_64 gluster-nagios-addons-0.2.10-2.el7rhgs.x86_64 glusterfs-libs-3.12.2-13.el7rhgs.x86_64 vdsm-gluster-4.19.43-2.3.el7rhgs.noarch glusterfs-fuse-3.12.2-13.el7rhgs.x86_64 glusterfs-geo-replication-3.12.2-13.el7rhgs.x86_64 glusterfs-events-3.12.2-13.el7rhgs.x86_64 [root@dhcp42-18 scripts]# How reproducible: ================ Always with constant lookups Steps to Reproduce: ==================== 1. Create and start a 3x3 dist-rep volume 2. Mount the volume on 2 different mount points 3. On one mntpt1 : touch file1 (Create file) --> on mntpt2, do a few lookups 4. On mntpt1: ln -s file1 400 (Create symlink file) --> on mntpt2, do a few lookups 5. On mntpt1: mv 400 sym_400 (Rename symlink file) --> on mntpt2, keep doing loopkups 6. On mntpt1 : mkdir 400 (Create directory with same name as symlink file )--> on mntpt2, do continuous look ups There were two symlinks on the slave instead of one. Actual results: =============== There were 2 symlinks on mntpt2 instead of one The directory that was created is seen on the slave only after 20-30 min Expected results: ================ There should be only 1 symlink on mntpt2 (the renamed symlink) and the directory created in the name of the original symlink (along with the file that was touched)
The systems were looked at and therefore, I'm putting a needinfo on Poornima to update her findings
The issue is the after rename, the old dentry is present in the inode-table and hence the name of new directory resolved to a wrong inode. This is not md-cache bug, the component should be changed to core. This is more likely bug in fuse/libglusterfs. Need to analyse this further.
*** Bug 1605237 has been marked as a duplicate of this bug. ***
Are the two names seen after mv, 400 and sym_400?
Rochelle, having discussed with Raghavendra we'd like to ask for repeating the test with fusedumps being taken. Note that fuse dumping have got some fixes and improvements since rghs-3.4.0 branch (most importantly, inclusion of timestamps). If you willing to test with a custom build and provide me the original SRPM from which the glusterfs rpms used for testing were built, I'll backport these changes and build the rpms for you with them. (The fusedump changes are isolated, won't have effect on anything else.)
A likely hypothesis for this bug is stale dentries (which can be confirmed once fusedumps are available) in inode table. If the hypothesis turns out to be true, its not a trivial fix to avoid stale dentries. Solutions I can think of are: * strict serialization of entry operations (lookup, rename etc). This approach is not feasible due to performance reasons. * Once ctime xlator is available, fuse can rely on ctime of parent to detect changes to parent directory and discard results of an older operation. For eg., it can identify lookup was issued before rename and hence not link stale dentry after rename. However, ctime xlator is not available by default. Without this trusting on mtime/ctime of stat is not correct as cluster.consistent-metadata won't be turned on always and hence stats could be obtained from different replica pairs.
Fusedumps requested available to validate hypothesis, how do you want to proceed?
The behavior described in this bug is confirmed. However, we can't even clearly say it's a bug, as it was never set clear what are the coherency guarantees among multiple clients we can or want to provide. (What we had is was just a constant uphill battle of best effort, case specific fixes.) Getting these things right would be a big effort for which there is no plan or resource in the foreseeable future. At the same time, real life impact is minimal. So we close it with "won't fix".
Upstreamed at https://github.com/gluster/glusterfs/issues/2560.