Bug 1724624

Summary:	LINK does not invalidate metadata cache of parent directory
Product:	[Community] GlusterFS	Reporter:	homma
Component:	upcall	Assignee:	bugs <bugs>
Status:	CLOSED UPSTREAM	QA Contact:
Severity:	high	Docs Contact:
Priority:	medium
Version:	mainline	CC:	bugs, khiremat, skoduri
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-03-12 12:44:09 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description homma 2019-06-27 12:30:10 UTC

Description of problem:
With cache invalidation enabled, a file creation by LINK does not invalidate cache of the parent directory.

Version-Release number of selected component (if applicable):
glusterfs-fuse.x86_64 5.6-1.el7 from centos-gluster5 repository

How reproducible:
Often (depending on the timing of cache expiration)

Steps to Reproduce:
1. Enable cache invalidation by 'set group group metadata-cache'.
2. Do the following on the gluster volume:
  On client#1: ls -l
  On client#2: ls -l
  On client#1: echo test > dir1/test
  On client#1: ln dir1/test dir2/test
  On client#1: rm dir1/test
  On client#1: ls -l
  On client#2: ls -l

Actual results:
Timestamp of dir2 on client#2 is not updated.
About ten minutes later, cache on client#2 expires and timestamp of dir2 is updated.

Expected results:
Timestamp of dir2 is updated immediately.

Additional info:
Dovecot detects changes of the mailbox by mtime of Maildir/new directory, this results in up to 10 minutes delay in arrival of new mails.

Comment 1 homma 2019-06-28 01:31:57 UTC

I saw the source code, and found that in up_link_cbk(), upcall_cache_invalidate() is called with flags=UP_UPDATE_CLIENT for the parent directory.
When upcall_cache_invalidate() is called with flags=UP_UPDATE_CLIENT (equals to UP_ATIME), cache-invalidation notification is not sent to the clients.
It also applies to up_unlink_cbk(), up_rename_cbk(), and up_rmdir_cbk(). 

In all cases, upcall_cache_invalidate() is called with UP_PARENT_DENTRY_FLAGS for the inode that is created/deleted/renamed, but notification will never be sent to clients that have cache entry of only the parent directory.

Comment 2 homma 2019-07-16 03:23:52 UTC

I have confirmed that the problem persists on release 6.3.

On release 6, metadata cache of directories are refreshed by 'ls -l' for 1-way distributed volumes, because of dht pass-through.
In that case, the problem is reproduced by the following steps:
  On client#1: stat dir1 dir2
  On client#2: stat dir1 dir2
  On client#1: echo test > dir1/test
  On client#1: ln dir1/test dir2/test
  On client#1: rm dir1/test
  On client#1: stat dir1 dir2
  On client#2: stat dir1 dir2

On client#2, mtime and ctime of dir2 are not updated until expiration of metadata cache.

Comment 4 Worker Ant 2020-03-12 12:44:09 UTC

This bug is moved to https://github.com/gluster/glusterfs/issues/933, and will be tracked there from now on. Visit GitHub issues URL for further details