Description of problem: "Upcall" feature enables clients to receive callback notifications from the brick process on the file/dir entries (which it may cache) in case of any change in the back-end by other clients. Feature page URL: http://www.gluster.org/community/documentation/index.php/Features/Upcall-infrastructure md-cache can make use of this feature to invalidate its cache entries. This bug is to track those changes.
With change http://review.gluster.org/#/c/10581 , cache-invalidation will be turned off in case if NFS-ganesha is disabled. We may need to avoid that once md-cache also uses the feature to invalidate its cache.
REVIEW: http://review.gluster.org/12951 (md-cache: Add cache invalidation support to invalidate the meta data cahe.) posted (#3) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/12996 (upcall: Add xattr invalidation) posted (#1) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/12951 (md-cache: Add cache invalidation support to invalidate the meta data cache) posted (#4) for review on master by Niels de Vos (ndevos)
REVIEW: http://review.gluster.org/12995 (upcall: add support to invalidate xattrs) posted (#2) for review on master by Niels de Vos (ndevos)
REVIEW: http://review.gluster.org/12996 (upcall: pass dict with xattrs on xattr invalidation) posted (#2) for review on master by Niels de Vos (ndevos)
REVIEW: http://review.gluster.org/12951 (md-cache: Add cache invalidation support to invalidate the meta data cache) posted (#5) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/12995 (upcall: add support to invalidate xattrs) posted (#3) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/12996 (upcall: pass dict with xattrs on xattr invalidation) posted (#3) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/12951 (md-cache: Add cache invalidation support to invalidate the meta data cache) posted (#6) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/12995 (upcall: add support to invalidate xattrs) posted (#4) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/12996 (upcall: pass dict with xattrs on xattr invalidation) posted (#4) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/12951 (md-cache: Add cache invalidation support to invalidate the meta data cache) posted (#7) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/12995 (upcall: add support to invalidate xattrs) posted (#5) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/12996 (upcall: pass dict with xattrs on xattr invalidation) posted (#5) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/12951 (md-cache: Add cache invalidation support to invalidate the meta data cache) posted (#8) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/12995 (upcall: add support to invalidate xattrs) posted (#6) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/12996 (upcall: pass dict with xattrs on xattr invalidation) posted (#6) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/13406 (md-cache: Add xattr caching support) posted (#1) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/13406 (md-cache: Add xattr caching support) posted (#2) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/12951 (md-cache: Add cache invalidation support to invalidate the meta data cache) posted (#9) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/12995 (upcall: add support to invalidate xattrs) posted (#7) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/12996 (upcall: pass dict with xattrs on xattr invalidation) posted (#7) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/13406 (md-cache: Add xattr caching support) posted (#3) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/12951 (md-cache: Add cache invalidation support to invalidate the meta data cache) posted (#10) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/12995 (upcall: add support to invalidate xattrs) posted (#8) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/12996 (upcall: pass dict with xattrs on xattr invalidation) posted (#8) for review on master by Poornima G (pgurusid)
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions
REVIEW: http://review.gluster.org/12995 (upcall: add support to invalidate xattrs) posted (#9) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/12995 (upcall: add support to invalidate xattrs) posted (#10) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/12995 (upcall: Add support to invalidate xattrs) posted (#11) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/12996 (upcall: pass dict with xattrs on xattr invalidation) posted (#9) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/12995 (upcall: Add support to invalidate xattrs) posted (#12) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/12996 (upcall: pass dict with xattrs on xattr invalidation) posted (#10) for review on master by Poornima G (pgurusid)
COMMIT: http://review.gluster.org/12995 committed in master by Niels de Vos (ndevos) ------ commit b2222c1e13d3bff17fa04b8f9b4870cefd457fe2 Author: Niels de Vos <ndevos> Date: Mon Dec 7 16:24:15 2015 +0000 upcall: Add support to invalidate xattrs When SElinux is used, clients should get a notification that the extended attributes have been updated. Other components (like md-cache) will be able to use this too. A big part of the implementation comes from Poornima through the first version of http://review.gluster.org/12996. Also moving the flags from upcall-cache-invalidation.h to the main libglusterfs upcall-utils.h file, so that other places can easily use them in future. Change-Id: I525345bed8f22d029524ff19ccaf726a2c905454 BUG: 1211863 Signed-off-by: Niels de Vos <ndevos> Signed-off-by: Poornima G <pgurusid> Reviewed-on: http://review.gluster.org/12995 Reviewed-by: soumya k <skoduri> Smoke: Gluster Build System <jenkins.com> NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.com>
REVIEW: http://review.gluster.org/12996 (upcall: pass dict with xattrs on xattr invalidation) posted (#11) for review on master by Poornima G (pgurusid)
COMMIT: http://review.gluster.org/12996 committed in master by Pranith Kumar Karampuri (pkarampu) ------ commit 8facd588f20ef8305b6f6b53da0f6d54d300093b Author: Poornima G <pgurusid> Date: Wed Dec 16 05:45:03 2015 -0500 upcall: pass dict with xattrs on xattr invalidation In case of xattr invalidation, return a dict containing the updated xattrs. [ndevos: move chunks to change 12995 and only address the xattrs-dict here] Change-Id: I8733f06a519a9a0f24be1bb4b2c38c9c9dce0ce2 BUG: 1211863 Signed-off-by: Poornima G <pgurusid> Reviewed-on: http://review.gluster.org/12996 Smoke: Gluster Build System <jenkins.com> CentOS-regression: Gluster Build System <jenkins.com> NetBSD-regression: NetBSD Build System <jenkins.org> Reviewed-by: Niels de Vos <ndevos> Reviewed-by: soumya k <skoduri> Tested-by: soumya k <skoduri>
REVIEW: http://review.gluster.org/12951 (md-cache: Add cache invalidation support to invalidate the meta data cache) posted (#11) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/14824 (md-cache: Cache gluster-samba metadata) posted (#1) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/14824 (md-cache: Cache gluster-samba metadata) posted (#2) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/12951 (md-cache: Add cache invalidation support to invalidate the meta data cache) posted (#12) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/14824 (md-cache: Cache gluster-samba metadata) posted (#3) for review on master by Poornima G (pgurusid)
COMMIT: http://review.gluster.org/14824 committed in master by Raghavendra G (rgowdapp) ------ commit 01d6b17bac704a320bc0549ae063ee7f4bf3748b Author: Poornima G <pgurusid> Date: Wed Jun 29 00:25:39 2016 -0400 md-cache: Cache gluster-samba metadata Change-Id: I0a95f4897440c5bf6f54612d9c232e015c8bf983 BUG: 1211863 Signed-off-by: Poornima G <pgurusid> Reviewed-on: http://review.gluster.org/14824 NetBSD-regression: NetBSD Build System <jenkins.org> Reviewed-by: Prashanth Pai <ppai> CentOS-regression: Gluster Build System <jenkins.org> Smoke: Gluster Build System <jenkins.org> Reviewed-by: Raghavendra G <rgowdapp>
REVIEW: http://review.gluster.org/12951 (md-cache: Add cache invalidation support to invalidate the meta data cache) posted (#13) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/12951 (md-cache: Add cache invalidation support to invalidate the meta data cache) posted (#14) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/14879 ("md-cache: Enable caching of stat fetched from readdirp) posted (#1) for review on master by Poornima G (pgurusid)
COMMIT: http://review.gluster.org/14879 committed in master by Jeff Darcy (jdarcy) ------ commit 62f826de85489f47da506e0d20e9ed349278d597 Author: Poornima G <pgurusid> Date: Fri Jul 8 14:25:35 2016 +0530 "md-cache: Enable caching of stat fetched from readdirp Patch http://review.gluster.org/11894 removed readdirp fop for md-cache, but there is no mention of exact xlator which was failing because of this. As mentioned by Rafi(author of patch 11894) tiering and svc doesn't really need this as the inode_ctx is populated in readdirp_cbk. Hence reverting this commit. This reverts commit c8c9308134ae4ce24c630a1b0ccfcf4e8f9b0fe7. Change-Id: Ib8d00b3f129596f3a54984f839199175f5c9b55b BUG: 1211863 Signed-off-by: Poornima G <pgurusid> Reviewed-on: http://review.gluster.org/14879 CentOS-regression: Gluster Build System <jenkins.org> NetBSD-regression: NetBSD Build System <jenkins.org> Smoke: Gluster Build System <jenkins.org> Reviewed-by: Zhou Zhengping <johnzzpcrystal> Reviewed-by: Jeff Darcy <jdarcy>
REVIEW: http://review.gluster.org/12951 (md-cache: Add cache invalidation support to invalidate the meta data cache) posted (#15) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/12951 (md-cache: Add cache invalidation support to invalidate the meta data cache) posted (#16) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/14912 (tier: properly update cached subvol during readdirp response) posted (#1) for review on master by mohammed rafi kc (rkavunga)
REVIEW: http://review.gluster.org/12951 (md-cache: Add cache invalidation support to invalidate the meta data cache) posted (#17) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/12951 (md-cache: Add cache invalidation support to invalidate the meta data cache) posted (#18) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/12951 (md-cache: Add cache invalidation support to invalidate the meta data cache) posted (#19) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/12951 (md-cache: Add cache invalidation support to invalidate the meta data cache) posted (#20) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/12951 (md-cache: Add cache invalidation support to invalidate the meta data cache) posted (#21) for review on master by Poornima G (pgurusid)
COMMIT: http://review.gluster.org/12951 committed in master by Raghavendra G (rgowdapp) ------ commit 1f97d7101b3313ce647638310e1028da8dac6785 Author: Poornima G <pgurusid> Date: Fri Dec 11 05:12:07 2015 -0500 md-cache: Add cache invalidation support to invalidate the meta data cache Problem: md-cache currently updates its stat in cbks of selected fops. The default cache time is 1 second, if this is increasd to reap the benefits of caching, we may end up with stale cache for long time, as there is no logic yet to notify md-cache of backend changes by another client. Solution: Use the existing upcall mechanism to invalidate the cache. For this feature to work, "features.cache-invalidation" volume option should be enabled. This patch as is doesn't improve any performance, the benifit of the patch is that it provides coherency for stat cache, hence the cache timeout can be quite longer which in turn can improve the performance. Change-Id: I2dbb0afa7b5e4a5a248f910188e0918e02f18692 BUG: 1211863 Signed-off-by: Poornima G <pgurusid> Reviewed-on: http://review.gluster.org/12951 Smoke: Gluster Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> NetBSD-regression: NetBSD Build System <jenkins.org> Reviewed-by: Raghavendra G <rgowdapp>
REVIEW: http://review.gluster.org/14971 (md-cache: fix indention to silence Coverity) posted (#1) for review on master by Niels de Vos (ndevos)
COMMIT: http://review.gluster.org/14971 committed in master by Vijay Bellur (vbellur) ------ commit 370197f6e8413c0c4c13571f1c5b613bfa1e50d5 Author: Niels de Vos <ndevos> Date: Thu Jul 21 08:12:04 2016 +0200 md-cache: fix indention to silence Coverity Coverity complains about the incorrect indention: *** CID 1357867: Control flow issues (NESTING_INDENT_MISMATCH) ... 2566 if (conf->mdc_invalidation) 2567 ret = mdc_invalidate (this, data); >>> CID 1357867: Control flow issues (NESTING_INDENT_MISMATCH) >>> This 'if' statement is indented to column 25, as if it were nested within the preceding parent statement, but it is not. 2568 if (default_notify (this, event, data) != 0) 2569 ret = -1; 2570 break; ... Even when md-cache does not have cache-invalidation on, we need to pass the upcall to the next xlator. Change-Id: I6d2a174eb54e3df270920aae9673b5010c235f25 CID: 1357867 BUG: 1211863 Signed-off-by: Niels de Vos <ndevos> Reviewed-on: http://review.gluster.org/14971 Smoke: Gluster Build System <jenkins.org> Reviewed-by: Prashanth Pai <ppai> Reviewed-by: Poornima G <pgurusid> Reviewed-by: Raghavendra G <rgowdapp> CentOS-regression: Gluster Build System <jenkins.org> NetBSD-regression: NetBSD Build System <jenkins.org> Reviewed-by: Vijay Bellur <vbellur>
REVIEW: http://review.gluster.org/15002 (md-cache: Register the list of xattrs with cache-invalidation) posted (#1) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15002 (md-cache: Register the list of xattrs with cache-invalidation) posted (#2) for review on master by Poornima G (pgurusid)
TODOs: 1. Implement cache size limit for md-cache 2. Implement register for notification in gfapi 3. IPC: Implement the way md-cache can communicate the list of xattr to upcall. - implement dht_ipc, afr_ipc, ec_ipc, shard_ipc - Enhance upcall to store the xattrs list seperately for each client, and enable each client to register and unregister its own set of xattrs. For better debugability 1. Add logging in md-cache, option to enable md-cache trace logging 2. Md-cache, and upcall integrate with statedump 3. Gfapi client perf profiling ability Test cases for md-cache: 1. Add test cases for testing cache-invalidation, specific cases 2. Run the regression with cache inval on and large timeout
REVIEW: http://review.gluster.org/15043 (md-cache/upcall: In case of mode bit change invalidate xattr) posted (#1) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15045 (md-cache: Fix wrong cache time update for xattrs) posted (#1) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15043 (md-cache/upcall: In case of mode bit change invalidate xattr) posted (#2) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15043 (md-cache/upcall: In case of mode bit change invalidate xattr) posted (#3) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15069 (md-cache: Add logging to increase debuggability) posted (#1) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15069 (md-cache: Add logging to increase debuggability) posted (#2) for review on master by Poornima G (pgurusid)
COMMIT: http://review.gluster.org/15043 committed in master by Jeff Darcy (jdarcy) ------ commit 5ae0a5d1e92175c28cd5470b890e99ff4eac0673 Author: Poornima G <pgurusid> Date: Fri Jul 29 12:20:11 2016 +0530 md-cache/upcall: In case of mode bit change invalidate xattr When the mode bits are changed, the ACL entries also do get affected. Currently in upcall, setattr invalidates only the stat info. With this patch, if mode bits are changed, the upcall will invalidate all the xattrs. Change-Id: Iccda2e1a7440ee845aa5442bf51970f74d9b0862 BUG: 1211863 Signed-off-by: Poornima G <pgurusid> Reviewed-on: http://review.gluster.org/15043 Smoke: Gluster Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> Reviewed-by: Niels de Vos <ndevos> NetBSD-regression: NetBSD Build System <jenkins.org> Reviewed-by: Prashanth Pai <ppai> Reviewed-by: Jeff Darcy <jdarcy>
REVIEW: http://review.gluster.org/15069 (md-cache: Add logging to increase debuggability) posted (#3) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15002 (md-cache: Register the list of xattrs with cache-invalidation) posted (#3) for review on master by Poornima G (pgurusid)
COMMIT: http://review.gluster.org/15069 committed in master by Raghavendra G (rgowdapp) ------ commit 2b0e83e908b3be2043e92a974ba92ae942bff4d1 Author: Poornima G <pgurusid> Date: Tue Aug 2 14:51:23 2016 +0530 md-cache: Add logging to increase debuggability Change-Id: I147d16ec3c20d3372892fdd5f62010e52f82f8bd BUG: 1211863 Signed-off-by: Poornima G <pgurusid> Reviewed-on: http://review.gluster.org/15069 Smoke: Gluster Build System <jenkins.org> NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> Reviewed-by: Vijay Bellur <vbellur> Reviewed-by: Raghavendra G <rgowdapp>
REVIEW: http://review.gluster.org/15002 (md-cache: Register the list of xattrs with cache-invalidation) posted (#4) for review on master by Poornima G (pgurusid)
(In reply to Poornima G from comment #61) > TODOs: > 1. Implement cache size limit for md-cache > 2. Implement register for notification in gfapi > 3. IPC: Implement the way md-cache can communicate the list of xattr to > upcall. > - implement dht_ipc, afr_ipc, ec_ipc, shard_ipc > - Enhance upcall to store the xattrs list seperately for each client, and > enable each client to register and unregister its own set of xattrs. 4. Transaction id framework to eliminate duplicate invalidations in case of replica and EC. 5. Display upcall invalidations also in the profile info 6. Add support for lower xlators to indicate md-cache to not cache > For better debugability > 1. Add logging in md-cache, option to enable md-cache trace logging > 2. Md-cache, and upcall integrate with statedump > 3. Gfapi client perf profiling ability > > Test cases for md-cache: > 1. Add test cases for testing cache-invalidation, specific cases > 2. Run the regression with cache inval on and large timeout
REVIEW: http://review.gluster.org/15185 (md-cache: Add cache hit and miss counters) posted (#1) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15193 (io-stats: Add stats for upcall notifications) posted (#1) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15002 (md-cache: Register the list of xattrs with cache-invalidation) posted (#5) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15002 (md-cache: Register the list of xattrs with cache-invalidation) posted (#6) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15185 (md-cache: Add cache hit and miss counters) posted (#2) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15185 (md-cache: Add cache hit and miss counters) posted (#3) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15225 (dht: Implement ipc fop) posted (#1) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15185 (md-cache: Add cache hit and miss counters) posted (#4) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15002 (md-cache: Register the list of xattrs with cache-invalidation) posted (#7) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15225 (dht: Implement ipc fop) posted (#2) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15314 (md-cache: Do not use features.cache-invalidation for both md-cache and upcall) posted (#1) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15002 (md-cache: Register the list of xattrs with cache-invalidation) posted (#8) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15225 (dht: Implement ipc fop) posted (#3) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15324 (md-cache: Process all the cache invalidation flags) posted (#1) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15193 (io-stats: Add stats for upcall notifications) posted (#2) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15314 (md-cache: Do not use features.cache-invalidation for both md-cache and upcall) posted (#2) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15193 (io-stats: Add stats for upcall notifications) posted (#3) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15193 (io-stats: Add stats for upcall notifications) posted (#4) for review on master by Poornima G (pgurusid)
COMMIT: http://review.gluster.org/15185 committed in master by Raghavendra G (rgowdapp) ------ commit 3cc7f6588c281846f8c590553da03dd16f150e8a Author: Poornima G <pgurusid> Date: Wed Aug 17 12:55:37 2016 +0530 md-cache: Add cache hit and miss counters These counters can be accessed either by .meta interface or statedump. From meta: cat on the private file in md-cache directory. Eg: cat /mnt/glusterfs/0/.meta/graphs/active/patchy-md-cache/private [performance/md-cache.patchy-md-cache] stat_hit_count = 2 stat_miss_count = 8 xattr_hit_count = 4 xattr_miss_count = 3 nameless_lookup_count = 1 negative_lookup_count = 0 stat_invalidations_recieved = 1 xattr_invalidations_recieved = 1 Change-Id: Ib62a8822f263a9f75858b15832d0119fbe382629 BUG: 1211863 Signed-off-by: Poornima G <pgurusid> Reviewed-on: http://review.gluster.org/15185 Smoke: Gluster Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> NetBSD-regression: NetBSD Build System <jenkins.org> Reviewed-by: Raghavendra G <rgowdapp>
COMMIT: http://review.gluster.org/15314 committed in master by Raghavendra G (rgowdapp) ------ commit 3f5273e19af2eaa7bc33c6abaf6b10850f97dcc0 Author: Poornima G <pgurusid> Date: Thu Aug 25 10:25:24 2016 +0530 md-cache: Do not use features.cache-invalidation for both md-cache and upcall Currently, the volume set option features.cache-invalidation enables upcall feature on server side and md-cache cache-invalidation on client side. There are multiple problems that can arise from this: 1. The scenario when user wants to, enable upcall for nfs-ganesha setup, but do not want to enable md-cache cache-invalidation, as the nfs-clients have already cached the metadata and upcall is used to to invalidate the nfs-client cache. In this case, users should have a way of disabling md-cache invalidation without disabling upcall. 2. Upcall requires a op-version of GD_OP_VERSION_3_7_0, where as md-cache invalidation requires an op version of GD_OP_VERSION_3_9_0. Consider a setup where the servers are in op-version GD_OP_VERSION_3_7_0, and th clients are in op-version GD_OP_VERSION_3_9_0. if there is one single volume set option, user can enable this feature in this setup. But it can lead to stale xattr cache as the xattr invalidation was introduced in upcall only in release 3.8. Hence, we should not be able to enable md-cache invalidation, if all the servers and clients are not on opversion >= GD_OP_VERSION_3_9_0. To solve the above mentioned issues, we have seperate volume options for enabling md-cache invalidation and upcall. But this can lead to issues when user enable md-cache invalidation and forgets to enable upcall. Probably in the next release, these can be enables by default. Change-Id: Ie70eff97fe12fcb623eec8f4f5861ac065bf483e BUG: 1211863 Signed-off-by: Poornima G <pgurusid> Reviewed-on: http://review.gluster.org/15314 NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> Smoke: Gluster Build System <jenkins.org> Reviewed-by: soumya k <skoduri> Reviewed-by: Raghavendra G <rgowdapp>
COMMIT: http://review.gluster.org/15225 committed in master by Raghavendra G (rgowdapp) ------ commit b85c648a6b236ca03494cb61b97e2e703be0950c Author: Poornima G <pgurusid> Date: Mon Jul 18 21:19:34 2016 +0530 dht: Implement ipc fop ipc is used by md-cache to communicate the list of xattrs that it is caching, to the upcall xlator. Hence implement this in dht, such that it winds to all the bricks if the ipc op is GF_IPC_MDC_TARGET_UPCALL. The ips should not fail if any of the bricks is down, as md-cache will replay the ipc late when the brick comes back up. Change-Id: Ica551a550c04cbb1240c0d211fe831c2e5eb6017 BUG: 1211863 Signed-off-by: Poornima G <pgurusid> Reviewed-on: http://review.gluster.org/15225 CentOS-regression: Gluster Build System <jenkins.org> Smoke: Gluster Build System <jenkins.org> NetBSD-regression: NetBSD Build System <jenkins.org> Reviewed-by: Raghavendra G <rgowdapp>
COMMIT: http://review.gluster.org/15045 committed in master by Raghavendra G (rgowdapp) ------ commit f8b51bef8820142264bdca9cfe0d7106fb045c2a Author: Poornima G <pgurusid> Date: Fri Jul 29 15:03:47 2016 +0530 md-cache: Fix wrong cache time update for xattrs In md-cache, the cache has two times: 1. Time when the stat was last fetched for that inode 2. Time when the xattrs were last fetched for that inode. This time should not be updated when only one of the xattrs is updated. If, its updated when one of the cached xattr is changed, it can so happen that the other xattrs have past their cache timeout, but are still served from cache. Solution: Do not update the xattr cache time, when one of the xattrs being cached is changed. With this, we may end up in cache timeout though it was updated recently, but it is not a harm. The other way is to have timeout for every xattr that is being cached. Its more complicated, and may be not worth it, as we have lot of lookup fops, that are overloaded to get all the xattrs. Change-Id: Id77e547f403fc792348f1ea56b468b9260a5a34f BUG: 1211863 Signed-off-by: Poornima G <pgurusid> Reviewed-on: http://review.gluster.org/15045 Smoke: Gluster Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> NetBSD-regression: NetBSD Build System <jenkins.org> Reviewed-by: Raghavendra G <rgowdapp>
REVIEW: http://review.gluster.org/15324 (md-cache: Process all the cache invalidation flags) posted (#2) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15002 (md-cache: Register the list of xattrs with cache-invalidation) posted (#9) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15334 (md-cache:Send inode invalidate to Fuse when there is unlink/rename) posted (#1) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15002 (md-cache: Register the list of xattrs with cache-invalidation) posted (#10) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15002 (md-cache: Register the list of xattrs with cache-invalidation) posted (#11) for review on master by Poornima G (pgurusid)
COMMIT: http://review.gluster.org/15002 committed in master by Raghavendra G (rgowdapp) ------ commit 8f053f9d7270f1c6d50c0b3ab5d020503ceeb31a Author: Poornima G <pgurusid> Date: Mon Jul 11 15:04:55 2016 +0530 md-cache: Register the list of xattrs with cache-invalidation Issue: md-cache caches a specified list of xattrs, and when cache invalidation is enabled, it makes sense to recieve invalidation only when those xattrs are modified by other clients. But the current implementation of upcall is that, it will send invalidation when any of the on-disk xattrs is modified. Solution: md-cache sends a list of xattrs that it is interested in, to upcall by issuing an ipc(). The challenge here is to make sure everytime a brick goes offline and comes back up, the ipc() needs to be issued to the bricks. Hence ipc() is sent from md-cache every time there is a CHILD_UP/CHILD_MODIFIED event. TODO: There will be patches following, in cluster xlators, to implement ipc fop. Change-Id: I6efcf3df474f5ce6eabd3d6694c00c7bd89bc25d BUG: 1211863 Signed-off-by: Poornima G <pgurusid> Reviewed-on: http://review.gluster.org/15002 Smoke: Gluster Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> Reviewed-by: Rajesh Joseph <rjoseph> NetBSD-regression: NetBSD Build System <jenkins.org> Reviewed-by: Prashanth Pai <ppai> Reviewed-by: Raghavendra G <rgowdapp>
COMMIT: http://review.gluster.org/15324 committed in master by Raghavendra G (rgowdapp) ------ commit fe929224c47d5c82da5650e9e1041645a8d7f244 Author: Poornima G <pgurusid> Date: Thu Aug 25 15:43:29 2016 +0530 md-cache: Process all the cache invalidation flags Currently, md-cache only processes IATT_UPDATE_FLAGS, UP_XATTR and UP_XATTR_RM. We also need to process UP_RENAME_FLAGS, UP_FORGET, UP_PARENT_DENTRY_FLAGS and UP_NLINK_FLAGS. Otherwise the files unlinked or renamed will not be reflected on other mounts. Change-Id: Icb8b03da51482c3fc2e2a7292d16d56e11a341d9 BUG: 1211863 Signed-off-by: Poornima G <pgurusid> Reviewed-on: http://review.gluster.org/15324 Smoke: Gluster Build System <jenkins.org> NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> Reviewed-by: Raghavendra G <rgowdapp>
REVIEW: http://review.gluster.org/15378 (afr: Add IPC fop) posted (#1) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15378 (afr: Add IPC fop) posted (#2) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15378 (afr: Implement IPC fop) posted (#3) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15419 (tests: Fix one of the md-cache test cases) posted (#1) for review on master by Poornima G (pgurusid)
COMMIT: http://review.gluster.org/15419 committed in master by Vijay Bellur (vbellur) ------ commit 0fd7d0e1c78fdbedfcdb085445c4b0be3c1a97a9 Author: Poornima G <pgurusid> Date: Wed Sep 7 15:47:14 2016 +0530 tests: Fix one of the md-cache test cases Verify if the unlink, rename and other ops are reflected both on the current mount and other mounts. Change-Id: I5a296cdd557194dcf487e65ee4a14bbeaf4be690 BUG: 1211863 Signed-off-by: Poornima G <pgurusid> Reviewed-on: http://review.gluster.org/15419 Smoke: Gluster Build System <jenkins.org> NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> Reviewed-by: Vijay Bellur <vbellur>
REVIEW: http://review.gluster.org/15378 (afr: Implement IPC fop) posted (#4) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15398 (md-cache, afr: Reduce the window of stale read) posted (#2) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15378 (afr: Implement IPC fop) posted (#5) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15398 (md-cache, afr: Reduce the window of stale read) posted (#3) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15387 (ec: Implement ipc fop) posted (#4) for review on master by Poornima G (pgurusid)
COMMIT: http://review.gluster.org/15387 committed in master by Pranith Kumar Karampuri (pkarampu) ------ commit 359b72a57b7c92fc2a11236ac05f5d740db2f540 Author: Poornima G <pgurusid> Date: Fri Sep 2 12:47:15 2016 +0530 ec: Implement ipc fop The ipc will be wound to all the bricks, but for it to be successfull, the fop should succeed on minimum number of bricks. Change-Id: I3f8cb6a349e87bafd0773583def9d4e3765aa140 BUG: 1211863 Signed-off-by: Poornima G <pgurusid> Reviewed-on: http://review.gluster.org/15387 NetBSD-regression: NetBSD Build System <jenkins.org> Smoke: Gluster Build System <jenkins.org> Reviewed-by: Ashish Pandey <aspandey> CentOS-regression: Gluster Build System <jenkins.org> Reviewed-by: Pranith Kumar Karampuri <pkarampu>
REVIEW: http://review.gluster.org/15378 (afr: Implement IPC fop) posted (#6) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15398 (md-cache, afr: Reduce the window of stale read) posted (#4) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15378 (afr: Implement IPC fop) posted (#7) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15398 (md-cache, afr: Reduce the window of stale read) posted (#5) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15378 (afr: Implement IPC fop) posted (#8) for review on master by Pranith Kumar Karampuri (pkarampu)
COMMIT: http://review.gluster.org/15378 committed in master by Pranith Kumar Karampuri (pkarampu) ------ commit 9ab5b52dee5be45458fdb5446d3cbf6a1a5306a6 Author: Poornima G <pgurusid> Date: Mon Aug 22 12:30:43 2016 +0530 afr: Implement IPC fop Currently ipc() is not implemented in afr. md-cache and upcall uses ipc to register the list of xattrs, [1] for more details. For the ipc op GF_IPC_TARGET_UPCALL, it has to be wound to all the replica subvolumes. ipc() is failed when any of the subvolumes fails with other than ENOTCONN or all of the subvolumes are down. [1] http://review.gluster.org/#/c/15002/ Change-Id: I0f651330eafda64e4d922043fe53bd0014536247 BUG: 1211863 Signed-off-by: Poornima G <pgurusid> Reviewed-on: http://review.gluster.org/15378 Tested-by: Pranith Kumar Karampuri <pkarampu> Smoke: Gluster Build System <jenkins.org> NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> Reviewed-by: Pranith Kumar Karampuri <pkarampu>
REVIEW: http://review.gluster.org/15398 (md-cache, afr: Reduce the window of stale read) posted (#6) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15398 (md-cache, afr: Reduce the window of stale read) posted (#7) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/15398 (md-cache, afr: Reduce the window of stale read) posted (#8) for review on master by Pranith Kumar Karampuri (pkarampu)
COMMIT: http://review.gluster.org/15398 committed in master by Pranith Kumar Karampuri (pkarampu) ------ commit 8d8eded58cd5431a7000a70337444b828cb400d8 Author: Poornima G <pgurusid> Date: Sun Sep 4 08:27:47 2016 +0530 md-cache, afr: Reduce the window of stale read Problem: Consider a replica setup, where one mount writes data to a file and the other mount reads the file. In afr, read operations are not transaction based, a brick(read subvolume) is chosen as a part of lookup or other operations, read is always wound only to the read subvolume, even if there was write from a different client that failed on this brick. This stale read continues until there is a lookup or any write operation from the mount point. Currently, this is not a major issue, as a lookup is issued before every read and it will switch the read subvolume to a correct one. But with the plan of increasing md-cache timeout to 600s, the stale read problem will be more pronounced, i.e. stale read can continue for 600s(or more if cascaded with readdirp), as there will be no lookups. Solution: Afr doesn't have any built-in solution for stale read(without affecting the performance). The solution that came up, was to use upcall. When a file on any brick is marked bad for the first time, upcall sends a notification to all the clients that had recently accessed the file. The solution has 2 parts: - Identifying when a file is marked bad, on any of the bricks, for the first time - Client side actions on recieving the notifications Identifying when a file is marked bad on any of the bricks for the first time: ----------------------------------------------------------------------------- The idea is to track xattrop in upcall. xattrop currently comes with 2 afr xattrs - afr dirty bit and afr pending xattrs. Dirty xattr is set to 1 before every write, and is unset if write succeeds. In certain scenarios, dirty xattr can be 0 and still the file could be bad copy. Hence do not track dirty xattr. Pending xattr is set on the good copy, indicating the other bricks that have bad copy. It is still not as simple as, notifying when any of the pending xattrs change. It could lead to flood of notifcations, in case the other brick is completely down or consistantly failing. Hence it is important to notify only once, the first time a good copy is marked bad. Client side actions on recieving pending xattr change, notification: -------------------------------------------------------------------- md-cache will invalidate the cache of that file, so that further lookup is passed down to afr and hence update the read subvolume. Invalidating only in md-cache is not enough, consider the folling oder of opertaions: - pending xattr invalidation - invalidate md-cache - readdirp on the bad read subvolume - fill md-cache - lookup (served from md-cache) - read - wound to the old read subvol. Hence, along with invalidating md-cache, it is very important to reset the read subvolume for that file, in afr. Design Credit: Anuradha Talur, Ravishankar N 1. xattrop doesn't carry info saying post op/pre op. 2. Pre xattrop will have 0 value for all pending xattrs, the cbk of pre xattrop carries the on-disk xattr value. Non zero indicated healing is required. 3. Post xattrop will have non zero value for any of the pending xattrs, if the fop failed on any of the bricks. Change-Id: I469cbc111714c433984fe1c922be2ef113c25804 BUG: 1211863 Signed-off-by: Poornima G <pgurusid> Reviewed-on: http://review.gluster.org/15398 Reviewed-by: Pranith Kumar Karampuri <pkarampu> Tested-by: Pranith Kumar Karampuri <pkarampu> Smoke: Gluster Build System <jenkins.org> NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.10.0, please open a new bug report. glusterfs-3.10.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://lists.gluster.org/pipermail/gluster-users/2017-February/030119.html [2] https://www.gluster.org/pipermail/gluster-users/