Bug 1226717

Summary: racy condition in nfs/auth-cache feature
Product: [Community] GlusterFS Reporter: Niels de Vos <ndevos>
Component: nfsAssignee: Niels de Vos <ndevos>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: high Docs Contact:
Priority: high    
Version: mainlineCC: bugs, gluster-bugs
Target Milestone: ---Keywords: Reopened, Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
URL: http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/11052/focus=11109
Whiteboard:
Fixed In Version: glusterfs-3.8rc2 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1242515 (view as bug list) Environment:
Last Closed: 2016-06-16 13:06:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1228157, 1235628, 1235939    
Bug Blocks: 1242515    

Description Niels de Vos 2015-05-31 21:16:43 UTC
Description of problem:
The auth-cache feature contains a function called auth_cache_purge(). This function replaces the auth_cache->cache_dict with a new dictionary that should contain fresh caches. The placing is triggered by the _mnt3_auth_param_refresh_thread().

There is no locking of the actual auth_cache_entry structures, and auth_cache_purge() can cause the free'ing of these entries while other threads are using them.

It is very rare to notice a problem, because the auth_cache_entry structures are used only very briefly. A chance for corruption is really small. Our regression tests seem to have hit this issue only once or twice in the last few months.

Version-Release number of selected component (if applicable):
3.7 and mainline

How reproducible:
extremely difficult. 

Additional info:
http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/11052/focus=11109

Comment 1 Anand Avati 2015-05-31 21:41:55 UTC
REVIEW: http://review.gluster.org/11021 (nfs: add a gf_lock_t for the auth_cache->cache_dict) posted (#1) for review on master by Niels de Vos (ndevos)

Comment 2 Anand Avati 2015-05-31 21:41:57 UTC
REVIEW: http://review.gluster.org/11022 (core: add "struct gf_ref" for common refcounting strutures) posted (#1) for review on master by Niels de Vos (ndevos)

Comment 3 Anand Avati 2015-05-31 21:41:59 UTC
REVIEW: http://review.gluster.org/11023 (nfs: refcount each auth_cache_entry and related data_t) posted (#1) for review on master by Niels de Vos (ndevos)

Comment 4 Anand Avati 2015-05-31 23:14:24 UTC
REVIEW: http://review.gluster.org/11021 (nfs: add a gf_lock_t for the auth_cache->cache_dict) posted (#2) for review on master by Niels de Vos (ndevos)

Comment 5 Anand Avati 2015-05-31 23:14:27 UTC
REVIEW: http://review.gluster.org/11022 (core: add "struct gf_ref" for common refcounting strutures) posted (#2) for review on master by Niels de Vos (ndevos)

Comment 6 Anand Avati 2015-05-31 23:14:55 UTC
REVIEW: http://review.gluster.org/11023 (nfs: refcount each auth_cache_entry and related data_t) posted (#2) for review on master by Niels de Vos (ndevos)

Comment 7 Anand Avati 2015-06-04 10:48:55 UTC
REVIEW: http://review.gluster.org/11021 (nfs: add a gf_lock_t for the auth_cache->cache_dict) posted (#3) for review on master by Niels de Vos (ndevos)

Comment 8 Anand Avati 2015-06-04 10:49:02 UTC
REVIEW: http://review.gluster.org/11023 (nfs: refcount each auth_cache_entry and related data_t) posted (#3) for review on master by Niels de Vos (ndevos)

Comment 9 Anand Avati 2015-06-04 13:45:11 UTC
REVIEW: http://review.gluster.org/11021 (nfs: add a gf_lock_t for the auth_cache->cache_dict) posted (#4) for review on master by Niels de Vos (ndevos)

Comment 10 Anand Avati 2015-06-04 13:45:15 UTC
REVIEW: http://review.gluster.org/11023 (nfs: refcount each auth_cache_entry and related data_t) posted (#4) for review on master by Niels de Vos (ndevos)

Comment 11 Anand Avati 2015-06-04 23:15:06 UTC
REVIEW: http://review.gluster.org/11021 (nfs: add a gf_lock_t for the auth_cache->cache_dict) posted (#5) for review on master by Niels de Vos (ndevos)

Comment 12 Anand Avati 2015-06-04 23:15:10 UTC
REVIEW: http://review.gluster.org/11023 (nfs: refcount each auth_cache_entry and related data_t) posted (#5) for review on master by Niels de Vos (ndevos)

Comment 13 Anand Avati 2015-06-18 16:09:08 UTC
REVIEW: http://review.gluster.org/11021 (nfs: add a gf_lock_t for the auth_cache->cache_dict) posted (#6) for review on master by Niels de Vos (ndevos)

Comment 14 Anand Avati 2015-06-18 16:09:11 UTC
REVIEW: http://review.gluster.org/11023 (nfs: refcount each auth_cache_entry and related data_t) posted (#6) for review on master by Niels de Vos (ndevos)

Comment 15 Anand Avati 2015-06-18 18:04:43 UTC
REVIEW: http://review.gluster.org/11023 (nfs: refcount each auth_cache_entry and related data_t) posted (#7) for review on master by Niels de Vos (ndevos)

Comment 16 Anand Avati 2015-06-18 18:08:07 UTC
REVIEW: http://review.gluster.org/11023 (nfs: refcount each auth_cache_entry and related data_t) posted (#8) for review on master by Niels de Vos (ndevos)

Comment 17 Anand Avati 2015-06-23 21:41:39 UTC
REVIEW: http://review.gluster.org/11021 (nfs: add a gf_lock_t for the auth_cache->cache_dict) posted (#7) for review on master by Niels de Vos (ndevos)

Comment 18 Anand Avati 2015-06-28 14:05:46 UTC
REVIEW: http://review.gluster.org/11021 (nfs: add a gf_lock_t for the auth_cache->cache_dict) posted (#8) for review on master by Niels de Vos (ndevos)

Comment 19 Anand Avati 2015-06-28 17:37:40 UTC
COMMIT: http://review.gluster.org/11023 committed in master by Niels de Vos (ndevos) 
------
commit 7b51bd636fc5e5e1ae48a4e7cba48d0d20878d15
Author: Niels de Vos <ndevos>
Date:   Sun May 31 16:27:44 2015 +0200

    nfs: refcount each auth_cache_entry and related data_t
    
    This makes sure that all the auth_cache_entry structures are only free'd
    when there is no reference to it anymore. When it is free'd, the
    associated data_t from the auth_cache->cache_dict gets unref'd too.
    
    Upon calling auth_cache_purge(), the auth_cache->cache_dict will free
    each auth_cache_entry in a secure way.
    
    Change-Id: If097cc11838e43599040f5414f82b30fc0fd40c6
    BUG: 1226717
    Signed-off-by: Niels de Vos <ndevos>
    Reviewed-on: http://review.gluster.org/11023
    Reviewed-by: Xavier Hernandez <xhernandez>
    Tested-by: Gluster Build System <jenkins.com>
    Tested-by: NetBSD Build System <jenkins.org>

Comment 20 Nagaprasad Sathyanarayana 2015-10-25 15:08:02 UTC
Fix for this BZ is already present in a GlusterFS release. You can find clone of this BZ, fixed in a GlusterFS release and closed. Hence closing this mainline BZ as well.

Comment 21 Niels de Vos 2016-06-16 13:06:59 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report.

glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user