Bug 1191030
Summary: | Use rcu to protect concurrent access to data structures in GlusterD | |||
---|---|---|---|---|
Product: | [Community] GlusterFS | Reporter: | Kaushal <kaushal> | |
Component: | glusterd | Assignee: | Kaushal <kaushal> | |
Status: | CLOSED CURRENTRELEASE | QA Contact: | ||
Severity: | medium | Docs Contact: | ||
Priority: | high | |||
Version: | mainline | CC: | bugs, gluster-bugs, sasundar, stoynsheron15 | |
Target Milestone: | --- | |||
Target Release: | --- | |||
Hardware: | Unspecified | |||
OS: | Unspecified | |||
Whiteboard: | ||||
Fixed In Version: | glusterfs-3.7.0 | Doc Type: | Bug Fix | |
Doc Text: | Story Points: | --- | ||
Clone Of: | ||||
: | 1211207 (view as bug list) | Environment: | ||
Last Closed: | 2015-05-14 17:29:06 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: | ||||
Bug Depends On: | 1205186 | |||
Bug Blocks: | 1211207 |
Description
Kaushal
2015-02-10 10:11:15 UTC
REVIEW: http://review.gluster.org/9605 (Add liburcu to build system) posted (#2) for review on master by Kaushal M (kaushal) REVIEW: http://review.gluster.org/9624 (glusterd: Replace libglusterfs lists with liburcu lists) posted (#1) for review on master by Kaushal M (kaushal) REVIEW: http://review.gluster.org/9605 (Add liburcu to build system) posted (#3) for review on master by Niels de Vos (ndevos) REVIEW: http://review.gluster.org/9605 (build, glusterd: Add liburcu to build system) posted (#4) for review on master by Kaushal M (kaushal) REVIEW: http://review.gluster.org/9624 (glusterd: Replace libglusterfs lists with liburcu lists) posted (#2) for review on master by Kaushal M (kaushal) REVIEW: http://review.gluster.org/9695 (glusterd: Protect the peer list and peerinfos with RCU.) posted (#1) for review on master by Kaushal M (kaushal) REVIEW: http://review.gluster.org/9605 (build, glusterd: Add liburcu to build system) posted (#5) for review on master by Kaushal M (kaushal) REVIEW: http://review.gluster.org/9624 (glusterd: Replace libglusterfs lists with liburcu lists) posted (#3) for review on master by Kaushal M (kaushal) REVIEW: http://review.gluster.org/9695 (glusterd: Protect the peer list and peerinfos with RCU.) posted (#2) for review on master by Kaushal M (kaushal) COMMIT: http://review.gluster.org/9605 committed in master by Krishnan Parthasarathi (kparthas) ------ commit 30357bcfcd4a7c92e0e59ff58a26d48c8416c564 Author: Kaushal M <kaushal> Date: Tue Jan 6 15:12:59 2015 +0530 build, glusterd: Add liburcu to build system This patch adds liburcu related checks to the build system and updates the spec file to require 'userspace-rcu'. liburcu >= 0.7 is required to build GlusterFS, but 0.8 and above is preferred. For cases when liburcu 0.7.x is the available version, some function definitions (currently just one) from liburcu-0.8.6 have been made available in /contrib/userspace-rcu/. This change was developed on the git branch at [1]. This commit is a combination of the following commits on the development branch. a5cd6bd Add userspace-rcu checks to configure.ac fe5ced3 Add URCU libs to glusterd libtool flags 1e43302 Add local definition of cds_list_add_tail_rcu for liburcu-0.7 98da755 Move local definition of cds_list_add_tail_rcu into contrib 8c44dfd Update spec file to include userspace-rcu0466e33 Rename rculist-additional.h to rculist-extra.h 947c7b3 Add rculist-extra.h to dist 19f32ad Address review comments 9605/1 [1]: https://github.com/kshlm/glusterfs/tree/urcu Change-Id: Ifbb617d0dacce8fa01214f894badb9d8cdcaf56f BUG: 1191030 Signed-off-by: Kaushal M <kaushal> Reviewed-on: http://review.gluster.org/9605 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Atin Mukherjee <amukherj> Reviewed-by: Niels de Vos <ndevos> Reviewed-by: Krishnan Parthasarathi <kparthas> Tested-by: Krishnan Parthasarathi <kparthas> REVIEW: http://review.gluster.org/9624 (glusterd: Replace libglusterfs lists with liburcu lists) posted (#4) for review on master by Kaushal M (kaushal) REVIEW: http://review.gluster.org/9624 (glusterd: Replace libglusterfs lists with liburcu lists) posted (#5) for review on master by Kaushal M (kaushal) REVIEW: http://review.gluster.org/9695 (glusterd: Protect the peer list and peerinfos with RCU.) posted (#3) for review on master by Kaushal M (kaushal) REVIEW: http://review.gluster.org/9624 (glusterd: Replace libglusterfs lists with liburcu lists) posted (#6) for review on master by Atin Mukherjee (amukherj) REVIEW: http://review.gluster.org/9695 (glusterd: Protect the peer list and peerinfos with RCU.) posted (#4) for review on master by Atin Mukherjee (amukherj) REVIEW: http://review.gluster.org/9755 (glusterd: protect vol_list and volinfo with RCU) posted (#1) for review on master by Atin Mukherjee (amukherj) REVIEW: http://review.gluster.org/9624 (glusterd: Replace libglusterfs lists with liburcu lists) posted (#7) for review on master by Kaushal M (kaushal) REVIEW: http://review.gluster.org/9695 (glusterd: Protect the peer list and peerinfos with RCU.) posted (#5) for review on master by Kaushal M (kaushal) COMMIT: http://review.gluster.org/9624 committed in master by Krishnan Parthasarathi (kparthas) ------ commit 673ba2659cebe22ee30c43f9fb080f330150f55e Author: Kaushal M <kaushal> Date: Tue Jan 6 18:23:41 2015 +0530 glusterd: Replace libglusterfs lists with liburcu lists This patch replaces usage of the libglusterfs lists data structures and API in glusterd with the lists data structures and API from liburcu. The liburcu data structes and APIs are a drop-in replacement for libglusterfs lists. All usages have been changed to keep the code consistent, and free from confusion. NOTE: glusterd_conf_t->xprt_list still uses the libglusterfs data structures and API, as it holds rpc_transport_t objects, which is not a part of glusterd and is not being changed in this patch. This change was developed on the git branch at [1]. This commit is a combination of the following commits on the development branch. 6dac576 Replace libglusterfs lists with liburcu lists a51b5ab Fix compilation issues d98a06f Fix merge issues a5d918e Remove merge remnant 1cca113 More style cleanup 1917be3 Address review comments on 9624/1 8d10f13 Use cds_lists for glusterd_svc_t 524ad5d Add rculist header in glusterd-conn-helper.c 646f294 glusterd: add list_add_order API honouring rcu [1]: https://github.com/kshlm/glusterfs/tree/urcu Change-Id: Ic613c5b6e496a677b9d3de15fc042a0492109fb0 BUG: 1191030 Signed-off-by: Kaushal M <kaushal> Signed-off-by: Krishnan Parthasarathi <kparthas> Reviewed-on: http://review.gluster.org/9624 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Atin Mukherjee <amukherj> Reviewed-by: Gaurav Kumar Garg <ggarg> Reviewed-by: Anand Nekkunti <anekkunt> REVIEW: http://review.gluster.org/9695 (glusterd: Protect the peer list and peerinfos with RCU.) posted (#6) for review on master by Kaushal M (kaushal) REVIEW: http://review.gluster.org/9695 (glusterd: Protect the peer list and peerinfos with RCU.) posted (#7) for review on master by Kaushal M (kaushal) http://review.gluster.org/9695 implements most of the protection for peerinfos and the peers list. But it still leaves peerinfo protection during a volume transaction, especially the transaction peers list, open. REVIEW: http://review.gluster.org/9695 (glusterd: Protect the peer list and peerinfos with RCU.) posted (#8) for review on master by Kaushal M (kaushal) REVIEW: http://review.gluster.org/9695 (glusterd: Protect the peer list and peerinfos with RCU.) posted (#9) for review on master by Kaushal M (kaushal) Need to also address the issue the glusterd_quorum_count macro. REVIEW: http://review.gluster.org/9695 (glusterd: Protect the peer list and peerinfos with RCU.) posted (#10) for review on master by Kaushal M (kaushal) COMMIT: http://review.gluster.org/9695 committed in master by Krishnan Parthasarathi (kparthas) ------ commit c7785f78420c94220954eef538ed4698713ebcdb Author: Kaushal M <kaushal> Date: Thu Jan 8 19:24:59 2015 +0530 glusterd: Protect the peer list and peerinfos with RCU. The peer list and the peerinfo objects are now protected using RCU. Design patterns described in the Paul McKenney's RCU dissertation [1] (sections 5 and 6) have been used to convert existing non-RCU protected code to RCU protected code. Currently, we are only targetting guaranteeing the existence of the peerinfo objects, ie., we are only looking to protect deletes, not all updaters. We chose this, as protecting all updates is a much more complex task. The steps used to accomplish this are, 1. Remove all long lived direct references to peerinfo objects (apart from the peerinfo list). This includes references in glusterd_peerctx_t (RPC), glusterd_friend_sm_event_t (friend state machine) and others. This way no one has a reference to deleted peerinfo object. 2. Replace the direct references with indirect references, ie., use peer uuid and peer hostname as indirect references to the peerinfo object. Any reader or updater now uses the indirect references to get to the actual peerinfo object, using glusterd_peerinfo_find. Cases where a peerinfo cannot be found are handled gracefully. 3. The readers get and use the peerinfo object only within a RCU read critical section. This prevents the object from being deleted/freed when in actual use. 4. The deletion of a peerinfo object is done in a ordered manner (glusterd_peerinfo_destroy). The object is first removed from the peerinfo list using an atomic list remove, but the list head is not reset to allow existing list readers to complete correctly. We wait for readers to complete, before resetting the list head. This removes the object from the list completely. After this no new readers can get a reference to the object, and it can be freed. This change was developed on the git branch at [2]. This commit is a combination of the following commits on the development branch. d7999b9 Protect the glusterd_conf_t->peers_list with RCU. 0da85c4 Synchronize before INITing peerinfo list head after removing from list. 32ec28a Add missing rcu_read_unlock 8fed0b8 Correctly exit read critical section once peer is found. 63db857 Free peerctx only on rpc destruction 56eff26 Cleanup style issues e5f38b0 Indirection for events and friend_sm 3c84ac4 In __glusterd_probe_cbk goto unlock only if peer already exists 141d855 Address review comments on 9695/1 aaeefed Protection during peer updates 6eda33d Revert "Synchronize before INITing peerinfo list head after removing from list." f69db96 Remove unneeded line b43d2ec Address review comments on 9695/4 7781921 Address review comments on 9695/5 eb6467b Add some missing semi-colons 328a47f Remove synchronize_rcu from glusterd_friend_sm_transition_state 186e429 Run part of glusterd_friend_remove in critical section 55c0a2e Fix gluster (peer status/ pool list) with no peers 93f8dcf Use call_rcu to free peerinfo c36178c Introduce composite struct, gd_rcu_head [1]: http://www.rdrop.com/~paulmck/RCU/RCUdissertation.2004.07.14e1.pdf [2]: https://github.com/kshlm/glusterfs/tree/urcu Change-Id: Ic1480e59c86d41d25a6a3d159aa3e11fbb3cbc7b BUG: 1191030 Signed-off-by: Kaushal M <kaushal> Reviewed-on: http://review.gluster.org/9695 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Atin Mukherjee <amukherj> Reviewed-by: Anand Nekkunti <anekkunt> Reviewed-by: Krishnan Parthasarathi <kparthas> Tested-by: Krishnan Parthasarathi <kparthas> REVIEW: http://review.gluster.org/9922 (glusterd: Remove compilation warning) posted (#1) for review on master by Kaushal M (kaushal) COMMIT: http://review.gluster.org/9922 committed in master by Kaushal M (kaushal) ------ commit 1250003956bc9fe7f030f0541f1c823cb282a07a Author: Kaushal M <kaushal> Date: Tue Mar 17 12:06:25 2015 +0530 glusterd: Remove compilation warning In glusterd_peerinfo_destroy, cast the passed 'strcut rcu_head *' pointer to 'gd_rcu_head *' before use in caa_container_of() to prevent the incompatible-pointer compilation warning. Also, refactor peerinfo->head to peerinfo->rcu_head to reduce confusion when reading code. This change was developed on the git branch at [1]. This commit is a combination of the following commits on the development branch. aa4a0bc Rename peerinfo->head to peerinfo->rcu_head c79144b Cast struct rcu_head * to gd_rcu_head * to prevent warning 1d222c3 More head -> rcu_head renames [1]: https://github.com/kshlm/glusterfs/tree/urcu BUG: 1191030 Change-Id: I7ede02090413839563ce44fdf6289697b28777e7 Signed-off-by: Kaushal M <kaushal> Reviewed-on: http://review.gluster.org/9922 Reviewed-by: Atin Mukherjee <amukherj> Tested-by: Gluster Build System <jenkins.com> REVIEW: http://review.gluster.org/9978 (glusterd: Prevent possible deadlock due to glusterd_quorum_count) posted (#1) for review on master by Kaushal M (kaushal) REVIEW: http://review.gluster.org/9978 (glusterd: Prevent possible deadlock due to glusterd_quorum_count) posted (#2) for review on master by Kaushal M (kaushal) REVIEW: http://review.gluster.org/9979 (glusterd: Prevent possible dealock in glusterd_friend_remove) posted (#1) for review on master by Kaushal M (kaushal) Created a new bug to track the TODOs remaining for GlusterFS-3.7.0 release. Removing this bug as a blocker on glusterfs-3.7.0. This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report. glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939 [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report. glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939 [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report. glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939 [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report. glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939 [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user The internet is flooded with websites that offer college essays and research papers for sale. These websites https://12hoursessay.com/sale.html are a boon for students who want to buy essays, but don't know how to write them themselves. These sites offer custom written papers on any topic you can think of and in any format. They also offer free samples so you can test the quality of their work before you buy anything from them. If you are looking for someone who will write your paper, then these websites are worth checking out. |