REVIEW: http://review.gluster.org/14148 (gfapi: Fix the synctask deadlock) posted (#1) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/14148 (gfapi: Fix the hang caused, when a graph switch is triggered, while async io is going on) posted (#2) for review on master by Poornima G (pgurusid)
REVIEW: http://review.gluster.org/14148 (gfapi: Fix a deadlock caused by graph switch while aio in progress) posted (#3) for review on master by Poornima G (pgurusid)
COMMIT: http://review.gluster.org/14148 committed in master by Pranith Kumar Karampuri (pkarampu) ------ commit 4b2b9a971e09b0c17018f4a5dd24fa7d64e94bf5 Author: Poornima G <pgurusid> Date: Fri Apr 29 12:24:24 2016 -0400 gfapi: Fix a deadlock caused by graph switch while aio in progress RCA: Currently async nature is achieved by submitting a syncop operation to synctask threads. Consider a scenario where the graph switch is triggered, the next write fop checks for the next available graph and sets fs->migration_in_progess and triggers the migration of fds and other things, which can cause some syncop_lookup operation. While this fop (on synctask thread) is waiting for syncop_lookup to return, lets say there are another 17 write async calls submitted, all these writes are blocked waiting for fs->migration_in_progress to be unset, hence all the 16 synctask threads are blocked waiting for fs->migration_in_progress to be unset. Now the syncop_lookup returns, but there are no synctask threads to process the lookup_cbk. If this syncop_lookup doesn't return, then fs->migration_in_progress can not be unset by the first fop. Thus causing a deadlock. To fix this deadlock, changing all the async APIs to use STACK_WIND, instead of syntask to achieve async nature. glfs_preadv_async is already implemented using STACK_WIND, now changing all the other async APIs also to do the same. This patch as such will not reduce the performance of async IO, the only thing that can affect is that, in case of write, the buf passed by application is copied onto iobuf in the same thread wheras before it was being copied in synctask thread. Since, the syncop + graph switch logic (lock across fops) is not a good candidate for synctask, changing the async APIs to use STACK_WIND Change-Id: Idf665cae0a8e27697fbfc5ec8d93a6d6bae3a4f1 BUG: 1332156 Signed-off-by: Poornima G <pgurusid> Reviewed-on: http://review.gluster.org/14148 Smoke: Gluster Build System <jenkins.com> NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.com> Reviewed-by: Raghavendra Talur <rtalur> Reviewed-by: Rajesh Joseph <rjoseph> Reviewed-by: Pranith Kumar Karampuri <pkarampu>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.9.0, please open a new bug report. glusterfs-3.9.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://lists.gluster.org/pipermail/gluster-users/2016-November/029281.html [2] https://www.gluster.org/pipermail/gluster-users/