Description of problem: glusterfsd is keeping fd open in index xlator Version-Release number of selected component (if applicable): How reproducible: Alwaus Steps to Reproduce: 1.Setup 100 volumes(test1..test100) and Enable brick multiplex 2.Start test1 volume 3.Run below command for j in {1..10}; do sleep 120;if ! ls -lrt /proc/`pidof glusterfsd`/fd; then break; fi;if ls -lrth /proc/`pidof glusterfsd`/fd | grep ".gluster" | grep -v "pid" | grep -v "log"; then break; fi;for i in {2..100}; do gluster --mode=script v start test${i} ; done;sleep 120;for i in {2..100}; do gluster --mode=script v stop test${i};done; done Actual results: glusterfsd is keeping fd open in index xlator for down brick Expected results: glusterfsd should not consume fd for a detach brick Additional info:
REVIEW: https://review.gluster.org/21379 (core: glusterfsd keeping fd open in index xlator) posted (#4) for review on master by Pranith Kumar Karampuri
There is one another way to reproduce the same diff --git a/xlators/protocol/server/src/server-helpers.c b/xlators/protocol/server/src/server-helpers.c index c492ab164..29af4a946 100644 --- a/xlators/protocol/server/src/server-helpers.c +++ b/xlators/protocol/server/src/server-helpers.c @@ -249,6 +249,7 @@ server_connection_cleanup_flush_cbk (call_frame_t *frame, void *cookie, fd = frame->local; client = frame->root->client; + sleep (5); fd_unref (fd); frame->local = NULL; Put sleep in code before doing fd_unref in server_connection_cleanup_flush_cbk and compile the code and follow the steps Steps: 1) start glusterd and set brick-mux to on 2) Create 2 plain replicate volumes and set open-behind off on the volume 3) Mount one of the volumes and on the mount execute "exec >a" 4) confirm that the file is opened on the bricks 5) execute "gluster volume stop <volname>" 6) Wait for a minute just to be on safer side and check "ls /proc/<pid-of-brick>/fd" It shows the file 'a' Regards Mohit Agrawal
COMMIT: https://review.gluster.org/21379 committed in master by "Amar Tumballi" <amarts> with a commit message- core: glusterfsd keeping fd open in index xlator Problem: At the time of processing GF_EVENT_PARENT_DOWN at brick xlator, it forwards the event to next xlator only while xlator ensures no stub is in progress. At io-thread xlator it decreases stub_cnt before the process a stub and notify EVENT to next xlator Solution: Introduce a new counter to save stub_cnt and decrease the counter after process the stub completely at io-thread xlator. To avoid brick crash at the time of call xlator_mem_cleanup move only brick xlator if detach brick name has found in the graph Note: Thanks to pranith for sharing a simple reproducer to reproduce the same fixes bz#1637934 Change-Id: I1a694a001f7a5417e8771e3adf92c518969b6baa Signed-off-by: Mohit Agrawal <moagrawal>
REVIEW: https://review.gluster.org/21480 (tests: brick-mux-fd-cleanup.t should be under core directory) posted (#1) for review on master by Atin Mukherjee
REVIEW: https://review.gluster.org/21480 (tests: brick-mux-fd-cleanup.t should be under core directory) posted (#2) for review on master by Amar Tumballi
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-6.0, please open a new bug report. glusterfs-6.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] https://lists.gluster.org/pipermail/announce/2019-March/000120.html [2] https://www.gluster.org/pipermail/gluster-users/