Bug 1626787

a flush request was indeed stuck in write-behind.

<raghug> [xlator.performance.write-behind.wb_inode]
<raghug> path=/SAS_workAB39000020E2_rhs-client44.lab.eng.blr.redhat.com/TD_5/FCST/process/SUBG/gp_h_filter.sas7bdat.lck
<raghug> inode=0x7f4f9c463c40
<raghug> gfid=59e84e6d-76f5-40f0-8888-8d3745291011
<raghug> window_conf=1048576
<raghug> window_current=65536
<raghug> transit-size=0
<raghug> dontsync=0
<raghug> [.WRITE]
<raghug> unique=17337841
<raghug> refcount=1
<raghug> wound=no
<raghug> generation-number=6
<raghug> req->op_ret=65536
<raghug> req->op_errno=0
<raghug> sync-attempts=0
<raghug> sync-in-progress=no
<raghug> size=65536
<raghug> offset=65536
<raghug> lied=-1
<raghug> append=0
<raghug> fulfilled=0
<raghug> go=-1
<raghug> [.FLUSH]
<raghug> unique=17337847
<raghug> refcount=1
<raghug> wound=no
<raghug> generation-number=9
<raghug> req->op_ret=0
<raghug> req->op_errno=0
<raghug> sync-attempts=0

REVIEW: https://review.gluster.org/21123 (performance/write-behind: remove the request from wip queue in wb_fulfill_request) posted (#2) for review on master by Raghavendra G

COMMIT: https://review.gluster.org/21123 committed in master by "Raghavendra G" <rgowdapp> with a commit message- performance/write-behind: remove the request from wip queue in wb_fulfill_request

The bug is very similar to bz 1379655 and the fix too very similar to
commit a8b2a981881221925bb5edfe7bb65b25ad855c04.

Before this patch, a request is removed from wip queue only when ref
count of request hits 0. Though, wb_fulfill_request does an unref,
it need not be the last unref and hence the request may survive in
wip queue till the last unref. Let,

T1: the time at which wb_fulfill_request is invoked
T2: the time at which last unref is done on request

Let's consider a case of T2 > T1. In the time window between T1 and
T2, any other request (waiter) conflicting with request in liability
queue (blocker - basically a write which has been lied) is blocked
from winding. If T2 happens to be when wb_do_unwinds is invoked, no
further processing of request list happens and "waiter" would get
blocked forever. An example imaginary sequence of events is given
below:

1. A write request w1 is picked up for winding in __wb_pick_winds
   and w1 is moved to wip queue. Let's call this
   invocation of wb_process_queue by wb_writev as PQ1. Note w1 is not
   unwound.

2. A dependent write (w2) hits write-behind and is unwound followed by
   a flush (f1) request. Since the liability queue
   of inode is not empty, w2 and f1 are not picked for unwinding. Let's call
   the invocation of wb_process_queue by wb_flush as PQ2. Note that
   invocation of wb_process_queue by w2 doesn't wind w2 instead
   unwinds it after which we hit PQ2

3. PQ2 continues and picks w1 for fulfilling and invokes
   wb_fulfill. As part of successful wb_fulfill_cbk,
   wb_fulfill_request (w1) is invoked. But, w1 is not freed (and hence
   not removed from wip queue) as w1 is not unwound _yet_ and a
   ref remains (PQ1 has not invoked wb_do_unwinds _yet_).

4. wb_fulfill_cbk (triggered by PQ2) invokes a wb_process_queue (let's
   say PQ3). w2 is not picked up for winding in PQ3 as w1 is still in wip
   queue. At this time, PQ2 and PQ3 are complete.

5. PQ1 continues, unwinds w1 and does last unref on w1 and w1 is freed
   (and removed from wip queue). Since PQ1 didn't invoke
   wb_fulfill on any other write requests, there won't be any future
   codepaths that would invoke wb_process_queue and w2 is stuck
   forever. This will prevent f2 too and hence close syscall is hung

With this fix, w1 is removed from liability queue in step 3 above and
PQ3 winds w2 in step 4 (as there are no requests conflicting with w2
in liability queue during execution of PQ3). Once w2 is complete, f1
is resumed.

Change-Id: Ia972fad0858dc4abccdc1227cb4d880f85b3b89b
Signed-off-by: Raghavendra G <rgowdapp>
Fixes: bz#1626787

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-5.0, please open a new bug report.

glusterfs-5.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] https://lists.gluster.org/pipermail/announce/2018-October/000115.html
[2] https://www.gluster.org/pipermail/gluster-users/