Description of problem: ======================= "ls" command on the fuse mount gets blocked when O_SYNC writes was in progress on the same mount, when barrier was enabled Version-Release number of selected component (if applicable): ============================================================== mainline How reproducible: ================= Always Steps to Reproduce: =================== 1. Enable barrier 2. Start O_SYNC writes on the fuse mount 3. perform "ls" on the same mount Actual results: =============== "ls" command got blocked till the barrier was disabled Expected results: ================= "ls" command should not get blocked Additional info: ================= While repeating the test with other barrier class fops {unlink, rmdir, removexattr, fremovexattr, fsync, rename, O_SYNC write }, I see that "ls" command got blocked except for truncate and ftruncate calls, till barrier was enabled.
REVIEW: http://review.gluster.org/7872 (dht: pass xdata to xlators above) posted (#1) for review on master by Krishnan Parthasarathi (kparthas)
REVIEW: http://review.gluster.org/7873 (barrier: enable FUSE direct-io-mode for barriered fds) posted (#1) for review on master by Krishnan Parthasarathi (kparthas)
REVIEW: http://review.gluster.org/7872 (dht: pass xdata to xlators above.) posted (#2) for review on master by Krishnan Parthasarathi (kparthas)
REVIEW: http://review.gluster.org/7873 (barrier: enable FUSE direct-io-mode for barriered fds) posted (#2) for review on master by Krishnan Parthasarathi (kparthas)
REVIEW: http://review.gluster.org/7872 (dht: pass xdata to xlators above.) posted (#3) for review on master by Krishnan Parthasarathi (kparthas)
REVIEW: http://review.gluster.org/7873 (barrier: enable FUSE direct-io-mode for barriered fds) posted (#3) for review on master by Krishnan Parthasarathi (kparthas)
REVIEW: http://review.gluster.org/7872 (dht: pass xdata to xlators above) posted (#4) for review on master by Krishnan Parthasarathi (kparthas)
REVIEW: http://review.gluster.org/7873 (barrier: enable FUSE direct-io-mode for barriered fds) posted (#4) for review on master by Krishnan Parthasarathi (kparthas)
Root cause analysis: -------------------- Problem: In a GlusterFS volume, when barrier is enabled, accessed via FUSE mount point, "ls --color" hangs when O_SYNC writes are performed on the same mount concurrently. Cause: In the FUSE kernel module, readdirplus attempts an attribute update on a file on which write is under-way, with either the new file size or the new mtime (possibly caused by the same write, or otherwise) and the page invalidation is waiting for the on-going write to complete before returning. The on-going write would not complete until barrier is disabled. This causes "ls --color" to hang. Fix: To prevent readdirp from being blocked on the page invalidation due to the barrier'd O_SYNC write, we set direct-io-mode flag for the fd which prevents page caching in the FUSE kernel module.
REVIEW: http://review.gluster.org/7872 (dht: pass xdata to xlators above.) posted (#5) for review on master by Krishnan Parthasarathi (kparthas)
REVIEW: http://review.gluster.org/7873 (barrier: enable FUSE direct-io-mode for barriered fds) posted (#5) for review on master by Krishnan Parthasarathi (kparthas)
REVIEW: http://review.gluster.org/7872 (dht: pass xdata to xlators above.) posted (#6) for review on master by Atin Mukherjee (amukherj)
REVIEW: http://review.gluster.org/7873 (barrier: enable FUSE direct-io-mode for barriered fds) posted (#6) for review on master by Atin Mukherjee (amukherj)
REVIEW: http://review.gluster.org/7872 (dht: pass xdata to xlators above.) posted (#7) for review on master by Atin Mukherjee (amukherj)
REVIEW: http://review.gluster.org/7873 (barrier: enable FUSE direct-io-mode for barriered fds) posted (#7) for review on master by Atin Mukherjee (amukherj)
REVIEW: http://review.gluster.org/7999 (libgfapi : glfs_create()/glfs_open() should set fd->flgas before syncop_open()) posted (#1) for review on master by Atin Mukherjee (amukherj)
REVIEW: http://review.gluster.org/7872 (dht: pass xdata to xlators above.) posted (#8) for review on master by Atin Mukherjee (amukherj)
REVIEW: http://review.gluster.org/7999 (libgfapi : glfs_create()/glfs_open() should set fd->flgas before syncop_open()) posted (#2) for review on master by Atin Mukherjee (amukherj)
REVIEW: http://review.gluster.org/7999 (libgfapi : glfs_create()/glfs_open() should set fd->flags before syncop_open()) posted (#3) for review on master by Krishnan Parthasarathi (kparthas)
REVIEW: http://review.gluster.org/7872 (dht: pass xdata to xlators above) posted (#9) for review on master by Atin Mukherjee (amukherj)
REVIEW: http://review.gluster.org/7873 (barrier: enable FUSE direct-io-mode for barriered fds) posted (#8) for review on master by Atin Mukherjee (amukherj)
REVIEW: http://review.gluster.org/7999 (libgfapi : glfs_create()/glfs_open() should set fd->flags before syncop_open()) posted (#4) for review on master by Atin Mukherjee (amukherj)
REVIEW: http://review.gluster.org/7999 (libgfapi : glfs_create()/glfs_open() should set fd->flgas before syncop_open()) posted (#5) for review on master by Atin Mukherjee (amukherj)
REVIEW: http://review.gluster.org/7999 (libgfapi : glfs_create()/glfs_open() should set fd->flags before syncop_open()) posted (#6) for review on master by Atin Mukherjee (amukherj)
REVIEW: http://review.gluster.org/7872 (dht: pass xdata to xlators above.) posted (#10) for review on master by Atin Mukherjee (amukherj)
REVIEW: http://review.gluster.org/7999 (libgfapi : glfs_create()/glfs_open() should set fd->flags before syncop_open()) posted (#7) for review on master by Atin Mukherjee (amukherj)
REVIEW: http://review.gluster.org/7872 (dht: pass xdata to xlators above) posted (#11) for review on master by Atin Mukherjee (amukherj)
REVIEW: http://review.gluster.org/7873 (barrier: enable FUSE direct-io-mode for barriered fds) posted (#9) for review on master by Atin Mukherjee (amukherj)
REVIEW: http://review.gluster.org/7999 (libgfapi : glfs_create()/glfs_open() should set fd->flags before syncop_open()) posted (#8) for review on master by Atin Mukherjee (amukherj)
REVIEW: http://review.gluster.org/7872 (dht: pass xdata to xlators above.) posted (#12) for review on master by Atin Mukherjee (amukherj)
REVIEW: http://review.gluster.org/7872 (dht: pass xdata to xlators above) posted (#13) for review on master by Atin Mukherjee (amukherj)
REVIEW: http://review.gluster.org/7873 (barrier: enable FUSE direct-io-mode for barriered fds) posted (#10) for review on master by Atin Mukherjee (amukherj)
REVIEW: http://review.gluster.org/7999 (libgfapi : glfs_create()/glfs_open() should set fd->flags before syncop_open()) posted (#9) for review on master by Atin Mukherjee (amukherj)
REVIEW: http://review.gluster.org/7872 (dht: pass xdata to xlators above.) posted (#14) for review on master by Atin Mukherjee (amukherj)
REVIEW: http://review.gluster.org/7999 (libgfapi : glfs_create()/glfs_h_creat()/glfs_open()/glfs_h_open() should set fd->flags before syncop_open()) posted (#10) for review on master by Atin Mukherjee (amukherj)
REVIEW: http://review.gluster.org/7873 (barrier: enable FUSE direct-io-mode for barriered fds) posted (#11) for review on master by Atin Mukherjee (amukherj)
REVIEW: http://review.gluster.org/7873 (barrier: enable FUSE direct-io-mode for barriered fds) posted (#12) for review on master by Atin Mukherjee (amukherj)
REVIEW: http://review.gluster.org/7873 (barrier: enable FUSE direct-io-mode for barriered fds) posted (#13) for review on master by Atin Mukherjee (amukherj)
COMMIT: http://review.gluster.org/7872 committed in master by Vijay Bellur (vbellur) ------ commit 187a7a926b6b1b57060046a030a5d9a44000fc4b Author: Krishnan Parthasarathi <kparthas> Date: Mon May 26 17:18:17 2014 +0530 dht: pass xdata to xlators above. Change-Id: I96e9feb88443fcd7da40c33c0e8c4e2645b1fcf3 BUG: 1096047 Signed-off-by: Krishnan Parthasarathi <kparthas> Reviewed-on: http://review.gluster.org/7872 Reviewed-by: Shyamsundar Ranganathan <srangana> Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Vijay Bellur <vbellur>
REVIEW: http://review.gluster.org/7999 (libgfapi : glfs_create()/glfs_h_creat()/glfs_open()/glfs_h_open() should set fd->flags before syncop_open()) posted (#11) for review on master by Atin Mukherjee (amukherj)
COMMIT: http://review.gluster.org/7999 committed in master by Anand Avati (avati) ------ commit 67a6f402d4a4af2e7d9b8d80b25c94f1a08ef029 Author: Atin Mukherjee <amukherj> Date: Fri Jun 6 12:51:57 2014 +0530 libgfapi : glfs_create()/glfs_h_creat()/glfs_open()/glfs_h_open() should set fd->flags before syncop_open() glfs_create() and glfs_open() do not set fd->flags before calling syncop_open(). This patch addresses this problem and ensure the flags are set in fd before invoking syncop_open() Change-Id: I9ef3243b1de610e1dd1a3e37b66fc2f763a865f9 BUG: 1096047 Signed-off-by: Atin Mukherjee <amukherj> Reviewed-on: http://review.gluster.org/7999 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Krishnan Parthasarathi <kparthas> Reviewed-by: Humble Devassy Chirammal <humble.devassy> Reviewed-by: Anand Avati <avati>
A beta release for GlusterFS 3.6.0 has been released. Please verify if the release solves this bug report for you. In case the glusterfs-3.6.0beta1 release does not have a resolution for this issue, leave a comment in this bug and move the status to ASSIGNED. If this release fixes the problem for you, leave a note and change the status to VERIFIED. Packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update (possibly an "updates-testing" repository) infrastructure for your distribution. [1] http://supercolony.gluster.org/pipermail/gluster-users/2014-September/018836.html [2] http://supercolony.gluster.org/pipermail/gluster-users/
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.6.1, please reopen this bug report. glusterfs-3.6.1 has been announced [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://supercolony.gluster.org/pipermail/gluster-users/2014-November/019410.html [2] http://supercolony.gluster.org/mailman/listinfo/gluster-users