+++ This bug was initially created as a clone of Bug #1161903 +++ Description of problem: A disperse volume, Different client can not "ls -al " in a same directory at the sametime In client-1 mountpoint , exec cmd "for((i=0;i<1000;i++));do ls -al;done",In the other client-2 mountpoint's smae directory, cmd "for((i=0;i<1000;i++));do ls -al;done" or "touch newfile" or "mkdir newdirectory" is blocked before client-1's cmd(1000 ls -al loops) is over. [root@localhost test]# gluster volume info Volume Name: test Type: Distributed-Disperse Volume ID: 433248ee-24f5-44e3-b334-488743850e45 Status: Started Number of Bricks: 2 x (2 + 1) = 6 Transport-type: tcp Bricks: Brick1: 10.10.101.111:/sda Brick2: 10.10.101.111:/sdb Brick3: 10.10.101.111:/sdc Brick4: 10.10.101.111:/sdd Brick5: 10.10.101.111:/sde Brick6: 10.10.101.111:/sdf Version-Release number of selected component (if applicable): 3.6.0 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: In the other client's same directory, ls, touch, mkdir is blocked Expected results: In the other client's same directory, ls, touch, mkdir should be ok or be blocked a short time. Additional info: --- Additional comment from Niels de Vos on 2014-11-11 13:52:26 CET --- Have you tried this also on other types of volumes? Is this only affecting a disperse volume? --- Additional comment from jiademing on 2014-11-12 02:36:07 CET --- Yes, only affecting a disperse volume, I tried to turn off the gf_timer_call_after() wthen ec_unlock in ec_common.c's ec_unlock_timer_add(), then can execute "for((i=0;i<1000;i++));do ls -al;done" in different client at the same time. In my opinion, the af_timer_call_after in ec_unlock is optimize for one client, but maybe it is bad for many clients. (In reply to Niels de Vos from comment #1) > Have you tried this also on other types of volumes? Is this only affecting a > disperse volume? --- Additional comment from Xavier Hernandez on 2014-11-12 18:23:25 CET --- Yes, this is a method to minimize lock/unlock calls. I'll try to find a good solution to minimize the multiple client problem. --- Additional comment from jiademing on 2014-11-17 02:30:07 CET --- yes, I will pay close attention to this problem, thanks. (In reply to Xavier Hernandez from comment #3) > Yes, this is a method to minimize lock/unlock calls. I'll try to find a good > solution to minimize the multiple client problem.
REVIEW: http://review.gluster.org/10845 (cluster/ec: Forced unlock when lock contention is detected) posted (#1) for review on master by Xavier Hernandez (xhernandez)
The previous patch should be half of the solution. Another patch will be sent to add wider support in the locks xlator.
REVIEW: http://review.gluster.org/10845 (cluster/ec: Forced unlock when lock contention is detected) posted (#2) for review on master by Pranith Kumar Karampuri (pkarampu)
REVIEW: http://review.gluster.org/10880 (features/locks: Handle virtual getxattrs in more fops) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu)
REVIEW: http://review.gluster.org/10845 (cluster/ec: Forced unlock when lock contention is detected) posted (#3) for review on master by Xavier Hernandez (xhernandez)
REVIEW: http://review.gluster.org/10845 (cluster/ec: Forced unlock when lock contention is detected) posted (#4) for review on master by Xavier Hernandez (xhernandez)
REVIEW: http://review.gluster.org/10845 (cluster/ec: Forced unlock when lock contention is detected) posted (#5) for review on master by Xavier Hernandez (xhernandez)
REVIEW: http://review.gluster.org/10845 (cluster/ec: Forced unlock when lock contention is detected) posted (#6) for review on master by Pranith Kumar Karampuri (pkarampu)
REVIEW: http://review.gluster.org/10880 (features/locks: Handle virtual getxattrs in more fops) posted (#2) for review on master by Pranith Kumar Karampuri (pkarampu)
REVIEW: http://review.gluster.org/10845 (cluster/ec: Forced unlock when lock contention is detected) posted (#7) for review on master by Pranith Kumar Karampuri (pkarampu)
REVIEW: http://review.gluster.org/10845 (cluster/ec: Forced unlock when lock contention is detected) posted (#8) for review on master by Pranith Kumar Karampuri (pkarampu)
REVIEW: http://review.gluster.org/10852 (cluster/ec: Forced unlock when lock contention is detected) posted (#4) for review on master by Xavier Hernandez (xhernandez)
REVIEW: http://review.gluster.org/10852 (cluster/ec: Forced unlock when lock contention is detected) posted (#5) for review on master by Xavier Hernandez (xhernandez)
REVIEW: http://review.gluster.org/10852 (cluster/ec: Forced unlock when lock contention is detected) posted (#6) for review on master by Xavier Hernandez (xhernandez)
REVIEW: http://review.gluster.org/10852 (cluster/ec: Forced unlock when lock contention is detected) posted (#7) for review on master by Xavier Hernandez (xhernandez)
REVIEW: http://review.gluster.org/10852 (cluster/ec: Forced unlock when lock contention is detected) posted (#8) for review on master by Pranith Kumar Karampuri (pkarampu)
REVIEW: http://review.gluster.org/10852 (cluster/ec: Forced unlock when lock contention is detected) posted (#9) for review on master by Xavier Hernandez (xhernandez)
REVIEW: http://review.gluster.org/10852 (cluster/ec: Forced unlock when lock contention is detected) posted (#10) for review on master by Pranith Kumar Karampuri (pkarampu)
REVIEW: http://review.gluster.org/10930 (cluster/ec: Remove ec tests from bad tests) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu)
COMMIT: http://review.gluster.org/10880 committed in master by Pranith Kumar Karampuri (pkarampu) ------ commit d37cb5aee7af8582d0343d2732c153226955945d Author: Pranith Kumar K <pkarampu> Date: Tue May 19 20:53:30 2015 +0530 features/locks: Handle virtual getxattrs in more fops With this patch getxattr of inodelk/entrylk counts can be requested in readv/writev/create/unlink/opendir. Change-Id: If7430317ad478a3c753eb33bdf89046cb001a904 BUG: 1165041 Signed-off-by: Pranith Kumar K <pkarampu> Reviewed-on: http://review.gluster.org/10880 Tested-by: NetBSD Build System Reviewed-by: Krutika Dhananjay <kdhananj>
COMMIT: http://review.gluster.org/10852 committed in master by Pranith Kumar Karampuri (pkarampu) ------ commit 3b666b40efbed157e8c5991f29b345d93b28c659 Author: Xavier Hernandez <xhernandez> Date: Wed May 20 15:17:35 2015 +0200 cluster/ec: Forced unlock when lock contention is detected EC uses an eager lock mechanism to optimize multiple read/write requests on the same entry or inode. This increases performance but can have adverse results when other clients try to access the same entry/inode. To solve this, this patch adds a functionality to detect when this happens and force an earlier release to not block other clients. The method consists on requesting GF_GLUSTERFS_INODELK_COUNT and GF_GLUSTERFS_ENTRYLK_COUNT for all fops that take a lock. When this count is greater than one, the lock is marked to be released. All fops already waiting for this lock will be executed normally before releasing the lock, but new requests that also require it will be blocked and restarted after the lock has been released and reacquired again. Another problem was that some operations did correctly lock the parent of an entry when needed, but got the size and version xattrs from the entry instead of the parent. This patch solves this problem by binding all queries of size and version to each lock and replacing all entrylk calls by inodelk ones to remove concurrent updates on directory metadata. This also allows rename to correctly update source and destination directories. Change-Id: I2df0b22bc6f407d49f3cbf0733b0720015bacfbd BUG: 1165041 Signed-off-by: Xavier Hernandez <xhernandez> Reviewed-on: http://review.gluster.org/10852 Tested-by: NetBSD Build System Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Pranith Kumar Karampuri <pkarampu>
REVIEW: http://review.gluster.org/10930 (cluster/ec: Remove ec tests from bad tests) posted (#2) for review on master by Pranith Kumar Karampuri (pkarampu)
REVIEW: http://review.gluster.org/10930 (tests: Remove tests from bad tests) posted (#3) for review on master by Ravishankar N (ravishankar)
REVIEW: http://review.gluster.org/10930 (tests: Remove tests from bad tests) posted (#4) for review on master by Ravishankar N (ravishankar)
REVIEW: http://review.gluster.org/10930 (tests: Remove tests from bad tests) posted (#5) for review on master by Vijay Bellur (vbellur)
Fix for this BZ is already present in a GlusterFS release. You can find clone of this BZ, fixed in a GlusterFS release and closed. Hence closing this mainline BZ as well.
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report. glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/ [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user