Red Hat Bugzilla – Bug 913662
Need to increase throughput of mgmt operations using synctask framework
Last modified: 2015-11-03 18:05:03 EST
Description of problem:
Volume operations using synctask framework issue management rpc ops in a serial manner. This approach will not scale with increasing no. of peers in the cluster.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
CHANGE: http://review.gluster.org/4558 (synctask: support for (assymetric) counted barriers) merged in master by Anand Avati (firstname.lastname@example.org)
CHANGE: http://review.gluster.org/4580 (volgen: Use bind-address option for bricks when option set on glusterd) merged in master by Vijay Bellur (email@example.com)
CHANGE: http://review.gluster.org/4570 (glusterd: Increasing throughput of synctask based mgmt ops.) merged in master by Vijay Bellur (firstname.lastname@example.org)
I am seeing volume commands fail occasionally on my 6 node 3x2 setups. Here is what I am seeing:
# gluster volume heal healtest info split-brain
# echo $?
This happened at just about 20:32:07. In the etc-glusterfs-glusterd.vol.log I see:
[2013-02-27 20:32:08.081243] E [glusterd-utils.c:278:glusterd_lock] 0-glusterd: Unable to get lock for uuid: c3538fdf-c16e-4eaf-9650-6cf0caf7478b, lock held by: 647b8ff3-8110-4eab-956b-df78235fb192
[2013-02-27 20:32:08.081254] E [glusterd-handler.c:470:glusterd_op_txn_begin] 0-management: Unable to acquire local lock, ret: -1
Is this a symptom of what this bug was opened on or is this a different issue?
This bug was opened to track the code changes for how (internal rpc) requests were sent to peers in cluster, during a volume operation.
The above observation is not a symptom of this bug. While you open a new bug to track what you observe, please provide the output of "gluster system:: fsm log". Run the command on the machine whose uuid is (still) holding the lock.
CHANGE: http://review.gluster.org/4636 (synctask: support for (assymetric) counted barriers) merged in release-3.4 by Anand Avati (email@example.com)
CHANGE: http://review.gluster.org/4637 (volgen: Use bind-address option for bricks when option set on glusterd) merged in release-3.4 by Anand Avati (firstname.lastname@example.org)