Bug 913662

Summary: Need to increase throughput of mgmt operations using synctask framework
Product: [Community] GlusterFS Reporter: krishnan parthasarathi <kparthas>
Component: glusterdAssignee: krishnan parthasarathi <kparthas>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: mainlineCC: bturner, gluster-bugs, jdarcy, nsathyan, vbellur
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: glusterfs-3.4.0 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-07-24 13:18:18 EDT Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Bug Depends On:    
Bug Blocks: 895528    

Description krishnan parthasarathi 2013-02-21 13:21:14 EST
Description of problem:
Volume operations using synctask framework issue management rpc ops in a serial manner. This approach will not scale with increasing no. of peers in the cluster.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
Actual results:

Expected results:

Additional info:
Comment 1 Vijay Bellur 2013-02-22 01:57:19 EST
CHANGE: http://review.gluster.org/4558 (synctask: support for (assymetric) counted barriers) merged in master by Anand Avati (avati@redhat.com)
Comment 2 Vijay Bellur 2013-02-26 12:04:49 EST
CHANGE: http://review.gluster.org/4580 (volgen: Use bind-address option for bricks when option set on glusterd) merged in master by Vijay Bellur (vbellur@redhat.com)
Comment 3 Vijay Bellur 2013-02-26 12:06:27 EST
CHANGE: http://review.gluster.org/4570 (glusterd: Increasing throughput of synctask based mgmt ops.) merged in master by Vijay Bellur (vbellur@redhat.com)
Comment 4 Ben Turner 2013-02-28 16:06:46 EST
I am seeing volume commands fail occasionally on my 6 node 3x2 setups.  Here is what I am seeing:

# gluster volume heal healtest info split-brain
operation failed
# echo $?

This happened at just about 20:32:07.  In the etc-glusterfs-glusterd.vol.log I see:

[2013-02-27 20:32:08.081243] E [glusterd-utils.c:278:glusterd_lock] 0-glusterd: Unable to get lock for uuid: c3538fdf-c16e-4eaf-9650-6cf0caf7478b, lock held by: 647b8ff3-8110-4eab-956b-df78235fb192
[2013-02-27 20:32:08.081254] E [glusterd-handler.c:470:glusterd_op_txn_begin] 0-management: Unable to acquire local lock, ret: -1

Is this a symptom of what this bug was opened on or is this a different issue?
Comment 5 krishnan parthasarathi 2013-02-28 23:45:16 EST
This bug was opened to track the code changes for how (internal rpc) requests were sent to peers in cluster, during a volume operation.
The above observation is not a symptom of this bug. While you open a new bug to track what you observe, please provide the output of "gluster system:: fsm log". Run the command on the machine whose uuid is (still) holding the lock.
Comment 6 Vijay Bellur 2013-03-07 00:32:54 EST
CHANGE: http://review.gluster.org/4636 (synctask: support for (assymetric) counted barriers) merged in release-3.4 by Anand Avati (avati@redhat.com)
Comment 7 Vijay Bellur 2013-03-07 01:05:54 EST
CHANGE: http://review.gluster.org/4637 (volgen: Use bind-address option for bricks when option set on glusterd) merged in release-3.4 by Anand Avati (avati@redhat.com)