Bug 1412907

Summary: [RHV-RHGS]: Volume status, remove-brick commands stopped working because of some lock issue
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Byreddy <bsrirama>
Component: glusterdAssignee: Atin Mukherjee <amukherj>
Status: CLOSED DUPLICATE QA Contact: Byreddy <bsrirama>
Severity: high Docs Contact:
Priority: unspecified    
Version: rhgs-3.2CC: rhinduja, rhs-bugs, sasundar, storage-qa-internal, vbellur
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-02-24 06:39:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Byreddy 2017-01-13 05:51:30 UTC 
Description of problem:
=======================
Volume status, remove-brick commands stopped working because of some lock issue.


Version-Release number of selected component (if applicable):
=============================================================
glusterfs-3.8.4-11


How reproducible:
=================
One time ( i don't think it will reproduce every time, but chances are there in production env hitting and looks like this issue happened because of parallel commands,  parallel commands may come from RHEV UI and from admin )


Steps to Reproduce:
====================
1.Have RHV-RHGS SETUP with 3 rhgs nodes and 2 clients (hosts)
2.create a 2 *3 volume 
3.create some Application VMs using the vm storage created in step-2.
4.Start running some IO inside the App VMs.
5. When IO is happening, do remove-brick -> add-brick operations ( keep doing )to check rebalance is not interrupting the App VMs running.

When i triggered the remove-brick from RHEV UI, command didn't responded.


Actual results:
===============
Volume status, remove-brick commands stopped working because of some lock issue


Expected results:
=================
if parallel commands are received from diff interfaces, we should handle properly with out any lock issue getting.


Additional info:
Comment 2 Byreddy 2017-01-13 05:53:07 UTC 
glusterd log:
=------------=
[2017-01-12 11:39:56.653410] I [MSGID: 106499] [glusterd-handler.c:4362:__glusterd_handle_status_volume] 0-management: Received status volume req for volume Dis-Rep1
[2017-01-12 11:39:56.971582] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/xl
ator/mgmt/glusterd.so(+0x2e155) [0x7fa9ec493155] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4c5f) [0x7fa9ec539c5f] ) 0-management: Lock for Dis-Rep1 held by 6799e2bf-ca97-42fa
-95c7-6c58d435275c
[2017-01-12 11:39:56.971655] E [MSGID: 106119] [glusterd-op-sm.c:3908:glusterd_op_ac_lock] 0-management: Unable to acquire lock for Dis-Rep1
[2017-01-12 11:39:56.971905] E [MSGID: 106376] [glusterd-op-sm.c:7909:glusterd_op_sm] 0-management: handler returned: -1
[2017-01-12 11:39:56.972288] E [MSGID: 106275] [glusterd-rpc-ops.c:878:glusterd_mgmt_v3_lock_peers_cbk_fn] 0-management: Received mgmt_v3 lock RJT from uuid: 6109621f-e6c8-46ed-81f5-e28aaacd
e138
[2017-01-12 11:39:56.974541] E [MSGID: 106275] [glusterd-rpc-ops.c:878:glusterd_mgmt_v3_lock_peers_cbk_fn] 0-management: Received mgmt_v3 lock RJT from uuid: 8cd39ea1-cf36-4dda-aab2-5117f061
0387
[2017-01-12 11:39:56.978401] W [glusterd-locks.c:686:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/
xlator/mgmt/glusterd.so(+0x2de42) [0x7fa9ec492e42] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4ffa) [0x7fa9ec539ffa] ) 0-management: Lock owner mismatch. Lock for vol Dis-Rep1
 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c
[2017-01-12 11:39:56.978458] E [MSGID: 106118] [glusterd-op-sm.c:3971:glusterd_op_ac_unlock] 0-management: Unable to release lock for Dis-Rep1
[2017-01-12 11:39:58.586776] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/xl
ator/mgmt/glusterd.so(+0x2e155) [0x7fa9ec493155] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4c5f) [0x7fa9ec539c5f] ) 0-management: Lock for Dis-Rep1 held by 6799e2bf-ca97-42fa
-95c7-6c58d435275c
[2017-01-12 11:39:58.591038] W [glusterd-locks.c:686:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/
xlator/mgmt/glusterd.so(+0x2de42) [0x7fa9ec492e42] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4ffa) [0x7fa9ec539ffa] ) 0-management: Lock owner mismatch. Lock for vol Dis-Rep1
 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c
[2017-01-12 11:39:58.591105] E [MSGID: 106118] [glusterd-op-sm.c:3971:glusterd_op_ac_unlock] 0-management: Unable to release lock for Dis-Rep1
[2017-01-12 11:39:58.591276] E [MSGID: 106376] [glusterd-op-sm.c:7909:glusterd_op_sm] 0-management: handler returned: 1
[2017-01-12 11:40:56.828857] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/xl
ator/mgmt/glusterd.so(+0x2e155) [0x7fa9ec493155] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4c5f) [0x7fa9ec539c5f] ) 0-management: Lock for Dis-Rep1 held by 6799e2bf-ca97-42fa
-95c7-6c58d435275c
[2017-01-12 11:40:56.831310] W [glusterd-locks.c:686:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2de42) [0x7fa9ec492e42] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4ffa) [0x7fa9ec539ffa] ) 0-management: Lock owner mismatch. Lock for vol Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c
The message "E [MSGID: 106376] [glusterd-op-sm.c:7909:glusterd_op_sm] 0-management: handler returned: -1" repeated 3 times between [2017-01-12 11:39:56.971905] and [2017-01-12 11:40:56.829280]
[2017-01-12 11:40:56.831391] E [MSGID: 106118] [glusterd-op-sm.c:3971:glusterd_op_ac_unlock] 0-management: Unable to release lock for Dis-Rep1
[2017-01-12 11:41:18.242282] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2e155) [0x7fa9ec493155] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4c5f) [0x7fa9ec539c5f] ) 0-management: Lock for Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c
The message "E [MSGID: 106119] [glusterd-op-sm.c:3908:glusterd_op_ac_lock] 0-management: Unable to acquire lock for Dis-Rep1" repeated 3 times between [2017-01-12 11:39:56.971655] and [2017-01-12 11:41:18.242353]
[2017-01-12 11:41:18.242619] E [MSGID: 106376] [glusterd-op-sm.c:7909:glusterd_op_sm] 0-management: handler returned: -1
[2017-01-12 11:41:18.244615] W [glusterd-locks.c:686:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2de42) [0x7fa9ec492e42] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4ffa) [0x7fa9ec539ffa] ) 0-management: Lock owner mismatch. Lock for vol Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c
[2017-01-12 11:41:44.853829] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xcfa20) [0x7fa9ec534a20] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xcf950) [0x7fa9ec534950] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4c5f) [0x7fa9ec539c5f] ) 0-management: Lock for Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c
[2017-01-12 11:41:44.853341] I [MSGID: 106499] [glusterd-handler.c:4362:__glusterd_handle_status_volume] 0-management: Received status volume req for volume Dis-Rep1
[2017-01-12 11:41:44.853882] E [MSGID: 106119] [glusterd-syncop.c:1851:gd_sync_task_begin] 0-management: Unable to acquire lock for Dis-Rep1
held by 6799e2bf-ca97-42fa-95c7-6c58d435275c
[2017-01-12 11:44:58.020745] I [MSGID: 106499] [glusterd-handler.c:4362:__glusterd_handle_status_volume] 0-management: Received status volume req for volume Dis-Rep1
[2017-01-12 11:44:58.021980] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xcfa20) [0x7fa9ec534a20] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xcf950) [0x7fa9ec534950] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4c5f) [0x7fa9ec539c5f] ) 0-management: Lock for Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c
[2017-01-12 11:44:58.022020] E [MSGID: 106119] [glusterd-syncop.c:1851:gd_sync_task_begin] 0-management: Unable to acquire lock for Dis-Rep1
[2017-01-12 11:44:59.751963] I [MSGID: 106499] [glusterd-handler.c:4362:__glusterd_handle_status_volume] 0-management: Received status volume req for volume Dis-Rep1
[2017-01-12 11:44:59.752522] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xcfa20) [0x7fa9ec534a20] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xcf950) [0x7fa9ec534950] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4c5f) [0x7fa9ec539c5f] ) 0-management: Lock for Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c
[2017-01-12 11:44:59.752554] E [MSGID: 106119] [glusterd-syncop.c:1851:gd_sync_task_begin] 0-management: Unable to acquire lock for Dis-Rep1
[2017-01-12 11:45:11.748739] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2e155) [0x7fa9ec493155] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4c5f) [0x7fa9ec539c5f] ) 0-management: Lock for Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c
[2017-01-12 11:45:11.751801] W [glusterd-locks.c:686:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2de42) [0x7fa9ec492e42] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4ffa) [0x7fa9ec539ffa] ) 0-management: Lock owner mismatch. Lock for vol Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c
[2017-01-12 11:43:58.584700] E [MSGID: 106376] [glusterd-op-sm.c:7909:glusterd_op_sm] 0-management: handler returned: 1
The message "E [MSGID: 106119] [glusterd-op-sm.c:3908:glusterd_op_ac_lock] 0-management: Unable to acquire lock for Dis-Rep1" repeated 2 times between [2017-01-12 11:43:56.009493] and [2017-01-12 11:45:11.748844]
The message "E [MSGID: 106118] [glusterd-op-sm.c:3971:glusterd_op_ac_unlock] 0-management: Unable to release lock for Dis-Rep1" repeated 2 times between [2017-01-12 11:43:56.011189] and [2017-01-12 11:45:11.751851]
The message "E [MSGID: 106376] [glusterd-op-sm.c:7909:glusterd_op_sm] 0-management: handler returned: -1" repeated 3 times between [2017-01-12 11:43:56.009666] and [2017-01-12 11:45:11.752042]
[2017-01-12 11:46:00.933853] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2e155) [0x7fa9ec493155] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4c5f) [0x7fa9ec539c5f] ) 0-management: Lock for Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c
[2017-01-12 11:46:00.933978] E [MSGID: 106119] [glusterd-op-sm.c:3908:glusterd_op_ac_lock] 0-management: Unable to acquire lock for Dis-Rep1
[2017-01-12 11:46:00.934150] E [MSGID: 106376] [glusterd-op-sm.c:7909:glusterd_op_sm] 0-management: handler returned: -1
[2017-01-12 11:46:00.936730] W [glusterd-locks.c:686:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2de42) [0x7fa9ec492e42] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4ffa) [0x7fa9ec539ffa] ) 0-management: Lock owner mismatch. Lock for vol Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c
[2017-01-12 11:46:00.936814] E [MSGID: 106118] [glusterd-op-sm.c:3971:glusterd_op_ac_unlock] 0-management: Unable to release lock for Dis-Rep1
[2017-01-12 11:46:00.937024] E [MSGID: 106376] [glusterd-op-sm.c:7909:glusterd_op_sm] 0-management: handler returned: 1
[2017-01-12 11:47:01.360797] I [MSGID: 106499] [glusterd-handler.c:4362:__glusterd_handle_status_volume] 0-management: Received status volume req for volume Dis-Rep1
[2017-01-12 11:47:01.361395] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xcfa20) [0x7fa9ec534a20] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xcf950) [0x7fa9ec534950] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4c5f) [0x7fa9ec539c5f] ) 0-management: Lock for Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c
[2017-01-12 11:47:01.361437] E [MSGID: 106119] [glusterd-syncop.c:1851:gd_sync_task_begin] 0-management: Unable to acquire lock for Dis-Rep1
[2017-01-12 11:48:01.731810] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2e155) [0x7fa9ec493155] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4c5f) [0x7fa9ec539c5f] ) 0-management: Lock for Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c
[2017-01-12 11:48:01.731901] E [MSGID: 106119] [glusterd-op-sm.c:3908:glusterd_op_ac_lock] 0-management: Unable to acquire lock for Dis-Rep1
[2017-01-12 11:48:01.732153] E [MSGID: 106376] [glusterd-op-sm.c:7909:glusterd_op_sm] 0-management: handler returned: -1
[2017-01-12 11:48:01.734751] W [glusterd-locks.c:686:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2de42) [0x7fa9ec492e42] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4ffa) [0x7fa9ec539ffa] ) 0-management: Lock owner mismatch. Lock for vol Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c
[2017-01-12 11:48:01.734835] E [MSGID: 106118] [glusterd-op-sm.c:3971:glusterd_op_ac_unlock] 0-management: Unable to release lock for Dis-Rep1
[2017-01-12 11:48:01.735036] E [MSGID: 106376] [glusterd-op-sm.c:7909:glusterd_op_sm] 0-management: handler returned: 1
(END)
Comment 10 Atin Mukherjee 2017-02-24 06:39:51 UTC 
See https://bugzilla.redhat.com/show_bug.cgi?id=1425681#c4 . I am closing this bug as duplicate of BZ 1425681

*** This bug has been marked as a duplicate of bug 1425681 ***