Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be unavailable on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1412907 - [RHV-RHGS]: Volume status, remove-brick commands stopped working because of some lock issue
Summary: [RHV-RHGS]: Volume status, remove-brick commands stopped working because of s...
Keywords:
Status: CLOSED DUPLICATE of bug 1425681
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterd
Version: rhgs-3.2
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: ---
Assignee: Atin Mukherjee
QA Contact: Byreddy
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-13 05:51 UTC by Byreddy
Modified: 2017-02-24 06:39 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-02-24 06:39:51 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Byreddy 2017-01-13 05:51:30 UTC
Description of problem:
=======================
Volume status, remove-brick commands stopped working because of some lock issue.


Version-Release number of selected component (if applicable):
=============================================================
glusterfs-3.8.4-11


How reproducible:
=================
One time ( i don't think it will reproduce every time, but chances are there in production env hitting and looks like this issue happened because of parallel commands,  parallel commands may come from RHEV UI and from admin )


Steps to Reproduce:
====================
1.Have RHV-RHGS SETUP with 3 rhgs nodes and 2 clients (hosts)
2.create a 2 *3 volume 
3.create some Application VMs using the vm storage created in step-2.
4.Start running some IO inside the App VMs.
5. When IO is happening, do remove-brick -> add-brick operations ( keep doing )to check rebalance is not interrupting the App VMs running.

When i triggered the remove-brick from RHEV UI, command didn't responded.


Actual results:
===============
Volume status, remove-brick commands stopped working because of some lock issue


Expected results:
=================
if parallel commands are received from diff interfaces, we should handle properly with out any lock issue getting.


Additional info:

Comment 2 Byreddy 2017-01-13 05:53:07 UTC
glusterd log:
=------------=
[2017-01-12 11:39:56.653410] I [MSGID: 106499] [glusterd-handler.c:4362:__glusterd_handle_status_volume] 0-management: Received status volume req for volume Dis-Rep1
[2017-01-12 11:39:56.971582] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/xl
ator/mgmt/glusterd.so(+0x2e155) [0x7fa9ec493155] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4c5f) [0x7fa9ec539c5f] ) 0-management: Lock for Dis-Rep1 held by 6799e2bf-ca97-42fa
-95c7-6c58d435275c
[2017-01-12 11:39:56.971655] E [MSGID: 106119] [glusterd-op-sm.c:3908:glusterd_op_ac_lock] 0-management: Unable to acquire lock for Dis-Rep1
[2017-01-12 11:39:56.971905] E [MSGID: 106376] [glusterd-op-sm.c:7909:glusterd_op_sm] 0-management: handler returned: -1
[2017-01-12 11:39:56.972288] E [MSGID: 106275] [glusterd-rpc-ops.c:878:glusterd_mgmt_v3_lock_peers_cbk_fn] 0-management: Received mgmt_v3 lock RJT from uuid: 6109621f-e6c8-46ed-81f5-e28aaacd
e138
[2017-01-12 11:39:56.974541] E [MSGID: 106275] [glusterd-rpc-ops.c:878:glusterd_mgmt_v3_lock_peers_cbk_fn] 0-management: Received mgmt_v3 lock RJT from uuid: 8cd39ea1-cf36-4dda-aab2-5117f061
0387
[2017-01-12 11:39:56.978401] W [glusterd-locks.c:686:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/
xlator/mgmt/glusterd.so(+0x2de42) [0x7fa9ec492e42] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4ffa) [0x7fa9ec539ffa] ) 0-management: Lock owner mismatch. Lock for vol Dis-Rep1
 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c
[2017-01-12 11:39:56.978458] E [MSGID: 106118] [glusterd-op-sm.c:3971:glusterd_op_ac_unlock] 0-management: Unable to release lock for Dis-Rep1
[2017-01-12 11:39:58.586776] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/xl
ator/mgmt/glusterd.so(+0x2e155) [0x7fa9ec493155] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4c5f) [0x7fa9ec539c5f] ) 0-management: Lock for Dis-Rep1 held by 6799e2bf-ca97-42fa
-95c7-6c58d435275c
[2017-01-12 11:39:58.591038] W [glusterd-locks.c:686:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/
xlator/mgmt/glusterd.so(+0x2de42) [0x7fa9ec492e42] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4ffa) [0x7fa9ec539ffa] ) 0-management: Lock owner mismatch. Lock for vol Dis-Rep1
 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c
[2017-01-12 11:39:58.591105] E [MSGID: 106118] [glusterd-op-sm.c:3971:glusterd_op_ac_unlock] 0-management: Unable to release lock for Dis-Rep1
[2017-01-12 11:39:58.591276] E [MSGID: 106376] [glusterd-op-sm.c:7909:glusterd_op_sm] 0-management: handler returned: 1
[2017-01-12 11:40:56.828857] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/xl
ator/mgmt/glusterd.so(+0x2e155) [0x7fa9ec493155] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4c5f) [0x7fa9ec539c5f] ) 0-management: Lock for Dis-Rep1 held by 6799e2bf-ca97-42fa
-95c7-6c58d435275c
[2017-01-12 11:40:56.831310] W [glusterd-locks.c:686:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2de42) [0x7fa9ec492e42] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4ffa) [0x7fa9ec539ffa] ) 0-management: Lock owner mismatch. Lock for vol Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c
The message "E [MSGID: 106376] [glusterd-op-sm.c:7909:glusterd_op_sm] 0-management: handler returned: -1" repeated 3 times between [2017-01-12 11:39:56.971905] and [2017-01-12 11:40:56.829280]
[2017-01-12 11:40:56.831391] E [MSGID: 106118] [glusterd-op-sm.c:3971:glusterd_op_ac_unlock] 0-management: Unable to release lock for Dis-Rep1
[2017-01-12 11:41:18.242282] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2e155) [0x7fa9ec493155] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4c5f) [0x7fa9ec539c5f] ) 0-management: Lock for Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c
The message "E [MSGID: 106119] [glusterd-op-sm.c:3908:glusterd_op_ac_lock] 0-management: Unable to acquire lock for Dis-Rep1" repeated 3 times between [2017-01-12 11:39:56.971655] and [2017-01-12 11:41:18.242353]
[2017-01-12 11:41:18.242619] E [MSGID: 106376] [glusterd-op-sm.c:7909:glusterd_op_sm] 0-management: handler returned: -1
[2017-01-12 11:41:18.244615] W [glusterd-locks.c:686:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2de42) [0x7fa9ec492e42] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4ffa) [0x7fa9ec539ffa] ) 0-management: Lock owner mismatch. Lock for vol Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c
[2017-01-12 11:41:44.853829] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xcfa20) [0x7fa9ec534a20] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xcf950) [0x7fa9ec534950] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4c5f) [0x7fa9ec539c5f] ) 0-management: Lock for Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c
[2017-01-12 11:41:44.853341] I [MSGID: 106499] [glusterd-handler.c:4362:__glusterd_handle_status_volume] 0-management: Received status volume req for volume Dis-Rep1
[2017-01-12 11:41:44.853882] E [MSGID: 106119] [glusterd-syncop.c:1851:gd_sync_task_begin] 0-management: Unable to acquire lock for Dis-Rep1
held by 6799e2bf-ca97-42fa-95c7-6c58d435275c
[2017-01-12 11:44:58.020745] I [MSGID: 106499] [glusterd-handler.c:4362:__glusterd_handle_status_volume] 0-management: Received status volume req for volume Dis-Rep1
[2017-01-12 11:44:58.021980] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xcfa20) [0x7fa9ec534a20] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xcf950) [0x7fa9ec534950] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4c5f) [0x7fa9ec539c5f] ) 0-management: Lock for Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c
[2017-01-12 11:44:58.022020] E [MSGID: 106119] [glusterd-syncop.c:1851:gd_sync_task_begin] 0-management: Unable to acquire lock for Dis-Rep1
[2017-01-12 11:44:59.751963] I [MSGID: 106499] [glusterd-handler.c:4362:__glusterd_handle_status_volume] 0-management: Received status volume req for volume Dis-Rep1
[2017-01-12 11:44:59.752522] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xcfa20) [0x7fa9ec534a20] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xcf950) [0x7fa9ec534950] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4c5f) [0x7fa9ec539c5f] ) 0-management: Lock for Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c
[2017-01-12 11:44:59.752554] E [MSGID: 106119] [glusterd-syncop.c:1851:gd_sync_task_begin] 0-management: Unable to acquire lock for Dis-Rep1
[2017-01-12 11:45:11.748739] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2e155) [0x7fa9ec493155] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4c5f) [0x7fa9ec539c5f] ) 0-management: Lock for Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c
[2017-01-12 11:45:11.751801] W [glusterd-locks.c:686:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2de42) [0x7fa9ec492e42] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4ffa) [0x7fa9ec539ffa] ) 0-management: Lock owner mismatch. Lock for vol Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c
[2017-01-12 11:43:58.584700] E [MSGID: 106376] [glusterd-op-sm.c:7909:glusterd_op_sm] 0-management: handler returned: 1
The message "E [MSGID: 106119] [glusterd-op-sm.c:3908:glusterd_op_ac_lock] 0-management: Unable to acquire lock for Dis-Rep1" repeated 2 times between [2017-01-12 11:43:56.009493] and [2017-01-12 11:45:11.748844]
The message "E [MSGID: 106118] [glusterd-op-sm.c:3971:glusterd_op_ac_unlock] 0-management: Unable to release lock for Dis-Rep1" repeated 2 times between [2017-01-12 11:43:56.011189] and [2017-01-12 11:45:11.751851]
The message "E [MSGID: 106376] [glusterd-op-sm.c:7909:glusterd_op_sm] 0-management: handler returned: -1" repeated 3 times between [2017-01-12 11:43:56.009666] and [2017-01-12 11:45:11.752042]
[2017-01-12 11:46:00.933853] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2e155) [0x7fa9ec493155] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4c5f) [0x7fa9ec539c5f] ) 0-management: Lock for Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c
[2017-01-12 11:46:00.933978] E [MSGID: 106119] [glusterd-op-sm.c:3908:glusterd_op_ac_lock] 0-management: Unable to acquire lock for Dis-Rep1
[2017-01-12 11:46:00.934150] E [MSGID: 106376] [glusterd-op-sm.c:7909:glusterd_op_sm] 0-management: handler returned: -1
[2017-01-12 11:46:00.936730] W [glusterd-locks.c:686:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2de42) [0x7fa9ec492e42] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4ffa) [0x7fa9ec539ffa] ) 0-management: Lock owner mismatch. Lock for vol Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c
[2017-01-12 11:46:00.936814] E [MSGID: 106118] [glusterd-op-sm.c:3971:glusterd_op_ac_unlock] 0-management: Unable to release lock for Dis-Rep1
[2017-01-12 11:46:00.937024] E [MSGID: 106376] [glusterd-op-sm.c:7909:glusterd_op_sm] 0-management: handler returned: 1
[2017-01-12 11:47:01.360797] I [MSGID: 106499] [glusterd-handler.c:4362:__glusterd_handle_status_volume] 0-management: Received status volume req for volume Dis-Rep1
[2017-01-12 11:47:01.361395] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xcfa20) [0x7fa9ec534a20] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xcf950) [0x7fa9ec534950] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4c5f) [0x7fa9ec539c5f] ) 0-management: Lock for Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c
[2017-01-12 11:47:01.361437] E [MSGID: 106119] [glusterd-syncop.c:1851:gd_sync_task_begin] 0-management: Unable to acquire lock for Dis-Rep1
[2017-01-12 11:48:01.731810] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2e155) [0x7fa9ec493155] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4c5f) [0x7fa9ec539c5f] ) 0-management: Lock for Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c
[2017-01-12 11:48:01.731901] E [MSGID: 106119] [glusterd-op-sm.c:3908:glusterd_op_ac_lock] 0-management: Unable to acquire lock for Dis-Rep1
[2017-01-12 11:48:01.732153] E [MSGID: 106376] [glusterd-op-sm.c:7909:glusterd_op_sm] 0-management: handler returned: -1
[2017-01-12 11:48:01.734751] W [glusterd-locks.c:686:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2de42) [0x7fa9ec492e42] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4ffa) [0x7fa9ec539ffa] ) 0-management: Lock owner mismatch. Lock for vol Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c
[2017-01-12 11:48:01.734835] E [MSGID: 106118] [glusterd-op-sm.c:3971:glusterd_op_ac_unlock] 0-management: Unable to release lock for Dis-Rep1
[2017-01-12 11:48:01.735036] E [MSGID: 106376] [glusterd-op-sm.c:7909:glusterd_op_sm] 0-management: handler returned: 1
(END)

Comment 10 Atin Mukherjee 2017-02-24 06:39:51 UTC
See https://bugzilla.redhat.com/show_bug.cgi?id=1425681#c4 . I am closing this bug as duplicate of BZ 1425681

*** This bug has been marked as a duplicate of bug 1425681 ***


Note You need to log in before you can comment on or make changes to this bug.