Description of problem: ======================= Volume status, remove-brick commands stopped working because of some lock issue. Version-Release number of selected component (if applicable): ============================================================= glusterfs-3.8.4-11 How reproducible: ================= One time ( i don't think it will reproduce every time, but chances are there in production env hitting and looks like this issue happened because of parallel commands, parallel commands may come from RHEV UI and from admin ) Steps to Reproduce: ==================== 1.Have RHV-RHGS SETUP with 3 rhgs nodes and 2 clients (hosts) 2.create a 2 *3 volume 3.create some Application VMs using the vm storage created in step-2. 4.Start running some IO inside the App VMs. 5. When IO is happening, do remove-brick -> add-brick operations ( keep doing )to check rebalance is not interrupting the App VMs running. When i triggered the remove-brick from RHEV UI, command didn't responded. Actual results: =============== Volume status, remove-brick commands stopped working because of some lock issue Expected results: ================= if parallel commands are received from diff interfaces, we should handle properly with out any lock issue getting. Additional info:
glusterd log: =------------= [2017-01-12 11:39:56.653410] I [MSGID: 106499] [glusterd-handler.c:4362:__glusterd_handle_status_volume] 0-management: Received status volume req for volume Dis-Rep1 [2017-01-12 11:39:56.971582] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/xl ator/mgmt/glusterd.so(+0x2e155) [0x7fa9ec493155] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4c5f) [0x7fa9ec539c5f] ) 0-management: Lock for Dis-Rep1 held by 6799e2bf-ca97-42fa -95c7-6c58d435275c [2017-01-12 11:39:56.971655] E [MSGID: 106119] [glusterd-op-sm.c:3908:glusterd_op_ac_lock] 0-management: Unable to acquire lock for Dis-Rep1 [2017-01-12 11:39:56.971905] E [MSGID: 106376] [glusterd-op-sm.c:7909:glusterd_op_sm] 0-management: handler returned: -1 [2017-01-12 11:39:56.972288] E [MSGID: 106275] [glusterd-rpc-ops.c:878:glusterd_mgmt_v3_lock_peers_cbk_fn] 0-management: Received mgmt_v3 lock RJT from uuid: 6109621f-e6c8-46ed-81f5-e28aaacd e138 [2017-01-12 11:39:56.974541] E [MSGID: 106275] [glusterd-rpc-ops.c:878:glusterd_mgmt_v3_lock_peers_cbk_fn] 0-management: Received mgmt_v3 lock RJT from uuid: 8cd39ea1-cf36-4dda-aab2-5117f061 0387 [2017-01-12 11:39:56.978401] W [glusterd-locks.c:686:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/ xlator/mgmt/glusterd.so(+0x2de42) [0x7fa9ec492e42] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4ffa) [0x7fa9ec539ffa] ) 0-management: Lock owner mismatch. Lock for vol Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c [2017-01-12 11:39:56.978458] E [MSGID: 106118] [glusterd-op-sm.c:3971:glusterd_op_ac_unlock] 0-management: Unable to release lock for Dis-Rep1 [2017-01-12 11:39:58.586776] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/xl ator/mgmt/glusterd.so(+0x2e155) [0x7fa9ec493155] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4c5f) [0x7fa9ec539c5f] ) 0-management: Lock for Dis-Rep1 held by 6799e2bf-ca97-42fa -95c7-6c58d435275c [2017-01-12 11:39:58.591038] W [glusterd-locks.c:686:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/ xlator/mgmt/glusterd.so(+0x2de42) [0x7fa9ec492e42] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4ffa) [0x7fa9ec539ffa] ) 0-management: Lock owner mismatch. Lock for vol Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c [2017-01-12 11:39:58.591105] E [MSGID: 106118] [glusterd-op-sm.c:3971:glusterd_op_ac_unlock] 0-management: Unable to release lock for Dis-Rep1 [2017-01-12 11:39:58.591276] E [MSGID: 106376] [glusterd-op-sm.c:7909:glusterd_op_sm] 0-management: handler returned: 1 [2017-01-12 11:40:56.828857] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/xl ator/mgmt/glusterd.so(+0x2e155) [0x7fa9ec493155] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4c5f) [0x7fa9ec539c5f] ) 0-management: Lock for Dis-Rep1 held by 6799e2bf-ca97-42fa -95c7-6c58d435275c [2017-01-12 11:40:56.831310] W [glusterd-locks.c:686:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2de42) [0x7fa9ec492e42] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4ffa) [0x7fa9ec539ffa] ) 0-management: Lock owner mismatch. Lock for vol Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c The message "E [MSGID: 106376] [glusterd-op-sm.c:7909:glusterd_op_sm] 0-management: handler returned: -1" repeated 3 times between [2017-01-12 11:39:56.971905] and [2017-01-12 11:40:56.829280] [2017-01-12 11:40:56.831391] E [MSGID: 106118] [glusterd-op-sm.c:3971:glusterd_op_ac_unlock] 0-management: Unable to release lock for Dis-Rep1 [2017-01-12 11:41:18.242282] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2e155) [0x7fa9ec493155] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4c5f) [0x7fa9ec539c5f] ) 0-management: Lock for Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c The message "E [MSGID: 106119] [glusterd-op-sm.c:3908:glusterd_op_ac_lock] 0-management: Unable to acquire lock for Dis-Rep1" repeated 3 times between [2017-01-12 11:39:56.971655] and [2017-01-12 11:41:18.242353] [2017-01-12 11:41:18.242619] E [MSGID: 106376] [glusterd-op-sm.c:7909:glusterd_op_sm] 0-management: handler returned: -1 [2017-01-12 11:41:18.244615] W [glusterd-locks.c:686:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2de42) [0x7fa9ec492e42] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4ffa) [0x7fa9ec539ffa] ) 0-management: Lock owner mismatch. Lock for vol Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c [2017-01-12 11:41:44.853829] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xcfa20) [0x7fa9ec534a20] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xcf950) [0x7fa9ec534950] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4c5f) [0x7fa9ec539c5f] ) 0-management: Lock for Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c [2017-01-12 11:41:44.853341] I [MSGID: 106499] [glusterd-handler.c:4362:__glusterd_handle_status_volume] 0-management: Received status volume req for volume Dis-Rep1 [2017-01-12 11:41:44.853882] E [MSGID: 106119] [glusterd-syncop.c:1851:gd_sync_task_begin] 0-management: Unable to acquire lock for Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c [2017-01-12 11:44:58.020745] I [MSGID: 106499] [glusterd-handler.c:4362:__glusterd_handle_status_volume] 0-management: Received status volume req for volume Dis-Rep1 [2017-01-12 11:44:58.021980] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xcfa20) [0x7fa9ec534a20] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xcf950) [0x7fa9ec534950] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4c5f) [0x7fa9ec539c5f] ) 0-management: Lock for Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c [2017-01-12 11:44:58.022020] E [MSGID: 106119] [glusterd-syncop.c:1851:gd_sync_task_begin] 0-management: Unable to acquire lock for Dis-Rep1 [2017-01-12 11:44:59.751963] I [MSGID: 106499] [glusterd-handler.c:4362:__glusterd_handle_status_volume] 0-management: Received status volume req for volume Dis-Rep1 [2017-01-12 11:44:59.752522] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xcfa20) [0x7fa9ec534a20] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xcf950) [0x7fa9ec534950] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4c5f) [0x7fa9ec539c5f] ) 0-management: Lock for Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c [2017-01-12 11:44:59.752554] E [MSGID: 106119] [glusterd-syncop.c:1851:gd_sync_task_begin] 0-management: Unable to acquire lock for Dis-Rep1 [2017-01-12 11:45:11.748739] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2e155) [0x7fa9ec493155] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4c5f) [0x7fa9ec539c5f] ) 0-management: Lock for Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c [2017-01-12 11:45:11.751801] W [glusterd-locks.c:686:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2de42) [0x7fa9ec492e42] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4ffa) [0x7fa9ec539ffa] ) 0-management: Lock owner mismatch. Lock for vol Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c [2017-01-12 11:43:58.584700] E [MSGID: 106376] [glusterd-op-sm.c:7909:glusterd_op_sm] 0-management: handler returned: 1 The message "E [MSGID: 106119] [glusterd-op-sm.c:3908:glusterd_op_ac_lock] 0-management: Unable to acquire lock for Dis-Rep1" repeated 2 times between [2017-01-12 11:43:56.009493] and [2017-01-12 11:45:11.748844] The message "E [MSGID: 106118] [glusterd-op-sm.c:3971:glusterd_op_ac_unlock] 0-management: Unable to release lock for Dis-Rep1" repeated 2 times between [2017-01-12 11:43:56.011189] and [2017-01-12 11:45:11.751851] The message "E [MSGID: 106376] [glusterd-op-sm.c:7909:glusterd_op_sm] 0-management: handler returned: -1" repeated 3 times between [2017-01-12 11:43:56.009666] and [2017-01-12 11:45:11.752042] [2017-01-12 11:46:00.933853] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2e155) [0x7fa9ec493155] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4c5f) [0x7fa9ec539c5f] ) 0-management: Lock for Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c [2017-01-12 11:46:00.933978] E [MSGID: 106119] [glusterd-op-sm.c:3908:glusterd_op_ac_lock] 0-management: Unable to acquire lock for Dis-Rep1 [2017-01-12 11:46:00.934150] E [MSGID: 106376] [glusterd-op-sm.c:7909:glusterd_op_sm] 0-management: handler returned: -1 [2017-01-12 11:46:00.936730] W [glusterd-locks.c:686:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2de42) [0x7fa9ec492e42] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4ffa) [0x7fa9ec539ffa] ) 0-management: Lock owner mismatch. Lock for vol Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c [2017-01-12 11:46:00.936814] E [MSGID: 106118] [glusterd-op-sm.c:3971:glusterd_op_ac_unlock] 0-management: Unable to release lock for Dis-Rep1 [2017-01-12 11:46:00.937024] E [MSGID: 106376] [glusterd-op-sm.c:7909:glusterd_op_sm] 0-management: handler returned: 1 [2017-01-12 11:47:01.360797] I [MSGID: 106499] [glusterd-handler.c:4362:__glusterd_handle_status_volume] 0-management: Received status volume req for volume Dis-Rep1 [2017-01-12 11:47:01.361395] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xcfa20) [0x7fa9ec534a20] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xcf950) [0x7fa9ec534950] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4c5f) [0x7fa9ec539c5f] ) 0-management: Lock for Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c [2017-01-12 11:47:01.361437] E [MSGID: 106119] [glusterd-syncop.c:1851:gd_sync_task_begin] 0-management: Unable to acquire lock for Dis-Rep1 [2017-01-12 11:48:01.731810] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2e155) [0x7fa9ec493155] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4c5f) [0x7fa9ec539c5f] ) 0-management: Lock for Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c [2017-01-12 11:48:01.731901] E [MSGID: 106119] [glusterd-op-sm.c:3908:glusterd_op_ac_lock] 0-management: Unable to acquire lock for Dis-Rep1 [2017-01-12 11:48:01.732153] E [MSGID: 106376] [glusterd-op-sm.c:7909:glusterd_op_sm] 0-management: handler returned: -1 [2017-01-12 11:48:01.734751] W [glusterd-locks.c:686:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3cc7f) [0x7fa9ec4a1c7f] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2de42) [0x7fa9ec492e42] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4ffa) [0x7fa9ec539ffa] ) 0-management: Lock owner mismatch. Lock for vol Dis-Rep1 held by 6799e2bf-ca97-42fa-95c7-6c58d435275c [2017-01-12 11:48:01.734835] E [MSGID: 106118] [glusterd-op-sm.c:3971:glusterd_op_ac_unlock] 0-management: Unable to release lock for Dis-Rep1 [2017-01-12 11:48:01.735036] E [MSGID: 106376] [glusterd-op-sm.c:7909:glusterd_op_sm] 0-management: handler returned: 1 (END)
See https://bugzilla.redhat.com/show_bug.cgi?id=1425681#c4 . I am closing this bug as duplicate of BZ 1425681 *** This bug has been marked as a duplicate of bug 1425681 ***