Description of problem: Unable to perform gluster vol status, start, delete on the cluster Version-Release number of selected component (if applicable): 3.8.4.21 How reproducible: Steps to Reproduce: 1. Created the cluster and enabled the Brick multiplexing 2. Created ec and dist-rep volumes. 3. attached bricks to dist-rep volume and removed the bricks twice from the same volume 4. Stopped the volume and it is succeeded 5. Volume delete failed as it acquired the lock and it won't release the lock. Actual results: [root@dhcp37-135 ~]# gluster vol delete testvol Deleting volume will erase all information about the volume. Do you want to continue? (y/n) y volume delete: testvol: failed: Another transaction is in progress for testvol. Please try again after sometime. Expected results: gluster operations should not fail should succeed Additional info: glusterd log [2017-04-12 04:51:33.521131] I [run.c:191:runner_log] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x36d85) [0x7eff3edb6d85] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4735) [0x7eff3ee54735] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7eff4a2c8105] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/stop/pre/S29CTDB-teardown.sh --volname=testvol-ec --last=yes [2017-04-12 04:51:33.529529] E [run.c:191:runner_log] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x36d85) [0x7eff3edb6d85] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd4696) [0x7eff3ee54696] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7eff4a2c8105] ) 0-management: Failed to execute script: /var/lib/glusterd/hooks/1/stop/pre/S30samba-stop.sh --volname=testvol-ec --last=yes [2017-04-12 04:51:33.529677] I [socket.c:3492:socket_submit_request] 0-management: not connected (priv->connected = -1) [2017-04-12 04:51:33.529734] W [rpc-clnt.c:1694:rpc_clnt_submit] 0-management: failed to submit rpc-request (XID: 0x8 Program: brick operations, ProgVers: 2, Proc: 1) to rpc-transport (management) [2017-04-12 04:51:33.529780] W [rpc-clnt.c:1694:rpc_clnt_submit] 0-management: failed to submit rpc-request (XID: 0x9 Program: brick operations, ProgVers: 2, Proc: 1) to rpc-transport (management) [2017-04-12 04:51:33.529805] W [rpc-clnt.c:1694:rpc_clnt_submit] 0-management: failed to submit rpc-request (XID: 0xa Program: brick operations, ProgVers: 2, Proc: 1) to rpc-transport (management) [2017-04-12 04:51:33.529823] W [rpc-clnt.c:1694:rpc_clnt_submit] 0-management: failed to submit rpc-request (XID: 0xb Program: brick operations, ProgVers: 2, Proc: 1) to rpc-transport (management) [2017-04-12 04:51:33.600864] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped [2017-04-12 04:51:33.601001] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: nfs service is stopped [2017-04-12 04:51:33.601382] I [MSGID: 106568] [glusterd-proc-mgmt.c:87:glusterd_proc_stop] 0-management: Stopping glustershd daemon running in pid: 19756 [2017-04-12 04:51:34.601633] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: glustershd service is stopped [2017-04-12 04:51:34.601786] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped [2017-04-12 04:51:34.601814] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: bitd service is stopped [2017-04-12 04:51:34.601868] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped [2017-04-12 04:51:34.601889] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: scrub service is stopped [2017-04-12 04:51:34.609297] W [glusterd-handler.c:5675:__glusterd_brick_rpc_notify] 0-management: got disconnect from stale rpc on /bricks/brick0/testvol_brick0 [2017-04-12 04:59:40.379360] I [MSGID: 106499] [glusterd-handler.c:4369:__glusterd_handle_status_volume] 0-management: Received status volume req for volume testvol [2017-04-12 04:59:40.379522] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd39d0) [0x7eff3ee539d0] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd3900) [0x7eff3ee53900] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd8c0f) [0x7eff3ee58c0f] ) 0-management: Lock for testvol held by 2352d844-ab16-4c77-86e5-6bfcfcc8b3b2 [2017-04-12 04:59:40.379562] E [MSGID: 106119] [glusterd-syncop.c:1864:gd_sync_task_begin] 0-management: Unable to acquire lock for testvol [2017-04-12 04:59:40.380043] I [MSGID: 106499] [glusterd-handler.c:4369:__glusterd_handle_status_volume] 0-management: Received status volume req for volume testvol-ec [2017-04-12 04:59:40.381881] E [MSGID: 106301] [glusterd-syncop.c:1315:gd_stage_op_phase] 0-management: Staging of operation 'Volume Status' failed on localhost : Volume testvol-ec is not started [2017-04-12 05:12:28.555698] I [MSGID: 106499] [glusterd-handler.c:4369:__glusterd_handle_status_volume] 0-management: Received status volume req for volume testvol [2017-04-12 05:12:28.555849] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd39d0) [0x7eff3ee539d0] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd3900) [0x7eff3ee53900] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd8c0f) [0x7eff3ee58c0f] ) 0-management: Lock for testvol held by 2352d844-ab16-4c77-86e5-6bfcfcc8b3b2 [2017-04-12 05:12:28.555877] E [MSGID: 106119] [glusterd-syncop.c:1864:gd_sync_task_begin] 0-management: Unable to acquire lock for testvol [2017-04-12 05:12:28.556369] I [MSGID: 106499] [glusterd-handler.c:4369:__glusterd_handle_status_volume] 0-management: Received status volume req for volume testvol-ec [2017-04-12 05:12:28.558175] E [MSGID: 106301] [glusterd-syncop.c:1315:gd_stage_op_phase] 0-management: Staging of operation 'Volume Status' failed on localhost : Volume testvol-ec is not started [2017-04-12 05:12:37.079892] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd39d0) [0x7eff3ee539d0] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd3900) [0x7eff3ee53900] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd8c0f) [0x7eff3ee58c0f] ) 0-management: Lock for testvol held by 2352d844-ab16-4c77-86e5-6bfcfcc8b3b2 [2017-04-12 05:12:37.079953] E [MSGID: 106119] [glusterd-syncop.c:1864:gd_sync_task_begin] 0-management: Unable to acquire lock for testvol [2017-04-12 05:18:31.518899] W [glusterd-locks.c:572:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd39d0) [0x7eff3ee539d0] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd3900) [0x7eff3ee53900] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd8c0f) [0x7eff3ee58c0f] ) 0-management: Lock for testvol held by 2352d844-ab16-4c77-86e5-6bfcfcc8b3b2
upstream patch : https://review.gluster.org/#/c/17055/
downstream patch : https://code.engineering.redhat.com/gerrit/#/c/103636
Build : 3.8.4-24 Followed the steps provided in the bug. After volume stop, able to delete the volume without any issue. Hence marking the bug as verified
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:2774