Hide Forgot
Description of problem: I'm unable to to delete a volume which is been forcefully stopped. I did the forcefull stop because the rebalance was running. Version-Release number of selected component (if applicable): glusterfs-3.3.0qa28 How reproducible: 1/1 Steps to Reproduce: 1. Create and start a dht volume with a single brick. 2. Now create some data on the mountpoint. 3. Add one more brick and do rebalance after some time rebalance completes. 4. Now run volume stop, it says rebalance is running which itself is an issue. 5. Now run volume stop force and then volume delete. Actual results: [root@RHEL6 /]# gluster v delete hosdu --mode=script Deleting volume hosdu has been unsuccessful Expected results: volume delete should successful and should delete the /etc/glusterd/vols/<volname> dir. Additional info: I see lot of connection attempt failed errors in socket.c and error saying rmdir on /etc/glusterd/vols/<volname> failed because directory is not empty. Entries from glusterd log.[2012-03-15 09:45:21.555835] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused) [2012-03-15 09:45:22.556108] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused) [2012-03-15 09:45:24.556481] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused) [2012-03-15 09:45:25.556704] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused) [2012-03-15 09:45:26.220948] I [glusterd-volume-ops.c:420:glusterd_handle_cli_delete_volume] 0-glusterd: Received delete vol reqfor volume hosdu [2012-03-15 09:45:26.220972] I [glusterd-utils.c:267:glusterd_lock] 0-glusterd: Cluster lock held by 89063a01-3f85-430d-af9d-1bd4d7712143 [2012-03-15 09:45:26.220986] I [glusterd-handler.c:453:glusterd_op_txn_begin] 0-management: Acquired local lock [2012-03-15 09:45:26.221515] I [glusterd-rpc-ops.c:551:glusterd3_1_cluster_lock_cbk] 0-glusterd: Received ACC from uuid: 13e5badf-357c-43e4-a92e-0d5d0936ca72 [2012-03-15 09:45:26.221554] I [glusterd-rpc-ops.c:551:glusterd3_1_cluster_lock_cbk] 0-glusterd: Received ACC from uuid: 0702ef84-167d-48bc-8505-d2ff55aa70f8 [2012-03-15 09:45:26.221577] I [glusterd-rpc-ops.c:551:glusterd3_1_cluster_lock_cbk] 0-glusterd: Received ACC from uuid: 7daf5457-2532-4fbb-b93f-8ca1189f28ad [2012-03-15 09:45:26.222373] I [glusterd-op-sm.c:1831:glusterd_op_ac_send_stage_op] 0-glusterd: Sent op req to 3 peers [2012-03-15 09:45:26.222573] I [glusterd-rpc-ops.c:880:glusterd3_1_stage_op_cbk] 0-glusterd: Received ACC from uuid: 7daf5457-2532-4fbb-b93f-8ca1189f28ad [2012-03-15 09:45:26.222673] I [glusterd-rpc-ops.c:880:glusterd3_1_stage_op_cbk] 0-glusterd: Received ACC from uuid: 13e5badf-357c-43e4-a92e-0d5d0936ca72 [2012-03-15 09:45:26.222708] I [glusterd-rpc-ops.c:880:glusterd3_1_stage_op_cbk] 0-glusterd: Received ACC from uuid: 0702ef84-167d-48bc-8505-d2ff55aa70f8 [2012-03-15 09:45:26.223139] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Removed cksum [2012-03-15 09:45:26.223173] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Removed info [2012-03-15 09:45:26.223201] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Removed hosdu.10.1.11.113.data-bricks-hosdu_brick1.vol [2012-03-15 09:45:26.223231] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Removed rbstate [2012-03-15 09:45:26.223263] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Removed run [2012-03-15 09:45:26.223292] I [glusterd-store.c:990:glusterd_store_delete_volume] 0-: errno:39 (Directory not empty) [2012-03-15 09:45:26.223304] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Failed to remove rebalance [2012-03-15 09:45:26.223336] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Removed hosdu.10.1.11.114.data-bricks-hosdu_brick2.vol [2012-03-15 09:45:26.223372] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Removed trusted-hosdu-fuse.vol [2012-03-15 09:45:26.223402] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Removed hosdu-fuse.vol [2012-03-15 09:45:26.238525] E [glusterd-store.c:1009:glusterd_store_delete_volume] 0-: Failed to rmdir: /etc/glusterd/vols/hosdu, err: Directory not empty [2012-03-15 09:45:26.238563] E [glusterd-op-sm.c:1914:glusterd_op_ac_send_commit_op] 0-: Commit failed [2012-03-15 09:45:26.238941] I [glusterd-rpc-ops.c:610:glusterd3_1_cluster_unlock_cbk] 0-glusterd: Received ACC from uuid: 7daf5457-2532-4fbb-b93f-8ca1189f28ad [2012-03-15 09:45:26.238988] I [glusterd-rpc-ops.c:610:glusterd3_1_cluster_unlock_cbk] 0-glusterd: Received ACC from uuid: 13e5badf-357c-43e4-a92e-0d5d0936ca72 [2012-03-15 09:45:26.239016] I [glusterd-rpc-ops.c:610:glusterd3_1_cluster_unlock_cbk] 0-glusterd: Received ACC from uuid: 0702ef84-167d-48bc-8505-d2ff55aa70f8 [2012-03-15 09:45:26.239029] I [glusterd-op-sm.c:2286:glusterd_op_txn_complete] 0-glusterd: Cleared local lock [2012-03-15 09:45:27.557057] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused) [2012-03-15 09:45:28.557253] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused) [2012-03-15 09:45:30.557538] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused) [2012-03-15 09:45:31.557749] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused) [2012-03-15 09:45:33.558022] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused) [2012-03-15 09:45:34.558229] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused) After unsuccessful volume delete volume info still lists the volume.
Can you please check if this issue still exists?
I still see this happening with glusterfs-3.3.0qa32, but only when volume is stopped forcefully when rebalance is running. Steps to reproduce to issue. 1. Create and start a distribute volume. 2. Now run rebalance and while rebalance is running run volume stop force. 3. Now run volume delete. Some times it says volume delete unsuccessful or operation failed on a node. 4. glusterd logs had following errors. [2012-03-29 08:49:48.235392] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused) [2012-03-29 08:49:50.235854] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused) [2012-03-29 08:49:51.236213] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused) [2012-03-29 08:49:51.236352] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused) [2012-03-29 08:49:52.835834] I [glusterd-handler.c:495:glusterd_handle_cluster_lock] 0-glusterd: Received LOCK from uuid: cdbe4f48-7b13-48bc-9eb9-bb0a57acadd4 [2012-03-29 08:49:52.835895] I [glusterd-utils.c:267:glusterd_lock] 0-glusterd: Cluster lock held by cdbe4f48-7b13-48bc-9eb9-bb0a57acadd4 [2012-03-29 08:49:52.835956] I [glusterd-handler.c:1313:glusterd_op_lock_send_resp] 0-glusterd: Responded, ret: 0 [2012-03-29 08:49:52.837468] I [glusterd-handler.c:540:glusterd_req_ctx_create] 0-glusterd: Received op from uuid: cdbe4f48-7b13-48bc-9eb9-bb0a57acadd4 [2012-03-29 08:49:52.837574] I [glusterd-handler.c:1415:glusterd_op_stage_send_resp] 0-glusterd: Responded to stage, ret: 0 [2012-03-29 08:49:52.839092] I [glusterd-handler.c:540:glusterd_req_ctx_create] 0-glusterd: Received op from uuid: cdbe4f48-7b13-48bc-9eb9-bb0a57acadd4 [2012-03-29 08:49:52.839454] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Removed info [2012-03-29 08:49:52.839513] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Removed hosdu-fuse.vol [2012-03-29 08:49:52.839560] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Removed hosdu.172.17.251.66.data-bricks-hosdu_brick2.vol [2012-03-29 08:49:52.839607] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Removed trusted-hosdu-fuse.vol [2012-03-29 08:49:52.839654] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Removed hosdu.172.17.251.63.data-bricks-hosdu_brick1.vol [2012-03-29 08:49:52.839699] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Removed cksum [2012-03-29 08:49:52.839764] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Removed hosdu.172.17.251.64.data-bricks-hosdu_brick4.vol [2012-03-29 08:49:52.839813] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Removed rbstate [2012-03-29 08:49:52.839860] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Removed run [2012-03-29 08:49:52.839896] I [glusterd-store.c:990:glusterd_store_delete_volume] 0-: errno:39 (Directory not empty) [2012-03-29 08:49:52.839912] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Failed to remove rebalance [2012-03-29 08:49:52.839948] E [glusterd-store.c:1009:glusterd_store_delete_volume] 0-: Failed to rmdir: /etc/glusterd/vols/hosdu, err: Directory not empty [2012-03-29 08:49:52.839966] E [glusterd-op-sm.c:2442:glusterd_op_ac_commit_op] 0-management: Commit failed: -1 [2012-03-29 08:49:52.840023] I [glusterd-handler.c:1456:glusterd_op_commit_send_resp] 0-glusterd: Responded to commit, ret: 0 [2012-03-29 08:49:52.840553] I [glusterd-handler.c:1357:glusterd_handle_cluster_unlock] 0-glusterd: Received UNLOCK from uuid: cdbe4f48-7b13-48bc-9eb9-bb0a57acadd4 [2012-03-29 08:49:52.840620] I [glusterd-handler.c:1333:glusterd_op_unlock_send_resp] 0-glusterd: Responded to unlock, ret: 0 [2012-03-29 08:49:53.236864] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused) [2012-03-29 08:49:54.237257] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused) [2012-03-29 08:49:54.237389] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused) [2012-03-29 08:49:56.237908] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused) [2012-03-29 08:49:57.238302] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused) [2012-03-29 08:49:57.238403] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused) [2012-03-29 08:49:59.238944] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused) [2012-03-29 08:50:00.239332] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused) [2012-03-29 08:50:00.239438] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused) [2012-03-29 08:50:02.239951] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused)
This is fixed on mainline. Please up-grade to release 3.3beta2