This service will be undergoing maintenance at 00:00 UTC, 2016-08-01. It is expected to last about 1 hours
Bug 803717 - [glusterfs-3.3.0qa28] - Unable to delete the volume after stop force
[glusterfs-3.3.0qa28] - Unable to delete the volume after stop force
Status: CLOSED UPSTREAM
Product: GlusterFS
Classification: Community
Component: glusterd (Show other bugs)
pre-release
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: shishir gowda
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-03-15 09:57 EDT by M S Vishwanath Bhat
Modified: 2016-05-31 21:55 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-04-18 03:19:41 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description M S Vishwanath Bhat 2012-03-15 09:57:37 EDT
Description of problem:
I'm unable to to delete a volume which is been forcefully stopped. I did the forcefull stop because the rebalance was running.

Version-Release number of selected component (if applicable):
glusterfs-3.3.0qa28

How reproducible:
1/1

Steps to Reproduce:
1. Create and start a dht volume with a single brick.
2. Now create some data on the mountpoint.
3. Add one more brick and do rebalance after some time rebalance completes.
4. Now run volume stop, it says rebalance is running which itself is an issue.
5. Now run volume stop force and then volume delete.
  
Actual results:
[root@RHEL6 /]# gluster v delete hosdu --mode=script
Deleting volume hosdu has been unsuccessful


Expected results:
volume delete should successful and should delete the /etc/glusterd/vols/<volname> dir.

Additional info:

I see lot of connection attempt failed errors in socket.c and error saying rmdir on /etc/glusterd/vols/<volname> failed because directory is not empty.

Entries from glusterd log.[2012-03-15 09:45:21.555835] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-03-15 09:45:22.556108] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-03-15 09:45:24.556481] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-03-15 09:45:25.556704] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-03-15 09:45:26.220948] I [glusterd-volume-ops.c:420:glusterd_handle_cli_delete_volume] 0-glusterd: Received delete vol reqfor volume hosdu
[2012-03-15 09:45:26.220972] I [glusterd-utils.c:267:glusterd_lock] 0-glusterd: Cluster lock held by 89063a01-3f85-430d-af9d-1bd4d7712143
[2012-03-15 09:45:26.220986] I [glusterd-handler.c:453:glusterd_op_txn_begin] 0-management: Acquired local lock
[2012-03-15 09:45:26.221515] I [glusterd-rpc-ops.c:551:glusterd3_1_cluster_lock_cbk] 0-glusterd: Received ACC from uuid: 13e5badf-357c-43e4-a92e-0d5d0936ca72
[2012-03-15 09:45:26.221554] I [glusterd-rpc-ops.c:551:glusterd3_1_cluster_lock_cbk] 0-glusterd: Received ACC from uuid: 0702ef84-167d-48bc-8505-d2ff55aa70f8
[2012-03-15 09:45:26.221577] I [glusterd-rpc-ops.c:551:glusterd3_1_cluster_lock_cbk] 0-glusterd: Received ACC from uuid: 7daf5457-2532-4fbb-b93f-8ca1189f28ad
[2012-03-15 09:45:26.222373] I [glusterd-op-sm.c:1831:glusterd_op_ac_send_stage_op] 0-glusterd: Sent op req to 3 peers
[2012-03-15 09:45:26.222573] I [glusterd-rpc-ops.c:880:glusterd3_1_stage_op_cbk] 0-glusterd: Received ACC from uuid: 7daf5457-2532-4fbb-b93f-8ca1189f28ad
[2012-03-15 09:45:26.222673] I [glusterd-rpc-ops.c:880:glusterd3_1_stage_op_cbk] 0-glusterd: Received ACC from uuid: 13e5badf-357c-43e4-a92e-0d5d0936ca72
[2012-03-15 09:45:26.222708] I [glusterd-rpc-ops.c:880:glusterd3_1_stage_op_cbk] 0-glusterd: Received ACC from uuid: 0702ef84-167d-48bc-8505-d2ff55aa70f8
[2012-03-15 09:45:26.223139] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Removed cksum
[2012-03-15 09:45:26.223173] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Removed info
[2012-03-15 09:45:26.223201] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Removed hosdu.10.1.11.113.data-bricks-hosdu_brick1.vol
[2012-03-15 09:45:26.223231] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Removed rbstate
[2012-03-15 09:45:26.223263] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Removed run
[2012-03-15 09:45:26.223292] I [glusterd-store.c:990:glusterd_store_delete_volume] 0-: errno:39 (Directory not empty)
[2012-03-15 09:45:26.223304] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Failed to remove rebalance
[2012-03-15 09:45:26.223336] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Removed hosdu.10.1.11.114.data-bricks-hosdu_brick2.vol
[2012-03-15 09:45:26.223372] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Removed trusted-hosdu-fuse.vol
[2012-03-15 09:45:26.223402] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Removed hosdu-fuse.vol
[2012-03-15 09:45:26.238525] E [glusterd-store.c:1009:glusterd_store_delete_volume] 0-: Failed to rmdir: /etc/glusterd/vols/hosdu, err: Directory not empty
[2012-03-15 09:45:26.238563] E [glusterd-op-sm.c:1914:glusterd_op_ac_send_commit_op] 0-: Commit failed
[2012-03-15 09:45:26.238941] I [glusterd-rpc-ops.c:610:glusterd3_1_cluster_unlock_cbk] 0-glusterd: Received ACC from uuid: 7daf5457-2532-4fbb-b93f-8ca1189f28ad
[2012-03-15 09:45:26.238988] I [glusterd-rpc-ops.c:610:glusterd3_1_cluster_unlock_cbk] 0-glusterd: Received ACC from uuid: 13e5badf-357c-43e4-a92e-0d5d0936ca72
[2012-03-15 09:45:26.239016] I [glusterd-rpc-ops.c:610:glusterd3_1_cluster_unlock_cbk] 0-glusterd: Received ACC from uuid: 0702ef84-167d-48bc-8505-d2ff55aa70f8
[2012-03-15 09:45:26.239029] I [glusterd-op-sm.c:2286:glusterd_op_txn_complete] 0-glusterd: Cleared local lock
[2012-03-15 09:45:27.557057] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-03-15 09:45:28.557253] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-03-15 09:45:30.557538] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-03-15 09:45:31.557749] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-03-15 09:45:33.558022] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-03-15 09:45:34.558229] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused)


After unsuccessful volume delete volume info still lists the volume.
Comment 1 shishir gowda 2012-03-27 00:13:04 EDT
Can you please check if this issue still exists?
Comment 2 M S Vishwanath Bhat 2012-03-29 08:57:29 EDT
I still see this happening with glusterfs-3.3.0qa32, but only when volume is stopped forcefully when rebalance is running. 

Steps to reproduce to issue.

1. Create and start a distribute volume.
2. Now run rebalance and while rebalance is running run volume stop force.
3. Now run volume delete. Some times it says volume delete unsuccessful or operation failed on a node.
4. glusterd logs had following errors.


[2012-03-29 08:49:48.235392] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-03-29 08:49:50.235854] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-03-29 08:49:51.236213] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-03-29 08:49:51.236352] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-03-29 08:49:52.835834] I [glusterd-handler.c:495:glusterd_handle_cluster_lock] 0-glusterd: Received LOCK from uuid: cdbe4f48-7b13-48bc-9eb9-bb0a57acadd4
[2012-03-29 08:49:52.835895] I [glusterd-utils.c:267:glusterd_lock] 0-glusterd: Cluster lock held by cdbe4f48-7b13-48bc-9eb9-bb0a57acadd4
[2012-03-29 08:49:52.835956] I [glusterd-handler.c:1313:glusterd_op_lock_send_resp] 0-glusterd: Responded, ret: 0
[2012-03-29 08:49:52.837468] I [glusterd-handler.c:540:glusterd_req_ctx_create] 0-glusterd: Received op from uuid: cdbe4f48-7b13-48bc-9eb9-bb0a57acadd4
[2012-03-29 08:49:52.837574] I [glusterd-handler.c:1415:glusterd_op_stage_send_resp] 0-glusterd: Responded to stage, ret: 0
[2012-03-29 08:49:52.839092] I [glusterd-handler.c:540:glusterd_req_ctx_create] 0-glusterd: Received op from uuid: cdbe4f48-7b13-48bc-9eb9-bb0a57acadd4
[2012-03-29 08:49:52.839454] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Removed info
[2012-03-29 08:49:52.839513] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Removed hosdu-fuse.vol
[2012-03-29 08:49:52.839560] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Removed hosdu.172.17.251.66.data-bricks-hosdu_brick2.vol
[2012-03-29 08:49:52.839607] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Removed trusted-hosdu-fuse.vol
[2012-03-29 08:49:52.839654] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Removed hosdu.172.17.251.63.data-bricks-hosdu_brick1.vol
[2012-03-29 08:49:52.839699] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Removed cksum
[2012-03-29 08:49:52.839764] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Removed hosdu.172.17.251.64.data-bricks-hosdu_brick4.vol
[2012-03-29 08:49:52.839813] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Removed rbstate
[2012-03-29 08:49:52.839860] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Removed run
[2012-03-29 08:49:52.839896] I [glusterd-store.c:990:glusterd_store_delete_volume] 0-: errno:39 (Directory not empty)
[2012-03-29 08:49:52.839912] I [glusterd-store.c:994:glusterd_store_delete_volume] 0-: Failed to remove rebalance
[2012-03-29 08:49:52.839948] E [glusterd-store.c:1009:glusterd_store_delete_volume] 0-: Failed to rmdir: /etc/glusterd/vols/hosdu, err: Directory not empty
[2012-03-29 08:49:52.839966] E [glusterd-op-sm.c:2442:glusterd_op_ac_commit_op] 0-management: Commit failed: -1
[2012-03-29 08:49:52.840023] I [glusterd-handler.c:1456:glusterd_op_commit_send_resp] 0-glusterd: Responded to commit, ret: 0
[2012-03-29 08:49:52.840553] I [glusterd-handler.c:1357:glusterd_handle_cluster_unlock] 0-glusterd: Received UNLOCK from uuid: cdbe4f48-7b13-48bc-9eb9-bb0a57acadd4
[2012-03-29 08:49:52.840620] I [glusterd-handler.c:1333:glusterd_op_unlock_send_resp] 0-glusterd: Responded to unlock, ret: 0
[2012-03-29 08:49:53.236864] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-03-29 08:49:54.237257] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-03-29 08:49:54.237389] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-03-29 08:49:56.237908] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-03-29 08:49:57.238302] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-03-29 08:49:57.238403] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-03-29 08:49:59.238944] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-03-29 08:50:00.239332] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-03-29 08:50:00.239438] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused)
[2012-03-29 08:50:02.239951] E [socket.c:2115:socket_connect] 0-management: connection attempt failed (Connection refused)
Comment 3 shishir gowda 2012-04-18 03:19:41 EDT
This is fixed on mainline.
Please up-grade to release 3.3beta2

Note You need to log in before you can comment on or make changes to this bug.