Bug 1401984

This needs more time. Too late for 3.4.

(In reply to Prasanth from comment #0)
> Description of problem:
> 
> Provide a "force" option in heketi-cli to allow the user to forcefully
> delete/flush any entries from heketi DB
> 
> Currently there is no way for an admin/user to delete a volume or other
> entries from heketi database if:
> 
> * they somehow endup in a situation where the '#heketi volume list' shows
> some volumes present in the DB whereas on performing a '#gluster volume
> list' in the gluster backend clearly shows that "No volumes present in
> cluster".
> * they want to easily flush out a whole test cluster (including volumes,
> devices, nodes, etc) created using heket-cli for various reasons, rather
> than manually cleaning up one by one to remove a cluster ID from heketi
> * they want to create a gluster cluster using heketi-cli but later decides
> to manage the cluster on it's own rather than heketi managing it
> * may be a few more use cases as well :)
> 
> 
> Version-Release number of selected component (if applicable):
> heketi-client-3.1.0-3.el7rhgs.x86_64

Hi Prasanth,

I feel scope of this big is out of heketi.

Point 1-
If there is any way where heketi db has an entry for a volume and its not present in backend gluster(without going to backend manually) then its a bug. It will not reach in such a state IMHO.

Point 2-
They can have a small script which can do this for them. Giving a command to delete whole cluster is not required unless a solid reason. 

Point 3-
I am not sure Why we need support for this? If someone whats to move out of heketi and do manage gluster cluster then they can just stop the heketi service and go ahead with the backend Old volumes will still be there and new volumes, management of old volume has to taken care by the admin.

Let me know If I am missing something.

--
Ashiq

Ashiq, are we planning this fix for CNS 3.5 release considering that, heketidbstorage volume has to be deleted for db migration ?

(In reply to Humble Chirammal from comment #3)
> Ashiq, are we planning this fix for CNS 3.5 release considering that,
> heketidbstorage volume has to be deleted for db migration ?

Humble,

As there is a discussion happening upstream about the secret size(1 MB). I did a small test with 150 volumes in 3 nodes with 4 disks each. The DB size was 1 MB. In this case, secret can not hold the db. As we support multiple gluster cluster from same heketi instance. 1MB can be easily reached. I will start a discussion downstream then we can come to a conclusion. I would like us to keep the volume intact, It is just my opinion. 

--
Ashiq

(In reply to Mohamed Ashiq from comment #2)
> (In reply to Prasanth from comment #0)
> > Description of problem:
> > 
> > Provide a "force" option in heketi-cli to allow the user to forcefully
> > delete/flush any entries from heketi DB
> > 
> > Currently there is no way for an admin/user to delete a volume or other
> > entries from heketi database if:
> > 
> > * they somehow endup in a situation where the '#heketi volume list' shows
> > some volumes present in the DB whereas on performing a '#gluster volume
> > list' in the gluster backend clearly shows that "No volumes present in
> > cluster".
> > * they want to easily flush out a whole test cluster (including volumes,
> > devices, nodes, etc) created using heket-cli for various reasons, rather
> > than manually cleaning up one by one to remove a cluster ID from heketi
> > * they want to create a gluster cluster using heketi-cli but later decides
> > to manage the cluster on it's own rather than heketi managing it
> > * may be a few more use cases as well :)
> > 
> > 
> > Version-Release number of selected component (if applicable):
> > heketi-client-3.1.0-3.el7rhgs.x86_64
> 
> Hi Prasanth,
> 
> I feel scope of this big is out of heketi.
> 
> Point 1-
> If there is any way where heketi db has an entry for a volume and its not
> present in backend gluster(without going to backend manually) then its a
> bug. It will not reach in such a state IMHO.
> 
> Point 2-
> They can have a small script which can do this for them. Giving a command to
> delete whole cluster is not required unless a solid reason. 
> 
> Point 3-
> I am not sure Why we need support for this? If someone whats to move out of
> heketi and do manage gluster cluster then they can just stop the heketi
> service and go ahead with the backend Old volumes will still be there and
> new volumes, management of old volume has to taken care by the admin.
> 
> Let me know If I am missing something.
> 
> --
> Ashiq

Today, the only way to remove device or a node is when they are reachable and clean up can actually be done on the device or node one is trying to clean. However there might be cases where node or device is not reachable or for some corner case, device or node might end up with partial information which prevents further action on that node or device. --force option becomes handy here, where irrespective of the state of node/device user has control to totally remove the information from heketi's view.

One such example I can quote here is the consequence of CNS system after hitting https://bugzilla.redhat.com/show_bug.cgi?id=1435238.

quick summary:
For some reason, heketi takes a long time to create a volume and before heketi could create a volume, the pvc request was deleted and heketi's device information ends up in weird state.

[root@dhcp46-202 ~]# heketi-cli device info fe3c78e3d576382ccb5acf8267db4623
Device Id: fe3c78e3d576382ccb5acf8267db4623
Name: /dev/sdg
State: online
Size (GiB): 99
Used (GiB): 10
Free (GiB): 89
Bricks:
Id:251c4f4e4b0f941529166a5bc5a3b064   Size (GiB):1       Path: 
Id:5c85d7ca2d7e17b99e7334c39dc2211b   Size (GiB):9       Path: /var/lib/heketi/mounts/vg_fe3c78e3d576382ccb5acf8267db4623/brick_5c85d7ca2d7e17b99e7334c39dc2211b/brick

Now this device or the space used cannot be removed unless the partial entry is cleaned up. We don't have a way today to forcefully remove such devices/nodes.

Hence, I'd say having a --force option with a caution makes sense and is very much needed.