Unable to delet pvc from a block hosting volume which is very low in available sapce

Description of problem:
++++++++++++++++++++++++
This bug was seen after all the issues observed in https://bugzilla.redhat.com/show_bug.cgi?id=1622645.

There were 2 block-hosting volumes (BHV) on the setup with free space <1GB. 

[root@dhcp42-137 nitin]# heketi-cli volume list
Id:4805e6aa548e1db5713a46c55ca51f95    Cluster:a7d48ca7363896a8b06665c0fe1bdd8c    Name:vol_4805e6aa548e1db5713a46c55ca51f95 [block]
Id:868297e82df2a131b228a20a746813da    Cluster:a7d48ca7363896a8b06665c0fe1bdd8c    Name:vol_868297e82df2a131b228a20a746813da [block]   <------------no space left
Id:d47e1a3101c13a1ab3edffa1ce765d3a    Cluster:a7d48ca7363896a8b06665c0fe1bdd8c    Name:heketidbstorage
Id:ed54bbe8770a4597f8c350f044177bca    Cluster:a7d48ca7363896a8b06665c0fe1bdd8c    Name:vol_ed54bbe8770a4597f8c350f044177bca [block]   < <------------no space left

Hence, to free up some space on vol_868297e82df2a131b228a20a746813da, tried deleting a block-device  new1 of 2 GB in size. But the pvc deletion failed with following error message
Similarly, tried deleting bk7 present in vol_ed54bbe8770a4597f8c350f044177bca to free up its space, but delete failed.


Thus if we wont be able to delete any volume to free up space, please let us know the workaround which can make us re-use all the existing pvcs from above 2 volumes. 

The block devices are not getting mounted on any app pod due to I/O error issue. Hence, being unable to free up space, these pvcs are no more usable.



Heketi logs
===============

[kubeexec] ERROR 2018/08/27 13:02:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_868297e82df2a131b228a20a746813da/blk_glusterfs_new1_9ab8b509-a8ee-11e8-af29-0a580a82003b --json] on glusterfs-storage-r9b56: Err[command terminated with exit code 255]: Stdout [{ "FAILED ON": [ "10.70.41.217", "10.70.42.223", "10.70.42.84" ], "RESULT": "FAIL" }
]: Stderr []
[cmdexec] ERROR 2018/08/27 13:02:44 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blk_glusterfs_new1_9ab8b509-a8ee-11e8-af29-0a580a82003b: Unable to execute command on glusterfs-storage-r9b56: 
[heketi] ERROR 2018/08/27 13:02:44 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-r9b56: 
[heketi] ERROR 2018/08/27 13:02:44 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Delete Block Volume Failed: Unable to execute command on glusterfs-storage-r9b56: 
[asynch




Gluster-block logs
========================

[2018-08-27 13:02:06.517202] INFO: delete cli request, volume=vol_868297e82df2a131b228a20a746813da blockname=blk_glusterfs_new1_9ab8b509-a8ee-11e8-af29-0a580a82003b [at block_svc_routines.c+4493 :<block_delete_cli_1_svc_st>]
[2018-08-27 13:02:07.577576] ERROR: glfs_write(blk_glusterfs_new1_9ab8b509-a8ee-11e8-af29-0a580a82003b): on volume vol_868297e82df2a131b228a20a746813da failed[No space left on device] [at block_svc_routines.c+1027 :<glusterBlockDeleteRemote>]
[2018-08-27 13:02:09.578476] ERROR: glfs_write(blk_glusterfs_new1_9ab8b509-a8ee-11e8-af29-0a580a82003b): on volume vol_868297e82df2a131b228a20a746813da failed[No space left on device] [at block_svc_routines.c+1027 :<glusterBlockDeleteRemote>]
[2018-08-27 13:02:11.578597] ERROR: glfs_write(blk_glusterfs_new1_9ab8b509-a8ee-11e8-af29-0a580a82003b): on volume vol_868297e82df2a131b228a20a746813da failed[No space left on device] [at block_svc_routines.c+1027 :<glusterBlockDeleteRemote>]
[2018-08-27 13:02:11.578926] ERROR: failed to delete config on 10.70.41.217 Failed to update transaction log for vol_868297e82df2a131b228a20a746813da/blk_glusterfs_new1_9ab8b509-a8ee-11e8-af29-0a580a82003b[No space left on device]: on volume vol_868297e82df2a131b228a20a746813da on host 10.70.41.217 [at block_svc_routines.c+1137 :<glusterBlockCollectAttemptSuccess>]
[2018-08-27 13:02:11.578999] ERROR: failed to delete config on 10.70.42.223 Failed to update transaction log for vol_868297e82df2a131b228a20a746813da/blk_glusterfs_new1_9ab8b509-a8ee-11e8-af29-0a580a82003b[No space left on device]: on volume vol_868297e82df2a131b228a20a746813da on host 10.70.42.223 [at block_svc_routines.c+1137 :<glusterBlockCollectAttemptSuccess>]
[2018-08-27 13:02:11.579055] ERROR: failed to delete config on 10.70.42.84 Failed to update transaction log for vol_868297e82df2a131b228a20a746813da/blk_glusterfs_new1_9ab8b509-a8ee-11e8-af29-0a580a82003b[No space left on device]: on volume vol_868297e82df2a131b228a20a746813da on host 10.70.42.84 [at block_svc_routines.c+1137 :<glusterBlockCollectAttemptSuccess>]
[2018-08-27 13:02:11.580962] WARNING: glusterBlockDeleteRemoteAsync: return -1 failed in remote async delete for block blk_glusterfs_new1_9ab8b509-a8ee-11e8-af29-0a580a82003b on volume vol_868297e82df2a131b228a20a746813da [at block_svc_routines.c+2884 :<glusterBlockCleanUp>]
[2018-08-27 13:02:11.581023] WARNING: glusterBlockCleanUp: return -1 on block blk_glusterfs_new1_9ab8b509-a8ee-11e8-af29-0a580a82003b for volume vol_868297e82df2a131b228a20a746813da [at block_svc_routines.c+4566 :<block_delete_cli_1_svc_st>]
[2018-08-27 13:02:12.457280] INFO: delete cli request, volume=vol_868297e82df2a131b228a20a746813da blockname=blk_glusterfs_new1_9ab8b509-a8ee-11e8-af29-0a580a82003b [at block_svc_routines.c+4493 :<block_delete_cli_1_svc_st>]
[2018-08-27 13:02:13.581507] ERROR: glfs_write(blk_glusterfs_new1_9ab8b509-a8ee-11e8-af29-0a580a82003b): on volume vol_868297e82df2a131b228a20a746813da failed[No space left on device] [at block_svc_routines.c+1027 :<glusterBlockDeleteRemote>]



[2018-08-27 16:40:32.393150] INFO: delete cli request, volume=vol_ed54bbe8770a4597f8c350f044177bca blockname=blk_glusterfs_bk7_c68b4b19-a7ad-11e8-8d82-0a580a80000a [at block_svc_routines.c+4493 :<block_delete_cli_1_svc_st>]
[2018-08-27 16:40:32.455315] ERROR: glfs_write(blk_glusterfs_bk7_c68b4b19-a7ad-11e8-8d82-0a580a80000a): on volume vol_ed54bbe8770a4597f8c350f044177bca failed[No space left on device] [at block_svc_routines.c+1027 :<glusterBlockDeleteRemote>]
[2018-08-27 16:40:32.463565] ERROR: glfs_write(blk_glusterfs_bk7_c68b4b19-a7ad-11e8-8d82-0a580a80000a): on volume vol_ed54bbe8770a4597f8c350f044177bca failed[No space left on device] [at block_svc_routines.c+1027 :<glusterBlockDeleteRemote>]
[2018-08-27 16:40:32.468168] ERROR: glfs_write(blk_glusterfs_bk7_c68b4b19-a7ad-11e8-8d82-0a580a80000a): on volume vol_ed54bbe8770a4597f8c350f044177bca failed[No space left on device] [at block_svc_routines.c+1027 :<glusterBlockDeleteRemote>]
[2018-08-27 16:40:32.468699] ERROR: failed to delete config on 10.70.41.217 Failed to update transaction log for vol_ed54bbe8770a4597f8c350f044177bca/blk_glusterfs_bk7_c68b4b19-a7ad-11e8-8d82-0a580a80000a[No space left on device]: on volume vol_ed54bbe8770a4597f8c350f044177bca on host 10.70.41.217 [at block_svc_routines.c+1137 :<glusterBlockCollectAttemptSuccess>]
[2018-08-27 16:40:32.468779] ERROR: failed to delete config on 10.70.42.223 Failed to update transaction log for vol_ed54bbe8770a4597f8c350f044177bca/blk_glusterfs_bk7_c68b4b19-a7ad-11e8-8d82-0a580a80000a[No space left on device]: on volume vol_ed54bbe8770a4597f8c350f044177bca on host 10.70.42.223 [at block_svc_routines.c+1137 :<glusterBlockCollectAttemptSuccess>]
[2018-08-27 16:40:32.468824] ERROR: failed to delete config on 10.70.42.84 Failed to update transaction log for vol_ed54bbe8770a4597f8c350f044177bca/blk_glusterfs_bk7_c68b4b19-a7ad-11e8-8d82-0a580a80000a[No space left on device]: on volume vol_ed54bbe8770a4597f8c350f044177bca on host 10.70.42.84 [at block_svc_routines.c+1137 :<glusterBlockCollectAttemptSuccess>]
[2018-08-27 16:40:32.474244] WARNING: glusterBlockDeleteRemoteAsync: return -1 failed in remote async delete for block blk_glusterfs_bk7_c68b4b19-a7ad-11e8-8d82-0a580a80000a on volume vol_ed54bbe8770a4597f8c350f044177bca [at block_svc_routines.c+2884 :<glusterBlockCleanUp>]





Blockvolumes carved out of vol_ed54bbe8770a4597f8c350f044177bca
===================================================================

[root@dhcp42-137 nitin]# heketi-cli blockvolume list
Id:3b11d9e57425783493c6e9c5a75999be    Cluster:a7d48ca7363896a8b06665c0fe1bdd8c    Name:blk_glusterfs_new1_9ab8b509-a8ee-11e8-af29-0a580a82003b
Id:71d2db06910ca90e2f8954cbd305d22f    Cluster:a7d48ca7363896a8b06665c0fe1bdd8c    Name:blk_glusterfs_new2_d0c6a56a-a8ee-11e8-af29-0a580a82003b
Id:7bb3bbda58079462d2b9a0d3df8b91a4    Cluster:a7d48ca7363896a8b06665c0fe1bdd8c    Name:blk_glusterfs_bk10_f0a6713a-a7ae-11e8-8d82-0a580a80000a

Blockvolumes carved out of vol_ed54bbe8770a4597f8c350f044177bca
======================================================

[root@dhcp42-137 upg-issue]# heketi-cli blockvolume list
Id:07c369cdd292c6b399641c64d451f9e9    Cluster:a7d48ca7363896a8b06665c0fe1bdd8c    Name:blk_glusterfs_bk6_c63592b8-a7ad-11e8-8d82-0a580a80000a
Id:09ebe607a084eb0f758033de1f889b4b    Cluster:a7d48ca7363896a8b06665c0fe1bdd8c    Name:blk_glusterfs_bk1_18c419fe-a7ab-11e8-8d82-0a580a80000a
Id:0a44d2a0253de4d8e1df96ea5d232d77    Cluster:a7d48ca7363896a8b06665c0fe1bdd8c    Name:blk_glusterfs_bk8_c90c051e-a7ad-11e8-8d82-0a580a80000a
Id:6626924142c35a5cae8dae958709064e    Cluster:a7d48ca7363896a8b06665c0fe1bdd8c    Name:blk_glusterfs_bk5_700297c7-a7ad-11e8-8d82-0a580a80000a
Id:8b77db1fe147fd03c546fe186bfdddae    Cluster:a7d48ca7363896a8b06665c0fe1bdd8c    Name:blk_glusterfs_bk2_6d6884db-a7ad-11e8-8d82-0a580a80000a
Id:9c64d1155a4cc362ec2b0c16aeb7681e    Cluster:a7d48ca7363896a8b06665c0fe1bdd8c    Name:blk_glusterfs_bk9_c9864871-a7ad-11e8-8d82-0a580a80000a
Id:a1cfbba6d6081e29fdea6822d8a1b52d    Cluster:a7d48ca7363896a8b06665c0fe1bdd8c    Name:blk_glusterfs_bk4_6f8896e7-a7ad-11e8-8d82-0a580a80000a
Id:b1388acdb898cd5a3f98676f20172530    Cluster:a7d48ca7363896a8b06665c0fe1bdd8c    Name:blk_glusterfs_bk3_6dbedcfb-a7ad-11e8-8d82-0a580a80000a
Id:e38b6a2a115d1b89ad8f6357aab7e636    Cluster:a7d48ca7363896a8b06665c0fe1bdd8c    Name:blk_glusterfs_bk7_c68b4b19-a7ad-11e8-8d82-0a580a80000a


Space usage from brick's df -kh
===================================
var-lib-heketi-mounts-vg_bb4e55003af42af5b6af507e0f8
[root@dhcp42-137 ~]# for i in `oc get pods -o wide| grep glusterfs|cut -d " " -f1` ; do echo $i; echo +++++++++++++++++++++++; oc exec $i -- df -kh|grep heketi; done
glusterfs-storage-59jt5
+++++++++++++++++++++++
/dev/mapper/vg_7cd0695cb3a720bb3af5524d2f1dd437-brick_42d291a6d1f16dfecc8f9c321ec89c31  2.0G   33M  2.0G   2% /var/lib/heketi/mounts/vg_7cd0695cb3a720bb3af5524d2f1dd437/brick_42d291a6d1f16dfecc8f9c321ec89c31
/dev/mapper/vg_49a8f6746a305fd3696f9aa9c1f32d6f-brick_9011b8f81ccc4c9cebf2a74e88ae70d6  100G  100G  180K 100% /var/lib/heketi/mounts/vg_49a8f6746a305fd3696f9aa9c1f32d6f/brick_9011b8f81ccc4c9cebf2a74e88ae70d6
/dev/mapper/vg_7cd0695cb3a720bb3af5524d2f1dd437-brick_a52072845f7ae8361e359a6f907f20cf  120G  120G  931M 100% /var/lib/heketi/mounts/vg_7cd0695cb3a720bb3af5524d2f1dd437/brick_a52072845f7ae8361e359a6f907f20cf
/dev/mapper/vg_7cd0695cb3a720bb3af5524d2f1dd437-brick_c5be1ff405a61f424ed2205d45c5d360  100G   99G  2.0G  99% /var/lib/heketi/mounts/vg_7cd0695cb3a720bb3af5524d2f1dd437/brick_c5be1ff405a61f424ed2205d45c5d360
glusterfs-storage-7s5h6
+++++++++++++++++++++++
/dev/mapper/vg_68893abe99daefd0dc9c7bf68f606902-brick_f21f7bd09d1c078c6f3191e811158bb2  2.0G   33M  2.0G   2% /var/lib/heketi/mounts/vg_68893abe99daefd0dc9c7bf68f606902/brick_f21f7bd09d1c078c6f3191e811158bb2
/dev/mapper/vg_68893abe99daefd0dc9c7bf68f606902-brick_4584491f8c346580734d7b1e710b08de  100G  100G  204K 100% /var/lib/heketi/mounts/vg_68893abe99daefd0dc9c7bf68f606902/brick_4584491f8c346580734d7b1e710b08de
/dev/mapper/vg_556c81099602cc71dca84fbe4926c6b0-brick_dad0c9b47e85ac26c7ca09ea1832bedc  120G  120G  931M 100% /var/lib/heketi/mounts/vg_556c81099602cc71dca84fbe4926c6b0/brick_dad0c9b47e85ac26c7ca09ea1832bedc
/dev/mapper/vg_68893abe99daefd0dc9c7bf68f606902-brick_a3f1f0d91f9ab6735b35d605e5ec7d98  100G   99G  2.0G  99% /var/lib/heketi/mounts/vg_68893abe99daefd0dc9c7bf68f606902/brick_a3f1f0d91f9ab6735b35d605e5ec7d98
glusterfs-storage-r9b56
+++++++++++++++++++++++
/dev/mapper/vg_4264459401ed48d0939685f2bd5472fb-brick_8724d3d50a9fc4c74f022e9e54b86874  2.0G   33M  2.0G   2% /var/lib/heketi/mounts/vg_4264459401ed48d0939685f2bd5472fb/brick_8724d3d50a9fc4c74f022e9e54b86874
/dev/mapper/vg_bb4e55003af42af5b6af507e0f81ee39-brick_7728805e80a10df134c6991134dee9f0  100G  100G  212K 100% /var/lib/heketi/mounts/vg_bb4e55003af42af5b6af507e0f81ee39/brick_7728805e80a10df134c6991134dee9f0
/dev/mapper/vg_bb4e55003af42af5b6af507e0f81ee39-brick_2fc66410598a5f3819d686d28fe3882d  120G  120G  931M 100% /var/lib/heketi/mounts/vg_bb4e55003af42af5b6af507e0f81ee39/brick_2fc66410598a5f3819d686d28fe3882d
/dev/mapper/vg_bb4e55003af42af5b6af507e0f81ee39-brick_c5a42e8b391e4594c274e9255b65a6a8  100G   99G  2.0G  99% /var/lib/heketi/mounts/vg_bb4e55003af42af5b6af507e0f81ee39/brick_c5a42e8b391e4594c274e9255b65a6a8
[root@dhcp42-137 ~]# 

Version-Release number of selected component (if applicable):
++++++++++++++++++++++++

How reproducible:
++++++++++++++++++++++++

Steps to Reproduce:
++++++++++++++++++++++++
1. Create a CNS 3.9 setup and use up all the space in the block hosting volume
2. Upgrade the setup to 3.10. Since the free space in BHV was 0 before upgrade , the reserved space in new heketi build will show as 0 GB instead of the actual 2GB(for 100GB BHV)
3. Try deleting some block devices from the above BHV. 
4. Check gluster block and heketi logs.


Actual results:
++++++++++++++++++++++++
Block device deletions from BHV's which are full is FAILING. The block pvcs do not get deleted from Gluster backend.
The entries from /etc/target/saveconfig.json are also intact.

Expected results:
++++++++++++++++++++++++
In a situaltion where we have already hit BZ#1622645 and are unable to mount any existing pvcs to app pods, we should be able to delete some pvcs from BHV's which are completely utilized. This will help in freeing up the space.