killing gluster-blockd during pvc-creation leads to spurious BV IDs using up complete free space of the BHV

This bug is somewhat similar to Bug 1624738(only exception - heketi-cli blockvolume list doesnt display the ghost block device IDs)

Description of problem:
++++++++++++++++++++++++++

A BHV along with some BVs already exist on the 4 node OCS Cluster setup( 1 node is kept disabled). Free size on BHV-vol_569900e02f6f3cfc05277daefe180984 is 77GB.

Started a loop to create 5 more block pvcs of 2 GB each and immediatelt killed gluster-blockd process on 2 gluster pods, so as not to meet the minimum host requirement for HA=3 pvcs.

As expected, pvcs went into pending state. But as seen from heketi logs, heketi still keeps trying to create those pvcs but fails to delete them as transactions fail.

This ultimately uses up all the 77 GB free space of the BHV and a new BHV is created(seen only on the gluster-backend). This leads to loss of space on the first BHV and BHV count mismatch between heketi and gluster.


Steps to reproduce
===================
1. A setup with pre-existing BHV and 10 BVs exist(block1-block10)

2. Start a loop to create extra BVs block{11-15} - Fri Oct  5 17:56:26 IST 2018

date && for i in  {11..15}; do ./pvc-create.sh block$i 2; done && date
Fri Oct  5 17:56:26 IST 2018


3. Immediately kill gluster-blockd on 2 pods

sh-4.2# systemctl stop gluster-blockd && date
Fri Oct  5 12:26:27 UTC 2018

sh-4.2# systemctl stop gluster-blockd && date
Fri Oct  5 12:26:27 UTC 2018

4. Check the used space of the BHV -vol_569900e02f6f3cfc05277daefe180984 at regular intervals. It keeps descreasing till no free space is left.

5. Check at the gluster backend, a new BHV is created, but is not listed in heketi-cli volume list. Thus a mismatch is seen.

# gluster v list|wc -l
15

# heketi-cli volume list|wc -l
14

New BHV in gluster = vol_4c48ee12e12ea6382069b2048569e208


6. Check for heketi pending operation count;

# heketi-cli server operations info 
Operation Counts:
  Total: 39
  In-Flight: 0
  New: 39
  Stale: 0




Issues seen:
=============

1.heketi-cli volume list displays 1 BHV but "gluster v list" displays 2 BHVs

2. heketi-cli blockvolume list displays only initial 10 BVs but gluster-block list of same BHV shows 11 BVs

3. the heketi-cli db dump lists IDs of many BVs which are not present in either heketi-cli/gluster backend

4. The df -kh on the glusterfs pods shows 77GB free on the same BHV whereas heketi shows only 1 GB free


5. heketi-cli blockvolume list displays 10 BHV but gluster-block list displays 11 BVs on the same BHV-vol_569900e02f6f3cfc05277daefe180984 


6. Even when the gluster-blockd is brought up, the pending 4 pvc creations never completed.


7. heketi logs still keep showing error messages for BV creates and db dump shows innumerable ghost IDS.

8. No new pvcs are now created as heketi keeps reporting similar error messages:

Started POST /blockvolumes
[heketi] WARNING 2018/10/05 13:41:59 Free size 1 is less than the requested block volume size 2
[heketi] WARNING 2018/10/05 13:41:59 temporarily rejecting block volume request: pending block-hosting-volume found



Setup Info
===========

# oc get pods|grep storage
glusterblock-storage-provisioner-dc-1-w66dm   1/1       Running   0          1d
glusterfs-storage-4fv4c                       1/1       Running   0          21h
glusterfs-storage-lbnv9                       1/1       Running   2          1d
glusterfs-storage-pfpsn                       1/1       Running   0          1d
glusterfs-storage-sdkqc                       1/1       Running   0          48m
heketi-storage-1-kp5kq                        1/1       Running   0          1d



Version-Release number of selected component (if applicable):
+++++++++++++++++++++++++++++++++++++++++++++++++

OC version = v3.11.16

Heketi version from heketi pod =
++++++++
sh-4.2# rpm -qa|grep heketi
heketi-client-7.0.0-14.el7rhgs.x86_64
heketi-7.0.0-14.el7rhgs.x86_64



Heketi client version from master node 
+++++
# rpm -qa|grep heketi
heketi-client-7.0.0-14.el7rhgs.x86_64
 

Gluster version
++++++

glusterfs-libs-3.12.2-18.1.el7rhgs.x86_64
glusterfs-3.12.2-18.1.el7rhgs.x86_64
glusterfs-api-3.12.2-18.1.el7rhgs.x86_64
python2-gluster-3.12.2-18.1.el7rhgs.x86_64
glusterfs-fuse-3.12.2-18.1.el7rhgs.x86_64
glusterfs-server-3.12.2-18.1.el7rhgs.x86_64
gluster-block-0.2.1-27.el7rhgs.x86_64
glusterfs-client-xlators-3.12.2-18.1.el7rhgs.x86_64
glusterfs-cli-3.12.2-18.1.el7rhgs.x86_64
glusterfs-geo-replication-3.12.2-18.1.el7rhgs.x86_64



sh-4.2# rpm -qa|grep tcmu-runner
tcmu-runner-1.2.0-25.el7rhgs.x86_64
sh-4.2# 




How reproducible:
++++++++++
2x2

Steps to Reproduce:
+++++++++++++++++
1. Create some block pvc , which in turn creates a heketi BHV and a heketi BVs
2. Start a for loop to create some more pvcs 
3. While pvc creation is not yet completed, kill the gluster-blockd process in 2 pods
4. The pvc creations go in pending state. Check heketi logs
5. Bring up the services in the pod - Mon Oct  1 12:03:53 UTC 2018
6. Check for mismatches and other issues. It is seen that ghost device IDs are listed in heketi-cli db dump and the db seems to have become inconsistent





Actual results:
+++++++++++

when we killed gluster-blockd on 2 pods(in a 4 node OCS cluster) while pvc creation(HA=3) is in progress(equivlent to having only 2 pods available for BV creation), the process resulted in a lot of mismatches and the content in heketi and gluster backend have become inconsistent.

Expected results:
++++++++++++

Till the time gluster-blockd was down, the pvc creations should stay in pending state. But once restored, the pending pvcs should have been created. No mismatch should have been seen in the gluster and hekti command outputs for both BHV and BVs.

In earlier heketi builds (before 7.0.0-11), similar test cases were run multiple times and none of these issues were seen.