Bug 1562485

Description of problem:
device remove operation hangs on a device which is removed from backend (virtual disk deleted on VMware env).

Refer steps to reproduce for the work flow.

Last successful operation seems to have happened an hour back. But, post this operation, the logs doesn't show any further operation.

[kubeexec] DEBUG 2018/03/31 09:20:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:244: Host: dhcp46-51.lab.eng.blr.redhat.com Pod: glusterfs-storage-npvzx Command: gluster --mode=script volume re
place-brick vol_bbef9b1e33a3c7c36552f1af5cd011c1 10.70.46.51:/var/lib/heketi/mounts/vg_12825d714782ee6542f226e69cfaa4f2/brick_026672f0fb485d2fc82e56c9d50a050a/brick 10.70.46.51:/var/lib/heketi/mounts/vg_654fd8c2
36965e6eaf4995a199315a12/brick_c7da79afa99bb587c42d612d7c3cd78c/brick commit force
Result: volume replace-brick: success: replace-brick commit force operation successful
[heketi] INFO 2018/03/31 09:20:52 Deleting brick 026672f0fb485d2fc82e56c9d50a050a


Version-Release number of selected component (if applicable):
rpm -qa | grep 'heketi'
heketi-6.0.0-7.1.el7rhgs.x86_64
python-heketi-6.0.0-7.1.el7rhgs.x86_64
heketi-client-6.0.0-7.1.el7rhgs.x86_64


How reproducible:
1/1

Steps to Reproduce:
1. keep creating volumes in a loop
2. while the creation is in progress, remove a disk on which volume is being created. [I removed the virtual disk from VMware env]
3. volume creation will fail
4. now, try to remove the device[12825d714782ee6542f226e69cfaa4f2] from heketi using heketi-cli device remove

Actual results:
device remove command is hung for more than an hour with no operations being performed by heketi (based on the log entries)

Expected results:
device remove should succeed

Additional info:
heketi logs shall be attached