Description of problem:

When heketi device remove command is run to remove a device, the command failed with an unexpected error message saying vg cannot be created due to insufficient space. This is not expected. 

Such error messages should be hidden from the user and a different device has to be identified to create bricks or remove device has to fail gracefully that there is no enough free space. In this case, there was enough space for the migration though.

                Id:75671a6256cac8d63fe5a19a0380f291   Name:/dev/sde            State:online    Size (GiB):199     Used (GiB):0       Free (GiB):199
                Id:f84d2265cb2bbc9e79c9c54bf962da5d   Name:/dev/sdd            State:online    Size (GiB):199     Used (GiB):195     Free (GiB):4
                Id:05f1db86a51e2eb2d0f1c67e77bc1def   Name:/dev/sdd            State:online    Size (GiB):199     Used (GiB):92      Free (GiB):107
                Id:39fc087fc120b9a8565fcfd06bef99e1   Name:/dev/sde            State:online    Size (GiB):199     Used (GiB):102     Free (GiB):97
                Id:cf96104a1630213d57e3529d33990926   Name:/dev/sdd            State:online    Size (GiB):199     Used (GiB):94      Free (GiB):105
                Id:fa3f91dd883ede7bfb40cbe514193715   Name:/dev/sde            State:online    Size (GiB):199     Used (GiB):100     Free (GiB):99
                Id:0ca7d795144a1f008ed3f6641b4d64f2   Name:/dev/sdd            State:online    Size (GiB):299     Used (GiB):34      Free (GiB):265
                Id:836af4c7981e140d7633eff47d7d8f85   Name:/dev/sde            State:online    Size (GiB):299     Used (GiB):275     Free (GiB):24
                Id:66db3b500225bf03d061a0efcae506bb   Name:/dev/sde            State:online    Size (GiB):299     Used (GiB):0       Free (GiB):299
                Id:727220b797fb8b1e83ae7632831e6692   Name:/dev/sdd            State:online    Size (GiB):299     Used (GiB):68      Free (GiB):231
                Id:6192b5e7021c765d30b32a081b063a34   Name:/dev/sde            State:online    Size (GiB):299     Used (GiB):223     Free (GiB):76
                Id:8d11bdd245f72a669bc29bbb3aa12c85   Name:/dev/sdd            State:online    Size (GiB):299     Used (GiB):90      Free (GiB):209


[root@dhcp46-202 claim_files]# heketi-cli device remove 836af4c7981e140d7633eff47d7d8f85
Error: Unable to execute command on glusterfs-pjl50:   Volume group "vg_66db3b500225bf03d061a0efcae506bb" has insufficient free space (25108 extents): 38400 required.

snippet of heketi log:

[kubeexec] ERROR 2017/03/09 15:14:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [lvcreate --poolmetadatasize 786432K -c 256K -L 157286400K -T vg_66db3b500225bf03d061a0efcae506bb/tp_45d71d15282b48c97f4bf3fe132ff4e6 -V 157286400K -n brick_45d71d15282b48c97f4bf3fe132ff4e6] on glusterfs-pjl50: Err[command terminated with exit code 5]: Stdout [  Using default stripesize 64.00 KiB.
]: Stderr [  Volume group "vg_66db3b500225bf03d061a0efcae506bb" has insufficient free space (25108 extents): 38400 required.
]
[kubeexec] ERROR 2017/03/09 15:14:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [umount /var/lib/heketi/mounts/vg_66db3b500225bf03d061a0efcae506bb/brick_45d71d15282b48c97f4bf3fe132ff4e6] on glusterfs-pjl50: Err[command terminated with exit code 32]: Stdout []: Stderr [umount: /var/lib/heketi/mounts/vg_66db3b500225bf03d061a0efcae506bb/brick_45d71d15282b48c97f4bf3fe132ff4e6: not mounted
]
[sshexec] ERROR 2017/03/09 15:14:45 /src/github.com/heketi/heketi/executors/sshexec/brick.go:134: Unable to execute command on glusterfs-pjl50: umount: /var/lib/heketi/mounts/vg_66db3b500225bf03d061a0efcae506bb/brick_45d71d15282b48c97f4bf3fe132ff4e6: not mounted
[kubeexec] ERROR 2017/03/09 15:14:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [lvremove -f vg_66db3b500225bf03d061a0efcae506bb/tp_45d71d15282b48c97f4bf3fe132ff4e6] on glusterfs-pjl50: Err[command terminated with exit code 5]: Stdout []: Stderr [  Failed to find logical volume "vg_66db3b500225bf03d061a0efcae506bb/tp_45d71d15282b48c97f4bf3fe132ff4e6"
]
[sshexec] ERROR 2017/03/09 15:14:45 /src/github.com/heketi/heketi/executors/sshexec/brick.go:143: Unable to execute command on glusterfs-pjl50:   Failed to find logical volume "vg_66db3b500225bf03d061a0efcae506bb/tp_45d71d15282b48c97f4bf3fe132ff4e6"
[heketi] ERROR 2017/03/09 15:14:46 /src/github.com/heketi/heketi/apps/glusterfs/brick_create.go:49: Unable to execute command on glusterfs-pjl50:   Volume group "vg_66db3b500225bf03d061a0efcae506bb" has insufficient free space (25108 extents): 38400 required.
[kubeexec] ERROR 2017/03/09 15:14:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [umount /var/lib/heketi/mounts/vg_66db3b500225bf03d061a0efcae506bb/brick_45d71d15282b48c97f4bf3fe132ff4e6] on glusterfs-pjl50: Err[command terminated with exit code 32]: Stdout []: Stderr [umount: /var/lib/heketi/mounts/vg_66db3b500225bf03d061a0efcae506bb/brick_45d71d15282b48c97f4bf3fe132ff4e6: mountpoint not found
]
[sshexec] ERROR 2017/03/09 15:14:46 /src/github.com/heketi/heketi/executors/sshexec/brick.go:134: Unable to execute command on glusterfs-pjl50: umount: /var/lib/heketi/mounts/vg_66db3b500225bf03d061a0efcae506bb/brick_45d71d15282b48c97f4bf3fe132ff4e6: mountpoint not found
[kubeexec] ERROR 2017/03/09 15:14:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [lvremove -f vg_66db3b500225bf03d061a0efcae506bb/tp_45d71d15282b48c97f4bf3fe132ff4e6] on glusterfs-pjl50: Err[command terminated with exit code 5]: Stdout []: Stderr [  Failed to find logical volume "vg_66db3b500225bf03d061a0efcae506bb/tp_45d71d15282b48c97f4bf3fe132ff4e6"
]
[sshexec] ERROR 2017/03/09 15:14:46 /src/github.com/heketi/heketi/executors/sshexec/brick.go:143: Unable to execute command on glusterfs-pjl50:   Failed to find logical volume "vg_66db3b500225bf03d061a0efcae506bb/tp_45d71d15282b48c97f4bf3fe132ff4e6"
[kubeexec] ERROR 2017/03/09 15:14:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [rmdir /var/lib/heketi/mounts/vg_66db3b500225bf03d061a0efcae506bb/brick_45d71d15282b48c97f4bf3fe132ff4e6] on glusterfs-pjl50: Err[command terminated with exit code 1]: Stdout []: Stderr [rmdir: failed to remove '/var/lib/heketi/mounts/vg_66db3b500225bf03d061a0efcae506bb/brick_45d71d15282b48c97f4bf3fe132ff4e6': No such file or directory
]
[sshexec] ERROR 2017/03/09 15:14:46 /src/github.com/heketi/heketi/executors/sshexec/brick.go:152: Unable to execute command on glusterfs-pjl50: rmdir: failed to remove '/var/lib/heketi/mounts/vg_66db3b500225bf03d061a0efcae506bb/brick_45d71d15282b48c97f4bf3fe132ff4e6': No such file or directory


Version-Release number of selected component (if applicable):

# rpm -qa | grep 'heketi'
heketi-client-4.0.0-1.el7rhgs.x86_64

How reproducible:
Yet to determine

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
heketi logs shall be attached