Bug 1430794 - unexpected error message seen while heketi device remove command is run - Volume group has insufficient free space
Summary: unexpected error message seen while heketi device remove command is run - Vol...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: heketi
Version: cns-3.5
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: CNS 3.5
Assignee: Raghavendra Talur
QA Contact: krishnaram Karthick
URL:
Whiteboard:
Depends On:
Blocks: 1415600
TreeView+ depends on / blocked
 
Reported: 2017-03-09 15:41 UTC by krishnaram Karthick
Modified: 2019-04-22 22:40 UTC (History)
7 users (show)

Fixed In Version: heketi-4.0.0-3.el7rhgs.x86_64.rpm
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-04-20 18:27:11 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2017:1112 normal SHIPPED_LIVE cns-deploy-tool bug fix and enhancement update 2017-04-20 22:25:47 UTC

Description krishnaram Karthick 2017-03-09 15:41:32 UTC
Description of problem:

When heketi device remove command is run to remove a device, the command failed with an unexpected error message saying vg cannot be created due to insufficient space. This is not expected. 

Such error messages should be hidden from the user and a different device has to be identified to create bricks or remove device has to fail gracefully that there is no enough free space. In this case, there was enough space for the migration though.

                Id:75671a6256cac8d63fe5a19a0380f291   Name:/dev/sde            State:online    Size (GiB):199     Used (GiB):0       Free (GiB):199
                Id:f84d2265cb2bbc9e79c9c54bf962da5d   Name:/dev/sdd            State:online    Size (GiB):199     Used (GiB):195     Free (GiB):4
                Id:05f1db86a51e2eb2d0f1c67e77bc1def   Name:/dev/sdd            State:online    Size (GiB):199     Used (GiB):92      Free (GiB):107
                Id:39fc087fc120b9a8565fcfd06bef99e1   Name:/dev/sde            State:online    Size (GiB):199     Used (GiB):102     Free (GiB):97
                Id:cf96104a1630213d57e3529d33990926   Name:/dev/sdd            State:online    Size (GiB):199     Used (GiB):94      Free (GiB):105
                Id:fa3f91dd883ede7bfb40cbe514193715   Name:/dev/sde            State:online    Size (GiB):199     Used (GiB):100     Free (GiB):99
                Id:0ca7d795144a1f008ed3f6641b4d64f2   Name:/dev/sdd            State:online    Size (GiB):299     Used (GiB):34      Free (GiB):265
                Id:836af4c7981e140d7633eff47d7d8f85   Name:/dev/sde            State:online    Size (GiB):299     Used (GiB):275     Free (GiB):24
                Id:66db3b500225bf03d061a0efcae506bb   Name:/dev/sde            State:online    Size (GiB):299     Used (GiB):0       Free (GiB):299
                Id:727220b797fb8b1e83ae7632831e6692   Name:/dev/sdd            State:online    Size (GiB):299     Used (GiB):68      Free (GiB):231
                Id:6192b5e7021c765d30b32a081b063a34   Name:/dev/sde            State:online    Size (GiB):299     Used (GiB):223     Free (GiB):76
                Id:8d11bdd245f72a669bc29bbb3aa12c85   Name:/dev/sdd            State:online    Size (GiB):299     Used (GiB):90      Free (GiB):209


[root@dhcp46-202 claim_files]# heketi-cli device remove 836af4c7981e140d7633eff47d7d8f85
Error: Unable to execute command on glusterfs-pjl50:   Volume group "vg_66db3b500225bf03d061a0efcae506bb" has insufficient free space (25108 extents): 38400 required.

snippet of heketi log:

[kubeexec] ERROR 2017/03/09 15:14:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [lvcreate --poolmetadatasize 786432K -c 256K -L 157286400K -T vg_66db3b500225bf03d061a0efcae506bb/tp_45d71d15282b48c97f4bf3fe132ff4e6 -V 157286400K -n brick_45d71d15282b48c97f4bf3fe132ff4e6] on glusterfs-pjl50: Err[command terminated with exit code 5]: Stdout [  Using default stripesize 64.00 KiB.
]: Stderr [  Volume group "vg_66db3b500225bf03d061a0efcae506bb" has insufficient free space (25108 extents): 38400 required.
]
[kubeexec] ERROR 2017/03/09 15:14:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [umount /var/lib/heketi/mounts/vg_66db3b500225bf03d061a0efcae506bb/brick_45d71d15282b48c97f4bf3fe132ff4e6] on glusterfs-pjl50: Err[command terminated with exit code 32]: Stdout []: Stderr [umount: /var/lib/heketi/mounts/vg_66db3b500225bf03d061a0efcae506bb/brick_45d71d15282b48c97f4bf3fe132ff4e6: not mounted
]
[sshexec] ERROR 2017/03/09 15:14:45 /src/github.com/heketi/heketi/executors/sshexec/brick.go:134: Unable to execute command on glusterfs-pjl50: umount: /var/lib/heketi/mounts/vg_66db3b500225bf03d061a0efcae506bb/brick_45d71d15282b48c97f4bf3fe132ff4e6: not mounted
[kubeexec] ERROR 2017/03/09 15:14:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [lvremove -f vg_66db3b500225bf03d061a0efcae506bb/tp_45d71d15282b48c97f4bf3fe132ff4e6] on glusterfs-pjl50: Err[command terminated with exit code 5]: Stdout []: Stderr [  Failed to find logical volume "vg_66db3b500225bf03d061a0efcae506bb/tp_45d71d15282b48c97f4bf3fe132ff4e6"
]
[sshexec] ERROR 2017/03/09 15:14:45 /src/github.com/heketi/heketi/executors/sshexec/brick.go:143: Unable to execute command on glusterfs-pjl50:   Failed to find logical volume "vg_66db3b500225bf03d061a0efcae506bb/tp_45d71d15282b48c97f4bf3fe132ff4e6"
[heketi] ERROR 2017/03/09 15:14:46 /src/github.com/heketi/heketi/apps/glusterfs/brick_create.go:49: Unable to execute command on glusterfs-pjl50:   Volume group "vg_66db3b500225bf03d061a0efcae506bb" has insufficient free space (25108 extents): 38400 required.
[kubeexec] ERROR 2017/03/09 15:14:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [umount /var/lib/heketi/mounts/vg_66db3b500225bf03d061a0efcae506bb/brick_45d71d15282b48c97f4bf3fe132ff4e6] on glusterfs-pjl50: Err[command terminated with exit code 32]: Stdout []: Stderr [umount: /var/lib/heketi/mounts/vg_66db3b500225bf03d061a0efcae506bb/brick_45d71d15282b48c97f4bf3fe132ff4e6: mountpoint not found
]
[sshexec] ERROR 2017/03/09 15:14:46 /src/github.com/heketi/heketi/executors/sshexec/brick.go:134: Unable to execute command on glusterfs-pjl50: umount: /var/lib/heketi/mounts/vg_66db3b500225bf03d061a0efcae506bb/brick_45d71d15282b48c97f4bf3fe132ff4e6: mountpoint not found
[kubeexec] ERROR 2017/03/09 15:14:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [lvremove -f vg_66db3b500225bf03d061a0efcae506bb/tp_45d71d15282b48c97f4bf3fe132ff4e6] on glusterfs-pjl50: Err[command terminated with exit code 5]: Stdout []: Stderr [  Failed to find logical volume "vg_66db3b500225bf03d061a0efcae506bb/tp_45d71d15282b48c97f4bf3fe132ff4e6"
]
[sshexec] ERROR 2017/03/09 15:14:46 /src/github.com/heketi/heketi/executors/sshexec/brick.go:143: Unable to execute command on glusterfs-pjl50:   Failed to find logical volume "vg_66db3b500225bf03d061a0efcae506bb/tp_45d71d15282b48c97f4bf3fe132ff4e6"
[kubeexec] ERROR 2017/03/09 15:14:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [rmdir /var/lib/heketi/mounts/vg_66db3b500225bf03d061a0efcae506bb/brick_45d71d15282b48c97f4bf3fe132ff4e6] on glusterfs-pjl50: Err[command terminated with exit code 1]: Stdout []: Stderr [rmdir: failed to remove '/var/lib/heketi/mounts/vg_66db3b500225bf03d061a0efcae506bb/brick_45d71d15282b48c97f4bf3fe132ff4e6': No such file or directory
]
[sshexec] ERROR 2017/03/09 15:14:46 /src/github.com/heketi/heketi/executors/sshexec/brick.go:152: Unable to execute command on glusterfs-pjl50: rmdir: failed to remove '/var/lib/heketi/mounts/vg_66db3b500225bf03d061a0efcae506bb/brick_45d71d15282b48c97f4bf3fe132ff4e6': No such file or directory


Version-Release number of selected component (if applicable):

# rpm -qa | grep 'heketi'
heketi-client-4.0.0-1.el7rhgs.x86_64

How reproducible:
Yet to determine

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
heketi logs shall be attached

Comment 8 krishnaram Karthick 2017-04-10 08:27:35 UTC
All the test cases were run on builds later than heketi-4.0.0-3.el7rhgs.x86_64.rpm and the issue reported in the bug was not seen during these runs. Hence, moving the bug to verified.

Comment 9 errata-xmlrpc 2017-04-20 18:27:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1112


Note You need to log in before you can comment on or make changes to this bug.