Bug 1430794
| Summary: | unexpected error message seen while heketi device remove command is run - Volume group has insufficient free space | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | krishnaram Karthick <kramdoss> |
| Component: | heketi | Assignee: | Raghavendra Talur <rtalur> |
| Status: | CLOSED ERRATA | QA Contact: | krishnaram Karthick <kramdoss> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | cns-3.5 | CC: | hchiramm, mliyazud, pprakash, rhs-bugs, rtalur, storage-qa-internal, vinug |
| Target Milestone: | --- | ||
| Target Release: | CNS 3.5 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | heketi-4.0.0-3.el7rhgs.x86_64.rpm | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-04-20 18:27:11 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1415600 | ||
All the test cases were run on builds later than heketi-4.0.0-3.el7rhgs.x86_64.rpm and the issue reported in the bug was not seen during these runs. Hence, moving the bug to verified. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:1112 |
Description of problem: When heketi device remove command is run to remove a device, the command failed with an unexpected error message saying vg cannot be created due to insufficient space. This is not expected. Such error messages should be hidden from the user and a different device has to be identified to create bricks or remove device has to fail gracefully that there is no enough free space. In this case, there was enough space for the migration though. Id:75671a6256cac8d63fe5a19a0380f291 Name:/dev/sde State:online Size (GiB):199 Used (GiB):0 Free (GiB):199 Id:f84d2265cb2bbc9e79c9c54bf962da5d Name:/dev/sdd State:online Size (GiB):199 Used (GiB):195 Free (GiB):4 Id:05f1db86a51e2eb2d0f1c67e77bc1def Name:/dev/sdd State:online Size (GiB):199 Used (GiB):92 Free (GiB):107 Id:39fc087fc120b9a8565fcfd06bef99e1 Name:/dev/sde State:online Size (GiB):199 Used (GiB):102 Free (GiB):97 Id:cf96104a1630213d57e3529d33990926 Name:/dev/sdd State:online Size (GiB):199 Used (GiB):94 Free (GiB):105 Id:fa3f91dd883ede7bfb40cbe514193715 Name:/dev/sde State:online Size (GiB):199 Used (GiB):100 Free (GiB):99 Id:0ca7d795144a1f008ed3f6641b4d64f2 Name:/dev/sdd State:online Size (GiB):299 Used (GiB):34 Free (GiB):265 Id:836af4c7981e140d7633eff47d7d8f85 Name:/dev/sde State:online Size (GiB):299 Used (GiB):275 Free (GiB):24 Id:66db3b500225bf03d061a0efcae506bb Name:/dev/sde State:online Size (GiB):299 Used (GiB):0 Free (GiB):299 Id:727220b797fb8b1e83ae7632831e6692 Name:/dev/sdd State:online Size (GiB):299 Used (GiB):68 Free (GiB):231 Id:6192b5e7021c765d30b32a081b063a34 Name:/dev/sde State:online Size (GiB):299 Used (GiB):223 Free (GiB):76 Id:8d11bdd245f72a669bc29bbb3aa12c85 Name:/dev/sdd State:online Size (GiB):299 Used (GiB):90 Free (GiB):209 [root@dhcp46-202 claim_files]# heketi-cli device remove 836af4c7981e140d7633eff47d7d8f85 Error: Unable to execute command on glusterfs-pjl50: Volume group "vg_66db3b500225bf03d061a0efcae506bb" has insufficient free space (25108 extents): 38400 required. snippet of heketi log: [kubeexec] ERROR 2017/03/09 15:14:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [lvcreate --poolmetadatasize 786432K -c 256K -L 157286400K -T vg_66db3b500225bf03d061a0efcae506bb/tp_45d71d15282b48c97f4bf3fe132ff4e6 -V 157286400K -n brick_45d71d15282b48c97f4bf3fe132ff4e6] on glusterfs-pjl50: Err[command terminated with exit code 5]: Stdout [ Using default stripesize 64.00 KiB. ]: Stderr [ Volume group "vg_66db3b500225bf03d061a0efcae506bb" has insufficient free space (25108 extents): 38400 required. ] [kubeexec] ERROR 2017/03/09 15:14:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [umount /var/lib/heketi/mounts/vg_66db3b500225bf03d061a0efcae506bb/brick_45d71d15282b48c97f4bf3fe132ff4e6] on glusterfs-pjl50: Err[command terminated with exit code 32]: Stdout []: Stderr [umount: /var/lib/heketi/mounts/vg_66db3b500225bf03d061a0efcae506bb/brick_45d71d15282b48c97f4bf3fe132ff4e6: not mounted ] [sshexec] ERROR 2017/03/09 15:14:45 /src/github.com/heketi/heketi/executors/sshexec/brick.go:134: Unable to execute command on glusterfs-pjl50: umount: /var/lib/heketi/mounts/vg_66db3b500225bf03d061a0efcae506bb/brick_45d71d15282b48c97f4bf3fe132ff4e6: not mounted [kubeexec] ERROR 2017/03/09 15:14:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [lvremove -f vg_66db3b500225bf03d061a0efcae506bb/tp_45d71d15282b48c97f4bf3fe132ff4e6] on glusterfs-pjl50: Err[command terminated with exit code 5]: Stdout []: Stderr [ Failed to find logical volume "vg_66db3b500225bf03d061a0efcae506bb/tp_45d71d15282b48c97f4bf3fe132ff4e6" ] [sshexec] ERROR 2017/03/09 15:14:45 /src/github.com/heketi/heketi/executors/sshexec/brick.go:143: Unable to execute command on glusterfs-pjl50: Failed to find logical volume "vg_66db3b500225bf03d061a0efcae506bb/tp_45d71d15282b48c97f4bf3fe132ff4e6" [heketi] ERROR 2017/03/09 15:14:46 /src/github.com/heketi/heketi/apps/glusterfs/brick_create.go:49: Unable to execute command on glusterfs-pjl50: Volume group "vg_66db3b500225bf03d061a0efcae506bb" has insufficient free space (25108 extents): 38400 required. [kubeexec] ERROR 2017/03/09 15:14:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [umount /var/lib/heketi/mounts/vg_66db3b500225bf03d061a0efcae506bb/brick_45d71d15282b48c97f4bf3fe132ff4e6] on glusterfs-pjl50: Err[command terminated with exit code 32]: Stdout []: Stderr [umount: /var/lib/heketi/mounts/vg_66db3b500225bf03d061a0efcae506bb/brick_45d71d15282b48c97f4bf3fe132ff4e6: mountpoint not found ] [sshexec] ERROR 2017/03/09 15:14:46 /src/github.com/heketi/heketi/executors/sshexec/brick.go:134: Unable to execute command on glusterfs-pjl50: umount: /var/lib/heketi/mounts/vg_66db3b500225bf03d061a0efcae506bb/brick_45d71d15282b48c97f4bf3fe132ff4e6: mountpoint not found [kubeexec] ERROR 2017/03/09 15:14:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [lvremove -f vg_66db3b500225bf03d061a0efcae506bb/tp_45d71d15282b48c97f4bf3fe132ff4e6] on glusterfs-pjl50: Err[command terminated with exit code 5]: Stdout []: Stderr [ Failed to find logical volume "vg_66db3b500225bf03d061a0efcae506bb/tp_45d71d15282b48c97f4bf3fe132ff4e6" ] [sshexec] ERROR 2017/03/09 15:14:46 /src/github.com/heketi/heketi/executors/sshexec/brick.go:143: Unable to execute command on glusterfs-pjl50: Failed to find logical volume "vg_66db3b500225bf03d061a0efcae506bb/tp_45d71d15282b48c97f4bf3fe132ff4e6" [kubeexec] ERROR 2017/03/09 15:14:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [rmdir /var/lib/heketi/mounts/vg_66db3b500225bf03d061a0efcae506bb/brick_45d71d15282b48c97f4bf3fe132ff4e6] on glusterfs-pjl50: Err[command terminated with exit code 1]: Stdout []: Stderr [rmdir: failed to remove '/var/lib/heketi/mounts/vg_66db3b500225bf03d061a0efcae506bb/brick_45d71d15282b48c97f4bf3fe132ff4e6': No such file or directory ] [sshexec] ERROR 2017/03/09 15:14:46 /src/github.com/heketi/heketi/executors/sshexec/brick.go:152: Unable to execute command on glusterfs-pjl50: rmdir: failed to remove '/var/lib/heketi/mounts/vg_66db3b500225bf03d061a0efcae506bb/brick_45d71d15282b48c97f4bf3fe132ff4e6': No such file or directory Version-Release number of selected component (if applicable): # rpm -qa | grep 'heketi' heketi-client-4.0.0-1.el7rhgs.x86_64 How reproducible: Yet to determine Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: heketi logs shall be attached