Bug 1590143
| Summary: | heketi device delete command hanged (as a vgremove command was waiting for an interactive prompt response) | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Apeksha <akhakhar> | ||||||
| Component: | heketi | Assignee: | John Mulligan <jmulligan> | ||||||
| Status: | CLOSED ERRATA | QA Contact: | Apeksha <akhakhar> | ||||||
| Severity: | high | Docs Contact: | |||||||
| Priority: | unspecified | ||||||||
| Version: | cns-3.10 | CC: | akrishna, hchiramm, jmulligan, kramdoss, pprakash, rhs-bugs, rtalur, sankarshan, storage-qa-internal, vinug | ||||||
| Target Milestone: | --- | ||||||||
| Target Release: | CNS 3.10 | ||||||||
| Hardware: | x86_64 | ||||||||
| OS: | Linux | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||
| Doc Text: |
Previously, Heketi executed LVM management operations on the underlying storage nodes such that the commands might prompt for input. Commands that triggered input prompts would never complete without administrative intervention. With this fix Heketi is changed to run the LVM operations for volume delete and will not prompt for input. Heketi operations that manage LVM storage will either succeed or fail .
|
Story Points: | --- | ||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2018-09-12 09:22:13 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Bug Depends On: | |||||||||
| Bug Blocks: | 1568862, 1594716 | ||||||||
| Attachments: |
|
||||||||
|
Description
Apeksha
2018-06-12 06:23:28 UTC
Created attachment 1450325 [details]
attaching heketi-pod logs
Created attachment 1450328 [details]
heketi-db-dump
heketi-pod logs at http://rhsqe-repo.lab.eng.blr.redhat.com/cns/bugs/BZ-1590143/ How long did the command hang for? Did it eventually time out or did you kill it?
Can we get a system into this state again? If we can I'd like to get a core dump of a running heketi to see exactly what operation the server was performing. In the attached db dump there are no pending operations in the db, which implies that either the server completed the operation before the db dump was taken, the cli never actually communicated the delete operation to the server, or the db was cleaned up manually and I doubt you would have done that.
The error triggering the log line:
[cmdexec] ERROR 2018/06/11 11:26:02 /src/github.com/heketi/heketi/executors/cmdexec/brick.go:123: Unable to execute command on glusterfs-storage-q25zn: umount: /var/lib/heketi/mounts/vg_e7a2abd8e004058324ce78235ae03927/brick_284a16bbc0e4bee4a3a124a0db224c8f: target is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
is an issue in gluster AFAIK around brick multiplexing (bz#1524336 if I'm not mistaken). However, this alone should not cause the command to hang.
(In reply to John Mulligan from comment #6) > How long did the command hang for? Did it eventually time out or did you > kill it? It hangs for more than 48 hours and then eventually i killed it. > Can we get a system into this state again? If we can I'd like to get a core > dump of a running heketi to see exactly what operation the server was > performing. In the attached db dump there are no pending operations in the > db, which implies that either the server completed the operation before the > db dump was taken, the cli never actually communicated the delete operation > to the server, or the db was cleaned up manually and I doubt you would have > done that. I dont have the setup now, but I have never manually cleared the db. Note i haven't seen this issue in CNS3.10. > The error triggering the log line: > [cmdexec] ERROR 2018/06/11 11:26:02 > /src/github.com/heketi/heketi/executors/cmdexec/brick.go:123: Unable to > execute command on glusterfs-storage-q25zn: umount: > /var/lib/heketi/mounts/vg_e7a2abd8e004058324ce78235ae03927/ > brick_284a16bbc0e4bee4a3a124a0db224c8f: target is busy. > (In some cases useful info about processes that use > the device is found by lsof(8) or fuser(1)) > > is an issue in gluster AFAIK around brick multiplexing (bz#1524336 if I'm > not mistaken). However, this alone should not cause the command to hang. Patch posted at : https://github.com/heketi/heketi/pull/1247 Fixed in version: rhgs-volmanager-rhel7:3.3.1-20 Havent hit this issue again in build -rhgs3/rhgs-volmanager-rhel7:3.3.1-22. Hence marking it as verified. Updated doc text in the Doc Text field. Please review for technical accuracy. have updated the doc text based on feedback. (In reply to Anjana from comment #22) > have updated the doc text based on feedback. Looks OK to me. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:2686 |