Bug 1435097
| Summary: | device remove command hangs when the node on which the device resides is not reachable | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | krishnaram Karthick <kramdoss> | ||||
| Component: | heketi | Assignee: | Raghavendra Talur <rtalur> | ||||
| Status: | CLOSED ERRATA | QA Contact: | krishnaram Karthick <kramdoss> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | cns-3.5 | CC: | hchiramm, kramdoss, madam, mliyazud, rhs-bugs, storage-qa-internal, vinug | ||||
| Target Milestone: | --- | ||||||
| Target Release: | CNS 3.5 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | heketi-4.0.0-6.el7rhgs | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2017-04-20 18:38:24 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1415600 | ||||||
| Attachments: |
|
||||||
Karthick, whats the behaviour when you issue volume create..etc when one node is down? This is not a regression. Unfortunately, kubeexec in heketi does not have a timeout and this behaviour will be seen in all ssh/gluster commands. Moving to next release. Running device disable and device remove on a failed node now works. Moving the bug to verified. p.s., adding a new node to cns might fail when one of the node is down, filed a separate bug for this case --> https://bugzilla.redhat.com/show_bug.cgi?id=1441675 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:1111 |
Created attachment 1265607 [details] heketi_logs Description of problem: When device remove command is run on a device whose node is not reachable, the command gets hung forever. There should be a timeout for such cases and cli should error out with a meaningful message. # heketi-cli device remove 0ce427039fd3bc8f03c43899383fc1ca [negroni] Started POST /devices/0ce427039fd3bc8f03c43899383fc1ca/state [negroni] Completed 202 Accepted in 487.498µs [asynchttp] INFO 2017/03/23 05:58:00 Started job 5ef8ee4fead4ec0f7b9d49cd218deec6 [heketi] INFO 2017/03/23 05:58:00 Replacing brick 2027c32314ae4316bd6877a7f2fc20ae on device 0ce427039fd3bc8f03c43899383fc1ca on node 76c04cd33916422802b3d14e6088ef75 [negroni] Started GET /queue/5ef8ee4fead4ec0f7b9d49cd218deec6 [negroni] Completed 200 OK in 101.92µs Version-Release number of selected component (if applicable): heketi-client-4.0.0-3.el7rhgs.x86_64 How reproducible: 1/1 Steps to Reproduce: 1. Have a 4 node CNS setup with some devices and volumes configured on these devices. For eg, node {1..4}, device {1..3} 2. suspend one of the VMs hosting the node. eg, suspend node 1 3. run remove device on node 1 device 1 which has volumes configured on it. Actual results: heketi-cli device remove command hangs Expected results: command should timeout with proper error message Additional info: