Created attachment 1265607 [details] heketi_logs Description of problem: When device remove command is run on a device whose node is not reachable, the command gets hung forever. There should be a timeout for such cases and cli should error out with a meaningful message. # heketi-cli device remove 0ce427039fd3bc8f03c43899383fc1ca [negroni] Started POST /devices/0ce427039fd3bc8f03c43899383fc1ca/state [negroni] Completed 202 Accepted in 487.498µs [asynchttp] INFO 2017/03/23 05:58:00 Started job 5ef8ee4fead4ec0f7b9d49cd218deec6 [heketi] INFO 2017/03/23 05:58:00 Replacing brick 2027c32314ae4316bd6877a7f2fc20ae on device 0ce427039fd3bc8f03c43899383fc1ca on node 76c04cd33916422802b3d14e6088ef75 [negroni] Started GET /queue/5ef8ee4fead4ec0f7b9d49cd218deec6 [negroni] Completed 200 OK in 101.92µs Version-Release number of selected component (if applicable): heketi-client-4.0.0-3.el7rhgs.x86_64 How reproducible: 1/1 Steps to Reproduce: 1. Have a 4 node CNS setup with some devices and volumes configured on these devices. For eg, node {1..4}, device {1..3} 2. suspend one of the VMs hosting the node. eg, suspend node 1 3. run remove device on node 1 device 1 which has volumes configured on it. Actual results: heketi-cli device remove command hangs Expected results: command should timeout with proper error message Additional info:
Karthick, whats the behaviour when you issue volume create..etc when one node is down?
This is not a regression. Unfortunately, kubeexec in heketi does not have a timeout and this behaviour will be seen in all ssh/gluster commands. Moving to next release.
Running device disable and device remove on a failed node now works. Moving the bug to verified. p.s., adding a new node to cns might fail when one of the node is down, filed a separate bug for this case --> https://bugzilla.redhat.com/show_bug.cgi?id=1441675
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:1111