Bug 1295431

Summary: Heketi on RHEL6: Device add operation crashes heketi server
Product: Red Hat Gluster Storage Reporter: Anush Shetty <ashetty>
Component: heketiAssignee: Luis Pabón <lpabon>
Status: CLOSED NOTABUG QA Contact: Anush Shetty <ashetty>
Severity: high Docs Contact:
Priority: unspecified    
Version: rhgs-3.1CC: ashetty, byarlaga, madam, rcyriac, vagarwal
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-01-07 09:28:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Anush Shetty 2016-01-04 13:22:46 UTC 
Description of problem: While running Heketi on RHEL6, adding a device to a Heketi cluster crashes the Heketi server. 


Version-Release number of selected component (if applicable): heketi-1.0.2-1.el6rhs.x86_64


How reproducible: Always


Steps to Reproduce:
1. Create a cluster, add a node, and then try add a device to the node
2.
3.

Actual results:

# heketi-cli device add -name=/dev/vdb -node=78cc8391d136fbaafc45e8c0c15563d6
Error: Get http://localhost:8081/queue/b7964d57ce86a1b68c2fb662ff54fb18: read tcp 127.0.0.1:8081: connection reset by peer

Additional info:

From heketi log:

[heketi] INFO 2016/01/05 02:52:56 Adding device /dev/vdb to node 78cc8391d136fbaafc45e8c0c15563d6
[negroni] Completed 202 Accepted in 1.867598ms
[asynchttp] INFO 2016/01/05 02:52:56 Started job b7964d57ce86a1b68c2fb662ff54fb18
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x5ba2e0]

goroutine 8 [running]:
github.com/heketi/heketi/executors/sshexec.(*SshExecutor).accessConnection(0x0, 0xc208096ea0, 0x1f)
        /builddir/build/BUILD/heketi-6563551111f7178b679866e85a0682325929d037/src/github.com/heketi/heketi/executors/sshexec/sshexec.go:97 +0x150
github.com/heketi/heketi/executors/sshexec.(*SshExecutor).sshExec(0x0, 0xc208096ea0, 0x1f, 0xc2080afd40, 0x2, 0x2, 0x5, 0x0, 0x0, 0x0, ...)
        /builddir/build/BUILD/heketi-6563551111f7178b679866e85a0682325929d037/src/github.com/heketi/heketi/executors/sshexec/sshexec.go:118 +0x89
github.com/heketi/heketi/executors/sshexec.(*SshExecutor).DeviceSetup(0x0, 0xc208096ea0, 0x1f, 0xc2080a4a70, 0x8, 0xc208097320, 0x20, 0x0, 0x0, 0x0)
        /builddir/build/BUILD/heketi-6563551111f7178b679866e85a0682325929d037/src/github.com/heketi/heketi/executors/sshexec/device.go:48 +0x470
github.com/heketi/heketi/apps/glusterfs.func·015(0x0, 0x0, 0x0, 0x0)
        /builddir/build/BUILD/heketi-6563551111f7178b679866e85a0682325929d037/src/github.com/heketi/heketi/apps/glusterfs/app_device.go:72 +0x159
github.com/heketi/heketi/rest.func·001()
        /builddir/build/BUILD/heketi-6563551111f7178b679866e85a0682325929d037/src/github.com/heketi/heketi/rest/asynchttp.go:128 +0x157
created by github.com/heketi/heketi/rest.(*AsyncHttpManager).AsyncHttpRedirectFunc
        /builddir/build/BUILD/heketi-6563551111f7178b679866e85a0682325929d037/src/github.com/heketi/heketi/rest/asynchttp.go:138 +0xfb

goroutine 1 [select]:
main.main()
        /builddir/build/BUILD/heketi-6563551111f7178b679866e85a0682325929d037/main.go:153 +0x1463
Comment 2 Luis Pabón 2016-01-05 05:07:19 UTC 
I'll take a look.
Comment 3 Luis Pabón 2016-01-05 06:21:34 UTC 
This is really weird.  The panic points to https://github.com/heketi/heketi/blob/master/executors/sshexec/sshexec.go#L97 which cannot be null.  It really is starting to feel like a compiler bug, but before I continue, I will try to reproduce.  It looks like rhel6 is using golang 1.3.3.
Comment 13 Anush Shetty 2016-01-07 07:11:49 UTC 
I reinstalled my RHEL6 VM, and now I am unable to reproduce it either. Will keep this bug open for a while until I try it for a few more times.
Comment 14 Anush Shetty 2016-01-07 09:28:14 UTC 
I haven't been able to reproduce the issue on a fresh RHEL6 machine. Closing this bug.