Description of problem:
=======================

Topology creation through heketi works for only 1 node. Either we have to reboot the client or the atomic hosts to get this works. Sometimes even this doesn't work.

Output:

[root@dhcp43-179 ~]# heketi-cli topology load --json=topology.json
Creating cluster ... ID: c8b9bbbd9fba899f79ca6c3347002bcf
	Creating node dhcp42-228.lab.eng.blr.redhat.com ... ID: 110d1469a207842e5bfec22c544c0144
		Adding device /dev/vdb ... [root@dhcp43-179 ~]# 
[root@dhcp43-179 ~]# 
[root@dhcp43-179 ~]# cat topology.json 
{
    "clusters": [
        {
            "nodes": [
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "dhcp42-228.lab.eng.blr.redhat.com"
                            ],
                            "storage": [
                                "10.70.42.228"
                            ]
                        },
                        "zone": 1
                    },
                    "devices": [
                        "/dev/vdb"
                    ]
                },
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "dhcp42-74.lab.eng.blr.redhat.com"
                            ],
                            "storage": [
                                "10.70.42.74"
                            ]
                        },
                        "zone": 1
                    },
                    "devices": [
                        "/dev/vdb"
                    ]
                },
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "dhcp43-207.lab.eng.blr.redhat.com"
                            ],
                            "storage": [
                                "10.70.43.207"
                            ]
                        },
                        "zone": 1
                    },
                    "devices": [
                        "/dev/vdb"
                    ]
                }
            ]
        }
    ]
}
[root@dhcp43-179 ~]# 


Docker logs:
============
-bash-4.2# docker logs -f 56ca62db40c2
Heketi 2.0.2
[heketi] INFO 2016/06/30 01:53:25 Loaded kubernetes executor
[heketi] INFO 2016/06/30 01:53:25 Loaded simple allocator
[heketi] INFO 2016/06/30 01:53:25 GlusterFS Application Loaded
Listening on port 8080
[negroni] Started POST /clusters
[negroni] Completed 201 Created in 3.28506ms
[negroni] Started POST /nodes
[heketi] INFO 2016/06/30 01:56:18 Adding node dhcp42-228.lab.eng.blr.redhat.com
[negroni] Completed 202 Accepted in 1.84486ms
[asynchttp] INFO 2016/06/30 01:56:18 Started job e2292d5d7d355f844aa9581fed5487f3
[negroni] Started GET /queue/e2292d5d7d355f844aa9581fed5487f3
[negroni] Completed 200 OK in 56.788µs
[heketi] INFO 2016/06/30 01:56:18 Added node 110d1469a207842e5bfec22c544c0144
[asynchttp] INFO 2016/06/30 01:56:18 Completed job e2292d5d7d355f844aa9581fed5487f3 in 2.443797ms
[negroni] Started GET /queue/e2292d5d7d355f844aa9581fed5487f3
[negroni] Completed 303 See Other in 73.192µs
[negroni] Started GET /nodes/110d1469a207842e5bfec22c544c0144
[negroni] Completed 200 OK in 426.178µs
[negroni] Started POST /devices
[heketi] INFO 2016/06/30 01:56:18 Adding device /dev/vdb to node 110d1469a207842e5bfec22c544c0144
[negroni] Completed 202 Accepted in 1.263767ms
[asynchttp] INFO 2016/06/30 01:56:18 Started job bc9e36d42988b4f2f51c7d36e6f83402
[negroni] Started GET /queue/bc9e36d42988b4f2f51c7d36e6f83402
[negroni] Completed 200 OK in 38.628µs
[kubeexec] ERROR 2016/06/30 01:56:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:204: dial tcp 10.70.42.228:8443: getsockopt: no route to host
[asynchttp] INFO 2016/06/30 01:56:18 Completed job bc9e36d42988b4f2f51c7d36e6f83402 in 2.555347ms
[negroni] Started GET /queue/bc9e36d42988b4f2f51c7d36e6f83402
[negroni] Completed 500 Internal Server Error in 52.55µs


Version-Release number of selected component (if applicable):
============================================================
OSE- 3.2
Atomic - 7.2.5

How reproducible:
=================
100%

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info: