Bug 1351439

Summary: heketi topology doesn't load for all the nodes
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Bhaskarakiran <byarlaga>
Component: heketiAssignee: Luis Pabón <lpabon>
Status: CLOSED DUPLICATE QA Contact: Jonathan Holloway <jholloway>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: unspecifiedCC: byarlaga, hchiramm, madam, mzywusko
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-07-06 16:42:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Bhaskarakiran 2016-06-30 05:57:58 UTC
Description of problem:
=======================

Topology creation through heketi works for only 1 node. Either we have to reboot the client or the atomic hosts to get this works. Sometimes even this doesn't work.

Output:

[root@dhcp43-179 ~]# heketi-cli topology load --json=topology.json
Creating cluster ... ID: c8b9bbbd9fba899f79ca6c3347002bcf
	Creating node dhcp42-228.lab.eng.blr.redhat.com ... ID: 110d1469a207842e5bfec22c544c0144
		Adding device /dev/vdb ... [root@dhcp43-179 ~]# 
[root@dhcp43-179 ~]# 
[root@dhcp43-179 ~]# cat topology.json 
{
    "clusters": [
        {
            "nodes": [
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "dhcp42-228.lab.eng.blr.redhat.com"
                            ],
                            "storage": [
                                "10.70.42.228"
                            ]
                        },
                        "zone": 1
                    },
                    "devices": [
                        "/dev/vdb"
                    ]
                },
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "dhcp42-74.lab.eng.blr.redhat.com"
                            ],
                            "storage": [
                                "10.70.42.74"
                            ]
                        },
                        "zone": 1
                    },
                    "devices": [
                        "/dev/vdb"
                    ]
                },
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "dhcp43-207.lab.eng.blr.redhat.com"
                            ],
                            "storage": [
                                "10.70.43.207"
                            ]
                        },
                        "zone": 1
                    },
                    "devices": [
                        "/dev/vdb"
                    ]
                }
            ]
        }
    ]
}
[root@dhcp43-179 ~]# 


Docker logs:
============
-bash-4.2# docker logs -f 56ca62db40c2
Heketi 2.0.2
[heketi] INFO 2016/06/30 01:53:25 Loaded kubernetes executor
[heketi] INFO 2016/06/30 01:53:25 Loaded simple allocator
[heketi] INFO 2016/06/30 01:53:25 GlusterFS Application Loaded
Listening on port 8080
[negroni] Started POST /clusters
[negroni] Completed 201 Created in 3.28506ms
[negroni] Started POST /nodes
[heketi] INFO 2016/06/30 01:56:18 Adding node dhcp42-228.lab.eng.blr.redhat.com
[negroni] Completed 202 Accepted in 1.84486ms
[asynchttp] INFO 2016/06/30 01:56:18 Started job e2292d5d7d355f844aa9581fed5487f3
[negroni] Started GET /queue/e2292d5d7d355f844aa9581fed5487f3
[negroni] Completed 200 OK in 56.788µs
[heketi] INFO 2016/06/30 01:56:18 Added node 110d1469a207842e5bfec22c544c0144
[asynchttp] INFO 2016/06/30 01:56:18 Completed job e2292d5d7d355f844aa9581fed5487f3 in 2.443797ms
[negroni] Started GET /queue/e2292d5d7d355f844aa9581fed5487f3
[negroni] Completed 303 See Other in 73.192µs
[negroni] Started GET /nodes/110d1469a207842e5bfec22c544c0144
[negroni] Completed 200 OK in 426.178µs
[negroni] Started POST /devices
[heketi] INFO 2016/06/30 01:56:18 Adding device /dev/vdb to node 110d1469a207842e5bfec22c544c0144
[negroni] Completed 202 Accepted in 1.263767ms
[asynchttp] INFO 2016/06/30 01:56:18 Started job bc9e36d42988b4f2f51c7d36e6f83402
[negroni] Started GET /queue/bc9e36d42988b4f2f51c7d36e6f83402
[negroni] Completed 200 OK in 38.628µs
[kubeexec] ERROR 2016/06/30 01:56:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:204: dial tcp 10.70.42.228:8443: getsockopt: no route to host
[asynchttp] INFO 2016/06/30 01:56:18 Completed job bc9e36d42988b4f2f51c7d36e6f83402 in 2.555347ms
[negroni] Started GET /queue/bc9e36d42988b4f2f51c7d36e6f83402
[negroni] Completed 500 Internal Server Error in 52.55µs


Version-Release number of selected component (if applicable):
============================================================
OSE- 3.2
Atomic - 7.2.5

How reproducible:
=================
100%

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 2 Humble Chirammal 2016-07-04 17:44:51 UTC
@Bhaskarakiran, unfortunately I was not able to understand this bug report. Can you please summarize the issue?

Comment 3 Humble Chirammal 2016-07-05 08:20:29 UTC
From the error logs, I can see :

---

[kubeexec] ERROR 2016/06/30 01:56:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:204: dial tcp 10.70.42.228:8443: getsockopt: no route to host
[asynchttp] INFO 2016/06/30 01:56:18 Completed job

---

iic, dhcp42-228.lab.eng.blr.redhat.com IP is 10.70.42.228 . Is this node "dhcp42-228.lab.eng.blr.redhat.com"  reachable ? Can you try to ping this node and check its availability?

Comment 4 Luis Pabón 2016-07-06 16:42:43 UTC
This is due to your disks not being clean.

*** This bug has been marked as a duplicate of bug 1350625 ***