Description of problem:
cns-deploy fails while trying to load the topology file. This issue is seen on multiple setups and manually trying to add a node fails too.
snippet of the log: (complete set of logs attached)
Determining heketi service URL ... OK
oc -n storage-project exec -it deploy-heketi-1-vtvkg -- heketi-cli -s http://localhost:8080 --user admin --secret '' topology load --json=/etc/heketi/topology.json 2>&1
Creating cluster ... ID: 327c9c54c115a7feb11476fd684f980d
Allowing file volumes on cluster.
Allowing block volumes on cluster.
Creating node dhcp47-174.lab.eng.blr.redhat.com ... Unable to create node: New Node doesn't have glusterd running
Creating node dhcp47-183.lab.eng.blr.redhat.com ... Unable to create node: New Node doesn't have glusterd running
Creating node dhcp46-133.lab.eng.blr.redhat.com ... Unable to create node: New Node doesn't have glusterd running
Error loading the cluster topology.
Please check the failed node or device and rerun this script.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. Run cns-deploy to setup cns
topology load fails
topology should get loaded and cns-deploy should succeed.
Created attachment 1316932 [details]
reason for error:
[kubeexec] ERROR 2017/08/22 22:22:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:303: Get https://172.30.0.1:443/api/v1/namespaces/storage-project/pods?labelSelector=glusterfs-node: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
[kubeexec] ERROR 2017/08/22 22:22:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:304: Failed to get list of pods
[negroni] Completed 400 Bad Request in 1.730791ms
This could be because:
1. the timeout change we made in heketi
2. setup has network/firewall issue
3. some labels on nodes are wrong.
Need to look further for more info.
This is because of patch https://github.com/heketi/heketi/pull/778
Have sent a revert for the same https://github.com/heketi/heketi/pull/840
Created attachment 1317840 [details]
logs from build - cns-deploy-5.0.0-23.el7rhgs.x86_64
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.