Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem: cns-deploy fails while trying to load the topology file. This issue is seen on multiple setups and manually trying to add a node fails too. snippet of the log: (complete set of logs attached) =================== Determining heketi service URL ... OK oc -n storage-project exec -it deploy-heketi-1-vtvkg -- heketi-cli -s http://localhost:8080 --user admin --secret '' topology load --json=/etc/heketi/topology.json 2>&1 Creating cluster ... ID: 327c9c54c115a7feb11476fd684f980d Allowing file volumes on cluster. Allowing block volumes on cluster. Creating node dhcp47-174.lab.eng.blr.redhat.com ... Unable to create node: New Node doesn't have glusterd running Creating node dhcp47-183.lab.eng.blr.redhat.com ... Unable to create node: New Node doesn't have glusterd running Creating node dhcp46-133.lab.eng.blr.redhat.com ... Unable to create node: New Node doesn't have glusterd running Error loading the cluster topology. Please check the failed node or device and rerun this script. Version-Release number of selected component (if applicable): cns-deploy-5.0.0-20.el7rhgs.x86_64 How reproducible: Always Steps to Reproduce: 1. Run cns-deploy to setup cns Actual results: topology load fails Expected results: topology should get loaded and cns-deploy should succeed. Additional info:
Created attachment 1316932 [details] cns logs
reason for error: [kubeexec] ERROR 2017/08/22 22:22:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:303: Get https://172.30.0.1:443/api/v1/namespaces/storage-project/pods?labelSelector=glusterfs-node: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) [kubeexec] ERROR 2017/08/22 22:22:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:304: Failed to get list of pods [negroni] Completed 400 Bad Request in 1.730791ms This could be because: 1. the timeout change we made in heketi 2. setup has network/firewall issue 3. some labels on nodes are wrong. Need to look further for more info.
This is because of patch https://github.com/heketi/heketi/pull/778 Have sent a revert for the same https://github.com/heketi/heketi/pull/840
Created attachment 1317840 [details] logs from build - cns-deploy-5.0.0-23.el7rhgs.x86_64
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:2881