Bug 1484217 - cns-deploy fails, failing to load the topology file
Summary: cns-deploy fails, failing to load the topology file
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: CNS-deployment
Version: cns-3.6
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: CNS 3.6
Assignee: Raghavendra Talur
QA Contact: krishnaram Karthick
Depends On:
Blocks: 1445448
TreeView+ depends on / blocked
Reported: 2017-08-23 03:42 UTC by krishnaram Karthick
Modified: 2018-12-14 11:19 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2017-10-11 07:12:11 UTC
Target Upstream Version:

Attachments (Terms of Use)
cns logs (38.84 KB, text/plain)
2017-08-23 03:47 UTC, krishnaram Karthick
no flags Details
logs from build - cns-deploy-5.0.0-23.el7rhgs.x86_64 (7.38 KB, text/plain)
2017-08-24 16:43 UTC, krishnaram Karthick
no flags Details

System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1483852 0 unspecified CLOSED cns-deployment fails during deploy-heketi, deploy-heketi url isn't reachable 2021-02-22 00:41:40 UTC
Red Hat Product Errata RHEA-2017:2881 0 normal SHIPPED_LIVE cns-deploy-tool bug fix and enhancement update 2017-10-11 11:11:43 UTC

Internal Links: 1483852

Description krishnaram Karthick 2017-08-23 03:42:34 UTC
Description of problem:
cns-deploy fails while trying to load the topology file. This issue is seen on multiple setups and manually trying to add a node fails too.

snippet of the log: (complete set of logs attached)
Determining heketi service URL ... OK
oc -n storage-project exec -it deploy-heketi-1-vtvkg -- heketi-cli -s http://localhost:8080 --user admin --secret '' topology load --json=/etc/heketi/topology.json 2>&1
Creating cluster ... ID: 327c9c54c115a7feb11476fd684f980d
Allowing file volumes on cluster.
Allowing block volumes on cluster.
Creating node dhcp47-174.lab.eng.blr.redhat.com ... Unable to create node: New Node doesn't have glusterd running
Creating node dhcp47-183.lab.eng.blr.redhat.com ... Unable to create node: New Node doesn't have glusterd running
Creating node dhcp46-133.lab.eng.blr.redhat.com ... Unable to create node: New Node doesn't have glusterd running
Error loading the cluster topology.
Please check the failed node or device and rerun this script.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Run cns-deploy to setup cns

Actual results:
topology load fails

Expected results:
topology should get loaded and cns-deploy should succeed.

Additional info:

Comment 4 krishnaram Karthick 2017-08-23 03:47:24 UTC
Created attachment 1316932 [details]
cns logs

Comment 5 Raghavendra Talur 2017-08-23 04:04:07 UTC
reason for error:

[kubeexec] ERROR 2017/08/22 22:22:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:303: Get net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
[kubeexec] ERROR 2017/08/22 22:22:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:304: Failed to get list of pods
[negroni] Completed 400 Bad Request in 1.730791ms

This could be because:
1. the timeout change we made in heketi
2. setup has network/firewall issue
3. some labels on nodes are wrong.

Need to look further for more info.

Comment 6 Raghavendra Talur 2017-08-23 05:52:51 UTC
This is because of patch https://github.com/heketi/heketi/pull/778

Have sent a revert for the same https://github.com/heketi/heketi/pull/840

Comment 12 krishnaram Karthick 2017-08-24 16:43:34 UTC
Created attachment 1317840 [details]
logs from build - cns-deploy-5.0.0-23.el7rhgs.x86_64

Comment 16 errata-xmlrpc 2017-10-11 07:12:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.