Bug 1741344

Summary:	Deploying RHHI.next fails during nodes cleanup.
Product:	Kubernetes-native Infrastructure	Reporter:	Alexander Chuzhoy <sasha>
Component:	Deployment	Assignee:	Angus Thomas <athomas>
Status:	CLOSED DUPLICATE	QA Contact:	Arik Chernetsky <achernet>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	unspecified	CC:	augol, kni-bugs, stbenjam
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-08-24 18:01:45 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Alexander Chuzhoy 2019-08-14 20:37:39 UTC

[cloud-user@r640-u05 dev-scripts]$ openstack baremetal node list
+--------------------------------------+--------------------+---------------+-------------+--------------------+-------------+
| UUID                                 | Name               | Instance UUID | Power State | Provisioning State | Maintenance |
+--------------------------------------+--------------------+---------------+-------------+--------------------+-------------+
| cc7b40a8-50af-4532-9758-33c74c355fe8 | openshift-master-2 | None          | power off   | clean failed       | True        |
| ff2be282-3015-4f62-b12b-3b486a594ec2 | openshift-master-0 | None          | power off   | clean failed       | True        |
| d327a79e-4dc2-4658-8329-1696d4f7ecf9 | openshift-master-1 | None          | power off   | clean failed       | True        |
+--------------------------------------+--------------------+---------------+-------------+--------------------+-------------+


The following error is in the log:
level=error
level=error msg="Error: could not make node available: Expected HTTP response code [202] when accessing [PUT http://localhost:6385/v1/nodes/cc7b40a8-50af-4532-9758-33c74c355fe8/states/provision], but got 409 instead"
level=error msg="{\"error_message\": \"{\\\"debuginfo\\\": null, \\\"faultcode\\\": \\\"Client\\\", \\\"faultstring\\\": \\\"Node cc7b40a8-50af-4532-9758-33c74c355fe8 is locked by host openshift-worker-0.qe2.kni.lab.eng.bos.redhat.com, please retry after the current operation is completed.\\\"}\"}"
level=error
level=error msg="  on ../../../tmp/openshift-install-870877658/masters/main.tf line 1, in resource \"ironic_node_v1\" \"openshift-master-host\":"
level=error msg="   1: resource \"ironic_node_v1\" \"openshift-master-host\" {"
level=error
level=error
level=error
level=error msg="Error: could not make node available: Expected HTTP response code [202] when accessing [PUT http://localhost:6385/v1/nodes/ff2be282-3015-4f62-b12b-3b486a594ec2/states/provision], but got 409 instead"
level=error msg="{\"error_message\": \"{\\\"debuginfo\\\": null, \\\"faultcode\\\": \\\"Client\\\", \\\"faultstring\\\": \\\"Node ff2be282-3015-4f62-b12b-3b486a594ec2 is locked by host openshift-worker-0.qe2.kni.lab.eng.bos.redhat.com, please retry after the current operation is completed.\\\"}\"}"
level=error
level=error msg="  on ../../../tmp/openshift-install-870877658/masters/main.tf line 1, in resource \"ironic_node_v1\" \"openshift-master-host\":"
level=error msg="   1: resource \"ironic_node_v1\" \"openshift-master-host\" {"
level=error
level=error
level=error
level=error msg="Error: could not make node available: Expected HTTP response code [202] when accessing [PUT http://localhost:6385/v1/nodes/d327a79e-4dc2-4658-8329-1696d4f7ecf9/states/provision], but got 409 instead"
level=error msg="{\"error_message\": \"{\\\"debuginfo\\\": null, \\\"faultcode\\\": \\\"Client\\\", \\\"faultstring\\\": \\\"Node d327a79e-4dc2-4658-8329-1696d4f7ecf9 is locked by host openshift-worker-0.qe2.kni.lab.eng.bos.redhat.com, please retry after the current operation is completed.\\\"}\"}"
level=error
level=error msg="  on ../../../tmp/openshift-install-870877658/masters/main.tf line 1, in resource \"ironic_node_v1\" \"openshift-master-host\":"
level=error msg="   1: resource \"ironic_node_v1\" \"openshift-master-host\" {"
level=error
level=error
level=fatal msg="failed to fetch Cluster: failed to generate asset \"Cluster\": failed to create cluster: failed to apply using Terraform"

Comment 2 Stephen Benjamin 2019-08-24 18:01:45 UTC

Thanks for report, I didn't see that someone had already created a BZ for this. I created and fixed it under BZ1743871.  It should be working in the latest CI images, please let me know if you still have any problems.  Thanks!

*** This bug has been marked as a duplicate of bug 1743871 ***