Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1973735

Summary:	Following an upgrade from 4.5.16 to 4.6.17 Baremetal host are suck in provisioning/deleting state
Product:	OpenShift Container Platform	Reporter:	arajapa
Component:	Bare Metal Hardware Provisioning	Assignee:	Zane Bitter <zbitter>
Bare Metal Hardware Provisioning sub component:	baremetal-operator	QA Contact:	Amit Ugol <augol>
Status:	CLOSED DUPLICATE	Docs Contact:
Severity:	high
Priority:	high	CC:	aos-bugs, beth.white, bhershbe, derekh, dtantsur, ealcaniz, omichael, racedoro, zbitter
Version:	4.6	Keywords:	Triaged
Target Milestone:	---
Target Release:	4.9.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-06-24 01:00:10 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description arajapa 2021-06-18 15:18:56 UTC

Description of problem:
Following an upgrade from 4.5.16 to 4.6.17, we are seeing 16 nodes are stuck in different inconsistent states, some nodes have no baremetal host assigned, some dont have node objects and/or machine objects.

We have a must-gather prior to the upgrade where we saw the machineset issue is pre-existing(13 nodes) to the upgrade but now there's 3 more additional nodes

Comment 12 Zane Bitter 2021-06-24 01:00:10 UTC

Summary of the issues encountered here:

* Hosts with their ironic Nodes in "clean failed" state during deprovisioning cannot progress and report the following error in the baremetal-operator log:

Reconciler error
action "deprovisioning" failed: failed to deprovision: failed to change provisioning state to "deleted": Bad request with: [PUT http://<HOST PROVISIONING IP>:6385/v1/nodes/8dacf0e4-5bff-476b-8bcd-9f782960d102/states/provision], error message: {"error_message": "{\"faultcode\": \"Client\", \"faultstring\": \"The requested action \\\"deleted\\\" can not be performed on node \\\"8dacf0e4-5bff-476b-8bcd-9f782960d102\\\" while it is in state \\\"clean failed\\\".\", \"debuginfo\": null}"}

Fixed upstream by https://github.com/metal3-io/baremetal-operator/commit/1f17aab6481a6c49db0c8c82bfb6c6518f0b8dc1#diff-f05fc4a0c9679815bc34bcef100c2a8d525688f2804a1128c5b9e1e575b76c29 - fix is present in 4.7. Workaround: delete the Pod associated with the metal3 Deployment (in openshift-machine-api namespace) to erase the ironic database and have the baremetal-operator rebuild it.


* Bug 1975518 (clone of bug 1946079) - when Nodes come up they may fail to get an IP address of the correct family due to Dracut changes.


* Bug 1975577 - The IPA image may fail to report IP addresses for both IPv4 and IPv6, resulting in failure to link Nodes and Machines when there is no IPv4 address recorded in the inspection data for the Host.
Marking this bug as a duplicate of that one, since this was the original issue.


* Bug 1975531 (clone of bug 1906448) - if hardware clocks on nodes running IPA are ahead of those on the cluster, communication back to Ironic fails due to SSL certificate validity.

*** This bug has been marked as a duplicate of bug 1975577 ***