Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2090631

Summary: IPI deployment on IBM cloud fails on masters "Unexpected exception performing deploy step install_coreos. NameError: name 'time' is not defined'"
Product: OpenShift Container Platform Reporter: Lubov <lshilin>
Component: Bare Metal Hardware ProvisioningAssignee: Tomas Sedovic <tsedovic>
Bare Metal Hardware Provisioning sub component: ironic QA Contact: Amit Ugol <augol>
Status: CLOSED DUPLICATE Docs Contact:
Severity: high    
Priority: unspecified CC: derekh
Version: 4.10Keywords: AutomationBlocker, Regression
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-05-26 08:28:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Lubov 2022-05-26 07:51:58 UTC
Description of problem:
Deployment of OCP 4.10 on real BM (HPE) setup is failing with report
time="2022-05-26T09:46:59+03:00" level=error msg="Error: cannot go from state 'deploy failed' to state 'manageable' , last error was 'Deploy step deploy.install_coreos failed on node e89bdc0f-a652-4379-9685-add2be0937da. Unexpected exception performing deploy step install_coreos. NameError: name 'time' is not defined'"
time="2022-05-26T09:46:59+03:00" level=error
time="2022-05-26T09:46:59+03:00" level=error msg="  on ../../tmp/openshift-install-masters-528184590/main.tf line 43, in resource \"ironic_deployment\" \"openshift-master-deployment\":"
time="2022-05-26T09:46:59+03:00" level=error msg="  43: resource \"ironic_deployment\" \"openshift-master-deployment\" {"
time="2022-05-26T09:46:59+03:00" level=error
time="2022-05-26T09:46:59+03:00" level=error
time="2022-05-26T09:46:59+03:00" level=error
time="2022-05-26T09:46:59+03:00" level=error msg="Error: cannot go from state 'deploy failed' to state 'manageable' , last error was 'Deploy step deploy.install_coreos failed on node 81a2e635-5138-440a-9436-eec6e950e48c. Unexpected exception performing deploy step install_coreos. NameError: name 'time' is not defined'"
time="2022-05-26T09:46:59+03:00" level=error
time="2022-05-26T09:46:59+03:00" level=error msg="  on ../../tmp/openshift-install-masters-528184590/main.tf line 43, in resource \"ironic_deployment\" \"openshift-master-deployment\":"
time="2022-05-26T09:46:59+03:00" level=error msg="  43: resource \"ironic_deployment\" \"openshift-master-deployment\" {"
time="2022-05-26T09:46:59+03:00" level=error
time="2022-05-26T09:46:59+03:00" level=error
time="2022-05-26T09:46:59+03:00" level=error
time="2022-05-26T09:46:59+03:00" level=error msg="Error: cannot go from state 'deploy failed' to state 'manageable' , last error was 'Deploy step deploy.install_coreos failed on node fa0abb35-5dbc-476b-b9af-bed14f9882e0. Unexpected exception performing deploy step install_coreos. NameError: name 'time' is not defined'"
time="2022-05-26T09:46:59+03:00" level=error
time="2022-05-26T09:46:59+03:00" level=error msg="  on ../../tmp/openshift-install-masters-528184590/main.tf line 43, in resource \"ironic_deployment\" \"openshift-master-deployment\":"
time="2022-05-26T09:46:59+03:00" level=error msg="  43: resource \"ironic_deployment\" \"openshift-master-deployment\" {"
time="2022-05-26T09:46:59+03:00" level=error
time="2022-05-26T09:46:59+03:00" level=error
time="2022-05-26T09:46:59+03:00" level=fatal msg="failed to fetch Cluster: failed to generate asset \"Cluster\": failed to create cluster: failed to apply Terraform: failed to complete the change"
time="2022-05-26T09:47:00+03:00" level=debug msg="OpenShift Installer 4.10.0-0.nightly-2022-05-25-203355"
time="2022-05-26T09:47:00+03:00" level=debug msg="Built from commit a4b45c9948014b7631beec9a33382dcb6bced684"
time="2022-05-26T09:47:00+03:00" level=info msg="Waiting up to 20m0s (until 10:07AM) for the Kubernetes API at https://api.ocp-edge1.lab.eng.tlv2.redhat.com:6443..."
time="2022-05-26T09:47:00+03:00" level=info msg="API v1.23.5+3afdacb up"
time="2022-05-26T09:47:00+03:00" level=info msg="Waiting up to 30m0s (until 10:17AM) for bootstrapping to complete..."
time="2022-05-26T10:17:00+03:00" level=info msg="Use the following commands to gather logs from the cluster"
time="2022-05-26T10:17:00+03:00" level=info msg="openshift-install gather bootstrap --help"
time="2022-05-26T10:17:00+03:00" level=error msg="Bootstrap failed to complete: timed out waiting for the condition"
time="2022-05-26T10:17:00+03:00" level=error msg="Failed to wait for bootstrapping to complete. This error usually happens when there is a problem with control plane hosts that prevents the control plane operators from creating the control plane."
time="2022-05-26T10:17:00+03:00" level=fatal msg="Bootstrap failed to complete"

Version-Release number of selected component (if applicable):
4.10.0-0.nightly-2022-05-25-203355

How reproducible:
100%

Steps to Reproduce:
Usual disconnected deploy 

Actual results:
see above

Expected results:
deployment should succeed

Additional info:
[kni@ocp-edge12 ~]$ oc get bmh -A
NAMESPACE               NAME                 STATE   CONSUMER                   ONLINE   ERROR   AGE
openshift-machine-api   openshift-master-0           ocp-edge1-mjztg-master-0   true             84m
openshift-machine-api   openshift-master-1           ocp-edge1-mjztg-master-1   true             84m
openshift-machine-api   openshift-master-2           ocp-edge1-mjztg-master-2   true             84m
openshift-machine-api   openshift-worker-0                                      true             84m
openshift-machine-api   openshift-worker-1                                      true             84m

[kni@ocp-edge12 ~]$ oc get machine -A
NAMESPACE               NAME                       PHASE   TYPE   REGION   ZONE   AGE
openshift-machine-api   ocp-edge1-mjztg-master-0                                  85m
openshift-machine-api   ocp-edge1-mjztg-master-1                                  85m
openshift-machine-api   ocp-edge1-mjztg-master-2                                  85m

[kni@ocp-edge12 ~]$ oc get nodes
No resources found

Comment 2 Derek Higgins 2022-05-26 08:28:46 UTC
A bad cherry-pick for bz#2053752 resulted in this. ("import time" wasn't needed in origin/main)

*** This bug has been marked as a duplicate of bug 2053752 ***