Bug 1843314 - Increase the timeout waiting for ironic node deployment
Summary: Increase the timeout waiting for ironic node deployment
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.5
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.6.0
Assignee: Stephen Benjamin
QA Contact: Amit Ugol
URL:
Whiteboard:
Depends On:
Blocks: 1846187
TreeView+ depends on / blocked
 
Reported: 2020-06-03 02:19 UTC by Alexander Chuzhoy
Modified: 2020-10-27 16:05 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-27 16:04:37 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift installer pull 3721 0 None closed Bug 1843314: baremetal: bump ironic timeout to 3600 seconds 2021-01-22 21:22:43 UTC
Red Hat Product Errata RHBA-2020:4196 0 None None None 2020-10-27 16:04:59 UTC

Description Alexander Chuzhoy 2020-06-03 02:19:27 UTC
Cluster version: 4.4.4

In our testbeds we occasionally see timeouts occuring during the phase when ironic is being brought up.

time="2020-06-02T20:56:12Z" level=debug msg="module.masters.ironic_node_v1.openshift-master-host[2]: Still creating... [34m0s elapsed]"
time="2020-06-02T20:56:12Z" level=debug msg="module.masters.ironic_node_v1.openshift-master-host[0]: Still creating... [34m0s elapsed]"
time="2020-06-02T20:56:17Z" level=error
time="2020-06-02T20:56:17Z" level=error msg="Error: could not contact API: timeout reached"

When I check the bootstrap VM - I see that it works fine, just took it a bit longer than 30 minutes to boot with the ignition config and to get the rhcos image.

Would it be possible to increase the timeout to say 1 hour.

Thanks.

Comment 1 Alexander Chuzhoy 2020-06-04 20:52:50 UTC
Just did some tests today on somewhat slow testbeds.
Successfully deployed 4.4.5 today.

As you can see from below it took a bit less than 27 mins:
time="2020-06-04T07:14:52Z" level=debug msg="module.masters.ironic_node_v1.openshift-master-host[0]: Creation complete after 26m49s [id=4d8b98b2-ff8e-400d-bf07-ef8778e6183b]"
time="2020-06-04T07:14:53Z" level=debug msg="module.masters.ironic_node_v1.openshift-master-host[2]: Creation complete after 26m50s [id=85acbf97-106e-4bee-83fa-4b855d69069c]"
time="2020-06-04T07:14:53Z" level=debug msg="module.masters.ironic_node_v1.openshift-master-host[1]: Creation complete after 26m50s [id=7b522fe0-91a8-4234-898f-3a837ea5ccd9]"


Then tried to install 4.5.0-0.nightly-2020-06-04-025914 it was timing out around 25 mins.

Comment 4 Alexander Chuzhoy 2020-06-17 03:48:04 UTC
Version:4.5.0-0.nightly-2020-06-16-014907


time="2020-06-17T00:40:08Z" level=debug msg="module.masters.ironic_node_v1.openshift-master-host[2]: Still creating... [25m10s elapsed]"
time="2020-06-17T00:40:08Z" level=debug msg="module.masters.ironic_node_v1.openshift-master-host[0]: Still creating... [25m10s elapsed]"
time="2020-06-17T00:40:18Z" level=debug msg="module.masters.ironic_node_v1.openshift-master-host[0]: Still creating... [25m20s elapsed]"
time="2020-06-17T00:40:18Z" level=debug msg="module.masters.ironic_node_v1.openshift-master-host[1]: Still creating... [25m20s elapsed]"
time="2020-06-17T00:40:18Z" level=debug msg="module.masters.ironic_node_v1.openshift-master-host[2]: Still creating... [25m20s elapsed]"
time="2020-06-17T00:40:28Z" level=debug msg="module.masters.ironic_node_v1.openshift-master-host[0]: Still creating... [25m30s elapsed]"
time="2020-06-17T00:40:28Z" level=debug msg="module.masters.ironic_node_v1.openshift-master-host[1]: Still creating... [25m30s elapsed]"
time="2020-06-17T00:40:28Z" level=debug msg="module.masters.ironic_node_v1.openshift-master-host[2]: Still creating... [25m30s elapsed]"
time="2020-06-17T00:40:34Z" level=debug msg="module.masters.ironic_node_v1.openshift-master-host[0]: Creation complete after 25m36s [id=3d0a33bb-f652-45cd-a7e3-2f3b6a5f3586]"
time="2020-06-17T00:40:35Z" level=debug msg="module.masters.ironic_node_v1.openshift-master-host[2]: Creation complete after 25m37s [id=6a8403e0-fd4b-4053-86c1-b2a2694215d2]"
time="2020-06-17T00:40:37Z" level=debug msg="module.masters.ironic_node_v1.openshift-master-host[1]: Creation complete after 25m38s [id=31c107c5-0df7-4ecc-94ee-92e6227ff4c

Thus it didn't time out after 25 mins.

Comment 6 errata-xmlrpc 2020-10-27 16:04:37 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196


Note You need to log in before you can comment on or make changes to this bug.