Bug 2068246 - Race between terraform and starting ironic conductor in a combined executable
Summary: Race between terraform and starting ironic conductor in a combined executable
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Bare Metal Hardware Provisioning
Version: 4.11
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.11.0
Assignee: Dmitry Tantsur
QA Contact: Amit Ugol
URL:
Whiteboard:
: 2071046 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-03-24 18:15 UTC by Dmitry Tantsur
Modified: 2022-04-26 10:27 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-04-25 07:45:55 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift ironic-image pull 265 0 None open Bug 2068246: Include fix for race in ironic combined executable 2022-04-07 16:34:33 UTC
OpenStack gerrit 835098 0 None NEW Start API after conductor is started in the combined executable 2022-03-24 18:15:32 UTC
OpenStack gerrit 836609 0 None NEW Start API after conductor is started in the combined executable 2022-04-07 12:11:27 UTC

Description Dmitry Tantsur 2022-03-24 18:15:33 UTC
I've noticed this in the CI: terraform tries to create an ironic node after the API starts but before the built-in conductor is ready, resulting in the following traceback in the ironic pod on bootstrap:

2022-03-24 16:29:50.528 1 ERROR ironic.api.method [req-ef4b0bb8-2be1-4b99-bac6-d4732ecdb4e9 - - - - -] Server-side error: "Cannot use 'none' RPC to connect to remote conductor 172.22.0.2". Detail: 
Traceback (most recent call last):

  File "/usr/lib/python3.6/site-packages/ironic/api/method.py", line 42, in callfunction
    result = f(self, *args, **kwargs)

  File "/usr/lib/python3.6/site-packages/ironic/api/method.py", line 109, in inner_body
    return function(*args, **kwargs)

  File "/usr/lib/python3.6/site-packages/ironic/common/args.py", line 379, in inner_check_args
    return function(*args, **kwargs_next)

  File "/usr/lib/python3.6/site-packages/ironic/api/controllers/v1/node.py", line 2493, in post
    new_node, topic)

  File "/usr/lib/python3.6/site-packages/ironic/conductor/rpcapi.py", line 314, in create_node
    cctxt = self._prepare_call(topic=topic, version='1.36')

  File "/usr/lib/python3.6/site-packages/ironic/conductor/rpcapi.py", line 213, in _prepare_call
    % host)

ironic.common.exception.ServiceUnavailable: Cannot use 'none' RPC to connect to remote conductor 172.22.0.2


I don't know how to reproduce this issue on purpose, but I think the fix is quite simple: only start API after the conductor is started.

Comment 1 Riccardo Pittau 2022-04-04 08:04:02 UTC
*** Bug 2071046 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.