Bug 1957980

The deployment fails with the following error:
~~~
Error: Agent API for node 32cbf50b-3cfd-49f9-a166-c5776a704529 returned HTTP status code 409 with error: Agent is busy: executing command execute_deploy_step.
~~~

The same deployment (same hosts, same install config) works when deploying 4.6.17

Version-Release number of selected component (if applicable):
4.6.9

How reproducible:
Fails every time, tested with both 4.7.6 and 4.7.9.


Actual results:
Checking on the half deployed host we eventually get to
~~~
May 06 13:48:21 master-0 ironic-python-agent[3208]: 2021-05-06 13:48:15.422 3208 DEBUG ironic_python_agent.ironic_api_client [-] Heartbeat: announcing callback URL https://172.22.0.15:9999, API version is 1.68 heartbeat /usr/lib/python3.6/site-packages/ironic_python_agent/ironic_api_client.py:150
May 06 13:48:21 master-0 ironic-python-agent[3208]: 2021-05-06 13:48:21.039 3208 INFO root [-] Asynchronous command execute_deploy_step completed: {'deploy_result': {'result': 'prepare_image: image (rhcos-47.83.202103251640-0-compressed.x86_64.qcow2) written to device /dev/sda root_uuid=0x00000000'}, 'deploy_step': 
{'interface': 'deploy', 'step': 'write_image', 'args': {'image_info': {'id': 'rhcos-47.83.202103251640-0-compressed.x86_64.qcow2', 'urls': ['http://172.22.0.2:80/images/rhcos-47.83.202103251640-0-openstack.x86_64.qcow2/rhcos-47.83.202103251640-0-compressed.x86_64.qcow2'], 'disk_format': None, 'container_format': Non
e, 'stream_raw_images': True, 'checksum': 'http://172.22.0.2:80/images/rhcos-47.83.202103251640-0-openstack.x86_64.qcow2/rhcos-47.83.202103251640-0-compressed.x86_64.qcow2.md5sum', 'node_uuid': 'a7046d77-3d11-400f-99cb-c1a20cb86678'}, 'configdrive': '<...>'}}}
May 06 13:48:21 master-0 rsyslogd[3212]: message too long (312102) with configured size 8096, begin of message is: 2021-05-06 13:47:18.864 3208 INFO root [-] Deploy step completed: {'interface':  [v8.1911.0-6.el8 try https://www.rsyslog.com/e/2445 ]
May 06 13:48:21 master-0 ironic-python-agent[3208]: 2021-05-06 13:48:21.154 3208 ERROR ironic_python_agent.agent [-] error sending heartbeat to http://172.22.0.2:6385: ironic_python_agent.errors.HeartbeatError: Error heartbeating to agent API: Error 400: Invalid or missing agent token received.
                                                    2021-05-06 13:48:21.154 3208 ERROR ironic_python_agent.agent Traceback (most recent call last):
                                                    2021-05-06 13:48:21.154 3208 ERROR ironic_python_agent.agent   File "/usr/lib/python3.6/site-packages/ironic_python_agent/agent.py", line 109, in do_heartbeat
                                                    2021-05-06 13:48:21.154 3208 ERROR ironic_python_agent.agent     generated_cert=self.agent.generated_cert,
                                                    2021-05-06 13:48:21.154 3208 ERROR ironic_python_agent.agent   File "/usr/lib/python3.6/site-packages/ironic_python_agent/ironic_api_client.py", line 163, in heartbeat
                                                    2021-05-06 13:48:21.154 3208 ERROR ironic_python_agent.agent     raise errors.HeartbeatError(error)
                                                    2021-05-06 13:48:21.154 3208 ERROR ironic_python_agent.agent ironic_python_agent.errors.HeartbeatError: Error heartbeating to agent API: Error 400: Invalid or missing agent token received.
                                                    2021-05-06 13:48:21.154 3208 ERROR ironic_python_agent.agent 
May 06 13:48:21 master-0 ironic-python-agent[3208]: 2021-05-06 13:48:21.237 3208 DEBUG ironic_python_agent.ironic_api_client [-] Heartbeat: announcing callback URL https://172.22.0.15:9999, API version is 1.68 heartbeat /usr/lib/python3.6/site-packages/ironic_python_agent/ironic_api_client.py:150
May 06 13:48:21 master-0 ironic-python-agent[3208]: 2021-05-06 13:48:21.500 3208 INFO ironic_python_agent.agent [-] sleeping before next heartbeat, interval: 156.01437867497847
May 06 13:48:21 master-0 ironic-python-agent[3208]: 2021-05-06 13:48:21.581 3208 ERROR ironic_python_agent.agent [-] error sending heartbeat to http://172.22.0.2:6385: ironic_python_agent.errors.HeartbeatError: Error heartbeating to agent API: Error 400: Invalid or missing agent token received.
                                                    2021-05-06 13:48:21.581 3208 ERROR ironic_python_agent.agent Traceback (most recent call last):
                                                    2021-05-06 13:48:21.581 3208 ERROR ironic_python_agent.agent   File "/usr/lib/python3.6/site-packages/ironic_python_agent/agent.py", line 109, in do_heartbeat
                                                    2021-05-06 13:48:21.581 3208 ERROR ironic_python_agent.agent     generated_cert=self.agent.generated_cert,
                                                    2021-05-06 13:48:21.581 3208 ERROR ironic_python_agent.agent   File "/usr/lib/python3.6/site-packages/ironic_python_agent/ironic_api_client.py", line 163, in heartbeat
                                                    2021-05-06 13:48:21.581 3208 ERROR ironic_python_agent.agent     raise errors.HeartbeatError(error)
                                                    2021-05-06 13:48:21.581 3208 ERROR ironic_python_agent.agent ironic_python_agent.errors.HeartbeatError: Error heartbeating to agent API: Error 400: Invalid or missing agent token received.
                                                    2021-05-06 13:48:21.581 3208 ERROR ironic_python_agent.agent 
May 06 13:48:21 master-0 ironic-python-agent[3208]: 2021-05-06 13:48:21.799 3208 INFO ironic_python_agent.agent [-] sleeping before next heartbeat, interval: 169.92832458453464
~~~

Expected results:
That it works just the same as 4.6 deployment.


Additional info:
bootstrap logs/data (`./openshift-install gather bootstrap --dir=<installation_directory>`) and while host is booted off ramdisk logs all attached to the case.