Bug 1881691

Created attachment 1715797 [details]
conductor log with details mentioned above

Followed the steps described here, on step 7.3:
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/director_installation_and_usage/creating-a-basic-overcloud-with-cli-tools#inspecting-the-hardware-of-nodes-basic

Called the following command:
 openstack overcloud node introspect --all-manageable --provide

Returned the following:
(undercloud) [stack@nfixundercloud ~]$ openstack overcloud node introspect --all-manageable --provide
Waiting for introspection to finish...
Waiting for messages on queue 'tripleo' with no timeout.
Introspection of node attempt failed:736d2086-d1c6-4261-93ee-6e8eed53032f.
Introspection of node attempt failed:3b9f9afd-69e3-4cb1-8d32-d59459fedd02.
Retrying 2 nodes that failed introspection. Attempt 1 of 3
Introspection of node attempt failed:3b9f9afd-69e3-4cb1-8d32-d59459fedd02.
Introspection of node attempt failed:736d2086-d1c6-4261-93ee-6e8eed53032f.
Retrying 2 nodes that failed introspection. Attempt 2 of 3
Introspection of node attempt failed:736d2086-d1c6-4261-93ee-6e8eed53032f.
Introspection of node attempt failed:3b9f9afd-69e3-4cb1-8d32-d59459fedd02.
Retrying 2 nodes that failed introspection. Attempt 3 of 3
Introspection of node attempt failed:736d2086-d1c6-4261-93ee-6e8eed53032f.
Introspection of node attempt failed:3b9f9afd-69e3-4cb1-8d32-d59459fedd02.
Retry limit reached with 2 nodes still failing introspection
{'result': 'Failure caused by error in tasks: send_message\n\n  send_message [task_ex_id=dca325ad-721d-450a-9439-ab18a116bc27] -> Workflow failed due to message status. Status:FAILED Message:Retry limit reached with 2 nodes still failing introspection\n    [wf_ex_id=54bd177a-875a-4c17-8367-6ce87d566f55, idx=0]: Workflow failed due to message status. Status:FAILED Message:Retry limit reached with 2 nodes still failing introspection\n', 'introspection_attempt': 2, 'introspected_nodes': {'3b9f9afd-69e3-4cb1-8d32-d59459fedd02': {'error': None, 'finished': False, 'finished_at': None, 'links': [{'href': 'http://192.168.200.2:13050/v1/introspection/3b9f9afd-69e3-4cb1-8d32-d59459fedd02', 'rel': 'self'}], 'started_at': '2020-09-22T20:33:00', 'state': 'waiting', 'uuid': '3b9f9afd-69e3-4cb1-8d32-d59459fedd02'}, '736d2086-d1c6-4261-93ee-6e8eed53032f': {'error': None, 'finished': False, 'finished_at': None, 'links': [{'href': 'http://192.168.200.2:13050/v1/introspection/736d2086-d1c6-4261-93ee-6e8eed53032f', 'rel': 'self'}], 'started_at': '2020-09-22T20:33:00', 'state': 'waiting', 'uuid': '736d2086-d1c6-4261-93ee-6e8eed53032f'}}, 'failed_introspection': ['3b9f9afd-69e3-4cb1-8d32-d59459fedd02', '736d2086-d1c6-4261-93ee-6e8eed53032f'], 'status': 'RUNNING', 'message': 'Retrying 2 nodes that failed introspection. Attempt 2 of 3 ', 'node_uuids': ['3b9f9afd-69e3-4cb1-8d32-d59459fedd02', '736d2086-d1c6-4261-93ee-6e8eed53032f']}
Exception introspecting nodes: {'result': 'Failure caused by error in tasks: send_message\n\n  send_message [task_ex_id=dca325ad-721d-450a-9439-ab18a116bc27] -> Workflow failed due to message status. Status:FAILED Message:Retry limit reached with 2 nodes still failing introspection\n    [wf_ex_id=54bd177a-875a-4c17-8367-6ce87d566f55, idx=0]: Workflow failed due to message status. Status:FAILED Message:Retry limit reached with 2 nodes still failing introspection\n', 'introspection_attempt': 2, 'introspected_nodes': {'3b9f9afd-69e3-4cb1-8d32-d59459fedd02': {'error': None, 'finished': False, 'finished_at': None, 'links': [{'href': 'http://192.168.200.2:13050/v1/introspection/3b9f9afd-69e3-4cb1-8d32-d59459fedd02', 'rel': 'self'}], 'started_at': '2020-09-22T20:33:00', 'state': 'waiting', 'uuid': '3b9f9afd-69e3-4cb1-8d32-d59459fedd02'}, '736d2086-d1c6-4261-93ee-6e8eed53032f': {'error': None, 'finished': False, 'finished_at': None, 'links': [{'href': 'http://192.168.200.2:13050/v1/introspection/736d2086-d1c6-4261-93ee-6e8eed53032f', 'rel': 'self'}], 'started_at': '2020-09-22T20:33:00', 'state': 'waiting', 'uuid': '736d2086-d1c6-4261-93ee-6e8eed53032f'}}, 'failed_introspection': ['3b9f9afd-69e3-4cb1-8d32-d59459fedd02', '736d2086-d1c6-4261-93ee-6e8eed53032f'], 'status': 'RUNNING', 'message': 'Retrying 2 nodes that failed introspection. Attempt 2 of 3 ', 'node_uuids': ['3b9f9afd-69e3-4cb1-8d32-d59459fedd02', '736d2086-d1c6-4261-93ee-6e8eed53032f']}


The ipmitool was updated to include the fix mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1831158
Running version: ipmitool-1.8.18-17.el8.x86_64

Setup includes:
1 x undercloud node (Linux RHEL 8)
1 x overcloud node  (135.121.21.39 = IPMI)
1 x undercloud node (135.121.21.45 = IPMI)

This is a sample of what is seen in the /var/log/containers/ironic/ironic-conductor.log
2020-09-22 17:10:06.621 8 DEBUG ironic.conductor.task_manager [req-b025487f-54a6-4ed4-ba38-5998496c3673 - - - - -] Successfully released shared lock for power state sync on node 736d2086-d1c6-4261-93ee-6e8eed53032f (lock was held 1.12 sec) release_resources /usr/lib/python3.6/site-packages/ironic/conductor/task_manager.py:356
2020-09-22 17:10:06.663 8 DEBUG oslo_concurrency.processutils [req-b025487f-54a6-4ed4-ba38-5998496c3673 - - - - -] CMD "ipmitool -I lanplus -H 135.121.21.39 -L ADMINISTRATOR -U ipdlab -R 1 -N 1 -f /tmp/tmpry77xrn0 power status" returned: 0 in 1.153s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:409
2020-09-22 17:10:06.663 8 DEBUG ironic.common.utils [req-b025487f-54a6-4ed4-ba38-5998496c3673 - - - - -] Execution completed, command line is "ipmitool -I lanplus -H 135.121.21.39 -L ADMINISTRATOR -U ipdlab -R 1 -N 1 -f /tmp/tmpry77xrn0 power status" execute /usr/lib/python3.6/site-packages/ironic/common/utils.py:77
2020-09-22 17:10:06.663 8 DEBUG ironic.common.utils [req-b025487f-54a6-4ed4-ba38-5998496c3673 - - - - -] Command stdout is: "Chassis Power is on
" execute /usr/lib/python3.6/site-packages/ironic/common/utils.py:78
2020-09-22 17:10:06.664 8 DEBUG ironic.common.utils [req-b025487f-54a6-4ed4-ba38-5998496c3673 - - - - -] Command stderr is: "Unable to Get Channel Cipher Suites
" execute /usr/lib/python3.6/site-packages/ironic/common/utils.py:79
2020-09-22 17:10:06.664 8 DEBUG ironic.conductor.task_manager [req-b025487f-54a6-4ed4-ba38-5998496c3673 - - - - -] Successfully released shared lock for power state sync on node 3b9f9afd-69e3-4cb1-8d32-d59459fedd02 (lock was held 1.17 sec) release_resources /usr/lib/python3.6/site-packages/ironic/conductor/task_manager.py:356