RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1881691 - Introspection of node attempt failed
Summary: Introspection of node attempt failed
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: ipmitool
Version: 8.2
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: rc
: 8.0
Assignee: kelly.griese
QA Contact: Rachel Sibley
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-09-22 21:43 UTC by kelly.griese
Modified: 2020-11-14 08:31 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-09-30 19:11:46 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
conductor log with details mentioned above (8.03 MB, text/plain)
2020-09-22 21:43 UTC, kelly.griese 		no flags Details
inspector log (8.51 MB, text/plain)
2020-09-22 21:44 UTC, kelly.griese 		no flags Details
app log (10.07 MB, text/plain)
2020-09-22 21:48 UTC, kelly.griese 		no flags Details
View All

Description kelly.griese 2020-09-22 21:43:18 UTC 
Created attachment 1715797 [details]
conductor log with details mentioned above

Followed the steps described here, on step 7.3:
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/director_installation_and_usage/creating-a-basic-overcloud-with-cli-tools#inspecting-the-hardware-of-nodes-basic

Called the following command:
 openstack overcloud node introspect --all-manageable --provide

Returned the following:
(undercloud) [stack@nfixundercloud ~]$ openstack overcloud node introspect --all-manageable --provide
Waiting for introspection to finish...
Waiting for messages on queue 'tripleo' with no timeout.
Introspection of node attempt failed:736d2086-d1c6-4261-93ee-6e8eed53032f.
Introspection of node attempt failed:3b9f9afd-69e3-4cb1-8d32-d59459fedd02.
Retrying 2 nodes that failed introspection. Attempt 1 of 3
Introspection of node attempt failed:3b9f9afd-69e3-4cb1-8d32-d59459fedd02.
Introspection of node attempt failed:736d2086-d1c6-4261-93ee-6e8eed53032f.
Retrying 2 nodes that failed introspection. Attempt 2 of 3
Introspection of node attempt failed:736d2086-d1c6-4261-93ee-6e8eed53032f.
Introspection of node attempt failed:3b9f9afd-69e3-4cb1-8d32-d59459fedd02.
Retrying 2 nodes that failed introspection. Attempt 3 of 3
Introspection of node attempt failed:736d2086-d1c6-4261-93ee-6e8eed53032f.
Introspection of node attempt failed:3b9f9afd-69e3-4cb1-8d32-d59459fedd02.
Retry limit reached with 2 nodes still failing introspection
{'result': 'Failure caused by error in tasks: send_message\n\n  send_message [task_ex_id=dca325ad-721d-450a-9439-ab18a116bc27] -> Workflow failed due to message status. Status:FAILED Message:Retry limit reached with 2 nodes still failing introspection\n    [wf_ex_id=54bd177a-875a-4c17-8367-6ce87d566f55, idx=0]: Workflow failed due to message status. Status:FAILED Message:Retry limit reached with 2 nodes still failing introspection\n', 'introspection_attempt': 2, 'introspected_nodes': {'3b9f9afd-69e3-4cb1-8d32-d59459fedd02': {'error': None, 'finished': False, 'finished_at': None, 'links': [{'href': 'http://192.168.200.2:13050/v1/introspection/3b9f9afd-69e3-4cb1-8d32-d59459fedd02', 'rel': 'self'}], 'started_at': '2020-09-22T20:33:00', 'state': 'waiting', 'uuid': '3b9f9afd-69e3-4cb1-8d32-d59459fedd02'}, '736d2086-d1c6-4261-93ee-6e8eed53032f': {'error': None, 'finished': False, 'finished_at': None, 'links': [{'href': 'http://192.168.200.2:13050/v1/introspection/736d2086-d1c6-4261-93ee-6e8eed53032f', 'rel': 'self'}], 'started_at': '2020-09-22T20:33:00', 'state': 'waiting', 'uuid': '736d2086-d1c6-4261-93ee-6e8eed53032f'}}, 'failed_introspection': ['3b9f9afd-69e3-4cb1-8d32-d59459fedd02', '736d2086-d1c6-4261-93ee-6e8eed53032f'], 'status': 'RUNNING', 'message': 'Retrying 2 nodes that failed introspection. Attempt 2 of 3 ', 'node_uuids': ['3b9f9afd-69e3-4cb1-8d32-d59459fedd02', '736d2086-d1c6-4261-93ee-6e8eed53032f']}
Exception introspecting nodes: {'result': 'Failure caused by error in tasks: send_message\n\n  send_message [task_ex_id=dca325ad-721d-450a-9439-ab18a116bc27] -> Workflow failed due to message status. Status:FAILED Message:Retry limit reached with 2 nodes still failing introspection\n    [wf_ex_id=54bd177a-875a-4c17-8367-6ce87d566f55, idx=0]: Workflow failed due to message status. Status:FAILED Message:Retry limit reached with 2 nodes still failing introspection\n', 'introspection_attempt': 2, 'introspected_nodes': {'3b9f9afd-69e3-4cb1-8d32-d59459fedd02': {'error': None, 'finished': False, 'finished_at': None, 'links': [{'href': 'http://192.168.200.2:13050/v1/introspection/3b9f9afd-69e3-4cb1-8d32-d59459fedd02', 'rel': 'self'}], 'started_at': '2020-09-22T20:33:00', 'state': 'waiting', 'uuid': '3b9f9afd-69e3-4cb1-8d32-d59459fedd02'}, '736d2086-d1c6-4261-93ee-6e8eed53032f': {'error': None, 'finished': False, 'finished_at': None, 'links': [{'href': 'http://192.168.200.2:13050/v1/introspection/736d2086-d1c6-4261-93ee-6e8eed53032f', 'rel': 'self'}], 'started_at': '2020-09-22T20:33:00', 'state': 'waiting', 'uuid': '736d2086-d1c6-4261-93ee-6e8eed53032f'}}, 'failed_introspection': ['3b9f9afd-69e3-4cb1-8d32-d59459fedd02', '736d2086-d1c6-4261-93ee-6e8eed53032f'], 'status': 'RUNNING', 'message': 'Retrying 2 nodes that failed introspection. Attempt 2 of 3 ', 'node_uuids': ['3b9f9afd-69e3-4cb1-8d32-d59459fedd02', '736d2086-d1c6-4261-93ee-6e8eed53032f']}


The ipmitool was updated to include the fix mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1831158
Running version: ipmitool-1.8.18-17.el8.x86_64

Setup includes:
1 x undercloud node (Linux RHEL 8)
1 x overcloud node  (135.121.21.39 = IPMI)
1 x undercloud node (135.121.21.45 = IPMI)

This is a sample of what is seen in the /var/log/containers/ironic/ironic-conductor.log
2020-09-22 17:10:06.621 8 DEBUG ironic.conductor.task_manager [req-b025487f-54a6-4ed4-ba38-5998496c3673 - - - - -] Successfully released shared lock for power state sync on node 736d2086-d1c6-4261-93ee-6e8eed53032f (lock was held 1.12 sec) release_resources /usr/lib/python3.6/site-packages/ironic/conductor/task_manager.py:356
2020-09-22 17:10:06.663 8 DEBUG oslo_concurrency.processutils [req-b025487f-54a6-4ed4-ba38-5998496c3673 - - - - -] CMD "ipmitool -I lanplus -H 135.121.21.39 -L ADMINISTRATOR -U ipdlab -R 1 -N 1 -f /tmp/tmpry77xrn0 power status" returned: 0 in 1.153s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:409
2020-09-22 17:10:06.663 8 DEBUG ironic.common.utils [req-b025487f-54a6-4ed4-ba38-5998496c3673 - - - - -] Execution completed, command line is "ipmitool -I lanplus -H 135.121.21.39 -L ADMINISTRATOR -U ipdlab -R 1 -N 1 -f /tmp/tmpry77xrn0 power status" execute /usr/lib/python3.6/site-packages/ironic/common/utils.py:77
2020-09-22 17:10:06.663 8 DEBUG ironic.common.utils [req-b025487f-54a6-4ed4-ba38-5998496c3673 - - - - -] Command stdout is: "Chassis Power is on
" execute /usr/lib/python3.6/site-packages/ironic/common/utils.py:78
2020-09-22 17:10:06.664 8 DEBUG ironic.common.utils [req-b025487f-54a6-4ed4-ba38-5998496c3673 - - - - -] Command stderr is: "Unable to Get Channel Cipher Suites
" execute /usr/lib/python3.6/site-packages/ironic/common/utils.py:79
2020-09-22 17:10:06.664 8 DEBUG ironic.conductor.task_manager [req-b025487f-54a6-4ed4-ba38-5998496c3673 - - - - -] Successfully released shared lock for power state sync on node 3b9f9afd-69e3-4cb1-8d32-d59459fedd02 (lock was held 1.17 sec) release_resources /usr/lib/python3.6/site-packages/ironic/conductor/task_manager.py:356
Comment 1 kelly.griese 2020-09-22 21:44:05 UTC 
Created attachment 1715798 [details]
inspector log
Comment 2 kelly.griese 2020-09-22 21:48:26 UTC 
Created attachment 1715799 [details]
app log
Note You need to log in before you can comment on or make changes to this bug.