Bug 1578011 - Introspection of failed node creates unnecessary and not useful information dump to the user. This should get cleaned up.
Summary: Introspection of failed node creates unnecessary and not useful information d...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-mistral
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: ---
Assignee: RHOS Maint
QA Contact: mlammon
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-05-14 15:47 UTC by mlammon
Modified: 2019-12-12 23:13 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-12-12 23:13:16 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description mlammon 2018-05-14 15:47:48 UTC
Description of problem:

Introspection of failed node creates unnecessary and not useful information dump to the user.   This should get cleaned up. 

Reproduce:
1. Install undercloud
2. Run introspection with a knowingly failed bare metal node
(in this case it was virtual and we block vbmc communication port on hypervisor)


(undercloud) [stack@undercloud-0 ~]$ ironic node-list
The "ironic" CLI is deprecated and will be removed in the S* release. Please use the "openstack baremetal" CLI instead.
+--------------------------------------+--------------+---------------+-------------+--------------------+-------------+
| UUID                                 | Name         | Instance UUID | Power State | Provisioning State | Maintenance |
+--------------------------------------+--------------+---------------+-------------+--------------------+-------------+
| 84afdc8d-5ef1-40d6-9699-ceed2038eb51 | ceph-0       | None          | power off   | manageable         | False       |
| 96c6189f-4cc7-4b74-b05a-ef9d4407bf7f | ceph-1       | None          | power off   | manageable         | False       |
| ea2a88ea-5c3b-429f-9eca-f5a2b4369964 | ceph-2       | None          | power off   | manageable         | False       |
| b8e38433-bdb2-4efa-8e9f-1c43f9a15404 | compute-0    | None          | power off   | manageable         | False       |
| e8a8dd6c-be5d-421b-938a-019f87997059 | controller-0 | None          | power off   | manageable         | False       |
| ca84b980-f81a-4a7e-a403-5826323ca778 | controller-1 | None          | power off   | manageable         | False       |
| 91fe16ea-fb0f-4e56-bdbd-738d99ecb892 | controller-2 | None          | power off   | manageable         | False       |
| 948f1601-7f8c-4874-8d44-ce26ae1bc8b8 | ironic-0     | None          | power off   | manageable         | False       |
| e7b8855a-37c0-429c-9343-13f0e7926572 | ironic-1     | None          | None        | manageable         | True        |
+--------------------------------------+--------------+---------------+-------------+--------------------+-------------+



(undercloud) [stack@undercloud-0 ~]$ openstack overcloud node introspect --all-manageable --provide
Waiting for introspection to finish...
Started Mistral Workflow tripleo.baremetal.v1.introspect_manageable_nodes. Execution ID: a5005fbb-3927-4fc6-b506-a9f6f86dc6b5
Waiting for messages on queue 'tripleo' with no timeout.
^[[OIntrospection of node 84afdc8d-5ef1-40d6-9699-ceed2038eb51 completed. Status:SUCCESS. Errors:None
Introspection of node 96c6189f-4cc7-4b74-b05a-ef9d4407bf7f completed. Status:SUCCESS. Errors:None
Introspection of node ea2a88ea-5c3b-429f-9eca-f5a2b4369964 completed. Status:SUCCESS. Errors:None
Introspection of node b8e38433-bdb2-4efa-8e9f-1c43f9a15404 completed. Status:SUCCESS. Errors:None
Introspection of node e8a8dd6c-be5d-421b-938a-019f87997059 completed. Status:SUCCESS. Errors:None
^[[I^[[OIntrospection of node ca84b980-f81a-4a7e-a403-5826323ca778 completed. Status:SUCCESS. Errors:None
Introspection of node 91fe16ea-fb0f-4e56-bdbd-738d99ecb892 completed. Status:SUCCESS. Errors:None
Introspection of node 948f1601-7f8c-4874-8d44-ce26ae1bc8b8 completed. Status:SUCCESS. Errors:None
Introspection of node e7b8855a-37c0-429c-9343-13f0e7926572 timed out.
Retrying 1 nodes that failed introspection. Attempt 2 of 3
^[[I^[[O^[[I^[[O^[[I^[[O^[[I^[[O^[[I^[[OIntrospection of node e7b8855a-37c0-429c-9343-13f0e7926572 timed out.
Retrying 1 nodes that failed introspection. Attempt 3 of 3



^[[I^[[O^[[I^[[O
^[[I^[[O^[[I^[[OIntrospection of node e7b8855a-37c0-429c-9343-13f0e7926572 timed out.
Retry limit reached with 1 nodes still failing introspection
{u'status': u'RUNNING', u'node_uuids': [u'e7b8855a-37c0-429c-9343-13f0e7926572'], u'failed_introspection': [u'e7b8855a-37c0-429c-9343-13f0e7926572'], u'result': None, u'introspected_nodes': {u'948f1601-7f8c-4874-8d44-ce26ae1bc8b8': {u'uuid': u'948f1601-7f8c-4874-8d44-ce26ae1bc8b8', u'links': [{u'href': u'http://192.168.24.1:5050/v1/introspection/948f1601-7f8c-4874-8d44-ce26ae1bc8b8', u'rel': u'self'}], u'finished_at': u'2018-05-11T20:12:35', u'state': u'finished', u'finished': True, u'error': None, u'started_at': u'2
018-05-11T20:07:15'}, u'b8e38433-bdb2-4efa-8e9f-1c43f9a15404': {u'uuid': u'b8e38433-bdb2-4efa-8e9f-1c43f9a15404', u'links': [{u'href': u'http://192.168.24.1:5050/v1/introspec
Introspection of node e7b8855a-37c0-429c-9343-13f0e7926572 timed out.
Retrying 1 nodes that failed introspection. Attempt 2 of 3
^[[I^[[O^[[I^[[O^[[I^[[O^[[I^[[O^[[I^[[OIntrospection of node e7b8855a-37c0-429c-9343-13f0e7926572 timed out.
Retrying 1 nodes that failed introspection. Attempt 3 of 3
018-05-11T20:07:15'}, u'b8e38433-bdb2-4efa-8e9f-1c43f9a15404': {u'uuid': u'b8e38433-bdb2-4efa-8e9f-1c43f9a15404', u'links': [{u'href': u'http://192.168.24.1:5050/v1/introspec
tion/b8e38433-bdb2-4efa-8e9f-1c43f9a15404', u'rel': u'self'}], u'finished_at': u'2018-05-11T20:09:17', u'state': u'finished', u'finished': True, u'error': None, u'started_at': u'2018-05-11T20:07:12'}, u'e7b8855a-37c0-429c-9343-13f0e7926572': {u'uuid': u'e7b8855a-37c0-429c-9343-13f0e7926572', u'links': [{u'href': u'http://192.168.24.1:5050/v1/introspection/e7b8855a-37c0-429c-9343-13f0e7926572', u'rel': u'self'}], u'finished_at': None, u'state': u'waiting', u'finished': False, u'error': None, u'started_at': u'2018-05-11T20:07:16'}, u'96c6189f-4cc7-4b74-b05a-ef9d4407bf7f': {u'uuid': u'96c6189f-4cc7-4b74-b05a-ef9d4407bf7f', u'links': [{u'href': u'http://192.168.24.1:5050/v1/introspection/96c6189f-4cc7-4b74-b05a-ef9d4407bf7f', u'rel': u'self'}], u'finished_at': u'2018-05-11T20:09:07', u'state': u'finished', u'finished': True, u'error': None, u'started_at': u'2018-05-11T20:07:10'}, u'91fe16ea-fb0f-4e56-bdbd-738d99ecb892': {u'uuid': u'91fe16ea-fb0f-4e56-bdbd-738d99ecb892', u'links': [{u'href': u'http://192.168.24.1:5050/v1/introspection/91fe16ea-fb0f-4e56-bdbd-738d99ecb892', u'rel': u'self'}], u'finished_at': u'2018-05-11T20:10:04', u'state': u'finished', u'finished': True, u'error': None, u'started_at': u'2018-05-11T20:07:14'}, u'ca84b980-f81a-4a7e-a403-5826323ca778': {u'uuid': u'ca84b980-f81a-4a7e-a403-5826323ca778', u'links': [{u'href': u'http://192.168.24.1:5050/v1/introspection/ca84b980-f81a-4a7e-a403-5826323ca778', u'rel': u'self'}], u'finished_at': u'2018-05-11T20:10:02', u'state': u'finished', u'finished': True, u'error': None, u'started_at': u'2018-05-11T20:07:13'}, u'ea2a88ea-5c3b-429f-9eca-f5a2b4369964': {u'uuid': u'ea2a88ea-5c3b-429f-9eca-f5a2b4369964', u'links': [{u'href': u'http://192.168.24.1:5050/v1/introspection/ea2a88ea-5c3b-429f-9eca-f5a2b4369964', u'rel': u'self'}], u'finished_at': u'2018-05-11T20:09:09', u'state': u'finished', u'finished': True, u'error': None, u'started_at': u'2018-05-11T20:07:11'}, u'e8a8dd6c-be5d-421b-938a-019f87997059': {u'uuid': u'e8a8dd6c-be5d-421b-938a-019f87997059', u'links': [{u'href': u'http://192.168.24.1:5050/v1/introspection/e8a8dd6c-be5d-421b-938a-019f87997059', u'rel': u'self'}], u'finished_at': u'2018-05-11T20:09:52', u'state': u'finished', u'finished': True, u'error': None, u'started_at': u'2018-05-11T20:07:13'}, u'84afdc8d-5ef1-40d6-9699-ceed2038eb51': {u'uuid': u'84afdc8d-5ef1-40d6-9699-ceed2038eb51', u'links': [{u'href': u'http://192.168.24.1:5050/v1/introspection/84afdc8d-5ef1-40d6-9699-ceed2038eb51', u'rel': u'self'}], u'finished_at': u'2018-05-11T20:08:57', u'state': u'finished', u'finished': True, u'error': None, u'started_at': u'2018-05-11T20:07:09'}}, u'message': u'Retrying 1 nodes that failed introspection. Attempt 2 of 3 ', u'introspection_attempt': 2}
Exception introspecting nodes: {u'status': u'RUNNING', u'node_uuids': [u'e7b8855a-37c0-429c-9343-13f0e7926572'], u'failed_introspection': [u'e7b8855a-37c0-429c-9343-13f0e7926572'], u'result': None, u'introspected_nodes': {u'948f1601-7f8c-4874-8d44-ce26ae1bc8b8': {u'uuid': u'948f1601-7f8c-4874-8d44-ce26ae1bc8b8', u'links': [{u'href': u'http://192.168.24.1:5050/v1/introspection/948f1601-7f8c-4874-8d44-ce26ae1bc8b8', u'rel': u'self'}], u'finished_at': u'2018-05-11T20:12:35', u'state': u'finished', u'finished': True, u'error': None, u'started_at': u'2018-05-11T20:07:15'}, u'b8e38433-bdb2-4efa-8e9f-1c43f9a15404': {u'uuid': u'b8e38433-bdb2-4efa-8e9f-1c43f9a15404', u'links': [{u'href': u'http://192.168.24.1:5050/v1/introspection/b8e38433-bdb2-4efa-8e9f-1c43f9a15404', u'rel': u'self'}], u'finished_at': u'2018-05-11T20:09:17', u'state': u'finished', u'finished': True, u'error': None, u'started_at': u'2018-05-11T20:07:12'}, u'e7b8855a-37c0-429c-9343-13f0e7926572': {u'uuid': u'e7b8855a-37c0-429c-9343-13f0e7926572', u'links': [{u'href': u'http://192.168.24.1:5050/v1/introspection/e7b8855a-37c0-429c-9343-13f0e7926572', u'rel': u'self'}], u'finished_at': None, u'state': u'waiting', u'finished': False, u'error': None, u'started_at': u'2018-05-11T20:07:16'}, u'96c6189f-4cc7-4b74-b05a-ef9d4407bf7f': {u'uuid': u'96c6189f-4cc7-4b74-b05a-ef9d4407bf7f', u'links': [{u'href': u'http://192.168.24.1:5050/v1/introspection/96c6189f-4cc7-4b74-b05a-ef9d4407bf7f', u'rel': u'self'}], u'finished_at': u'2018-05-11T20:09:07', u'state': u'finished', u'finished': True, u'error': None, u'started_at': u'2018-05-11T20:07:10'}, u'91fe16ea-fb0f-4e56-bdbd-738d99ecb892': {u'uuid': u'91fe16ea-fb0f-4e56-bdbd-738d99ecb892', u'links': [{u'href': u'http://192.168.24.1:5050/v1/introspection/91fe16ea-fb0f-4e56-bdbd-738d99ecb892', u'rel': u'self'}], u'finished_at': u'2018-05-11T20:10:04', u'state': u'finished', u'finished': True, u'error': None, u'started_at': u'2018-05-11T20:07:14'}, u'ca84b980-f81a-4a7e-a403-5826323ca778': {u'uuid': u'ca84b980-f81a-4a7e-a403-5826323ca778', u'links': [{u'href': u'http://192.168.24.1:5050/v1/introspection/ca84b980-f81a-4a7e-a403-5826323ca778', u'rel': u'self'}], u'finished_at': u'2018-05-11T20:10:02', u'state': u'finished', u'finished': True, u'error': None, u'started_at': u'2018-05-11T20:07:13'}, u'ea2a88ea-5c3b-429f-9eca-f5a2b4369964': {u'uuid': u'ea2a88ea-5c3b-429f-9eca-f5a2b4369964', u'links': [{u'href': u'http://192.168.24.1:5050/v1/introspection/ea2a88ea-5c3b-429f-9eca-f5a2b4369964', u'rel': u'self'}], u'finished_at': u'2018-05-11T20:09:09', u'state': u'finished', u'finished': True, u'error': None, u'started_at': u'2018-05-11T20:07:11'}, u'e8a8dd6c-be5d-421b-938a-019f87997059': {u'uuid': u'e8a8dd6c-be5d-421b-938a-019f87997059', u'links': [{u'href': u'http://192.168.24.1:5050/v1/introspection/e8a8dd6c-be5d-421b-938a-019f87997059', u'rel': u'self'}], u'finished_at': u'2018-05-11T20:09:52', u'state': u'finished', u'finished': True, u'error': None, u'started_at': u'2018-05-11T20:07:13'}, u'84afdc8d-5ef1-40d6-9699-ceed2038eb51': {u'uuid': u'84afdc8d-5ef1-40d6-9699-ceed2038eb51', u'links': [{u'href': u'http://192.168.24.1:5050/v1/introspection/84afdc8d-5ef1-40d6-9699-ceed2038eb51', u'rel': u'self'}], u'finished_at': u'2018-05-11T20:08:57', u'state': u'finished', u'finished': True, u'error': None, u'started_at': u'2018-05-11T20:07:09'}}, u'message': u'Retrying 1 nodes that failed introspection. Attempt 2 of 3 ', u'introspection_attempt': 2}

Comment 1 Bob Fournier 2018-05-14 16:28:37 UTC
Would like to see if Workflows DFG can take a look at this. Please send back to HardProv if this looks like it needs Ironic changes.

Comment 6 Dougal Matthews 2018-10-17 10:29:23 UTC
I think this bug would be resolved in tripleoclient.


Note You need to log in before you can comment on or make changes to this bug.