| Summary: | Introspection fails with 'mistral engine error' on Ocata if an 'ironic node-update' is done before introspection runs | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Ronelle Landy <rlandy> | ||||||||
| Component: | openstack-tripleo-common | Assignee: | Dougal Matthews <dmatthew> | ||||||||
| Status: | CLOSED EOL | QA Contact: | Dan Yasny <dyasny> | ||||||||
| Severity: | high | Docs Contact: | |||||||||
| Priority: | medium | ||||||||||
| Version: | 11.0 (Ocata) | CC: | atelang, bfournie, brad, dtantsur, mburns, rbrady, rlandy, slinaber, yrachman, zgreenbe | ||||||||
| Target Milestone: | --- | Keywords: | Reopened, Triaged, ZStream | ||||||||
| Target Release: | --- | ||||||||||
| Hardware: | Unspecified | ||||||||||
| OS: | Unspecified | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||||
| Doc Text: | Story Points: | --- | |||||||||
| Clone Of: | Environment: | ||||||||||
| Last Closed: | 2018-06-22 12:29:32 UTC | Type: | Bug | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Attachments: |
|
||||||||||
Are you sure that node-update is really related? I can think of no way it could affect Mistral workflows... Dmitry, I don't have any direct proof that node-update is related. CI only reproduces the error on one environment (and that environment happens to use disk hints and therefore requires update to run) but it could be some other feature of this environment that triggers the failure. If you have any other debugging steps to suggest, I can run through those to narrow down the problem. Sorry, this issue fall out of my radar. Is it still a problem? I have the suggested workaround:
>> sudo mistral-db-manage populate
in place in the CI code. I would have to take the workaround out to retest.
Hard to tell if this is just a quirk of one particular hardware setup or a more generic issue that shows up.
As we don't have a way to duplicate this or make any progress on it, and it doesn't appear to be happening over the last year I'd like to close this. Please reopen if the problem occurs again. Thanks. Hi, I'm reopening the following bug as I've encountered the same issue in my environment as well. My deployment as following: OSP11 with puddle 2017-12-05.1 Hybrid environment includes virtual nodes and baremetal nodes (virt undercloud, virt 3 controllers, and 1 bm compute). Deployed with an infrared tool. FYI, I've applied the proposed workaround in c#4, it worked for me as well. I'm attaching the sosreport, ironic's logs, and a full introspection execution error log. Thanks. Ziv Created attachment 1363285 [details]
sosreport
Created attachment 1363286 [details]
introspection error log
Created attachment 1363287 [details]
ironic's logs
> Invalid input [name=baremetal_introspection.introspect, class=NoneType, unexpected=[u\'uuid\'] Taken from the command output. That suggests that the Mistral action failed to correctly load. This can be worked around usually by running... > sudo mistral-db-manage populate After that you may need to restart Mistral too. OK, that's what I did, I've applied the W/A. But what is the root cause of the issue, please? Ziv - thanks for the logs. What I see in the ironic inspector log are those "Could not find node xxx in cache" errors: ironic/ironic-inspector.log:2017-12-05 10:52:14.115 26036 ERROR ironic_inspector.utils [-] Could not find node c92928f2-dc60-414d-8de3-0b5959970743 in cache ironic/ironic-inspector.log:2017-12-05 10:52:14.115 26036 DEBUG ironic_inspector.main [-] Returning error to client: Could not find node c92928f2-dc60-414d-8de3-0b5959970743 in cache error_response /usr/lib/python2.7/site-packages/ironic_inspector/main.py:79 ironic/ironic-inspector.log:2017-12-05 10:52:14.123 26036 ERROR ironic_inspector.utils [-] Could not find node cd1dad5c-80f3-4dae-8d97-981afccf6754 in cache ironic/ironic-inspector.log:2017-12-05 10:52:14.123 26036 DEBUG ironic_inspector.main [-] Returning error to client: Could not find node cd1dad5c-80f3-4dae-8d97-981afccf6754 in cache error_response /usr/lib/python2.7/site-packages/ironic_inspector/main.py:79 ironic/ironic-inspector.log:2017-12-05 10:52:14.129 26036 ERROR ironic_inspector.utils [-] Could not find node 1b4ec862-7244-4d6f-a33e-9eced28ac5b2 in cache ironic/ironic-inspector.log:2017-12-05 10:52:14.129 26036 DEBUG ironic_inspector.main [-] Returning error to client: Could not find node 1b4ec862-7244-4d6f-a33e-9eced28ac5b2 in cache error_response /usr/lib/python2.7/site-packages/ironic_inspector/main.py:79 This looks similar to https://bugzilla.redhat.com/show_bug.cgi?id=1519232, it has the same "Could not find node xxx in cache" errors. That one was closed because of a misconfiguration within the env". Is it the same issue here? The reason I closed it, is because in our full BM CI the introspection did pass eventually with no particular issue. For now, seems like the problem coexists in the hybrid environments only. Thanks Ziv. I think Mistral is out of sync and using older Ironic node IDs in the introspection bulk command, which is why Ironic generates all of these "Could not find node xxx in cache". The node IDs in the error message don't match the current Ironic node IDs listed in capture. [stack@undercloud-0 ~]$ openstack baremetal node list +--------------------------------------+------------------+---------------+-------------+--------------------+-------------+ | UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance | +--------------------------------------+------------------+---------------+-------------+--------------------+-------------+ | 78ed5643-3efd-4ced-9a8b-595dabc9f169 | computeovsdpdk-0 | None | power off | manageable | False | | b3878de0-6504-4f39-ae17-668a25b045c0 | controller-0 | None | power off | manageable | False | | b9899335-9c03-4832-bb81-7b333633ecc8 | controller-1 | None | power off | manageable | False | | 33ba313d-7f19-46ba-8e06-bc88ce3cbae4 | controller-2 | None | power off | manageable | False | +--------------------------------------+------------------+---------------+-------------+--------------------+-------------+ The logs and workaround point to this being a workflows issue, not an Ironic one. I'm moving this to the Workflows DFG to help answer the question in comment 11. OSP11 is now retired, see details at https://access.redhat.com/errata/product/191/ver=11/rhel---7/x86_64/RHBA-2018:1828 |
Description of problem: Reproducible on only one of the CI hardware environments. This environment requires disk hints and therefore an 'ironic node-update' needs to be run before introspection starts. Introspection fails with: (from: /var/log/mistral/engine.log) ***************************************** 2016-12-12 12:50:47.806 18707 ERROR mistral.engine.task_handler [-] Failed to run task [error=Invalid input [name=baremetal_introspection.introspect, class=NoneType, unexpected=[u'uuid']], wf=WorkflowExecution {'state_info': None, 'accepted': False, 'output': {}, 'name': u'tripleo.baremetal.v1.introspect', 'workflow_name': u'tripleo.baremetal.v1.introspect', 'state': 'RUNNING', 'created_at': '2016-12-12 12:50:47', 'runtime_context': {'index': 0}, 'updated_at': '2016-12-12 12:50:47', 'spec': {u'input': [u'node_uuids', {u'queue_name': u'tripleo'}], u'tasks': {u'send_message': {u'retry': u'count=5 delay=1', u'name': u'send_message', u'version': '2.0', u'action': u'zaqar.queue_post', u'input': {u'queue_name': u'<% $.queue_name %>', u'messages': {u'body': {u'type': u'tripleo.baremetal.v1.introspect', u'payload': {u'status': u"<% $.get('status', 'SUCCESS') %>", u'message': u"<% $.get('message', '') %>", u'execution': u'<% execution() %>', u'introspected_nodes': u"<% $.get('introspected_nodes', []) %>"}}}}, u'type': 'direct'}, u'set_status_failed_start_introspection': {u'version': '2.0', u'type': 'direct', u'name': u'set_status_failed_start_introspection', u'publish': {u'status': u'FAILED', u'message': u'<% task(start_introspection).result %>', u'introspected_nodes': []}, u'on-success': u'send_message'}, u'wait_for_introspection_to_finish': {u'name': u'wait_for_introspection_to_finish', u'on-success': u'send_message', u'publish': {u'status': u'SUCCESS', u'message': u'Successfully introspected nodes.', u'introspected_nodes': u'<% task(wait_for_introspection_to_finish).result %>'}, u'version': '2.0', u'action': u'baremetal_introspection.wait_for_finish', u'input': {u'uuids': u'<% $.node_uuids %>'}, u'type': 'direct'}, u'start_introspection': {u'with-items': u'uuid in <% $.node_uuids %>', u'name': u'start_introspection', u'on-error': u'set_status_failed_start_introspection', u'on-success': u'wait_for_introspection_to_finish', u'version': '2.0', u'action': u'baremetal_introspection.introspect uuid=<% $.uuid %>', u'type': 'direct'}}, u'description': u'Take a list of nodes and move them through introspection.', u'version': u'2.0', u'name': u'introspect'}, 'workflow_id': u'c8dcb78b-b727-4b8c-ae8b-4adce508695c', 'params': {'index': 0, 'task_execution_id': 'ba7849fa-9fee-4b2a-b51d-c319c234b402'}, 'context': {'openstack': {u'is_target': False, u'project_name': u'admin', u'user_id': u'aed139d0a2dd4603ac1811dd9e1ac588', u'region_name': None, u'roles': [u'admin'], u'auth_uri': u'http://10.8.147.1:5000/v3', u'auth_cacert': None, u'auth_token': u'gAAAAABYTp0lTPzIGrToxqnaTtJR0zJ3ZAiRbH-EvyOOWNMHbtORpRZeplKOihc4DP8v_KW2Z1VfVeU46bpvlRZyHP30bra-pxb2wS80zDjB6cOUgu-3jHh-jaU8OCkfPChqNuZSDPGWLDPusL14wcr1ReptDRoqCpIhdwhFWqkixVpXyB6e_84', u'expires_at': u'2016-12-12T16:50:45.000000Z', u'is_trust_scoped': False, u'service_catalog': {u'is_domain': False, u'methods': [u'password'], u'roles': [{u'id': u'93ac8d5cf04245d3acf90b0818a52eb6', u'name': u'admin'}], u'auth_token': u'gAAAAABYTp0lTPzIGrToxqnaTtJR0zJ3ZAiRbH-EvyOOWNMHbtORpRZeplKOihc4DP8v_KW2Z1VfVeU46bpvlRZyHP30bra-pxb2wS80zDjB6cOUgu-3jHh-jaU8OCkfPChqNuZSDPGWLDPusL14wcr1ReptDRoqCpIhdwhFWqkixVpXyB6e_84', u'expires_at': u'2016-12-12T16:50:45.000000Z', u'project': {u'domain': {u'id': u'default', u'name': u'Default'}, u'id': u'089d583354d74f86815e0b535f97ddeb', u'name': u'admin'}, u'catalog': [{u'endpoints': [{u'url': u'http://10.8.147.1:5050', u'interface': u'public', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'54bc453d3ef643d99b01779608fb312f'}, {u'url': u'http://10.8.147.1:5050', u'interface': u'internal', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'62c65def6e4d461990d68e2df7fa1cab'}, {u'url': u'http://10.8.147.1:5050', u'interface': u'admin', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'fa6413b1ed3b4caaa3365a5aa2dbbd09'}], u'type': u'baremetal-introspection', u'id': u'0763c34e3e4144899019dd1ca236febf', u'name': u'ironic-inspector'}, {u'endpoints': [{u'url': u'http://10.8.147.1:8080/v1/AUTH_089d583354d74f86815e0b535f97ddeb', u'interface': u'internal', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'33555740738f4d089a75ddb3aaa59fb4'}, {u'url': u'http://10.8.147.1:8080/v1/AUTH_089d583354d74f86815e0b535f97ddeb', u'interface': u'public', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'5bccd075ec6a4f399d77fa5f30dcc2f7'}, {u'url': u'http://10.8.147.1:8080', u'interface': u'admin', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'9bb1067de7ff47ff8cfcfafdcabf5efe'}], u'type': u'object-store', u'id': u'1673c7744f334fb08cd0131f8e830f70', u'name': u'swift'}, {u'endpoints': [{u'url': u'http://10.8.147.1:9696', u'interface': u'internal', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'6cbf467fb5c545fa8526f8be53ef6ccd'}, {u'url': u'http://10.8.147.1:9696', u'interface': u'admin', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'8e308179ecbb4532a96ec712208f706b'}, {u'url': u'http://10.8.147.1:9696', u'interface': u'public', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'9b73d8f592d6424380a9422c958196be'}], u'type': u'network', u'id': u'2172c2272b984665906004b557469472', u'name': u'neutron'}, {u'endpoints': [{u'url': u'http://10.8.147.1:6385', u'interface': u'public', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'6df34b7f1eec467bb14b3944c37059a6'}, {u'url': u'http://10.8.147.1:6385', u'interface': u'internal', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'c7f02ebb380c4186bde1d008c39513dc'}, {u'url': u'http://10.8.147.1:6385', u'interface': u'admin', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'ff3033285f3647b586100ae0db6f605f'}], u'type': u'baremetal', u'id': u'26d65f0d77674b06a637eb6a53bfb7b0', u'name': u'ironic'}, {u'endpoints': [{u'url': u'http://10.8.147.1:35357/v2.0', u'interface': u'admin', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'2e050847902a44c99151f4a4f71aab34'}, {u'url': u'http://10.8.147.1:5000/v2.0', u'interface': u'internal', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'5e9bb0f0964748ffa43c58cb38d45590'}, {u'url': u'http://10.8.147.1:5000/v2.0', u'interface': u'public', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'a7dec80e790c4ce0b2c3bd417c6c9fc6'}], u'type': u'identity', u'id': u'272b28f1684244c29593c9b63353a501', u'name': u'keystone'}, {u'endpoints': [{u'url': u'ws://10.8.147.1:9000', u'interface': u'public', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'6b588bcef9c842dfa8b3ace7431ce8ff'}, {u'url': u'ws://10.8.147.1:9000', u'interface': u'admin', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'a9fc05ebd6874f84b3abe6207eef4f38'}, {u'url': u'ws://10.8.147.1:9000', u'interface': u'internal', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'c230dafd206b4ca286c4df79518b0141'}], u'type': u'messaging-websocket', u'id': u'33dd7330c74042b38e5ee3caa6a6fae1', u'name': u'zaqar-websocket'}, {u'endpoints': [{u'url': u'http://10.8.147.1:8888', u'interface': u'internal', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'af9f6453f53c44d9a237baa3ee2870fd'}, {u'url': u'http://10.8.147.1:8888', u'interface': u'admin', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'b468ef18904248c484fd17ddf7e12e11'}, {u'url': u'http://10.8.147.1:8888', u'interface': u'public', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'df771240c9b742689cc55eb005dc772a'}], u'type': u'messaging', u'id': u'4ab09ba9a3ba4f4dbe80d437a09fe3c7', u'name': u'zaqar'}, {u'endpoints': [{u'url': u'http://10.8.147.1:8989/v2', u'interface': u'public', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'1bcad173369f47d6a258637bd6ecfd9a'}, {u'url': u'http://10.8.147.1:8989/v2', u'interface': u'internal', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'453f965aa888481aaf3d3949b28c7e35'}, {u'url': u'http://10.8.147.1:8989/v2', u'interface': u'admin', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'eca9aaa384e64246845212b00a4bb3b4'}], u'type': u'workflowv2', u'id': u'63a803930c2c4203b8f088644a6a8d67', u'name': u'mistral'}, {u'endpoints': [{u'url': u'http://10.8.147.1:9292', u'interface': u'internal', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'5cf9634030ac4a1084b9c94f24356986'}, {u'url': u'http://10.8.147.1:9292', u'interface': u'admin', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'725bd2f217084e98b6a2810c4c04eab9'}, {u'url': u'http://10.8.147.1:9292', u'interface': u'public', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'7f66ff1b1ab142caa1b27532d9888c00'}], u'type': u'image', u'id': u'826a229aa9e74ed1a7de521f90fb0d4b', u'name': u'glance'}, {u'endpoints': [{u'url': u'http://10.8.147.1:8777', u'interface': u'admin', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'4659dc3c2a4d4a1792ad5118182a2e13'}, {u'url': u'http://10.8.147.1:8777', u'interface': u'public', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'7e675da024fe4c4ab635dbd1535504c3'}, {u'url': u'http://10.8.147.1:8777', u'interface': u'internal', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'caf0218bd21948e985390caae52dc9df'}], u'type': u'metering', u'id': u'93b3af96a1604e9ea2c32e3016e41d24', u'name': u'ceilometer'}, {u'endpoints': [{u'url': u'http://10.8.147.1:8042', u'interface': u'internal', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'184dd7cd9da94353b512746a0d11e528'}, {u'url': u'http://10.8.147.1:8042', u'interface': u'admin', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'b0c47d4e0f3c4646b36f7a07aad5bc92'}, {u'url': u'http://10.8.147.1:8042', u'interface': u'public', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'e53db6a1b21b4765b1eb66364a699d26'}], u'type': u'alarming', u'id': u'a39fe5df7e1647088a8ec3f9a71cfcec', u'name': u'aodh'}, {u'endpoints': [{u'url': u'http://10.8.147.1:8774/v2.1', u'interface': u'internal', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'08653ccb172049ad89ca1926307a0be8'}, {u'url': u'http://10.8.147.1:8774/v2.1', u'interface': u'public', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'207e5abb177a45368abe20a784c27c20'}, {u'url': u'http://10.8.147.1:8774/v2.1', u'interface': u'admin', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'3f8748b060744293aad1a2f60d9e5724'}], u'type': u'compute', u'id': u'a67e06c4adde4abfb5ef7f43429bc337', u'name': u'nova'}, {u'endpoints': [{u'url': u'http://10.8.147.1:8004/v1/089d583354d74f86815e0b535f97ddeb', u'interface': u'admin', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'32eeb6b86a7948c49c0eba7447b0436c'}, {u'url': u'http://10.8.147.1:8004/v1/089d583354d74f86815e0b535f97ddeb', u'interface': u'internal', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'4bd0413b6b3e46b0b9450edcb5163889'}, {u'url': u'http://10.8.147.1:8004/v1/089d583354d74f86815e0b535f97ddeb', u'interface': u'public', u'region': u'regionOne', u'region_id': u'regionOne', u'id': u'6ca77385fd1d44128a3e66303b7450ab'}], u'type': u'orchestration', u'id': u'bf112b0a1d4742ef8c53e8303a4d9c59', u'name': u'heat'}], u'version': u'v3', u'user': {u'domain': {u'id': u'default', u'name': u'Default'}, u'id': u'aed139d0a2dd4603ac1811dd9e1ac588', u'name': u'admin', u'password_expires_at': None}, u'audit_ids': [u'PcF0fzdaR8aKqk6hhWqWNg'], u'issued_at': u'2016-12-12T12:50:45.000000Z'}, u'project_id': u'089d583354d74f86815e0b535f97ddeb', u'user_name': u'admin'}, '__execution': {'id': '78de6b67-3ffc-468b-a317-41c69eefde1d'}}, 'input': {u'queue_name': u'dc9e4d0b-7d77-4e77-8f34-cbde2126544d', u'node_uuids': [u'8dc1bedf-b1ca-4738-8721-a222b5458780', u'0d8beb91-7bb3-4051-ae41-8be439be60aa', u'c3ab5301-348a-4f25-8134-44badbb96e58', u'bb836b8f-ec8e-4ebd-b5cd-a2a7dee96451']}, 'scope': 'private', 'project_id': u'089d583354d74f86815e0b535f97ddeb', 'task_execution_id': 'ba7849fa-9fee-4b2a-b51d-c319c234b402', 'id': '78de6b67-3ffc-468b-a317-41c69eefde1d', 'description': 'sub-workflow execution'}, task=start_introspection]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/mistral/engine/task_handler.py", line 58, in run_task task.run() File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 154, in wrapper return f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/mistral/engine/tasks.py", line 270, in run self._run_new() File "/usr/lib/python2.7/site-packages/mistral/engine/tasks.py", line 293, in _run_new self._schedule_actions() File "/usr/lib/python2.7/site-packages/mistral/engine/tasks.py", line 465, in _schedule_actions action.validate_input(input_dict) File "/usr/lib/python2.7/site-packages/mistral/engine/actions.py", line 279, in validate_input e_utils.validate_input(self.action_def, input_dict) File "/usr/lib/python2.7/site-packages/mistral/engine/utils.py", line 58, in validate_input msg % tuple(msg_props) InputException: Invalid input [name=baremetal_introspection.introspect, class=NoneType, unexpected=[u'uuid']] ********************************** Version-Release number of selected component (if applicable): [stack@undercloud ~]$ rpm -qa | grep ironic-inspector openstack-ironic-inspector-4.2.1-0.20161207210344.c106d91.el7.centos.noarch python-ironic-inspector-client-1.9.0-0.20161116144913.6364bc9.el7.centos.noarch [stack@undercloud ~]$ rpm -qa | grep mistral openstack-mistral-executor-4.0.0-0.20161209115108.317e4f3.el7.centos.noarch puppet-mistral-10.0.0-0.20161209230917.17d6fcf.el7.centos.noarch python2-mistralclient-2.1.2-0.20161207154141.1dcfc2c.el7.centos.noarch openstack-mistral-common-4.0.0-0.20161209115108.317e4f3.el7.centos.noarch openstack-mistral-api-4.0.0-0.20161209115108.317e4f3.el7.centos.noarch python-mistral-tests-4.0.0-0.20161209115108.317e4f3.el7.centos.noarch openstack-mistral-engine-4.0.0-0.20161209115108.317e4f3.el7.centos.noarch python-openstack-mistral-4.0.0-0.20161209115108.317e4f3.el7.centos.noarch How reproducible: 100% reproducible on specific hardware Steps to Reproduce: 1. On a system that requires disk hints or any other node-update, install the undercloud 2. Register the nodes with ironic and then update the nodes 3. Run bulk introspection Actual results: Introspection fails with mistral error above Expected results: Introspection completes Additional info: ***** WORKAROUND *****: (from dmatthews) Execute: >> sudo mistral-db-manage populate after the node update and before introspection and introspection completes.