Bug 1536521 - Introspection of baremetal nodes fail with 'No hypervisor matching' problem
Summary: Introspection of baremetal nodes fail with 'No hypervisor matching' problem
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-common
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: ---
: 13.0 (Queens)
Assignee: Toure Dunnon
QA Contact: Alexander Chuzhoy
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-01-19 15:08 UTC by Yolanda Robla
Modified: 2018-11-13 22:28 UTC (History)
23 users (show)

Fixed In Version: openstack-tripleo-common-8.6.1-4.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-11-13 22:26:39 UTC


Attachments (Terms of Use)
mistral executor.log (812.88 KB, text/plain)
2018-01-26 01:25 UTC, Bob Fournier
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:3587 None None None 2018-11-13 22:28:09 UTC
OpenStack gerrit 559314 None stable/queens: MERGED tripleo-common: Fix error handling in set_provision_state/set_power_state workflows (Ice19306d4c4a2080b0337bc02a6ccee4a8... 2018-10-15 14:28:48 UTC
OpenStack gerrit 569811 None master: MERGED tripleo-common: Clean up node registration output. (I4b8f8d153a8e8c2eabb9efadb52cf685e1f3759f) 2018-10-15 14:28:43 UTC
Launchpad 1710685 None None None 2018-01-19 15:08:03 UTC

Description Yolanda Robla 2018-01-19 15:08:03 UTC
On the introspection step, on queens, an error message is constantly shown:

$ openstack overcloud node import --provide instackenv.json
Started Mistral Workflow tripleo.baremetal.v1.register_or_update. Execution ID: b49129a9-18ad-443d-b326-7a1b3d3d53b4
Waiting for messages on queue '1c7ccb01-462b-40fb-b824-19f61a82b920' with no timeout.

Nodes set to managed.
Successfully registered node UUID 3cc25cf3-064a-4bb7-ae89-848f03b4dec2
Successfully registered node UUID 0040bc4d-f601-403d-933a-87473a4cf6f0
Started Mistral Workflow tripleo.baremetal.v1.provide. Execution ID: 263d89dd-cab8-4a84-b5c4-c466dd3d28f1
Waiting for messages on queue '1c7ccb01-462b-40fb-b824-19f61a82b920' with no timeout.
[{u'cpu_info': u'', u'manager': {u'api': {u'server_groups': None, u'keypairs': None, u'servers': None, u'server_external_events': None, u'server_migrations': None, u'agents': None, u'instance_action': None, u'glance': None, u'hypervisor_stats': None, u'virtual_interfaces': None, u'flavors': None, u'availability_zones': None, u'user_id': None, u'cloudpipe': None, u'os_cache': False, u'quotas': None, u'migrations': None, u'usage': None, u'logger': None, u'project_id': None, u'neutron': None, u'quota_classes': None, u'project_name': None, u'aggregates': None, u'flavor_access': None, u'services': None, u'list_extensions': None, u'limits': None, u'hypervisors': None, u'cells': None, u'versions': None, u'client': None, u'hosts': None, u'volumes': None, u'assisted_volume_snapshots': None, u'certs': None}}, u'free_disk_gb': 0, u'id': 1, u'service': {u'host': u'undercloud-test.localdomain', u'disabled_reason': None, u'id': 6}, u'local_gb_used': 0, u'memory_mb_used': 0, u'current_workload': 0, u'state': u'up', u'status': u'enabled', u'host_ip': u'12.0.0.3', u'hypervisor_hostname': u'3cc25cf3-064a-4bb7-ae89-848f03b4dec2', u'hypervisor_version': 1, u'disk_available_least': 0, u'local_gb': 0, u'free_ram_mb': 0, u'vcpus_used': 0, u'hypervisor_type': u'ironic', u'x_openstack_request_ids': [u'req-56331e62-bab4-4a0b-8c3c-12fe27e4d1a9'], u'memory_mb': 0, u'vcpus': 0, u'running_vms': 0, u'_info': {u'status': u'enabled', u'service': {u'host': u'undercloud-test.localdomain', u'disabled_reason': None, u'id': 6}, u'vcpus_used': 0, u'hypervisor_type': u'ironic', u'local_gb_used': 0, u'host_ip': u'12.0.0.3', u'hypervisor_hostname': u'3cc25cf3-064a-4bb7-ae89-848f03b4dec2', u'memory_mb_used': 0, u'memory_mb': 0, u'current_workload': 0, u'vcpus': 0, u'state': u'up', u'cpu_info': u'', u'running_vms': 0, u'free_disk_gb': 0, u'hypervisor_version': 1, u'disk_available_least': 0, u'local_gb': 0, u'free_ram_mb': 0, u'id': 1}, u'_loaded': True}, u"Failed to run action [action_ex_id=4b0a56dc-478c-4106-a333-7f6c82286d18, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'0040bc4d-f601-403d-933a-87473a4cf6f0'}']\n NovaAction.hypervisors.find failed: No Hypervisor matching {u'hypervisor_hostname': u'0040bc4d-f601-403d-933a-87473a4cf6f0'}. (HTTP 404)"]

Initially it looked just as an ugly output, but in my tests, i have to stop the provide command, because it just hangs, and also it leaves mistral with lots of ERROR tasks:

openstack overcloud node provide --all-manageable 

[u"Failed to run action [action_ex_id=cdbec835-efc5-4e80-9590-992ace0deb31, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'8e24a698-3956-4ad8-8723-f0bd7d083237'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=8f0de763-3644-4c48-8ae0-cddde09b2ed4, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'8e24a698-3956-4ad8-8723-f0bd7d083237'}']\n NovaAction.hypervisors.find failed: "]
{u'status': u'FAILED', u'message': [u"Failed to run action [action_ex_id=8f0de763-3644-4c48-8ae0-cddde09b2ed4, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'8e24a698-3956-4ad8-8723-f0bd7d083237'}']\n NovaAction.hypervisors.find failed: "], u'result': None}
[u"Failed to run action [action_ex_id=8034ad13-563c-46b4-ab6c-e43fe883794a, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'8e24a698-3956-4ad8-8723-f0bd7d083237'}']\n NovaAction.hypervisors.find failed: "]

And also you can see on mistral:

mistral execution-list | grep "ERROR";
| ba40a872-9170-4b83-ae5b-c78d465051f3 | 87e20d75-4b86-438b-bb0d-7b574c41a4de | tripleo.validations.v1.copy_ssh_key | | <none> | ERROR | Failure caused by error i... | 2018-01-18 14:35:34 | 2018-01-18 14:35:36 |
| 1c0f01f1-0c73-4b4d-8779-c659dd8b9e1f | 17915115-e0b4-44ea-9b9b-c05e5dad5dfc | tripleo.baremetal.v1.provide_manageable_nodes | | <none> | ERROR | Failure caused by error i... | 2018-01-18 15:42:05 | 2018-01-18 15:58:21 |
| 41fc80fa-a6da-44e3-a66f-96325ee37755 | 2a29ba5b-eb15-4cae-9fad-da7149bbf54c | tripleo.baremetal.v1.provide | sub-workflow execution | f3e1bf76-ea9c-4d6a-a090-865f1eba9a58 | ERROR | None | 2018-01-18 15:42:06 | 2018-01-18 15:58:21 |
| b8aeb67f-6874-4721-a56b-56df59e17227 | b75ece3c-c0c9-40b4-98ec-dba1a19b8fe5 | tripleo.baremetal.v1.cellv2_discovery | sub-workflow execution | 04fcd7dc-cbf2-490a-975e-6b79b741cf0d | ERROR | None | 2018-01-18 15:42:09 | 2018-01-18 15:42:15 |
| 80bd4c22-9d4e-4fc2-b276-cee11876cdfa | b75ece3c-c0c9-40b4-98ec-dba1a19b8fe5 | tripleo.baremetal.v1.cellv2_discovery | sub-workflow execution | 04fcd7dc-cbf2-490a-975e-6b79b741cf0d | ERROR | None | 2018-01-18 15:42:44 | 2018-01-18 15:42:50 |

Comment 1 Bob Fournier 2018-01-24 20:44:03 UTC
In my experience, I've been seeing this error message also, but it seems harmless and doesn't cause any problems with further baremetal commands.

(undercloud) [stack@host01 ~]$ openstack overcloud node import --provide instackenv.json
Started Mistral Workflow tripleo.baremetal.v1.register_or_update. Execution ID: 3a524abb-4ba7-4029-aab0-50a46bb1f5a4
Waiting for messages on queue 'ba202ca1-57ea-4403-b157-b97e96040fbc' with no timeout.


Nodes set to managed.
Successfully registered node UUID 618a604d-cc41-4c10-945b-c5e6b5f01020
Successfully registered node UUID ccf4fa3c-9c57-4727-9b6a-396f54b63c1d
Started Mistral Workflow tripleo.baremetal.v1.provide. Execution ID: 954f600c-1032-4dc4-859b-dc63c8f0a1b8
Waiting for messages on queue 'ba202ca1-57ea-4403-b157-b97e96040fbc' with no timeout.
[u"Failed to run action [action_ex_id=55fe1277-dad9-44a0-bd9d-061625e4c1a4, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'618a604d-cc41-4c10-945b-c5e6b5f01020'}']\n NovaAction.hypervisors.find failed: No Hypervisor matching {u'hypervisor_hostname': u'618a604d-cc41-4c10-945b-c5e6b5f01020'}. (HTTP 404)", u"Failed to run action [action_ex_id=b71eb254-c675-490a-bcd6-fa3da612b462, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'ccf4fa3c-9c57-4727-9b6a-396f54b63c1d'}']\n NovaAction.hypervisors.find failed: No Hypervisor matching {u'hypervisor_hostname': u'ccf4fa3c-9c57-4727-9b6a-396f54b63c1d'}. (HTTP 404)"]

Successfully set nodes state to available.
(undercloud) [stack@host01 ~]$ openstack overcloud node provide --all-manageable
Started Mistral Workflow tripleo.baremetal.v1.provide_manageable_nodes. Execution ID: 62c1bb17-907d-487e-a9d1-d5cb5d7d921b
Waiting for messages on queue '76f08097-e567-40fa-9cc3-d2ed3f3e70df' with no timeout.

Successfully set nodes state to available.

========

Can we get more info on your setup and ironic logs?

That being said, it is a bug as we shouldn't be getting this failure on import, but I'm hesitant to flag it as a blocker unless we have evidence that its causing the "provide" command to fail.

Comment 2 Dmitry Tantsur 2018-01-25 17:01:16 UTC
Ironic logs are useless here, this failure has barely anything to do with ironic itself. I suspect it's either an issue in tripleo-common workflows or even overly extensive logging there. No time to look in depth right now, unfortunately. But is much be related to some post-providining validations of nova state.

Changing the component according to my quick guess.

Comment 3 Bob Fournier 2018-01-26 01:23:39 UTC
Workflows DFG - sending this bug your way to see if you can have a look. The problem is easy to reproduce, it happens every time that I run the import command and I can provide a system for debugging.  Also, to me it appears to be an innocuous - I haven't seen it as a symptom or other problems.  I know Yolanda has seen that the provides also fails but that can be due to other issues with node cleaning unrelated to this message.
The problem has also been occurring for me for a couple months but since it wasn't causing a problem I ignored it.

A couple things to note - the hypervisor that is being flagged as "No Hypervisor matching" actually corresponds to the node uuid of a baremetal node.  So its possible that the check is being done too early.

I've attached part of the mistral executor log that shows the stack trace.  

It looks like this action is causing the problem.
https://github.com/openstack/tripleo-common/blob/master/workbooks/baremetal.yaml#L1062

Anjali - I've removed the blocker flag '?' as this does not seem to be indicative of a problem besides the message.

Comment 4 Bob Fournier 2018-01-26 01:25:43 UTC
Created attachment 1386403 [details]
mistral executor.log

Comment 5 Bob Fournier 2018-01-26 14:46:23 UTC
Also, here are the baremetal node uuids to correlate with the mistral log.  This problem seems to only be occurring when using bsremetal nodes.

(undercloud) [stack@host01 ~]$ openstack baremetal node list
+--------------------------------------+-------+--------------------------------------+-------------+--------------------+-------------+
| UUID                                 | Name  | Instance UUID                        | Power State | Provisioning State | Maintenance |
+--------------------------------------+-------+--------------------------------------+-------------+--------------------+-------------+
| 618a604d-cc41-4c10-945b-c5e6b5f01020 | host2 | 5cbae7f7-7993-4032-bdde-a787bc6f84e9 | power on    | active             | False       |
| ccf4fa3c-9c57-4727-9b6a-396f54b63c1d | host3 | 21e69dbd-4f36-4114-b107-56d994918cd0 | power on    | active             | False       |
+--------------------------------------+-------+--------------------------------------+-------------+--------------------+-------------+

Comment 6 Piotr Kopec 2018-02-28 18:27:33 UTC
I am getting this on Pike as well when I use name and not baremetal's UUID in command *openstack  overcloud node introspect --provide*. When I use baremetal's UUID it does not occur. Hope that helps.

Comment 7 Piotr Kopec 2018-02-28 18:35:21 UTC
(In reply to Piotr Kopec from comment #6)
> I am getting this on Pike as well when I use name and not baremetal's UUID
> in command *openstack  overcloud node introspect --provide*. When I use
> baremetal's UUID it does not occur. Hope that helps.

undercloud hypervisros use UUID as Hypervisor Hostname

(undercloud) [stack@undercloud ~]$ openstack hypervisor list -c ID -c Hypervisor\ Hostname
+-----+--------------------------------------+
|  ID | Hypervisor Hostname                  |
+-----+--------------------------------------+
| 144 | 8316ad5f-2f1c-44aa-b4d8-826683663a54 |
| 145 | ca99d7af-0699-4401-8810-5c5392c9b22e |
| 146 | 9985176c-5ee0-47d6-b293-0bf5ae5d4243 |
| 147 | 66bfcbfd-9154-4378-98df-e161449cbd23 |
| 148 | 25db0ff9-8cc9-43d5-bcea-6f77edbfc3d9 |
| 149 | 4ab1aada-1709-4629-85ad-4597e4fb371a |
| 150 | 0a999565-2c5c-4642-a132-a47a44ff5521 |
| 151 | 89556cbc-4862-4226-8be4-db5a8ec85892 |
| 153 | ff08b5cf-31d9-4125-8f1d-b004981879fd |
| 156 | ddd66092-339e-4408-ba2a-ffd8d1500a38 |
| 158 | abf65ffc-65c9-4c4f-90b9-de79abead7d7 |
| 159 | f3e9359c-907b-4fec-bd49-89bbbec89b77 |
| 161 | 6eee60fb-7ca8-4d4b-aa53-e7da033c8f65 |
| 162 | d67c3bcb-7fbc-44a5-84d1-4aee2ff06521 |
+-----+--------------------------------------+

Comment 8 Yolanda Robla 2018-03-20 09:17:33 UTC
How is status of that bug? Will that be fixed for queens?

Comment 9 Yolanda Robla 2018-03-27 15:44:10 UTC
I continue hitting that on queens. That's not just a warning, is causing the openstack overcloud node introspect [NODE UUID] --provide to hang for extra time, having to kill the command manually.

Comment 10 Yolanda Robla 2018-03-27 15:50:32 UTC
When i abort the command, to stop cycling, and launch the overcloud deploy command to actually add the new node, it fails complaining about the same error:

(undercloud) [stack@undercloud-0 ~]$ bash overcloud_deploy.sh
Started Mistral Workflow tripleo.validations.v1.check_pre_deployment_validations. Execution ID: e8e2e03d-b380-444b-99eb-b94f56d42e6a
Waiting for messages on queue 'tripleo' with no timeout.
Removing the current plan files
Uploading new plan files
Started Mistral Workflow tripleo.plan_management.v1.update_deployment_plan. Execution ID: 28a79518-8d3b-4ca3-a442-e5be85001c5f
[u"Failed to run action [action_ex_id=eabc0848-9cce-4dde-aa7b-d1f841e0ffc5, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'5baab564-5f14-406e-bf2c-ad171333eb3d'}']\n NovaAction.hypervisors.find failed: "]
Plan updated.
Processing templates in the directory /tmp/tripleoclient-MlaIp0/tripleo-heat-templates
Started Mistral Workflow tripleo.plan_management.v1.get_deprecated_parameters. Execution ID: afc7e094-470f-4bcb-acba-e46aa74dcc7a
WARNING: Following parameters are deprecated and still defined. Deprecated parameters will be removed soon!
  HostCpusList
  OvercloudControlFlavor
WARNING: Following parameters are defined but not used in plan. Could be possible that parameter is valid but currently not used.
  ComputeKernelArgs
  TenantNetworkVlanID
  ControlPlaneDefaultRoute
  StorageMgmtNetworkVlanID
  ExternalAllocationPools
  TenantNetCidr
  NeutronPhysicalDevMappings
  EC2MetadataIp
  InternalApiNetCidr
  NeutronSupportedPCIVendorDevs
  InternalApiAllocationPools
  ExternalInterfaceDefaultRoute
  DnsServers
  StorageNetworkVlanID
  StorageAllocationPools
  StorageMgmtAllocationPools
  StorageNetCidr
  StorageMgmtNetCidr
  InternalApiNetworkVlanID
  NeutronSriovNumVFs
  ExternalNetCidr
  TenantAllocationPools
Deploying templates in the directory /tmp/tripleoclient-MlaIp0/tripleo-heat-templates
Started Mistral Workflow tripleo.deployment.v1.deploy_plan. Execution ID: eacbfef7-1abf-44c3-b224-738fff083a75
{u'execution': {u'created_at': u'2018-03-27 15:43:30',
                u'id': u'626d4508-b597-4e55-851b-0cfa7b85c018',
                u'input': {u'node_uuids': [u'5baab564-5f14-406e-bf2c-ad171333eb3d'],
                           u'queue_name': u'tripleo'},
                u'name': u'tripleo.baremetal.v1.cellv2_discovery',
                u'params': {u'index': 0,
                            u'namespace': u'',
                            u'root_execution_id': u'11863553-83a8-476d-9efc-e8fa3d7f799a',
                            u'task_execution_id': u'632b098f-a78b-4169-8b11-d13a07175a4c'},
                u'spec': {u'description': u'Run cell_v2 host discovery',
                          u'input': [u'node_uuids',
                                     {u'queue_name': u'tripleo'}],
                          u'name': u'cellv2_discovery',
                          u'tags': [u'tripleo-common-managed'],
                          u'tasks': {u'cell_v2_discover_hosts': {u'action': u'tripleo.baremetal.cell_v2_discover_hosts',
                                                                 u'name': u'cell_v2_discover_hosts',
                                                                 u'on-error': u'cell_v2_discover_hosts_failed',
                                                                 u'on-success': u'wait_for_nova_resources',
                                                                 u'type': u'direct',
                                                                 u'version': u'2.0'},
                                     u'cell_v2_discover_hosts_failed': {u'name': u'cell_v2_discover_hosts_failed',
                                                                        u'on-success': u'send_message',
                                                                        u'publish': {u'message': u'<% task(cell_v2_discover_hosts).result %>',
                                                                                     u'status': u'FAILED'},
                                                                        u'type': u'direct',
                                                                        u'version': u'2.0'},
                                     u'send_message': {u'action': u'zaqar.queue_post',
                                                       u'input': {u'messages': {u'body': {u'payload': {u'execution': u'<% execution() %>',
                                                                                                       u'message': u"<% $.get('message', '') %>",
                                                                                                       u'status': u"<% $.get('status', 'SUCCESS') %>"},
                                                                                          u'type': u'tripleo.baremetal.v1.cellv2_discovery'}},
                                                                  u'queue_name': u'<% $.queue_name %>'},
                                                       u'name': u'send_message',
                                                       u'on-success': [{u'fail': u'<% $.get(\'status\') = "FAILED" %>'}],
                                                       u'retry': u'count=5 delay=1',
                                                       u'type': u'direct',
                                                       u'version': u'2.0'},
                                     u'wait_for_nova_resources': {u'action': u'nova.hypervisors_find hypervisor_hostname=<% $.node_uuid %>',
                                                                  u'name': u'wait_for_nova_resources',
                                                                  u'on-error': u'wait_for_nova_resources_failed',
                                                                  u'on-success': u'send_message',
                                                                  u'type': u'direct',
                                                                  u'version': u'2.0',
                                                                  u'with-items': u'node_uuid in <% $.node_uuids %>'},
                                     u'wait_for_nova_resources_failed': {u'name': u'wait_for_nova_resources_failed',
                                                                         u'on-success': u'send_message',
                                                                         u'publish': {u'message': u'<% task(wait_for_nova_resources).result %>',
                                                                                      u'status': u'FAILED'},
                                                                         u'type': u'direct',
                                                                         u'version': u'2.0'}},
                          u'version': u'2.0'},
                u'updated_at': u'2018-03-27 15:43:30'},
 u'message': [u"Failed to run action [action_ex_id=cf2b7083-f5b5-4198-849e-90518eeb8275, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'5baab564-5f14-406e-bf2c-ad171333eb3d'}']\n NovaAction.hypervisors.find failed: "],
 u'status': u'FAILED'}

This is urgent as it's stopping adding new nodes after a 13 deploy.

Comment 11 Toure Dunnon 2018-04-18 16:00:05 UTC
A fix has been submitted upstream to address the workflow issue for ironic where the state of node is not tracked which can lead to partial registration.

Comment 12 Yolanda Robla 2018-04-25 06:44:05 UTC
I'm still seeing same issue, even with the patch applied:

(undercloud) [stack@undercloud-0 ~]$  openstack overcloud node introspect 090e64eb-6d9c-4e76-8ea3-f9dee663fc02 --provide
Waiting for introspection to finish...
Started Mistral Workflow tripleo.baremetal.v1.introspect. Execution ID: 56deb951-9d78-4232-ba85-6940510a7783
Waiting for messages on queue 'tripleo' with no timeout.
Introspection of node 090e64eb-6d9c-4e76-8ea3-f9dee663fc02 completed. Status:SUCCESS. Errors:None
Successfully introspected 1 node(s).
Started Mistral Workflow tripleo.baremetal.v1.provide. Execution ID: eaae1af9-6d92-4b3f-be3d-d4b5675fc8ee
Waiting for messages on queue 'tripleo' with no timeout.
[u"Failed to run action [action_ex_id=6d3e3407-613d-4cad-ad5e-6c29649b49ef, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=e64d9241-5e23-4140-b55e-ac708a244d58, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=31077220-4d66-418e-9bad-dc0aec091a77, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=6cc857bb-c264-4f0f-8c1a-0a4d6615fc05, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=3ca4759e-9ff7-4774-bc08-e3cd6689056f, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=d71d3a7e-5ed0-4478-ba92-e2b75ab87c83, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=2c267dbe-f5a3-438e-8dfe-343fd7bc4852, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=f2596cc0-6c63-441a-9d1d-27b2e8a7f15c, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=bb1a7721-5cfb-421c-8fea-f604da2ee7b3, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=38eaefe3-f806-46c6-a65b-fab892094231, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=a0e89834-5b41-4bdb-96c6-55b0afb07895, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=82dc5d83-a6a4-4d3f-bd38-c7c73f77d101, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=cc6e19eb-15ce-4f19-9024-6779a52338bf, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=a8a9c984-4501-4343-82e7-54927712d25d, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=741a0383-5ab7-4ced-991c-1b6ca40e1147, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=ffce6e50-1586-49c4-b478-d90eee51f16c, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=ad246642-3b5f-4fd9-8e78-f64247975e32, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=f4b49dd2-f5ff-41a5-85a5-96204b7a44ab, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=a4f97891-7c55-48f1-91df-e1093ab705c2, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]


[u"Failed to run action [action_ex_id=53256dbd-b363-4f05-b477-2b029659da16, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=08f14ec3-a9cd-4e1a-9e91-26caf61b38b6, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=0ff64675-4255-4e84-bccc-87320a3097b0, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=7bfd1fa3-3a7d-49a4-8f3e-561604377d28, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=78bb8a6c-6411-4444-ad8f-320999efa407, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=a5626184-99ac-44fd-9354-8dbb7af31630, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=4753d722-c46d-44d2-934a-248cfaa8060a, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=edc0a077-ad16-4274-bf7e-8df7f9670349, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=9f5464a9-81be-4009-a8d8-515184b6a642, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=4e986281-3c6f-4089-a93f-eb3fdddc317f, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=82436b8f-b415-45d3-8a8b-46a96d5d95eb, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
{u'status': u'FAILED', u'message': [u"Failed to run action [action_ex_id=82436b8f-b415-45d3-8a8b-46a96d5d95eb, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "], u'result': None}
Failed to set nodes to available state: Failed.
(undercloud) [stack@undercloud-0 ~]$

Comment 17 mlammon 2018-05-01 15:45:33 UTC
I am seeing this issue in latest osp puddle : 2018-04-26.3

Started Mistral Workflow tripleo.baremetal.v1.register_or_update. Execution ID: 079c99d3-8687-4093-93d0-75abe8635a59
Waiting for messages on queue 'tripleo' with no timeout.


4 node(s) successfully moved to the "manageable" state.
Successfully registered node UUID 2b30bb39-577e-4ab8-bc6f-871412ca7089
Successfully registered node UUID 3e18593d-944c-4ac9-89e8-3e06c849ef5c
Successfully registered node UUID 6c9c45f7-2295-4185-adf4-c513426974b7
Successfully registered node UUID 901ee5c7-f05e-4e59-9133-0a927a4d3316
Started Mistral Workflow tripleo.baremetal.v1.provide. Execution ID: cab4886e-b339-44c3-ace5-2025f7990f76
Waiting for messages on queue 'tripleo' with no timeout.
[u"Failed to run action [action_ex_id=5fdd4870-41d3-4f5c-a4b7-19972badbf45, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'2b30bb39-577e-4ab8-bc6f-871412ca7089'}']\n NovaAction.hypervisors.find failed: No Hypervisor matching {u'hypervisor_hostname': u'2b30bb39-577e-4ab8-bc6f-871412ca7089'}. (HTTP 404)", {u'cpu_info': u'', u'manager': {u'api': {u'server_groups': None, u'keypairs': None, u'servers': None, u'server_external_events': None, u'server_migrations': None, u'agents': None, u'instance_action': None, u'glance': None, u'hypervisor_stats': None, u'virtual_interfaces': None, u'flavors': None, u'availability_zones': None, u'user_id': None, u'cloudpipe': None, u'os_cache': False, u'quotas': None, u'migrations': None, u'usage': None, u'logger': None, u'project_id': None, u'neutron': None, u'quota_classes': None, u'project_name': None, u'aggregates': None, u'flavor_access': None, u'services': None, u'list_extensions': None, u'limits': None, u'hypervisors': None, u'cells': None, u'versions': None, u'client': None, u'hosts': None, u'volumes': None, u'assisted_volume_snapshots': None, u'certs': None}}, u'free_disk_gb': 0, u'id': 1, u'service': {u'host': u'undercloud.field.lab.eng.rdu.redhat.com', u'disabled_reason': None, u'id': 6}, u'local_gb_used': 0, u'memory_mb_used': 0, u'current_workload': 0, u'state': u'up', u'status': u'enabled', u'host_ip': u'10.12.50.145', u'hypervisor_hostname': u'3e18593d-944c-4ac9-89e8-3e06c849ef5c', u'hypervisor_version': 1, u'disk_available_least': 0, u'local_gb': 0, u'free_ram_mb': 0, u'vcpus_used': 0, u'hypervisor_type': u'ironic', u'x_openstack_request_ids': [u'req-5249c558-278b-4894-b836-59c667dd5034'], u'memory_mb': 0, u'vcpus': 0, u'running_vms': 0, u'_info': {u'status': u'enabled', u'service': {u'host': u'undercloud.field.lab.eng.rdu.redhat.com', u'disabled_reason': None, u'id': 6}, u'vcpus_used': 0, u'hypervisor_type': u'ironic', u'local_gb_used': 0, u'host_ip': u'10.12.50.145', u'hypervisor_hostname': u'3e18593d-944c-4ac9-89e8-3e06c849ef5c', u'memory_mb_used': 0, u'memory_mb': 0, u'current_workload': 0, u'vcpus': 0, u'state': u'up', u'cpu_info': u'', u'running_vms': 0, u'free_disk_gb': 0, u'hypervisor_version': 1, u'disk_available_least': 0, u'local_gb': 0, u'free_ram_mb': 0, u'id': 1}, u'_loaded': True}, {u'cpu_info': u'', u'manager': {u'api': {u'server_groups': None, u'keypairs': None, u'servers': None, u'server_external_events': None, u'server_migrations': None, u'agents': None, u'instance_action': None, u'glance': None, u'hypervisor_stats': None, u'virtual_interfaces': None, u'flavors': None, u'availability_zones': None, u'user_id': None, u'cloudpipe': None, u'os_cache': False, u'quotas': None, u'migrations': None, u'usage': None, u'logger': None, u'project_id': None, u'neutron': None, u'quota_classes': None, u'project_name': None, u'aggregates': None, u'flavor_access': None, u'services': None, u'list_extensions': None, u'limits': None, u'hypervisors': None, u'cells': None, u'versions': None, u'client': None, u'hosts': None, u'volumes': None, u'assisted_volume_snapshots': None, u'certs': None}}, u'free_disk_gb': None, u'id': 2, u'service': {u'host': u'undercloud.field.lab.eng.rdu.redhat.com', u'disabled_reason': None, u'id': 6}, u'local_gb_used': 0, u'memory_mb_used': 0, u'current_workload': None, u'state': u'up', u'status': u'enabled', u'host_ip': u'10.12.50.145', u'hypervisor_hostname': u'6c9c45f7-2295-4185-adf4-c513426974b7', u'hypervisor_version': 1, u'disk_available_least': 0, u'local_gb': 0, u'free_ram_mb': None, u'vcpus_used': 0, u'hypervisor_type': u'ironic', u'x_openstack_request_ids': [u'req-c09cc2f3-27d0-45bd-a360-1af43a0b7190'], u'memory_mb': 0, u'vcpus': 0, u'running_vms': None, u'_info': {u'status': u'enabled', u'service': {u'host': u'undercloud.field.lab.eng.rdu.redhat.com', u'disabled_reason': None, u'id': 6}, u'vcpus_used': 0, u'hypervisor_type': u'ironic', u'local_gb_used': 0, u'host_ip': u'10.12.50.145', u'hypervisor_hostname': u'6c9c45f7-2295-4185-adf4-c513426974b7', u'memory_mb_used': 0, u'memory_mb': 0, u'current_workload': None, u'vcpus': 0, u'state': u'up', u'cpu_info': u'', u'running_vms': None, u'free_disk_gb': None, u'hypervisor_version': 1, u'disk_available_least': 0, u'local_gb': 0, u'free_ram_mb': None, u'id': 2}, u'_loaded': True}, u"Failed to run action [action_ex_id=5a4d78b7-e88f-454d-aacb-f6e790b47988, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'901ee5c7-f05e-4e59-9133-0a927a4d3316'}']\n NovaAction.hypervisors.find failed: No Hypervisor matching {u'hypervisor_hostname': u'901ee5c7-f05e-4e59-9133-0a927a4d3316'}. (HTTP 404)"]

4 node(s) successfully moved to the "available" state.

(undercloud) [stack@undercloud ~]$ openstack workflow execution list
+--------------------------------------+--------------------------------------+-----------------------------------------------------+--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+---------+------------+---------------------+---------------------+
| ID                                   | Workflow ID                          | Workflow name                                       | Workflow namespace | Description                                                                                                                                                                                                                       | Task Execution ID                    | State   | State info | Created at          | Updated at          |
+--------------------------------------+--------------------------------------+-----------------------------------------------------+--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+---------+------------+---------------------+---------------------+
| bf75e696-8a78-4fd3-846e-47d34b68e898 | 86118777-658e-48a2-85b3-9f81d1596f0a | tripleo.plan_management.v1.create_deployment_plan   |                    |                                                                                                                                                                                                                                   | <none>                               | SUCCESS | None       | 2018-04-30 23:47:40 | 2018-04-30 23:48:40 |
| 7f781cd6-9afa-4748-b795-ad681c5957bb | 374dcfae-021f-4083-8d2b-cd88dde0cee5 | tripleo.validations.v1.copy_ssh_key                 |                    |                                                                                                                                                                                                                                   | <none>                               | SUCCESS | None       | 2018-04-30 23:48:42 | 2018-04-30 23:48:43 |
| 079c99d3-8687-4093-93d0-75abe8635a59 | 8324764f-ae0a-483e-a26a-25a348359583 | tripleo.baremetal.v1.register_or_update             |                    |                                                                                                                                                                                                                                   | <none>                               | SUCCESS | None       | 2018-04-30 23:52:14 | 2018-04-30 23:52:40 |
| cdd65043-4d8f-4f2d-8cde-12888ef8e7c4 | 59318d46-2e66-4152-8a26-8e9e4a76431b | tripleo.baremetal.v1.validate_nodes                 |                    | sub-workflow execution                                                                                                                                                                                                            | 5dc5c6ed-594b-4f44-8dc1-2b2cfaf5298e | SUCCESS | None       | 2018-04-30 23:52:14 | 2018-04-30 23:52:18 |
| 6fbfb372-3001-4cc3-94a3-f4d4c8a4b2d2 | 9a9fab1e-00d3-4e81-b3c7-09508448c4e8 | tripleo.baremetal.v1.set_node_state                 |                    | sub-workflow execution                                                                                                                                                                                                            | f80fe1bf-3c4d-49e0-9f02-9615f3076eca | SUCCESS | None       | 2018-04-30 23:52:25 | 2018-04-30 23:52:27 |
| ada78894-dd2d-491c-8efd-537a0e0309cc | 20d0d265-2c8f-4ac4-95bc-5e1ec95f83b9 | tripleo.baremetal.v1.manage                         |                    | sub-workflow execution                                                                                                                                                                                                            | 73f3e487-677c-4d16-bb13-ab4cf9cbaa5f | SUCCESS | None       | 2018-04-30 23:52:25 | 2018-04-30 23:52:34 |
| b062da09-54ed-45f5-8aef-0abe2fb5402f | 9a9fab1e-00d3-4e81-b3c7-09508448c4e8 | tripleo.baremetal.v1.set_node_state                 |                    | sub-workflow execution                                                                                                                                                                                                            | f80fe1bf-3c4d-49e0-9f02-9615f3076eca | SUCCESS | None       | 2018-04-30 23:52:25 | 2018-04-30 23:52:29 |
| ceba4fa3-b4fa-49f6-83a6-b103b9e23140 | 9a9fab1e-00d3-4e81-b3c7-09508448c4e8 | tripleo.baremetal.v1.set_node_state                 |                    | sub-workflow execution                                                                                                                                                                                                            | f80fe1bf-3c4d-49e0-9f02-9615f3076eca | SUCCESS | None       | 2018-04-30 23:52:25 | 2018-04-30 23:52:30 |
| d44b9916-0e81-48ec-8c05-18b388430548 | 9a9fab1e-00d3-4e81-b3c7-09508448c4e8 | tripleo.baremetal.v1.set_node_state                 |                    | sub-workflow execution                                                                                                                                                                                                            | f80fe1bf-3c4d-49e0-9f02-9615f3076eca | SUCCESS | None       | 2018-04-30 23:52:25 | 2018-04-30 23:52:30 |
| 397cc087-80b6-449b-8e66-5100bb861fb1 | 9a9fab1e-00d3-4e81-b3c7-09508448c4e8 | tripleo.baremetal.v1.set_node_state                 |                    | sub-workflow execution                                                                                                                                                                                                            | 5888c3d8-ee99-4036-b16b-d301fd161041 | SUCCESS | None       | 2018-04-30 23:52:40 | 2018-04-30 23:52:42 |
| 51366480-4194-49e6-bc0e-ecdd06159656 | 9a9fab1e-00d3-4e81-b3c7-09508448c4e8 | tripleo.baremetal.v1.set_node_state                 |                    | sub-workflow execution                                                                                                                                                                                                            | 5888c3d8-ee99-4036-b16b-d301fd161041 | SUCCESS | None       | 2018-04-30 23:52:40 | 2018-04-30 23:52:41 |
| 72609978-cbc8-487f-8f45-615f4e993124 | 9a9fab1e-00d3-4e81-b3c7-09508448c4e8 | tripleo.baremetal.v1.set_node_state                 |                    | sub-workflow execution                                                                                                                                                                                                            | 5888c3d8-ee99-4036-b16b-d301fd161041 | SUCCESS | None       | 2018-04-30 23:52:40 | 2018-04-30 23:52:41 |
| cab4886e-b339-44c3-ace5-2025f7990f76 | 294458bb-ebc4-4181-9378-ced70eed8f2b | tripleo.baremetal.v1.provide                        |                    |                                                                                                                                                                                                                                   | <none>                               | SUCCESS | None       | 2018-04-30 23:52:40 | 2018-04-30 23:53:44 |
| d3512a83-2de6-4c98-9d28-b321f0b32f83 | 9a9fab1e-00d3-4e81-b3c7-09508448c4e8 | tripleo.baremetal.v1.set_node_state                 |                    | sub-workflow execution                                                                                                                                                                                                            | 5888c3d8-ee99-4036-b16b-d301fd161041 | SUCCESS | None       | 2018-04-30 23:52:40 | 2018-04-30 23:52:41 |
| d42d8e38-fd0f-4031-8c28-0abf47f2bd96 | 060249b4-8055-4e47-ae53-d1ca4f8daf27 | tripleo.baremetal.v1.cellv2_discovery               |                    | sub-workflow execution                                                                                                                                                                                                            | faf41862-c7f8-46e2-b420-36298e01e416 | ERROR   | None       | 2018-04-30 23:52:43 | 2018-04-30 23:52:48 |
| eb45d48c-f594-40c9-b5cb-ffbf17c6afec | 060249b4-8055-4e47-ae53-d1ca4f8daf27 | tripleo.baremetal.v1.cellv2_discovery               |                    | sub-workflow execution                                                                                                                                                                                                            | faf41862-c7f8-46e2-b420-36298e01e416 | SUCCESS | None       | 2018-04-30 23:53:18 | 2018-04-30 23:53:23 |
| 0b43a3df-7708-4667-a7d8-cc518d08e320 | 7c6e8b24-689c-44b3-a35a-72b3d4c6259d | tripleo.baremetal.v1.set_power_state                |                    | sub-workflow execution                                                                                                                                                                                                            | 37940432-3f14-4846-94bf-c470ee449c37 | SUCCESS | None       | 2018-04-30 23:53:23 | 2018-04-30 23:53:42 |
| 14aeeebd-aa43-4eda-be93-7281aa002835 | 7c6e8b24-689c-44b3-a35a-72b3d4c6259d | tripleo.baremetal.v1.set_power_state                |                    | sub-workflow execution                                                                                                                                                                                                            | 37940432-3f14-4846-94bf-c470ee449c37 | SUCCESS | None       | 2018-04-30 23:53:23 | 2018-04-30 23:53:42 |
| 494f5ce5-0ff5-458a-b2ac-d5313c30391f | 7c6e8b24-689c-44b3-a35a-72b3d4c6259d | tripleo.baremetal.v1.set_power_state                |                    | sub-workflow execution                                                                                                                                                                                                            | 37940432-3f14-4846-94bf-c470ee449c37 | SUCCESS | None       | 2018-04-30 23:53:23 | 2018-04-30 23:53:43 |
| 8c734b1e-3f37-4e1b-9f10-f5d91f3c75b4 | 7c6e8b24-689c-44b3-a35a-72b3d4c6259d | tripleo.baremetal.v1.set_power_state                |                    | sub-workflow execution                                                                                                                                                                                                            | 37940432-3f14-4846-94bf-c470ee449c37 | SUCCESS | None       | 2018-04-30 23:53:23 | 2018-04-30 23:53:38 |
| 8f96316c-77cb-4ed5-bfe7-81fe22649979 | a258162f-550b-47b9-9811-73907d5b832f | tripleo.plan_management.v1.publish_ui_logs_to_swift |                    | {"triggered_by": {"type": "cron_trigger", "id": "09186f88-d080-442a-9ce7-58672cdbd622", "name": "publish-ui-logs-hourly"}, "description": "Workflow execution created by cron trigger '(09186f88-d080-442a-9ce7-58672cdbd622)'."} | <none>                               | SUCCESS | None       | 2018-05-01 00:06:40 | 2018-05-01 00:06:44 |
| 3e1e3db6-02d2-4319-a70b-d7878a415f12 | a258162f-550b-47b9-9811-73907d5b832f | tripleo.plan_management.v1.publish_ui_logs_to_swift |                    | {"triggered_by": {"type": "cron_trigger", "id": "09186f88-d080-442a-9ce7-58672cdbd622", "name": "publish-ui-logs-hourly"}, "description": "Workflow execution created by cron trigger '(09186f88-d080-442a-9ce7-58672cdbd622)'."} | <none>                               | SUCCESS | None       | 2018-05-01 01:07:47 | 2018-05-01 01:07:51 |
| 199bb331-17b3-4a15-b9c3-dcb2a999f9c1 | a258162f-550b-47b9-9811-73907d5b832f | tripleo.plan_management.v1.publish_ui_logs_to_swift |                    | {"triggered_by": {"type": "cron_trigger", "id": "09186f88-d080-442a-9ce7-58672cdbd622", "name": "publish-ui-logs-hourly"}, "description": "Workflow execution created by cron trigger '(09186f88-d080-442a-9ce7-58672cdbd622)'."} | <none>                               | SUCCESS | None       | 2018-05-01 02:09:05 | 2018-05-01 02:09:09 |
| 3bcb9623-f3a1-444c-9bff-ce7cadf5e6ef | a258162f-550b-47b9-9811-73907d5b832f | tripleo.plan_management.v1.publish_ui_logs_to_swift |                    | {"triggered_by": {"type": "cron_trigger", "id": "09186f88-d080-442a-9ce7-58672cdbd622", "name": "publish-ui-logs-hourly"}, "description": "Workflow execution created by cron trigger '(09186f88-d080-442a-9ce7-58672cdbd622)'."} | <none>                               | SUCCESS | None       | 2018-05-01 03:00:14 | 2018-05-01 03:00:17 |
| bd0d8107-caa1-45c6-ab5a-95fc02f4cad7 | a258162f-550b-47b9-9811-73907d5b832f | tripleo.plan_management.v1.publish_ui_logs_to_swift |                    | {"triggered_by": {"type": "cron_trigger", "id": "09186f88-d080-442a-9ce7-58672cdbd622", "name": "publish-ui-logs-hourly"}, "description": "Workflow execution created by cron trigger '(09186f88-d080-442a-9ce7-58672cdbd622)'."} | <none>                               | SUCCESS | None       | 2018-05-01 04:02:11 | 2018-05-01 04:02:15 |
| 247b3ae1-c186-4a9d-83cc-42262728a6dc | a258162f-550b-47b9-9811-73907d5b832f | tripleo.plan_management.v1.publish_ui_logs_to_swift |                    | {"triggered_by": {"type": "cron_trigger", "id": "09186f88-d080-442a-9ce7-58672cdbd622", "name": "publish-ui-logs-hourly"}, "description": "Workflow execution created by cron trigger '(09186f88-d080-442a-9ce7-58672cdbd622)'."} | <none>                               | SUCCESS | None       | 2018-05-01 05:03:36 | 2018-05-01 05:03:40 |
| 278105c7-8aec-49e4-a189-2ab649a86474 | a258162f-550b-47b9-9811-73907d5b832f | tripleo.plan_management.v1.publish_ui_logs_to_swift |                    | {"triggered_by": {"type": "cron_trigger", "id": "09186f88-d080-442a-9ce7-58672cdbd622", "name": "publish-ui-logs-hourly"}, "description": "Workflow execution created by cron trigger '(09186f88-d080-442a-9ce7-58672cdbd622)'."} | <none>                               | SUCCESS | None       | 2018-05-01 06:04:57 | 2018-05-01 06:05:01 |
| bc407e98-6bd3-4e2e-b710-bb1a3eea9a36 | a258162f-550b-47b9-9811-73907d5b832f | tripleo.plan_management.v1.publish_ui_logs_to_swift |                    | {"triggered_by": {"type": "cron_trigger", "id": "09186f88-d080-442a-9ce7-58672cdbd622", "name": "publish-ui-logs-hourly"}, "description": "Workflow execution created by cron trigger '(09186f88-d080-442a-9ce7-58672cdbd622)'."} | <none>                               | SUCCESS | None       | 2018-05-01 07:06:12 | 2018-05-01 07:06:16 |
| d4b6687c-f6ce-4312-bef7-bfd23afcb544 | a258162f-550b-47b9-9811-73907d5b832f | tripleo.plan_management.v1.publish_ui_logs_to_swift |                    | {"triggered_by": {"type": "cron_trigger", "id": "09186f88-d080-442a-9ce7-58672cdbd622", "name": "publish-ui-logs-hourly"}, "description": "Workflow execution created by cron trigger '(09186f88-d080-442a-9ce7-58672cdbd622)'."} | <none>                               | SUCCESS | None       | 2018-05-01 08:08:07 | 2018-05-01 08:08:11 |
| 303d3d0d-d361-459e-bc07-3261ca0b888c | a258162f-550b-47b9-9811-73907d5b832f | tripleo.plan_management.v1.publish_ui_logs_to_swift |                    | {"triggered_by": {"type": "cron_trigger", "id": "09186f88-d080-442a-9ce7-58672cdbd622", "name": "publish-ui-logs-hourly"}, "description": "Workflow execution created by cron trigger '(09186f88-d080-442a-9ce7-58672cdbd622)'."} | <none>                               | SUCCESS | None       | 2018-05-01 09:10:11 | 2018-05-01 09:10:15 |
| 9172f670-633b-4139-adb8-36c11d1cebbd | a258162f-550b-47b9-9811-73907d5b832f | tripleo.plan_management.v1.publish_ui_logs_to_swift |                    | {"triggered_by": {"type": "cron_trigger", "id": "09186f88-d080-442a-9ce7-58672cdbd622", "name": "publish-ui-logs-hourly"}, "description": "Workflow execution created by cron trigger '(09186f88-d080-442a-9ce7-58672cdbd622)'."} | <none>                               | SUCCESS | None       | 2018-05-01 10:01:47 | 2018-05-01 10:01:50 |
| 534a87ed-7d0c-413c-8d00-eed17b024f0b | a258162f-550b-47b9-9811-73907d5b832f | tripleo.plan_management.v1.publish_ui_logs_to_swift |                    | {"triggered_by": {"type": "cron_trigger", "id": "09186f88-d080-442a-9ce7-58672cdbd622", "name": "publish-ui-logs-hourly"}, "description": "Workflow execution created by cron trigger '(09186f88-d080-442a-9ce7-58672cdbd622)'."} | <none>                               | SUCCESS | None       | 2018-05-01 11:03:20 | 2018-05-01 11:03:24 |
| 21701be6-0935-4e93-8890-4664fbc2b4c7 | a258162f-550b-47b9-9811-73907d5b832f | tripleo.plan_management.v1.publish_ui_logs_to_swift |                    | {"triggered_by": {"type": "cron_trigger", "id": "09186f88-d080-442a-9ce7-58672cdbd622", "name": "publish-ui-logs-hourly"}, "description": "Workflow execution created by cron trigger '(09186f88-d080-442a-9ce7-58672cdbd622)'."} | <none>                               | SUCCESS | None       | 2018-05-01 12:05:38 | 2018-05-01 12:05:43 |
| 755317ad-b61c-49a6-a18f-601c0b7c20d8 | a258162f-550b-47b9-9811-73907d5b832f | tripleo.plan_management.v1.publish_ui_logs_to_swift |                    | {"triggered_by": {"type": "cron_trigger", "id": "09186f88-d080-442a-9ce7-58672cdbd622", "name": "publish-ui-logs-hourly"}, "description": "Workflow execution created by cron trigger '(09186f88-d080-442a-9ce7-58672cdbd622)'."} | <none>                               | SUCCESS | None       | 2018-05-01 13:07:26 | 2018-05-01 13:07:31 |
| 40ae71f9-4a0f-46ab-85f9-cf8a3595c73d | 59318d46-2e66-4152-8a26-8e9e4a76431b | tripleo.baremetal.v1.validate_nodes                 |                    | sub-workflow execution                                                                                                                                                                                                            | 62b5a18e-8af3-4e8f-a02e-50f9b89eabfc | SUCCESS | None       | 2018-05-01 13:20:19 | 2018-05-01 13:20:23 |
| 79b62d24-dbec-4e68-8246-622de89b0f56 | 8324764f-ae0a-483e-a26a-25a348359583 | tripleo.baremetal.v1.register_or_update             |                    |                                                                                                                                                                                                                                   | <none>                               | SUCCESS | None       | 2018-05-01 13:20:19 | 2018-05-01 13:20:40 |
| 13d5c475-5b62-4b6f-96a0-cb52fd4cfd0a | 9a9fab1e-00d3-4e81-b3c7-09508448c4e8 | tripleo.baremetal.v1.set_node_state                 |                    | sub-workflow execution                                                                                                                                                                                                            | f791d0c6-cd60-413c-a4b5-192c0d5b4d9b | SUCCESS | None       | 2018-05-01 13:20:29 | 2018-05-01 13:20:32 |
| 4239b080-f5b7-4b28-b357-059f7b499563 | 20d0d265-2c8f-4ac4-95bc-5e1ec95f83b9 | tripleo.baremetal.v1.manage                         |                    | sub-workflow execution                                                                                                                                                                                                            | 97ade1ca-31af-412c-b13f-1ae00ae05907 | SUCCESS | None       | 2018-05-01 13:20:29 | 2018-05-01 13:20:36 |
| 82e328fa-7c2b-4132-b680-b07fcba274cd | 9a9fab1e-00d3-4e81-b3c7-09508448c4e8 | tripleo.baremetal.v1.set_node_state                 |                    | sub-workflow execution                                                                                                                                                                                                            | f791d0c6-cd60-413c-a4b5-192c0d5b4d9b | SUCCESS | None       | 2018-05-01 13:20:29 | 2018-05-01 13:20:34 |
| b502b8a5-a112-452d-80a7-a4567d5b4b24 | 9a9fab1e-00d3-4e81-b3c7-09508448c4e8 | tripleo.baremetal.v1.set_node_state                 |                    | sub-workflow execution                                                                                                                                                                                                            | f791d0c6-cd60-413c-a4b5-192c0d5b4d9b | SUCCESS | None       | 2018-05-01 13:20:29 | 2018-05-01 13:20:34 |
| ebbea7af-e6a2-48f5-adf9-a7242d305497 | 9a9fab1e-00d3-4e81-b3c7-09508448c4e8 | tripleo.baremetal.v1.set_node_state                 |                    | sub-workflow execution                                                                                                                                                                                                            | f791d0c6-cd60-413c-a4b5-192c0d5b4d9b | SUCCESS | None       | 2018-05-01 13:20:29 | 2018-05-01 13:20:32 |
| 67976e6e-c079-4a78-b073-7ccb6f41387f | 9a9fab1e-00d3-4e81-b3c7-09508448c4e8 | tripleo.baremetal.v1.set_node_state                 |                    | sub-workflow execution                                                                                                                                                                                                            | 3d811172-2b93-49a7-b06c-f936419130d0 | SUCCESS | None       | 2018-05-01 13:20:40 | 2018-05-01 13:20:41 |
| 680e7de4-b26c-45fd-adb0-e24b8c5294ed | 9a9fab1e-00d3-4e81-b3c7-09508448c4e8 | tripleo.baremetal.v1.set_node_state                 |                    | sub-workflow execution                                                                                                                                                                                                            | 3d811172-2b93-49a7-b06c-f936419130d0 | SUCCESS | None       | 2018-05-01 13:20:40 | 2018-05-01 13:20:41 |
| 836751fd-03c8-4cfd-9729-e096a421ad27 | 9a9fab1e-00d3-4e81-b3c7-09508448c4e8 | tripleo.baremetal.v1.set_node_state                 |                    | sub-workflow execution                                                                                                                                                                                                            | 3d811172-2b93-49a7-b06c-f936419130d0 | SUCCESS | None       | 2018-05-01 13:20:40 | 2018-05-01 13:20:41 |
| b4990f9d-dbe8-4182-87c3-07351f2d3080 | 9a9fab1e-00d3-4e81-b3c7-09508448c4e8 | tripleo.baremetal.v1.set_node_state                 |                    | sub-workflow execution                                                                                                                                                                                                            | 3d811172-2b93-49a7-b06c-f936419130d0 | SUCCESS | None       | 2018-05-01 13:20:40 | 2018-05-01 13:20:41 |
| b5794849-5c18-46e0-8b3a-3f8a20c043e0 | 294458bb-ebc4-4181-9378-ced70eed8f2b | tripleo.baremetal.v1.provide                        |                    |                                                                                                                                                                                                                                   | <none>                               | SUCCESS | None       | 2018-05-01 13:20:40 | 2018-05-01 13:20:58 |
| 30a84084-bf97-4dd2-9420-a52354f8ac25 | 060249b4-8055-4e47-ae53-d1ca4f8daf27 | tripleo.baremetal.v1.cellv2_discovery               |                    | sub-workflow execution                                                                                                                                                                                                            | 26b8495d-19eb-43a8-bc5f-61651cbeef97 | SUCCESS | None       | 2018-05-01 13:20:42 | 2018-05-01 13:20:53 |
| 09e0e97c-e28d-4ce0-8689-11e074d3b200 | 7c6e8b24-689c-44b3-a35a-72b3d4c6259d | tripleo.baremetal.v1.set_power_state                |                    | sub-workflow execution                                                                                                                                                                                                            | 117e278f-f092-4b0f-84b1-bd341a516240 | SUCCESS | None       | 2018-05-01 13:20:54 | 2018-05-01 13:20:55 |
| 9aff7425-6a01-4fea-88fd-d767e4e118da | 7c6e8b24-689c-44b3-a35a-72b3d4c6259d | tripleo.baremetal.v1.set_power_state                |                    | sub-workflow execution                                                                                                                                                                                                            | 117e278f-f092-4b0f-84b1-bd341a516240 | SUCCESS | None       | 2018-05-01 13:20:54 | 2018-05-01 13:20:55 |
| eb77d861-1921-4240-83cd-360ae55fe184 | 7c6e8b24-689c-44b3-a35a-72b3d4c6259d | tripleo.baremetal.v1.set_power_state                |                    | sub-workflow execution                                                                                                                                                                                                            | 117e278f-f092-4b0f-84b1-bd341a516240 | SUCCESS | None       | 2018-05-01 13:20:54 | 2018-05-01 13:20:55 |
| fcacea22-9eeb-4a48-bcc1-5920cdc20b56 | 7c6e8b24-689c-44b3-a35a-72b3d4c6259d | tripleo.baremetal.v1.set_power_state                |                    | sub-workflow execution                                                                                                                                                                                                            | 117e278f-f092-4b0f-84b1-bd341a516240 | SUCCESS | None       | 2018-05-01 13:20:54 | 2018-05-01 13:20:55 |
| 2f696819-8340-4c13-90ec-edd6d74f0fe2 | a258162f-550b-47b9-9811-73907d5b832f | tripleo.plan_management.v1.publish_ui_logs_to_swift |                    | {"triggered_by": {"type": "cron_trigger", "id": "09186f88-d080-442a-9ce7-58672cdbd622", "name": "publish-ui-logs-hourly"}, "description": "Workflow execution created by cron trigger '(09186f88-d080-442a-9ce7-58672cdbd622)'."} | <none>                               | SUCCESS | None       | 2018-05-01 14:09:02 | 2018-05-01 14:09:05 |
| 81f127d7-c060-4041-901d-a53d6bf2ac04 | a258162f-550b-47b9-9811-73907d5b832f | tripleo.plan_management.v1.publish_ui_logs_to_swift |                    | {"triggered_by": {"type": "cron_trigger", "id": "09186f88-d080-442a-9ce7-58672cdbd622", "name": "publish-ui-logs-hourly"}, "description": "Workflow execution created by cron trigger '(09186f88-d080-442a-9ce7-58672cdbd622)'."} | <none>                               | SUCCESS | None       | 2018-05-01 15:00:35 | 2018-05-01 15:00:38 |
+--------------------------------------+--------------------------------------+-----------------------------------------------------+--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+---------+------------+---------------------+---------------------+

Comment 18 mlammon 2018-05-01 16:42:29 UTC
FYI, I have updated our ovirt staging drivers to use RPM instead of pip installed.

I seem to get this consistently now
openstack overcloud node import /home/stack/instackenv.json --provide
Started Mistral Workflow tripleo.baremetal.v1.register_or_update. Execution ID: b56983fb-98e4-422a-a4e3-d2a6c05286d6
Waiting for messages on queue 'tripleo' with no timeout.


4 node(s) successfully moved to the "manageable" state.
Successfully registered node UUID 2ee83985-b252-4210-bffe-c962cea150ec
Successfully registered node UUID 9fefef00-67bd-4ee3-bf97-e95819746d67
Successfully registered node UUID 80c6e914-5be8-42ab-ac55-bbe132f2d5e7
Successfully registered node UUID dbe9b853-a49e-4f1e-9476-a6744a38355f
Started Mistral Workflow tripleo.baremetal.v1.provide. Execution ID: b6558879-1887-4360-ae32-5d970cf0c3bd
Waiting for messages on queue 'tripleo' with no timeout.
[u"Failed to run action [action_ex_id=92ad8728-caa4-4cbe-9473-e993e87d3dc7, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'2ee83985-b252-4210-bffe-c962cea150ec'}']\n NovaAction.hypervisors.find failed: No Hypervisor matching {u'hypervisor_hostname': u'2ee83985-b252-4210-bffe-c962cea150ec'}. (HTTP 404)", u"Failed to run action [action_ex_id=49fd9725-a262-4fe4-a67d-3ab2d835c42f, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'9fefef00-67bd-4ee3-bf97-e95819746d67'}']\n NovaAction.hypervisors.find failed: No Hypervisor matching {u'hypervisor_hostname': u'9fefef00-67bd-4ee3-bf97-e95819746d67'}. (HTTP 404)", u"Failed to run action [action_ex_id=d68d7e28-6506-48e0-81b7-1335a0167ae8, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'80c6e914-5be8-42ab-ac55-bbe132f2d5e7'}']\n NovaAction.hypervisors.find failed: No Hypervisor matching {u'hypervisor_hostname': u'80c6e914-5be8-42ab-ac55-bbe132f2d5e7'}. (HTTP 404)", u"Failed to run action [action_ex_id=10a6175f-3f87-47f6-a65c-6db8997b4c69, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'dbe9b853-a49e-4f1e-9476-a6744a38355f'}']\n NovaAction.hypervisors.find failed: No Hypervisor matching {u'hypervisor_hostname': u'dbe9b853-a49e-4f1e-9476-a6744a38355f'}. (HTTP 404)"]

4 node(s) successfully moved to the "available" state.

Comment 19 mlammon 2018-05-01 18:56:30 UTC
Here is the import with the (--debug) but not seem any traceback
Using python-ovirt-engine-sdk4-4.2.6-1.el7ev.x86_64.rpm  for staging driver
and seeing error each import.



(undercloud) [stack@undercloud ~]$ openstack overcloud node import /home/stack/instackenv.json --provide --debug
START with options: [u'overcloud', u'node', u'import', u'/home/stack/instackenv.json', u'--provide', u'--debug']
options: Namespace(access_key='', access_secret='***', access_token='***', access_token_endpoint='', access_token_type='', aodh_endpoint='', application_credential_id='', application_credential_name='', application_credential_secret='***', auth_type='password', auth_url='http://172.16.0.1:5000/', cacert=None, cert='', client_id='', client_secret='***', cloud='', code='', consumer_key='', consumer_secret='***', debug=True, default_domain='default', default_domain_id='', default_domain_name='', deferred_help=False, discovery_endpoint='', domain_id='', domain_name='', endpoint='', identity_provider='', identity_provider_url='', insecure=None, inspector_api_version='1', inspector_url=None, interface='', key='', log_file=None, openid_scope='', os_alarming_api_version='2', os_baremetal_api_version='1.34', os_beta_command=False, os_compute_api_version='', os_container_infra_api_version='1', os_data_processing_api_version='1.1', os_data_processing_url='', os_database_api_version='1', os_dns_api_version='2', os_identity_api_version='3', os_image_api_version='', os_key_manager_api_version='1', os_loadbalancer_api_version='2.0', os_metrics_api_version='1', os_network_api_version='', os_object_api_version='', os_orchestration_api_version='1', os_project_id=None, os_project_name=None, os_queues_api_version='2', os_tripleoclient_api_version='1', os_volume_api_version='', os_workflow_api_version='2', passcode='', password='***', profile='', project_domain_id='', project_domain_name='Default', project_id='', project_name='admin', protocol='', redirect_uri='', region_name='', remote_project_domain_id='', remote_project_domain_name='', remote_project_id='', remote_project_name='', roles='', service_provider='', service_provider_endpoint='', service_provider_entity_id='', system_scope='', timing=False, token='***', trust_id='', url='', user='', user_domain_id='', user_domain_name='Default', user_id='', username='admin', verbose_level=3, verify=None)
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://172.16.0.1:5000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'loadbalancer_api_version': '2.0', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.34', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}}
defaults: {u'auth_type': 'password', u'status': u'active', u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', 'api_timeout': None, u'baremetal_api_version': u'1', u'image_api_version': u'2', u'container_infra_api_version': u'1', u'metering_api_version': u'2', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', 'cacert': None, u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', u'application_catalog_api_version': u'1', u'key_manager_api_version': u'v1', u'workflow_api_version': u'2', 'verify': True, u'identity_api_version': u'2.0', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'container_api_version': u'1', u'dns_api_version': u'2', u'object_store_api_version': u'1', u'interface': None, u'disable_vendor_agent': {}}
cloud cfg: {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://172.16.0.1:5000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'loadbalancer_api_version': '2.0', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.34', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}}
compute API version 2, cmd group openstack.compute.v2
network API version 2, cmd group openstack.network.v2
image API version 2, cmd group openstack.image.v2
volume API version 2, cmd group openstack.volume.v2
identity API version 3, cmd group openstack.identity.v3
object_store API version 1, cmd group openstack.object_store.v1
neutronclient API version 2, cmd group openstack.neutronclient.v2
orchestration API version 1, cmd group openstack.orchestration.v1
messaging API version 2, cmd group openstack.messaging.v2
baremetal API version 1.34, cmd group openstack.baremetal.v1
baremetal_introspection API version 1, cmd group openstack.baremetal_introspection.v1
workflow_engine API version 2, cmd group openstack.workflow_engine.v2
tripleoclient API version 1, cmd group openstack.tripleoclient.v1
dns API version 2, cmd group openstack.dns.v2
load_balancer API version 2.0, cmd group openstack.load_balancer.v2
database API version 1, cmd group openstack.database.v1
key_manager API version 1, cmd group openstack.key_manager.v1
alarming API version 2, cmd group openstack.alarming.v2
container_infra API version 1, cmd group openstack.container_infra.v1
data_processing API version 1.1, cmd group openstack.data_processing.v1
metric API version 1, cmd group openstack.metric.v1
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://172.16.0.1:5000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'loadbalancer_api_version': '2.0', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.34', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}}
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://172.16.0.1:5000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'loadbalancer_api_version': '2.0', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.34', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}}
command: overcloud node import -> tripleoclient.v1.overcloud_node.ImportNode (auth=True)
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://172.16.0.1:5000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'loadbalancer_api_version': '2.0', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.34', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}}
Using auth plugin: password
Using parameters {'username': 'admin', 'project_name': 'admin', 'user_domain_name': 'Default', 'auth_url': 'http://172.16.0.1:5000/', 'password': '***', 'project_domain_name': 'Default'}
Get auth_ref
REQ: curl -g -i -X GET http://172.16.0.1:5000/ -H "Accept: application/json" -H "User-Agent: osc-lib/1.9.0 keystoneauth1/3.4.0 python-requests/2.14.2 CPython/2.7.5"
Starting new HTTP connection (1): 172.16.0.1
http://172.16.0.1:5000 "GET / HTTP/1.1" 300 595
RESP: [300] Date: Tue, 01 May 2018 18:44:56 GMT Server: Apache Vary: X-Auth-Token Content-Length: 595 Keep-Alive: timeout=15, max=100 Connection: Keep-Alive Content-Type: application/json
RESP BODY: {"versions": {"values": [{"status": "stable", "updated": "2018-02-28T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.10", "links": [{"href": "http://172.16.0.1:5000/v3/", "rel": "self"}]}, {"status": "deprecated", "updated": "2016-08-04T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://172.16.0.1:5000/v2.0/", "rel": "self"}, {"href": "https://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}]}}

Making authentication request to http://172.16.0.1:5000/v3/auth/tokens
http://172.16.0.1:5000 "POST /v3/auth/tokens HTTP/1.1" 201 8032
{"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "1bc1d1ec75bb47b6a7a90985c40cf85d", "name": "admin"}], "expires_at": "2018-05-01T22:44:56.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "6aeec69946724398b297c03373b82a20", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://172.16.0.1:5050", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "543ff698a0e841a78c3eda1fe30455f2"}, {"url": "http://172.16.0.1:5050", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "c78115215c4047fd91592a8fee609049"}, {"url": "http://172.16.0.1:5050", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "ea2aec627d954ae587415644eb5ec871"}], "type": "baremetal-introspection", "id": "0bfc501a6ae04df0b586047348aaefd6", "name": "ironic-inspector"}, {"endpoints": [{"url": "http://172.16.0.1:8989/v2", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "63ab02f8cbaf4f46a3eec7bc5e75c094"}, {"url": "http://172.16.0.1:8989/v2", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "6425b42a43c04d7fbfd6730b443ddf95"}, {"url": "http://172.16.0.1:8989/v2", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "7c5ed023934d4da2a7db0bb546844934"}], "type": "workflowv2", "id": "3b91c312fee24808a971b951948a2ec0", "name": "mistral"}, {"endpoints": [{"url": "http://172.16.0.1:8000/v1/6aeec69946724398b297c03373b82a20", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "2a9dc1a1e3c8413abb140d040f3456b8"}, {"url": "http://172.16.0.1:8000/v1/6aeec69946724398b297c03373b82a20", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "b1cb6f7ea49e4cbf9a4e4f1aeeaed917"}, {"url": "http://172.16.0.1:8000/v1/6aeec69946724398b297c03373b82a20", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "c00c48b4c0a34e079d0c84381d4aac88"}], "type": "cloudformation", "id": "3f80b635fe18447384008b035b78c1de", "name": "heat-cfn"}, {"endpoints": [{"url": "http://172.16.0.1:5000", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "28de66398fd24e5485ef900196339b6f"}, {"url": "http://172.16.0.1:5000", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "84b57d30caed49e89c07dea4bd7ba91c"}, {"url": "http://172.16.0.1:35357", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "e971a3175dcd48358889c29cbfee53ec"}], "type": "identity", "id": "406ff4958f7a4ec48255ce7caa8eddf9", "name": "keystone"}, {"endpoints": [{"url": "http://172.16.0.1:8774/v2.1", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "3792eeedcf97418981c9352bcc6f2e1c"}, {"url": "http://172.16.0.1:8774/v2.1", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "63e40c28bac04df5b574dd10b0b919ed"}, {"url": "http://172.16.0.1:8774/v2.1", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "82636d1e23514e3f9545633ee3b22cee"}], "type": "compute", "id": "6161dca5ce47423fba47c3b37db0d714", "name": "nova"}, {"endpoints": [{"url": "http://172.16.0.1:8004/v1/6aeec69946724398b297c03373b82a20", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "413e05ec7dc346c7ae86713d81aa5f24"}, {"url": "http://172.16.0.1:8004/v1/6aeec69946724398b297c03373b82a20", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "b44fdd6e7ea34f6cb29155c10c07f515"}, {"url": "http://172.16.0.1:8004/v1/6aeec69946724398b297c03373b82a20", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "cf13a092b5134b1dbff68cccadca40cc"}], "type": "orchestration", "id": "ab2f972c1fd648b8a3119b9c811f24f0", "name": "heat"}, {"endpoints": [{"url": "ws://172.16.0.1:9000", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "280f9de09d9b48aca407a57896875b1e"}, {"url": "ws://172.16.0.1:9000", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "5bf9a8125d594e8582d40d9c9ff74c30"}, {"url": "ws://172.16.0.1:9000", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "a428df9c4bfd4e69935652fbc3426861"}], "type": "messaging-websocket", "id": "c7b9308512bb46be9f5278338680ca2b", "name": "zaqar-websocket"}, {"endpoints": [{"url": "http://172.16.0.1:9292", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "05d57b914b2642a2a3e748187c1fb554"}, {"url": "http://172.16.0.1:9292", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "1e886ecd03a844cd805f4885cf2690a5"}, {"url": "http://172.16.0.1:9292", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "75ac030c2d754ae49559bc0f8745ce3c"}], "type": "image", "id": "d0e94690e3364930960f8ad25f3a97d5", "name": "glance"}, {"endpoints": [{"url": "http://172.16.0.1:8888", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "0d72026835834028b72ef04a3efc41f2"}, {"url": "http://172.16.0.1:8888", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "21dbbfe453884ad09394c49a7b2916f7"}, {"url": "http://172.16.0.1:8888", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "6cbec0d1d1fb4b46a886c2b1a5e49731"}], "type": "messaging", "id": "d9c68cd8b61040158ecd6726a29a64a4", "name": "zaqar"}, {"endpoints": [{"url": "http://172.16.0.1:8778/placement", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "065d9d6375d04df7b2d5b9121ab6d168"}, {"url": "http://172.16.0.1:8778/placement", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "345a88ca6a494d4894a02c555d92db4a"}, {"url": "http://172.16.0.1:8778/placement", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "3665432921e14486935ccb54f4a74f78"}], "type": "placement", "id": "e4f7989d7f804b5aa634cb6da125039b", "name": "placement"}, {"endpoints": [{"url": "http://172.16.0.1:9696", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "6327c072fe44450a99ba768ec962afed"}, {"url": "http://172.16.0.1:9696", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "bf164f4e1e384250a7286a4e2e746b09"}, {"url": "http://172.16.0.1:9696", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "d11ce5393ce5426e8250d940c46d1a3e"}], "type": "network", "id": "e7b5c052cdd0453b9547ead0a1c45605", "name": "neutron"}, {"endpoints": [{"url": "http://172.16.0.1:8080/v1/AUTH_6aeec69946724398b297c03373b82a20", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "6adbb2dbee194d4a94aeef758e0bb925"}, {"url": "http://172.16.0.1:8080/v1/AUTH_6aeec69946724398b297c03373b82a20", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "df1a52bc7e8d4851b2d26018e137cedb"}, {"url": "http://172.16.0.1:8080", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "f64940e346ff46e68a020c3b449d92bb"}], "type": "object-store", "id": "f344feac93d94ee1933c021d545c3bf8", "name": "swift"}, {"endpoints": [{"url": "http://172.16.0.1:6385", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "1229230391464b75b76b507bbdc3056c"}, {"url": "http://172.16.0.1:6385", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "502d4f63d11c4a6ca230c6797283b02b"}, {"url": "http://172.16.0.1:6385", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "910b528d3a1d4318a384c55d4d39534a"}], "type": "baremetal", "id": "fbe5c11766e94e46b277ac0ba65f3a92", "name": "ironic"}], "user": {"domain": {"id": "default", "name": "Default"}, "password_expires_at": null, "name": "admin", "id": "946ce4bcc65c483b8995a2629d9b87ab"}, "audit_ids": ["_Q6RJj7zR9W86oooYPJVrg"], "issued_at": "2018-05-01T18:44:56.000000Z"}}
run(Namespace(env_file=<open file u'/home/stack/instackenv.json', mode 'r' at 0x7f66a3b6d5d0>, instance_boot_option='local', introspect=False, no_deploy_image=False, provide=True, run_validations=False))
take_action(Namespace(env_file=<open file u'/home/stack/instackenv.json', mode 'r' at 0x7f66a3b6d5d0>, instance_boot_option='local', introspect=False, no_deploy_image=False, provide=True, run_validations=False))
Instantiating workflow engine client: <class 'mistralclient.api.v2.client.Client'>
Making authentication request to http://172.16.0.1:5000/v3/auth/tokens
http://172.16.0.1:5000 "POST /v3/auth/tokens HTTP/1.1" 201 8032
{"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "1bc1d1ec75bb47b6a7a90985c40cf85d", "name": "admin"}], "expires_at": "2018-05-01T22:44:57.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "6aeec69946724398b297c03373b82a20", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://172.16.0.1:5050", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "543ff698a0e841a78c3eda1fe30455f2"}, {"url": "http://172.16.0.1:5050", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "c78115215c4047fd91592a8fee609049"}, {"url": "http://172.16.0.1:5050", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "ea2aec627d954ae587415644eb5ec871"}], "type": "baremetal-introspection", "id": "0bfc501a6ae04df0b586047348aaefd6", "name": "ironic-inspector"}, {"endpoints": [{"url": "http://172.16.0.1:8989/v2", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "63ab02f8cbaf4f46a3eec7bc5e75c094"}, {"url": "http://172.16.0.1:8989/v2", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "6425b42a43c04d7fbfd6730b443ddf95"}, {"url": "http://172.16.0.1:8989/v2", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "7c5ed023934d4da2a7db0bb546844934"}], "type": "workflowv2", "id": "3b91c312fee24808a971b951948a2ec0", "name": "mistral"}, {"endpoints": [{"url": "http://172.16.0.1:8000/v1/6aeec69946724398b297c03373b82a20", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "2a9dc1a1e3c8413abb140d040f3456b8"}, {"url": "http://172.16.0.1:8000/v1/6aeec69946724398b297c03373b82a20", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "b1cb6f7ea49e4cbf9a4e4f1aeeaed917"}, {"url": "http://172.16.0.1:8000/v1/6aeec69946724398b297c03373b82a20", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "c00c48b4c0a34e079d0c84381d4aac88"}], "type": "cloudformation", "id": "3f80b635fe18447384008b035b78c1de", "name": "heat-cfn"}, {"endpoints": [{"url": "http://172.16.0.1:5000", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "28de66398fd24e5485ef900196339b6f"}, {"url": "http://172.16.0.1:5000", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "84b57d30caed49e89c07dea4bd7ba91c"}, {"url": "http://172.16.0.1:35357", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "e971a3175dcd48358889c29cbfee53ec"}], "type": "identity", "id": "406ff4958f7a4ec48255ce7caa8eddf9", "name": "keystone"}, {"endpoints": [{"url": "http://172.16.0.1:8774/v2.1", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "3792eeedcf97418981c9352bcc6f2e1c"}, {"url": "http://172.16.0.1:8774/v2.1", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "63e40c28bac04df5b574dd10b0b919ed"}, {"url": "http://172.16.0.1:8774/v2.1", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "82636d1e23514e3f9545633ee3b22cee"}], "type": "compute", "id": "6161dca5ce47423fba47c3b37db0d714", "name": "nova"}, {"endpoints": [{"url": "http://172.16.0.1:8004/v1/6aeec69946724398b297c03373b82a20", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "413e05ec7dc346c7ae86713d81aa5f24"}, {"url": "http://172.16.0.1:8004/v1/6aeec69946724398b297c03373b82a20", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "b44fdd6e7ea34f6cb29155c10c07f515"}, {"url": "http://172.16.0.1:8004/v1/6aeec69946724398b297c03373b82a20", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "cf13a092b5134b1dbff68cccadca40cc"}], "type": "orchestration", "id": "ab2f972c1fd648b8a3119b9c811f24f0", "name": "heat"}, {"endpoints": [{"url": "ws://172.16.0.1:9000", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "280f9de09d9b48aca407a57896875b1e"}, {"url": "ws://172.16.0.1:9000", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "5bf9a8125d594e8582d40d9c9ff74c30"}, {"url": "ws://172.16.0.1:9000", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "a428df9c4bfd4e69935652fbc3426861"}], "type": "messaging-websocket", "id": "c7b9308512bb46be9f5278338680ca2b", "name": "zaqar-websocket"}, {"endpoints": [{"url": "http://172.16.0.1:9292", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "05d57b914b2642a2a3e748187c1fb554"}, {"url": "http://172.16.0.1:9292", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "1e886ecd03a844cd805f4885cf2690a5"}, {"url": "http://172.16.0.1:9292", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "75ac030c2d754ae49559bc0f8745ce3c"}], "type": "image", "id": "d0e94690e3364930960f8ad25f3a97d5", "name": "glance"}, {"endpoints": [{"url": "http://172.16.0.1:8888", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "0d72026835834028b72ef04a3efc41f2"}, {"url": "http://172.16.0.1:8888", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "21dbbfe453884ad09394c49a7b2916f7"}, {"url": "http://172.16.0.1:8888", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "6cbec0d1d1fb4b46a886c2b1a5e49731"}], "type": "messaging", "id": "d9c68cd8b61040158ecd6726a29a64a4", "name": "zaqar"}, {"endpoints": [{"url": "http://172.16.0.1:8778/placement", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "065d9d6375d04df7b2d5b9121ab6d168"}, {"url": "http://172.16.0.1:8778/placement", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "345a88ca6a494d4894a02c555d92db4a"}, {"url": "http://172.16.0.1:8778/placement", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "3665432921e14486935ccb54f4a74f78"}], "type": "placement", "id": "e4f7989d7f804b5aa634cb6da125039b", "name": "placement"}, {"endpoints": [{"url": "http://172.16.0.1:9696", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "6327c072fe44450a99ba768ec962afed"}, {"url": "http://172.16.0.1:9696", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "bf164f4e1e384250a7286a4e2e746b09"}, {"url": "http://172.16.0.1:9696", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "d11ce5393ce5426e8250d940c46d1a3e"}], "type": "network", "id": "e7b5c052cdd0453b9547ead0a1c45605", "name": "neutron"}, {"endpoints": [{"url": "http://172.16.0.1:8080/v1/AUTH_6aeec69946724398b297c03373b82a20", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "6adbb2dbee194d4a94aeef758e0bb925"}, {"url": "http://172.16.0.1:8080/v1/AUTH_6aeec69946724398b297c03373b82a20", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "df1a52bc7e8d4851b2d26018e137cedb"}, {"url": "http://172.16.0.1:8080", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "f64940e346ff46e68a020c3b449d92bb"}], "type": "object-store", "id": "f344feac93d94ee1933c021d545c3bf8", "name": "swift"}, {"endpoints": [{"url": "http://172.16.0.1:6385", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "1229230391464b75b76b507bbdc3056c"}, {"url": "http://172.16.0.1:6385", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "502d4f63d11c4a6ca230c6797283b02b"}, {"url": "http://172.16.0.1:6385", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "910b528d3a1d4318a384c55d4d39534a"}], "type": "baremetal", "id": "fbe5c11766e94e46b277ac0ba65f3a92", "name": "ironic"}], "user": {"domain": {"id": "default", "name": "Default"}, "password_expires_at": null, "name": "admin", "id": "946ce4bcc65c483b8995a2629d9b87ab"}, "audit_ids": ["w1QHF1v2RXiN3gysigxS2A"], "issued_at": "2018-05-01T18:44:57.000000Z"}}
Instantiating messaging websocket client: ws://172.16.0.1:9000
REQ: curl -g -i -X POST http://172.16.0.1:8989/v2/executions -H "User-Agent: osc-lib/1.9.0 keystoneauth1/3.4.0 python-requests/2.14.2 CPython/2.7.5" -H "content-type: application/json" -H "X-Auth-Token: {SHA1}982098c0c7678062eb57f7e5b961819cc9dc23c8" -d '{"input": "{\"instance_boot_option\": \"local\", \"nodes_json\": [{\"pm_password\": \"123456\", \"name\": \"osp13-controller-1\", \"capabilities\": \"profile:control,boot_option:local\", \"pm_type\": \"staging-ovirt\", \"pm_addr\": \"rhvm.field.lab.eng.rdu.redhat.com\", \"ovirt_vm_name\": \"osp13-controller-1\", \"mac\": [\"00:1a:4a:16:01:39\"], \"pm_vm_name\": \"osp13-controller-1\", \"memory\": \"4096\", \"disk\": \"40\", \"arch\": \"x86_64\", \"cpu\": \"2\", \"pm_user\": \"moran@internal\"}, {\"pm_password\": \"123456\", \"name\": \"osp13-controller-2\", \"capabilities\": \"profile:control,boot_option:local\", \"pm_type\": \"staging-ovirt\", \"pm_addr\": \"rhvm.field.lab.eng.rdu.redhat.com\", \"ovirt_vm_name\": \"osp13-controller-2\", \"mac\": [\"00:1a:4a:16:01:3a\"], \"pm_vm_name\": \"osp13-controller-2\", \"memory\": \"4096\", \"disk\": \"40\", \"arch\": \"x86_64\", \"cpu\": \"2\", \"pm_user\": \"moran@internal\"}, {\"pm_password\": \"123456\", \"name\": \"osp13-controller-3\", \"capabilities\": \"profile:control,boot_option:local\", \"pm_type\": \"staging-ovirt\", \"pm_addr\": \"rhvm.field.lab.eng.rdu.redhat.com\", \"ovirt_vm_name\": \"osp13-controller-3\", \"mac\": [\"00:1a:4a:16:01:3b\"], \"pm_vm_name\": \"osp13-controller-3\", \"memory\": \"4096\", \"disk\": \"40\", \"arch\": \"x86_64\", \"cpu\": \"2\", \"pm_user\": \"moran@internal\"}, {\"pm_password\": \"123456\", \"name\": \"osp13-compute-1\", \"capabilities\": \"profile:compute,boot_option:local\", \"pm_type\": \"staging-ovirt\", \"pm_addr\": \"rhvm.field.lab.eng.rdu.redhat.com\", \"ovirt_vm_name\": \"osp13-compute-1\", \"mac\": [\"00:1a:4a:16:01:3c\"], \"pm_vm_name\": \"osp13-compute-1\", \"memory\": \"4096\", \"disk\": \"40\", \"arch\": \"x86_64\", \"cpu\": \"2\", \"pm_user\": \"moran@internal\"}], \"ramdisk_name\": \"bm-deploy-ramdisk\", \"kernel_name\": \"bm-deploy-kernel\"}", "workflow_name": "tripleo.baremetal.v1.register_or_update", "description": ""}'
Starting new HTTP connection (1): 172.16.0.1
http://172.16.0.1:8989 "POST /v2/executions HTTP/1.1" 201 2433
RESP: [201] Content-Length: 2433 Content-Type: application/json Date: Tue, 01 May 2018 18:44:58 GMT Connection: keep-alive
RESP BODY: {"root_execution_id": null, "state_info": null, "description": "", "state": "RUNNING", "workflow_name": "tripleo.baremetal.v1.register_or_update", "task_execution_id": null, "updated_at": "2018-05-01 18:44:58", "workflow_id": "8324764f-ae0a-483e-a26a-25a348359583", "params": "{\"namespace\": \"\"}", "workflow_namespace": "", "output": "{}", "input": "{\"queue_name\": \"tripleo\", \"remove\": false, \"ramdisk_name\": \"bm-deploy-ramdisk\", \"kernel_name\": \"bm-deploy-kernel\", \"nodes_json\": [{\"pm_password\": \"123456\", \"name\": \"osp13-controller-1\", \"memory\": \"4096\", \"pm_addr\": \"rhvm.field.lab.eng.rdu.redhat.com\", \"cpu\": \"2\", \"mac\": [\"00:1a:4a:16:01:39\"], \"capabilities\": \"profile:control,boot_option:local\", \"pm_type\": \"staging-ovirt\", \"disk\": \"40\", \"arch\": \"x86_64\", \"ovirt_vm_name\": \"osp13-controller-1\", \"pm_user\": \"moran@internal\", \"pm_vm_name\": \"osp13-controller-1\"}, {\"pm_password\": \"123456\", \"name\": \"osp13-controller-2\", \"memory\": \"4096\", \"pm_addr\": \"rhvm.field.lab.eng.rdu.redhat.com\", \"cpu\": \"2\", \"mac\": [\"00:1a:4a:16:01:3a\"], \"capabilities\": \"profile:control,boot_option:local\", \"pm_type\": \"staging-ovirt\", \"disk\": \"40\", \"arch\": \"x86_64\", \"ovirt_vm_name\": \"osp13-controller-2\", \"pm_user\": \"moran@internal\", \"pm_vm_name\": \"osp13-controller-2\"}, {\"pm_password\": \"123456\", \"name\": \"osp13-controller-3\", \"memory\": \"4096\", \"pm_addr\": \"rhvm.field.lab.eng.rdu.redhat.com\", \"cpu\": \"2\", \"mac\": [\"00:1a:4a:16:01:3b\"], \"capabilities\": \"profile:control,boot_option:local\", \"pm_type\": \"staging-ovirt\", \"disk\": \"40\", \"arch\": \"x86_64\", \"ovirt_vm_name\": \"osp13-controller-3\", \"pm_user\": \"moran@internal\", \"pm_vm_name\": \"osp13-controller-3\"}, {\"pm_password\": \"123456\", \"name\": \"osp13-compute-1\", \"memory\": \"4096\", \"pm_addr\": \"rhvm.field.lab.eng.rdu.redhat.com\", \"cpu\": \"2\", \"mac\": [\"00:1a:4a:16:01:3c\"], \"capabilities\": \"profile:compute,boot_option:local\", \"pm_type\": \"staging-ovirt\", \"disk\": \"40\", \"arch\": \"x86_64\", \"ovirt_vm_name\": \"osp13-compute-1\", \"pm_user\": \"moran@internal\", \"pm_vm_name\": \"osp13-compute-1\"}], \"initial_state\": \"manageable\", \"instance_boot_option\": \"local\"}", "created_at": "2018-05-01 18:44:58", "project_id": "6aeec69946724398b297c03373b82a20", "id": "652b3ec5-0b1b-46a0-a388-dc1974a499d1"}

HTTP POST http://172.16.0.1:8989/v2/executions 201
Started Mistral Workflow tripleo.baremetal.v1.register_or_update. Execution ID: 652b3ec5-0b1b-46a0-a388-dc1974a499d1
Waiting for messages on queue 'tripleo' with no timeout.


4 node(s) successfully moved to the "manageable" state.
Successfully registered node UUID 1c2ed87a-5aeb-464a-8bef-3d9d9548b1b4
Successfully registered node UUID e602f940-33da-4939-8442-095dbe0435e1
Successfully registered node UUID dee9680c-2a95-4a49-871e-592d4c880b14
Successfully registered node UUID bacd39fc-f96c-459a-84e3-1f684fc2f4e8
Instantiating messaging websocket client: ws://172.16.0.1:9000
REQ: curl -g -i -X POST http://172.16.0.1:8989/v2/executions -H "User-Agent: osc-lib/1.9.0 keystoneauth1/3.4.0 python-requests/2.14.2 CPython/2.7.5" -H "content-type: application/json" -H "X-Auth-Token: {SHA1}982098c0c7678062eb57f7e5b961819cc9dc23c8" -d '{"input": "{\"node_uuids\": [\"1c2ed87a-5aeb-464a-8bef-3d9d9548b1b4\", \"e602f940-33da-4939-8442-095dbe0435e1\", \"dee9680c-2a95-4a49-871e-592d4c880b14\", \"bacd39fc-f96c-459a-84e3-1f684fc2f4e8\"]}", "workflow_name": "tripleo.baremetal.v1.provide", "description": ""}'
http://172.16.0.1:8989 "POST /v2/executions HTTP/1.1" 201 692
RESP: [201] Content-Length: 692 Content-Type: application/json Date: Tue, 01 May 2018 18:45:14 GMT Connection: keep-alive
RESP BODY: {"root_execution_id": null, "state_info": null, "description": "", "state": "RUNNING", "workflow_name": "tripleo.baremetal.v1.provide", "task_execution_id": null, "updated_at": "2018-05-01 18:45:14", "workflow_id": "294458bb-ebc4-4181-9378-ced70eed8f2b", "params": "{\"namespace\": \"\"}", "workflow_namespace": "", "output": "{}", "input": "{\"queue_name\": \"tripleo\", \"node_uuids\": [\"1c2ed87a-5aeb-464a-8bef-3d9d9548b1b4\", \"e602f940-33da-4939-8442-095dbe0435e1\", \"dee9680c-2a95-4a49-871e-592d4c880b14\", \"bacd39fc-f96c-459a-84e3-1f684fc2f4e8\"]}", "created_at": "2018-05-01 18:45:14", "project_id": "6aeec69946724398b297c03373b82a20", "id": "2f54e044-ba51-493d-a1e9-67ba24dbbcb2"}

HTTP POST http://172.16.0.1:8989/v2/executions 201
Started Mistral Workflow tripleo.baremetal.v1.provide. Execution ID: 2f54e044-ba51-493d-a1e9-67ba24dbbcb2
Waiting for messages on queue 'tripleo' with no timeout.
[u"Failed to run action [action_ex_id=153e5fb9-eb72-4687-9e54-d378e10a9a40, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'1c2ed87a-5aeb-464a-8bef-3d9d9548b1b4'}']\n NovaAction.hypervisors.find failed: No Hypervisor matching {u'hypervisor_hostname': u'1c2ed87a-5aeb-464a-8bef-3d9d9548b1b4'}. (HTTP 404)", u"Failed to run action [action_ex_id=26302869-1fcc-4148-8c53-03afaf5b4932, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'e602f940-33da-4939-8442-095dbe0435e1'}']\n NovaAction.hypervisors.find failed: No Hypervisor matching {u'hypervisor_hostname': u'e602f940-33da-4939-8442-095dbe0435e1'}. (HTTP 404)", u"Failed to run action [action_ex_id=ce2146d7-c985-4698-8a10-1fce0816a6ab, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'dee9680c-2a95-4a49-871e-592d4c880b14'}']\n NovaAction.hypervisors.find failed: No Hypervisor matching {u'hypervisor_hostname': u'dee9680c-2a95-4a49-871e-592d4c880b14'}. (HTTP 404)", u"Failed to run action [action_ex_id=8ef704cf-f9b1-49ce-93fa-65eab1b5bc06, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'bacd39fc-f96c-459a-84e3-1f684fc2f4e8'}']\n NovaAction.hypervisors.find failed: No Hypervisor matching {u'hypervisor_hostname': u'bacd39fc-f96c-459a-84e3-1f684fc2f4e8'}. (HTTP 404)"]

4 node(s) successfully moved to the "available" state.
clean_up ImportNode:
END return value: None

Comment 21 Toure Dunnon 2018-05-02 17:37:19 UTC
Mike could you give the output of: openstack hypervisor show bacd39fc-f96c-459a-84e3-1f684fc2f4e8

Comment 22 Toure Dunnon 2018-05-02 20:42:16 UTC
After working with Mike on this issue, the problem seems to stem from a false positive output from the message bus. We ran the node registration
and were able to complete the registration but received the described output but upon further investigation the process completed correctly.
Will check with Dmitry to confirm these findings from the ironic point of view.



(undercloud) [stack@undercloud ~]$ openstack overcloud node import /home/stack/instackenv.json --provide
Started Mistral Workflow tripleo.baremetal.v1.register_or_update. Execution ID: ad3f2576-3dd5-423e-bb7c-d3990be5d742
Waiting for messages on queue 'tripleo' with no timeout.


4 node(s) successfully moved to the "manageable" state.
Successfully registered node UUID ad7983b0-e229-4243-9c4f-10ba13f7f906
Successfully registered node UUID 618deb78-ccfd-4ae6-8c80-82d436a1b418
Successfully registered node UUID ab6ed0d6-4cb7-4df4-a675-cd8a906bd7d8
Successfully registered node UUID 7ced5cc5-f070-4b4e-997f-cf4a510bd641
Started Mistral Workflow tripleo.baremetal.v1.provide. Execution ID: 291ced5b-971c-4da5-9baf-7d83ee168a7a
Waiting for messages on queue 'tripleo' with no timeout.
[u"Failed to run action [action_ex_id=e431ec27-c1bb-48e9-9934-429ec6a1a99d, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervis
'}', params='{u'hypervisor_hostname': u'ad7983b0-e229-4243-9c4f-10ba13f7f906'}']\n NovaAction.hypervisors.find failed: No Hypervisor matching {u'hypervisor_hostname': u'ad7983b0-e229-4243-
a13f7f906'}. (HTTP 404)", u"Failed to run action [action_ex_id=2b485fcc-c2de-492a-b93f-7038ff7872c0, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client
name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'618deb78-ccfd-4ae6-8c80-82d436a1b418'}']\n NovaAction.hypervisors.find failed: No Hypervisor matching {u'hypervisor_hostnam
8deb78-ccfd-4ae6-8c80-82d436a1b418'}. (HTTP 404)", u"Failed to run action [action_ex_id=9d385d08-128a-4aff-b6ea-2c6ae7c6f264, action_cls='<class 'mistral.actions.action_factory.NovaAction'
ibutes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'ab6ed0d6-4cb7-4df4-a675-cd8a906bd7d8'}']\n NovaAction.hypervisors.find failed: No Hypervisor match
ypervisor_hostname': u'ab6ed0d6-4cb7-4df4-a675-cd8a906bd7d8'}. (HTTP 404)", u"Failed to run action [action_ex_id=6e2191e8-9375-425b-8c21-6fc2c7162763, action_cls='<class 'mistral.actions.a
ctory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'7ced5cc5-f070-4b4e-997f-cf4a510bd641'}']\n NovaAction.hypervisors.find fa
 Hypervisor matching {u'hypervisor_hostname': u'7ced5cc5-f070-4b4e-997f-cf4a510bd641'}. (HTTP 404)"]

4 node(s) successfully moved to the "available" state.
(undercloud) [stack@undercloud ~]$ mistral execution-get 291ced5b-971c-4da5-9baf-7d83ee168a7a
+--------------------+--------------------------------------+
| Field              | Value                                |
+--------------------+--------------------------------------+
| ID                 | 291ced5b-971c-4da5-9baf-7d83ee168a7a |
| Workflow ID        | 294458bb-ebc4-4181-9378-ced70eed8f2b |
| Workflow name      | tripleo.baremetal.v1.provide         |
| Workflow namespace |                                      |
| Description        |                                      |
| Task Execution ID  | <none>                               |
| State              | SUCCESS                              |
| State info         | None                                 |
| Created at         | 2018-05-02 20:28:59                  |
| Updated at         | 2018-05-02 20:29:46                  |
+--------------------+--------------------------------------+
(undercloud) [stack@undercloud ~]$ mistral execution-get ad3f2576-3dd5-423e-bb7c-d3990be5d742
+--------------------+-----------------------------------------+
| Field              | Value                                   |
+--------------------+-----------------------------------------+
| ID                 | ad3f2576-3dd5-423e-bb7c-d3990be5d742    |
| Workflow ID        | 8324764f-ae0a-483e-a26a-25a348359583    |
| Workflow name      | tripleo.baremetal.v1.register_or_update |
| Workflow namespace |                                         |
| Description        |                                         |
| Task Execution ID  | <none>                                  |
| State              | SUCCESS                                 |
| State info         | None                                    |
| Created at         | 2018-05-02 20:28:44                     |
| Updated at         | 2018-05-02 20:28:59                     |
+--------------------+-----------------------------------------+
(undercloud) [stack@undercloud ~]$

Comment 23 Dmitry Tantsur 2018-05-03 14:04:15 UTC
I cannot really comment from Ironic perspective since Ironic is not involved here, strictly speaking. But I suspect we should just remove logging inside of some retry loop.

Comment 28 Yolanda Robla 2018-05-28 12:35:17 UTC
It winally worked for me. Error was caused by another bug, that was causing a duplicate hypervisor problem, and then mistral was parsing that and showing that error. Now the bug was fixed so i don't have this problem anymore

Comment 29 Toure Dunnon 2018-05-29 13:28:42 UTC
The newly attached proposed patch cleans up the mistral defined workflow to prevent erroneous messages from cluttering the command line.

Comment 32 Brian Fife 2018-09-14 19:18:14 UTC
(In reply to Yolanda Robla from comment #28)
> It winally worked for me. Error was caused by another bug, that was causing
> a duplicate hypervisor problem, and then mistral was parsing that and
> showing that error. Now the bug was fixed so i don't have this problem
> anymore

I noticed this too, I am struggling with it right now:

>>> nova.hypervisors.findall(hypervisor_hostname='3d4a448b-4410-421a-af5c-4fc220a8abd2')
[<Hypervisor: 104>, <Hypervisor: 104>]

Do you have any details on which particular bug was fixed that solved this?

Comment 33 Lon Hohberger 2018-10-19 10:42:01 UTC
According to our records, this should be resolved by openstack-tripleo-common-8.6.3-13.el7ost.  This build is available now.

Comment 35 Dariusz Wojewódzki 2018-11-08 09:28:30 UTC
Hi,

I have Cust who hit a similar issue with openstack-tripleo-common - 8.6.3-13.
They managed to introspect the node but are stuck with making it available for deployment:

The node provide operation results with: NovaAction.hypervisors.find failed: 

$ openstack overcloud node provide 16ac74c7-803b-4958-a1b1-ebed29e82e7e
Started Mistral Workflow tripleo.baremetal.v1.provide. Execution ID: bc681531-352f-43f5-913d-228a291879af
Waiting for messages on queue 'tripleo' with no timeout.
[u"Failed to run action [action_ex_id=1259ef07-24e1-4cc7-a20f-9f007e73c7f9, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'16ac74c7-803b-4958-a1b1-ebed29e82e7e'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=4f740fcb-98d7-4305-ae8f-c691e844e76c, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'16ac74c7-803b-4958-a1b1-ebed29e82e7e'}']\n NovaAction.hypervisors.find failed: "]

Comment 36 Dariusz Wojewódzki 2018-11-08 11:48:42 UTC
(In reply to Yolanda Robla from comment #28)
> It winally worked for me. Error was caused by another bug, that was causing
> a duplicate hypervisor problem, and then mistral was parsing that and
> showing that error. Now the bug was fixed so i don't have this problem
> anymore

Hey Yolanda - Could You point me to that another bug that was causing
a duplicate hypervisor problem?

It seems my Customer hit the same issue. Every Hypervisor is doubled in the output of:
(undercloud) [stack@pndh-ostkdir02 environments]$ openstack hypervisor list --long

Comment 37 Yolanda Robla 2018-11-08 12:14:50 UTC
That is the problem i was hitting, with duplicated entries:

https://bugzilla.redhat.com/show_bug.cgi?id=1576953

But this was resolved for me long time ago, you should need to check versions of the packages.

Comment 38 Bob Fournier 2018-11-08 12:19:30 UTC
Dariusz - note that the fix for this bug is available in openstack-tripleo-common-8.6.1-4.el7ost which will be in the upcoming release of OSP-13z4.

You might want to do the hot fix route to get openstack-tripleo-common-8.6.1-4.el7ost if needed for your customer before the z4 release.

Comment 39 Jon Schlueter 2018-11-09 14:35:22 UTC
(In reply to Bob Fournier from comment #38)
> Dariusz - note that the fix for this bug is available in
> openstack-tripleo-common-8.6.1-4.el7ost which will be in the upcoming
> release of OSP-13z4.
> 
> You might want to do the hot fix route to get
> openstack-tripleo-common-8.6.1-4.el7ost if needed for your customer before
> the z4 release.

Actually the above mentioned version of tripleo-common is already released.

Latest released build is openstack-tripleo-common-8.6.3-13.el7ost and it should have the relevent fixes in it.

Comment 41 errata-xmlrpc 2018-11-13 22:26:39 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3587


Note You need to log in before you can comment on or make changes to this bug.