Bug 1536521
| Summary: | Introspection of baremetal nodes fail with 'No hypervisor matching' problem | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Yolanda Robla <yroblamo> | ||||
| Component: | openstack-tripleo-common | Assignee: | Toure Dunnon <tdunnon> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Alexander Chuzhoy <sasha> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | medium | ||||||
| Version: | 13.0 (Queens) | CC: | atelang, bfournie, brad, brian.fife, dmatthew, dtantsur, dwojewod, emacchi, jschluet, lbezdick, lmarsh, mbracho, mburns, mcornea, mlammon, pkopec.bugzilla, rbrady, rhel-osp-director-maint, slinaber, tdunnon, vcojot, yroblamo | ||||
| Target Milestone: | --- | Keywords: | TestOnly, Triaged, ZStream | ||||
| Target Release: | 13.0 (Queens) | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | openstack-tripleo-common-8.6.1-4.el7ost | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2018-11-13 22:26:39 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
In my experience, I've been seeing this error message also, but it seems harmless and doesn't cause any problems with further baremetal commands.
(undercloud) [stack@host01 ~]$ openstack overcloud node import --provide instackenv.json
Started Mistral Workflow tripleo.baremetal.v1.register_or_update. Execution ID: 3a524abb-4ba7-4029-aab0-50a46bb1f5a4
Waiting for messages on queue 'ba202ca1-57ea-4403-b157-b97e96040fbc' with no timeout.
Nodes set to managed.
Successfully registered node UUID 618a604d-cc41-4c10-945b-c5e6b5f01020
Successfully registered node UUID ccf4fa3c-9c57-4727-9b6a-396f54b63c1d
Started Mistral Workflow tripleo.baremetal.v1.provide. Execution ID: 954f600c-1032-4dc4-859b-dc63c8f0a1b8
Waiting for messages on queue 'ba202ca1-57ea-4403-b157-b97e96040fbc' with no timeout.
[u"Failed to run action [action_ex_id=55fe1277-dad9-44a0-bd9d-061625e4c1a4, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'618a604d-cc41-4c10-945b-c5e6b5f01020'}']\n NovaAction.hypervisors.find failed: No Hypervisor matching {u'hypervisor_hostname': u'618a604d-cc41-4c10-945b-c5e6b5f01020'}. (HTTP 404)", u"Failed to run action [action_ex_id=b71eb254-c675-490a-bcd6-fa3da612b462, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'ccf4fa3c-9c57-4727-9b6a-396f54b63c1d'}']\n NovaAction.hypervisors.find failed: No Hypervisor matching {u'hypervisor_hostname': u'ccf4fa3c-9c57-4727-9b6a-396f54b63c1d'}. (HTTP 404)"]
Successfully set nodes state to available.
(undercloud) [stack@host01 ~]$ openstack overcloud node provide --all-manageable
Started Mistral Workflow tripleo.baremetal.v1.provide_manageable_nodes. Execution ID: 62c1bb17-907d-487e-a9d1-d5cb5d7d921b
Waiting for messages on queue '76f08097-e567-40fa-9cc3-d2ed3f3e70df' with no timeout.
Successfully set nodes state to available.
========
Can we get more info on your setup and ironic logs?
That being said, it is a bug as we shouldn't be getting this failure on import, but I'm hesitant to flag it as a blocker unless we have evidence that its causing the "provide" command to fail.
Ironic logs are useless here, this failure has barely anything to do with ironic itself. I suspect it's either an issue in tripleo-common workflows or even overly extensive logging there. No time to look in depth right now, unfortunately. But is much be related to some post-providining validations of nova state. Changing the component according to my quick guess. Workflows DFG - sending this bug your way to see if you can have a look. The problem is easy to reproduce, it happens every time that I run the import command and I can provide a system for debugging. Also, to me it appears to be an innocuous - I haven't seen it as a symptom or other problems. I know Yolanda has seen that the provides also fails but that can be due to other issues with node cleaning unrelated to this message. The problem has also been occurring for me for a couple months but since it wasn't causing a problem I ignored it. A couple things to note - the hypervisor that is being flagged as "No Hypervisor matching" actually corresponds to the node uuid of a baremetal node. So its possible that the check is being done too early. I've attached part of the mistral executor log that shows the stack trace. It looks like this action is causing the problem. https://github.com/openstack/tripleo-common/blob/master/workbooks/baremetal.yaml#L1062 Anjali - I've removed the blocker flag '?' as this does not seem to be indicative of a problem besides the message. Created attachment 1386403 [details]
mistral executor.log
Also, here are the baremetal node uuids to correlate with the mistral log. This problem seems to only be occurring when using bsremetal nodes. (undercloud) [stack@host01 ~]$ openstack baremetal node list +--------------------------------------+-------+--------------------------------------+-------------+--------------------+-------------+ | UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance | +--------------------------------------+-------+--------------------------------------+-------------+--------------------+-------------+ | 618a604d-cc41-4c10-945b-c5e6b5f01020 | host2 | 5cbae7f7-7993-4032-bdde-a787bc6f84e9 | power on | active | False | | ccf4fa3c-9c57-4727-9b6a-396f54b63c1d | host3 | 21e69dbd-4f36-4114-b107-56d994918cd0 | power on | active | False | +--------------------------------------+-------+--------------------------------------+-------------+--------------------+-------------+ I am getting this on Pike as well when I use name and not baremetal's UUID in command *openstack overcloud node introspect --provide*. When I use baremetal's UUID it does not occur. Hope that helps. (In reply to Piotr Kopec from comment #6) > I am getting this on Pike as well when I use name and not baremetal's UUID > in command *openstack overcloud node introspect --provide*. When I use > baremetal's UUID it does not occur. Hope that helps. undercloud hypervisros use UUID as Hypervisor Hostname (undercloud) [stack@undercloud ~]$ openstack hypervisor list -c ID -c Hypervisor\ Hostname +-----+--------------------------------------+ | ID | Hypervisor Hostname | +-----+--------------------------------------+ | 144 | 8316ad5f-2f1c-44aa-b4d8-826683663a54 | | 145 | ca99d7af-0699-4401-8810-5c5392c9b22e | | 146 | 9985176c-5ee0-47d6-b293-0bf5ae5d4243 | | 147 | 66bfcbfd-9154-4378-98df-e161449cbd23 | | 148 | 25db0ff9-8cc9-43d5-bcea-6f77edbfc3d9 | | 149 | 4ab1aada-1709-4629-85ad-4597e4fb371a | | 150 | 0a999565-2c5c-4642-a132-a47a44ff5521 | | 151 | 89556cbc-4862-4226-8be4-db5a8ec85892 | | 153 | ff08b5cf-31d9-4125-8f1d-b004981879fd | | 156 | ddd66092-339e-4408-ba2a-ffd8d1500a38 | | 158 | abf65ffc-65c9-4c4f-90b9-de79abead7d7 | | 159 | f3e9359c-907b-4fec-bd49-89bbbec89b77 | | 161 | 6eee60fb-7ca8-4d4b-aa53-e7da033c8f65 | | 162 | d67c3bcb-7fbc-44a5-84d1-4aee2ff06521 | +-----+--------------------------------------+ How is status of that bug? Will that be fixed for queens? I continue hitting that on queens. That's not just a warning, is causing the openstack overcloud node introspect [NODE UUID] --provide to hang for extra time, having to kill the command manually. When i abort the command, to stop cycling, and launch the overcloud deploy command to actually add the new node, it fails complaining about the same error:
(undercloud) [stack@undercloud-0 ~]$ bash overcloud_deploy.sh
Started Mistral Workflow tripleo.validations.v1.check_pre_deployment_validations. Execution ID: e8e2e03d-b380-444b-99eb-b94f56d42e6a
Waiting for messages on queue 'tripleo' with no timeout.
Removing the current plan files
Uploading new plan files
Started Mistral Workflow tripleo.plan_management.v1.update_deployment_plan. Execution ID: 28a79518-8d3b-4ca3-a442-e5be85001c5f
[u"Failed to run action [action_ex_id=eabc0848-9cce-4dde-aa7b-d1f841e0ffc5, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'5baab564-5f14-406e-bf2c-ad171333eb3d'}']\n NovaAction.hypervisors.find failed: "]
Plan updated.
Processing templates in the directory /tmp/tripleoclient-MlaIp0/tripleo-heat-templates
Started Mistral Workflow tripleo.plan_management.v1.get_deprecated_parameters. Execution ID: afc7e094-470f-4bcb-acba-e46aa74dcc7a
WARNING: Following parameters are deprecated and still defined. Deprecated parameters will be removed soon!
HostCpusList
OvercloudControlFlavor
WARNING: Following parameters are defined but not used in plan. Could be possible that parameter is valid but currently not used.
ComputeKernelArgs
TenantNetworkVlanID
ControlPlaneDefaultRoute
StorageMgmtNetworkVlanID
ExternalAllocationPools
TenantNetCidr
NeutronPhysicalDevMappings
EC2MetadataIp
InternalApiNetCidr
NeutronSupportedPCIVendorDevs
InternalApiAllocationPools
ExternalInterfaceDefaultRoute
DnsServers
StorageNetworkVlanID
StorageAllocationPools
StorageMgmtAllocationPools
StorageNetCidr
StorageMgmtNetCidr
InternalApiNetworkVlanID
NeutronSriovNumVFs
ExternalNetCidr
TenantAllocationPools
Deploying templates in the directory /tmp/tripleoclient-MlaIp0/tripleo-heat-templates
Started Mistral Workflow tripleo.deployment.v1.deploy_plan. Execution ID: eacbfef7-1abf-44c3-b224-738fff083a75
{u'execution': {u'created_at': u'2018-03-27 15:43:30',
u'id': u'626d4508-b597-4e55-851b-0cfa7b85c018',
u'input': {u'node_uuids': [u'5baab564-5f14-406e-bf2c-ad171333eb3d'],
u'queue_name': u'tripleo'},
u'name': u'tripleo.baremetal.v1.cellv2_discovery',
u'params': {u'index': 0,
u'namespace': u'',
u'root_execution_id': u'11863553-83a8-476d-9efc-e8fa3d7f799a',
u'task_execution_id': u'632b098f-a78b-4169-8b11-d13a07175a4c'},
u'spec': {u'description': u'Run cell_v2 host discovery',
u'input': [u'node_uuids',
{u'queue_name': u'tripleo'}],
u'name': u'cellv2_discovery',
u'tags': [u'tripleo-common-managed'],
u'tasks': {u'cell_v2_discover_hosts': {u'action': u'tripleo.baremetal.cell_v2_discover_hosts',
u'name': u'cell_v2_discover_hosts',
u'on-error': u'cell_v2_discover_hosts_failed',
u'on-success': u'wait_for_nova_resources',
u'type': u'direct',
u'version': u'2.0'},
u'cell_v2_discover_hosts_failed': {u'name': u'cell_v2_discover_hosts_failed',
u'on-success': u'send_message',
u'publish': {u'message': u'<% task(cell_v2_discover_hosts).result %>',
u'status': u'FAILED'},
u'type': u'direct',
u'version': u'2.0'},
u'send_message': {u'action': u'zaqar.queue_post',
u'input': {u'messages': {u'body': {u'payload': {u'execution': u'<% execution() %>',
u'message': u"<% $.get('message', '') %>",
u'status': u"<% $.get('status', 'SUCCESS') %>"},
u'type': u'tripleo.baremetal.v1.cellv2_discovery'}},
u'queue_name': u'<% $.queue_name %>'},
u'name': u'send_message',
u'on-success': [{u'fail': u'<% $.get(\'status\') = "FAILED" %>'}],
u'retry': u'count=5 delay=1',
u'type': u'direct',
u'version': u'2.0'},
u'wait_for_nova_resources': {u'action': u'nova.hypervisors_find hypervisor_hostname=<% $.node_uuid %>',
u'name': u'wait_for_nova_resources',
u'on-error': u'wait_for_nova_resources_failed',
u'on-success': u'send_message',
u'type': u'direct',
u'version': u'2.0',
u'with-items': u'node_uuid in <% $.node_uuids %>'},
u'wait_for_nova_resources_failed': {u'name': u'wait_for_nova_resources_failed',
u'on-success': u'send_message',
u'publish': {u'message': u'<% task(wait_for_nova_resources).result %>',
u'status': u'FAILED'},
u'type': u'direct',
u'version': u'2.0'}},
u'version': u'2.0'},
u'updated_at': u'2018-03-27 15:43:30'},
u'message': [u"Failed to run action [action_ex_id=cf2b7083-f5b5-4198-849e-90518eeb8275, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'5baab564-5f14-406e-bf2c-ad171333eb3d'}']\n NovaAction.hypervisors.find failed: "],
u'status': u'FAILED'}
This is urgent as it's stopping adding new nodes after a 13 deploy.
A fix has been submitted upstream to address the workflow issue for ironic where the state of node is not tracked which can lead to partial registration. I'm still seeing same issue, even with the patch applied:
(undercloud) [stack@undercloud-0 ~]$ openstack overcloud node introspect 090e64eb-6d9c-4e76-8ea3-f9dee663fc02 --provide
Waiting for introspection to finish...
Started Mistral Workflow tripleo.baremetal.v1.introspect. Execution ID: 56deb951-9d78-4232-ba85-6940510a7783
Waiting for messages on queue 'tripleo' with no timeout.
Introspection of node 090e64eb-6d9c-4e76-8ea3-f9dee663fc02 completed. Status:SUCCESS. Errors:None
Successfully introspected 1 node(s).
Started Mistral Workflow tripleo.baremetal.v1.provide. Execution ID: eaae1af9-6d92-4b3f-be3d-d4b5675fc8ee
Waiting for messages on queue 'tripleo' with no timeout.
[u"Failed to run action [action_ex_id=6d3e3407-613d-4cad-ad5e-6c29649b49ef, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=e64d9241-5e23-4140-b55e-ac708a244d58, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=31077220-4d66-418e-9bad-dc0aec091a77, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=6cc857bb-c264-4f0f-8c1a-0a4d6615fc05, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=3ca4759e-9ff7-4774-bc08-e3cd6689056f, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=d71d3a7e-5ed0-4478-ba92-e2b75ab87c83, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=2c267dbe-f5a3-438e-8dfe-343fd7bc4852, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=f2596cc0-6c63-441a-9d1d-27b2e8a7f15c, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=bb1a7721-5cfb-421c-8fea-f604da2ee7b3, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=38eaefe3-f806-46c6-a65b-fab892094231, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=a0e89834-5b41-4bdb-96c6-55b0afb07895, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=82dc5d83-a6a4-4d3f-bd38-c7c73f77d101, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=cc6e19eb-15ce-4f19-9024-6779a52338bf, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=a8a9c984-4501-4343-82e7-54927712d25d, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=741a0383-5ab7-4ced-991c-1b6ca40e1147, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=ffce6e50-1586-49c4-b478-d90eee51f16c, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=ad246642-3b5f-4fd9-8e78-f64247975e32, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=f4b49dd2-f5ff-41a5-85a5-96204b7a44ab, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=a4f97891-7c55-48f1-91df-e1093ab705c2, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=53256dbd-b363-4f05-b477-2b029659da16, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=08f14ec3-a9cd-4e1a-9e91-26caf61b38b6, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=0ff64675-4255-4e84-bccc-87320a3097b0, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=7bfd1fa3-3a7d-49a4-8f3e-561604377d28, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=78bb8a6c-6411-4444-ad8f-320999efa407, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=a5626184-99ac-44fd-9354-8dbb7af31630, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=4753d722-c46d-44d2-934a-248cfaa8060a, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=edc0a077-ad16-4274-bf7e-8df7f9670349, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=9f5464a9-81be-4009-a8d8-515184b6a642, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=4e986281-3c6f-4089-a93f-eb3fdddc317f, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=82436b8f-b415-45d3-8a8b-46a96d5d95eb, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "]
{u'status': u'FAILED', u'message': [u"Failed to run action [action_ex_id=82436b8f-b415-45d3-8a8b-46a96d5d95eb, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'090e64eb-6d9c-4e76-8ea3-f9dee663fc02'}']\n NovaAction.hypervisors.find failed: "], u'result': None}
Failed to set nodes to available state: Failed.
(undercloud) [stack@undercloud-0 ~]$
I am seeing this issue in latest osp puddle : 2018-04-26.3
Started Mistral Workflow tripleo.baremetal.v1.register_or_update. Execution ID: 079c99d3-8687-4093-93d0-75abe8635a59
Waiting for messages on queue 'tripleo' with no timeout.
4 node(s) successfully moved to the "manageable" state.
Successfully registered node UUID 2b30bb39-577e-4ab8-bc6f-871412ca7089
Successfully registered node UUID 3e18593d-944c-4ac9-89e8-3e06c849ef5c
Successfully registered node UUID 6c9c45f7-2295-4185-adf4-c513426974b7
Successfully registered node UUID 901ee5c7-f05e-4e59-9133-0a927a4d3316
Started Mistral Workflow tripleo.baremetal.v1.provide. Execution ID: cab4886e-b339-44c3-ace5-2025f7990f76
Waiting for messages on queue 'tripleo' with no timeout.
[u"Failed to run action [action_ex_id=5fdd4870-41d3-4f5c-a4b7-19972badbf45, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'2b30bb39-577e-4ab8-bc6f-871412ca7089'}']\n NovaAction.hypervisors.find failed: No Hypervisor matching {u'hypervisor_hostname': u'2b30bb39-577e-4ab8-bc6f-871412ca7089'}. (HTTP 404)", {u'cpu_info': u'', u'manager': {u'api': {u'server_groups': None, u'keypairs': None, u'servers': None, u'server_external_events': None, u'server_migrations': None, u'agents': None, u'instance_action': None, u'glance': None, u'hypervisor_stats': None, u'virtual_interfaces': None, u'flavors': None, u'availability_zones': None, u'user_id': None, u'cloudpipe': None, u'os_cache': False, u'quotas': None, u'migrations': None, u'usage': None, u'logger': None, u'project_id': None, u'neutron': None, u'quota_classes': None, u'project_name': None, u'aggregates': None, u'flavor_access': None, u'services': None, u'list_extensions': None, u'limits': None, u'hypervisors': None, u'cells': None, u'versions': None, u'client': None, u'hosts': None, u'volumes': None, u'assisted_volume_snapshots': None, u'certs': None}}, u'free_disk_gb': 0, u'id': 1, u'service': {u'host': u'undercloud.field.lab.eng.rdu.redhat.com', u'disabled_reason': None, u'id': 6}, u'local_gb_used': 0, u'memory_mb_used': 0, u'current_workload': 0, u'state': u'up', u'status': u'enabled', u'host_ip': u'10.12.50.145', u'hypervisor_hostname': u'3e18593d-944c-4ac9-89e8-3e06c849ef5c', u'hypervisor_version': 1, u'disk_available_least': 0, u'local_gb': 0, u'free_ram_mb': 0, u'vcpus_used': 0, u'hypervisor_type': u'ironic', u'x_openstack_request_ids': [u'req-5249c558-278b-4894-b836-59c667dd5034'], u'memory_mb': 0, u'vcpus': 0, u'running_vms': 0, u'_info': {u'status': u'enabled', u'service': {u'host': u'undercloud.field.lab.eng.rdu.redhat.com', u'disabled_reason': None, u'id': 6}, u'vcpus_used': 0, u'hypervisor_type': u'ironic', u'local_gb_used': 0, u'host_ip': u'10.12.50.145', u'hypervisor_hostname': u'3e18593d-944c-4ac9-89e8-3e06c849ef5c', u'memory_mb_used': 0, u'memory_mb': 0, u'current_workload': 0, u'vcpus': 0, u'state': u'up', u'cpu_info': u'', u'running_vms': 0, u'free_disk_gb': 0, u'hypervisor_version': 1, u'disk_available_least': 0, u'local_gb': 0, u'free_ram_mb': 0, u'id': 1}, u'_loaded': True}, {u'cpu_info': u'', u'manager': {u'api': {u'server_groups': None, u'keypairs': None, u'servers': None, u'server_external_events': None, u'server_migrations': None, u'agents': None, u'instance_action': None, u'glance': None, u'hypervisor_stats': None, u'virtual_interfaces': None, u'flavors': None, u'availability_zones': None, u'user_id': None, u'cloudpipe': None, u'os_cache': False, u'quotas': None, u'migrations': None, u'usage': None, u'logger': None, u'project_id': None, u'neutron': None, u'quota_classes': None, u'project_name': None, u'aggregates': None, u'flavor_access': None, u'services': None, u'list_extensions': None, u'limits': None, u'hypervisors': None, u'cells': None, u'versions': None, u'client': None, u'hosts': None, u'volumes': None, u'assisted_volume_snapshots': None, u'certs': None}}, u'free_disk_gb': None, u'id': 2, u'service': {u'host': u'undercloud.field.lab.eng.rdu.redhat.com', u'disabled_reason': None, u'id': 6}, u'local_gb_used': 0, u'memory_mb_used': 0, u'current_workload': None, u'state': u'up', u'status': u'enabled', u'host_ip': u'10.12.50.145', u'hypervisor_hostname': u'6c9c45f7-2295-4185-adf4-c513426974b7', u'hypervisor_version': 1, u'disk_available_least': 0, u'local_gb': 0, u'free_ram_mb': None, u'vcpus_used': 0, u'hypervisor_type': u'ironic', u'x_openstack_request_ids': [u'req-c09cc2f3-27d0-45bd-a360-1af43a0b7190'], u'memory_mb': 0, u'vcpus': 0, u'running_vms': None, u'_info': {u'status': u'enabled', u'service': {u'host': u'undercloud.field.lab.eng.rdu.redhat.com', u'disabled_reason': None, u'id': 6}, u'vcpus_used': 0, u'hypervisor_type': u'ironic', u'local_gb_used': 0, u'host_ip': u'10.12.50.145', u'hypervisor_hostname': u'6c9c45f7-2295-4185-adf4-c513426974b7', u'memory_mb_used': 0, u'memory_mb': 0, u'current_workload': None, u'vcpus': 0, u'state': u'up', u'cpu_info': u'', u'running_vms': None, u'free_disk_gb': None, u'hypervisor_version': 1, u'disk_available_least': 0, u'local_gb': 0, u'free_ram_mb': None, u'id': 2}, u'_loaded': True}, u"Failed to run action [action_ex_id=5a4d78b7-e88f-454d-aacb-f6e790b47988, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'901ee5c7-f05e-4e59-9133-0a927a4d3316'}']\n NovaAction.hypervisors.find failed: No Hypervisor matching {u'hypervisor_hostname': u'901ee5c7-f05e-4e59-9133-0a927a4d3316'}. (HTTP 404)"]
4 node(s) successfully moved to the "available" state.
(undercloud) [stack@undercloud ~]$ openstack workflow execution list
+--------------------------------------+--------------------------------------+-----------------------------------------------------+--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+---------+------------+---------------------+---------------------+
| ID | Workflow ID | Workflow name | Workflow namespace | Description | Task Execution ID | State | State info | Created at | Updated at |
+--------------------------------------+--------------------------------------+-----------------------------------------------------+--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+---------+------------+---------------------+---------------------+
| bf75e696-8a78-4fd3-846e-47d34b68e898 | 86118777-658e-48a2-85b3-9f81d1596f0a | tripleo.plan_management.v1.create_deployment_plan | | | <none> | SUCCESS | None | 2018-04-30 23:47:40 | 2018-04-30 23:48:40 |
| 7f781cd6-9afa-4748-b795-ad681c5957bb | 374dcfae-021f-4083-8d2b-cd88dde0cee5 | tripleo.validations.v1.copy_ssh_key | | | <none> | SUCCESS | None | 2018-04-30 23:48:42 | 2018-04-30 23:48:43 |
| 079c99d3-8687-4093-93d0-75abe8635a59 | 8324764f-ae0a-483e-a26a-25a348359583 | tripleo.baremetal.v1.register_or_update | | | <none> | SUCCESS | None | 2018-04-30 23:52:14 | 2018-04-30 23:52:40 |
| cdd65043-4d8f-4f2d-8cde-12888ef8e7c4 | 59318d46-2e66-4152-8a26-8e9e4a76431b | tripleo.baremetal.v1.validate_nodes | | sub-workflow execution | 5dc5c6ed-594b-4f44-8dc1-2b2cfaf5298e | SUCCESS | None | 2018-04-30 23:52:14 | 2018-04-30 23:52:18 |
| 6fbfb372-3001-4cc3-94a3-f4d4c8a4b2d2 | 9a9fab1e-00d3-4e81-b3c7-09508448c4e8 | tripleo.baremetal.v1.set_node_state | | sub-workflow execution | f80fe1bf-3c4d-49e0-9f02-9615f3076eca | SUCCESS | None | 2018-04-30 23:52:25 | 2018-04-30 23:52:27 |
| ada78894-dd2d-491c-8efd-537a0e0309cc | 20d0d265-2c8f-4ac4-95bc-5e1ec95f83b9 | tripleo.baremetal.v1.manage | | sub-workflow execution | 73f3e487-677c-4d16-bb13-ab4cf9cbaa5f | SUCCESS | None | 2018-04-30 23:52:25 | 2018-04-30 23:52:34 |
| b062da09-54ed-45f5-8aef-0abe2fb5402f | 9a9fab1e-00d3-4e81-b3c7-09508448c4e8 | tripleo.baremetal.v1.set_node_state | | sub-workflow execution | f80fe1bf-3c4d-49e0-9f02-9615f3076eca | SUCCESS | None | 2018-04-30 23:52:25 | 2018-04-30 23:52:29 |
| ceba4fa3-b4fa-49f6-83a6-b103b9e23140 | 9a9fab1e-00d3-4e81-b3c7-09508448c4e8 | tripleo.baremetal.v1.set_node_state | | sub-workflow execution | f80fe1bf-3c4d-49e0-9f02-9615f3076eca | SUCCESS | None | 2018-04-30 23:52:25 | 2018-04-30 23:52:30 |
| d44b9916-0e81-48ec-8c05-18b388430548 | 9a9fab1e-00d3-4e81-b3c7-09508448c4e8 | tripleo.baremetal.v1.set_node_state | | sub-workflow execution | f80fe1bf-3c4d-49e0-9f02-9615f3076eca | SUCCESS | None | 2018-04-30 23:52:25 | 2018-04-30 23:52:30 |
| 397cc087-80b6-449b-8e66-5100bb861fb1 | 9a9fab1e-00d3-4e81-b3c7-09508448c4e8 | tripleo.baremetal.v1.set_node_state | | sub-workflow execution | 5888c3d8-ee99-4036-b16b-d301fd161041 | SUCCESS | None | 2018-04-30 23:52:40 | 2018-04-30 23:52:42 |
| 51366480-4194-49e6-bc0e-ecdd06159656 | 9a9fab1e-00d3-4e81-b3c7-09508448c4e8 | tripleo.baremetal.v1.set_node_state | | sub-workflow execution | 5888c3d8-ee99-4036-b16b-d301fd161041 | SUCCESS | None | 2018-04-30 23:52:40 | 2018-04-30 23:52:41 |
| 72609978-cbc8-487f-8f45-615f4e993124 | 9a9fab1e-00d3-4e81-b3c7-09508448c4e8 | tripleo.baremetal.v1.set_node_state | | sub-workflow execution | 5888c3d8-ee99-4036-b16b-d301fd161041 | SUCCESS | None | 2018-04-30 23:52:40 | 2018-04-30 23:52:41 |
| cab4886e-b339-44c3-ace5-2025f7990f76 | 294458bb-ebc4-4181-9378-ced70eed8f2b | tripleo.baremetal.v1.provide | | | <none> | SUCCESS | None | 2018-04-30 23:52:40 | 2018-04-30 23:53:44 |
| d3512a83-2de6-4c98-9d28-b321f0b32f83 | 9a9fab1e-00d3-4e81-b3c7-09508448c4e8 | tripleo.baremetal.v1.set_node_state | | sub-workflow execution | 5888c3d8-ee99-4036-b16b-d301fd161041 | SUCCESS | None | 2018-04-30 23:52:40 | 2018-04-30 23:52:41 |
| d42d8e38-fd0f-4031-8c28-0abf47f2bd96 | 060249b4-8055-4e47-ae53-d1ca4f8daf27 | tripleo.baremetal.v1.cellv2_discovery | | sub-workflow execution | faf41862-c7f8-46e2-b420-36298e01e416 | ERROR | None | 2018-04-30 23:52:43 | 2018-04-30 23:52:48 |
| eb45d48c-f594-40c9-b5cb-ffbf17c6afec | 060249b4-8055-4e47-ae53-d1ca4f8daf27 | tripleo.baremetal.v1.cellv2_discovery | | sub-workflow execution | faf41862-c7f8-46e2-b420-36298e01e416 | SUCCESS | None | 2018-04-30 23:53:18 | 2018-04-30 23:53:23 |
| 0b43a3df-7708-4667-a7d8-cc518d08e320 | 7c6e8b24-689c-44b3-a35a-72b3d4c6259d | tripleo.baremetal.v1.set_power_state | | sub-workflow execution | 37940432-3f14-4846-94bf-c470ee449c37 | SUCCESS | None | 2018-04-30 23:53:23 | 2018-04-30 23:53:42 |
| 14aeeebd-aa43-4eda-be93-7281aa002835 | 7c6e8b24-689c-44b3-a35a-72b3d4c6259d | tripleo.baremetal.v1.set_power_state | | sub-workflow execution | 37940432-3f14-4846-94bf-c470ee449c37 | SUCCESS | None | 2018-04-30 23:53:23 | 2018-04-30 23:53:42 |
| 494f5ce5-0ff5-458a-b2ac-d5313c30391f | 7c6e8b24-689c-44b3-a35a-72b3d4c6259d | tripleo.baremetal.v1.set_power_state | | sub-workflow execution | 37940432-3f14-4846-94bf-c470ee449c37 | SUCCESS | None | 2018-04-30 23:53:23 | 2018-04-30 23:53:43 |
| 8c734b1e-3f37-4e1b-9f10-f5d91f3c75b4 | 7c6e8b24-689c-44b3-a35a-72b3d4c6259d | tripleo.baremetal.v1.set_power_state | | sub-workflow execution | 37940432-3f14-4846-94bf-c470ee449c37 | SUCCESS | None | 2018-04-30 23:53:23 | 2018-04-30 23:53:38 |
| 8f96316c-77cb-4ed5-bfe7-81fe22649979 | a258162f-550b-47b9-9811-73907d5b832f | tripleo.plan_management.v1.publish_ui_logs_to_swift | | {"triggered_by": {"type": "cron_trigger", "id": "09186f88-d080-442a-9ce7-58672cdbd622", "name": "publish-ui-logs-hourly"}, "description": "Workflow execution created by cron trigger '(09186f88-d080-442a-9ce7-58672cdbd622)'."} | <none> | SUCCESS | None | 2018-05-01 00:06:40 | 2018-05-01 00:06:44 |
| 3e1e3db6-02d2-4319-a70b-d7878a415f12 | a258162f-550b-47b9-9811-73907d5b832f | tripleo.plan_management.v1.publish_ui_logs_to_swift | | {"triggered_by": {"type": "cron_trigger", "id": "09186f88-d080-442a-9ce7-58672cdbd622", "name": "publish-ui-logs-hourly"}, "description": "Workflow execution created by cron trigger '(09186f88-d080-442a-9ce7-58672cdbd622)'."} | <none> | SUCCESS | None | 2018-05-01 01:07:47 | 2018-05-01 01:07:51 |
| 199bb331-17b3-4a15-b9c3-dcb2a999f9c1 | a258162f-550b-47b9-9811-73907d5b832f | tripleo.plan_management.v1.publish_ui_logs_to_swift | | {"triggered_by": {"type": "cron_trigger", "id": "09186f88-d080-442a-9ce7-58672cdbd622", "name": "publish-ui-logs-hourly"}, "description": "Workflow execution created by cron trigger '(09186f88-d080-442a-9ce7-58672cdbd622)'."} | <none> | SUCCESS | None | 2018-05-01 02:09:05 | 2018-05-01 02:09:09 |
| 3bcb9623-f3a1-444c-9bff-ce7cadf5e6ef | a258162f-550b-47b9-9811-73907d5b832f | tripleo.plan_management.v1.publish_ui_logs_to_swift | | {"triggered_by": {"type": "cron_trigger", "id": "09186f88-d080-442a-9ce7-58672cdbd622", "name": "publish-ui-logs-hourly"}, "description": "Workflow execution created by cron trigger '(09186f88-d080-442a-9ce7-58672cdbd622)'."} | <none> | SUCCESS | None | 2018-05-01 03:00:14 | 2018-05-01 03:00:17 |
| bd0d8107-caa1-45c6-ab5a-95fc02f4cad7 | a258162f-550b-47b9-9811-73907d5b832f | tripleo.plan_management.v1.publish_ui_logs_to_swift | | {"triggered_by": {"type": "cron_trigger", "id": "09186f88-d080-442a-9ce7-58672cdbd622", "name": "publish-ui-logs-hourly"}, "description": "Workflow execution created by cron trigger '(09186f88-d080-442a-9ce7-58672cdbd622)'."} | <none> | SUCCESS | None | 2018-05-01 04:02:11 | 2018-05-01 04:02:15 |
| 247b3ae1-c186-4a9d-83cc-42262728a6dc | a258162f-550b-47b9-9811-73907d5b832f | tripleo.plan_management.v1.publish_ui_logs_to_swift | | {"triggered_by": {"type": "cron_trigger", "id": "09186f88-d080-442a-9ce7-58672cdbd622", "name": "publish-ui-logs-hourly"}, "description": "Workflow execution created by cron trigger '(09186f88-d080-442a-9ce7-58672cdbd622)'."} | <none> | SUCCESS | None | 2018-05-01 05:03:36 | 2018-05-01 05:03:40 |
| 278105c7-8aec-49e4-a189-2ab649a86474 | a258162f-550b-47b9-9811-73907d5b832f | tripleo.plan_management.v1.publish_ui_logs_to_swift | | {"triggered_by": {"type": "cron_trigger", "id": "09186f88-d080-442a-9ce7-58672cdbd622", "name": "publish-ui-logs-hourly"}, "description": "Workflow execution created by cron trigger '(09186f88-d080-442a-9ce7-58672cdbd622)'."} | <none> | SUCCESS | None | 2018-05-01 06:04:57 | 2018-05-01 06:05:01 |
| bc407e98-6bd3-4e2e-b710-bb1a3eea9a36 | a258162f-550b-47b9-9811-73907d5b832f | tripleo.plan_management.v1.publish_ui_logs_to_swift | | {"triggered_by": {"type": "cron_trigger", "id": "09186f88-d080-442a-9ce7-58672cdbd622", "name": "publish-ui-logs-hourly"}, "description": "Workflow execution created by cron trigger '(09186f88-d080-442a-9ce7-58672cdbd622)'."} | <none> | SUCCESS | None | 2018-05-01 07:06:12 | 2018-05-01 07:06:16 |
| d4b6687c-f6ce-4312-bef7-bfd23afcb544 | a258162f-550b-47b9-9811-73907d5b832f | tripleo.plan_management.v1.publish_ui_logs_to_swift | | {"triggered_by": {"type": "cron_trigger", "id": "09186f88-d080-442a-9ce7-58672cdbd622", "name": "publish-ui-logs-hourly"}, "description": "Workflow execution created by cron trigger '(09186f88-d080-442a-9ce7-58672cdbd622)'."} | <none> | SUCCESS | None | 2018-05-01 08:08:07 | 2018-05-01 08:08:11 |
| 303d3d0d-d361-459e-bc07-3261ca0b888c | a258162f-550b-47b9-9811-73907d5b832f | tripleo.plan_management.v1.publish_ui_logs_to_swift | | {"triggered_by": {"type": "cron_trigger", "id": "09186f88-d080-442a-9ce7-58672cdbd622", "name": "publish-ui-logs-hourly"}, "description": "Workflow execution created by cron trigger '(09186f88-d080-442a-9ce7-58672cdbd622)'."} | <none> | SUCCESS | None | 2018-05-01 09:10:11 | 2018-05-01 09:10:15 |
| 9172f670-633b-4139-adb8-36c11d1cebbd | a258162f-550b-47b9-9811-73907d5b832f | tripleo.plan_management.v1.publish_ui_logs_to_swift | | {"triggered_by": {"type": "cron_trigger", "id": "09186f88-d080-442a-9ce7-58672cdbd622", "name": "publish-ui-logs-hourly"}, "description": "Workflow execution created by cron trigger '(09186f88-d080-442a-9ce7-58672cdbd622)'."} | <none> | SUCCESS | None | 2018-05-01 10:01:47 | 2018-05-01 10:01:50 |
| 534a87ed-7d0c-413c-8d00-eed17b024f0b | a258162f-550b-47b9-9811-73907d5b832f | tripleo.plan_management.v1.publish_ui_logs_to_swift | | {"triggered_by": {"type": "cron_trigger", "id": "09186f88-d080-442a-9ce7-58672cdbd622", "name": "publish-ui-logs-hourly"}, "description": "Workflow execution created by cron trigger '(09186f88-d080-442a-9ce7-58672cdbd622)'."} | <none> | SUCCESS | None | 2018-05-01 11:03:20 | 2018-05-01 11:03:24 |
| 21701be6-0935-4e93-8890-4664fbc2b4c7 | a258162f-550b-47b9-9811-73907d5b832f | tripleo.plan_management.v1.publish_ui_logs_to_swift | | {"triggered_by": {"type": "cron_trigger", "id": "09186f88-d080-442a-9ce7-58672cdbd622", "name": "publish-ui-logs-hourly"}, "description": "Workflow execution created by cron trigger '(09186f88-d080-442a-9ce7-58672cdbd622)'."} | <none> | SUCCESS | None | 2018-05-01 12:05:38 | 2018-05-01 12:05:43 |
| 755317ad-b61c-49a6-a18f-601c0b7c20d8 | a258162f-550b-47b9-9811-73907d5b832f | tripleo.plan_management.v1.publish_ui_logs_to_swift | | {"triggered_by": {"type": "cron_trigger", "id": "09186f88-d080-442a-9ce7-58672cdbd622", "name": "publish-ui-logs-hourly"}, "description": "Workflow execution created by cron trigger '(09186f88-d080-442a-9ce7-58672cdbd622)'."} | <none> | SUCCESS | None | 2018-05-01 13:07:26 | 2018-05-01 13:07:31 |
| 40ae71f9-4a0f-46ab-85f9-cf8a3595c73d | 59318d46-2e66-4152-8a26-8e9e4a76431b | tripleo.baremetal.v1.validate_nodes | | sub-workflow execution | 62b5a18e-8af3-4e8f-a02e-50f9b89eabfc | SUCCESS | None | 2018-05-01 13:20:19 | 2018-05-01 13:20:23 |
| 79b62d24-dbec-4e68-8246-622de89b0f56 | 8324764f-ae0a-483e-a26a-25a348359583 | tripleo.baremetal.v1.register_or_update | | | <none> | SUCCESS | None | 2018-05-01 13:20:19 | 2018-05-01 13:20:40 |
| 13d5c475-5b62-4b6f-96a0-cb52fd4cfd0a | 9a9fab1e-00d3-4e81-b3c7-09508448c4e8 | tripleo.baremetal.v1.set_node_state | | sub-workflow execution | f791d0c6-cd60-413c-a4b5-192c0d5b4d9b | SUCCESS | None | 2018-05-01 13:20:29 | 2018-05-01 13:20:32 |
| 4239b080-f5b7-4b28-b357-059f7b499563 | 20d0d265-2c8f-4ac4-95bc-5e1ec95f83b9 | tripleo.baremetal.v1.manage | | sub-workflow execution | 97ade1ca-31af-412c-b13f-1ae00ae05907 | SUCCESS | None | 2018-05-01 13:20:29 | 2018-05-01 13:20:36 |
| 82e328fa-7c2b-4132-b680-b07fcba274cd | 9a9fab1e-00d3-4e81-b3c7-09508448c4e8 | tripleo.baremetal.v1.set_node_state | | sub-workflow execution | f791d0c6-cd60-413c-a4b5-192c0d5b4d9b | SUCCESS | None | 2018-05-01 13:20:29 | 2018-05-01 13:20:34 |
| b502b8a5-a112-452d-80a7-a4567d5b4b24 | 9a9fab1e-00d3-4e81-b3c7-09508448c4e8 | tripleo.baremetal.v1.set_node_state | | sub-workflow execution | f791d0c6-cd60-413c-a4b5-192c0d5b4d9b | SUCCESS | None | 2018-05-01 13:20:29 | 2018-05-01 13:20:34 |
| ebbea7af-e6a2-48f5-adf9-a7242d305497 | 9a9fab1e-00d3-4e81-b3c7-09508448c4e8 | tripleo.baremetal.v1.set_node_state | | sub-workflow execution | f791d0c6-cd60-413c-a4b5-192c0d5b4d9b | SUCCESS | None | 2018-05-01 13:20:29 | 2018-05-01 13:20:32 |
| 67976e6e-c079-4a78-b073-7ccb6f41387f | 9a9fab1e-00d3-4e81-b3c7-09508448c4e8 | tripleo.baremetal.v1.set_node_state | | sub-workflow execution | 3d811172-2b93-49a7-b06c-f936419130d0 | SUCCESS | None | 2018-05-01 13:20:40 | 2018-05-01 13:20:41 |
| 680e7de4-b26c-45fd-adb0-e24b8c5294ed | 9a9fab1e-00d3-4e81-b3c7-09508448c4e8 | tripleo.baremetal.v1.set_node_state | | sub-workflow execution | 3d811172-2b93-49a7-b06c-f936419130d0 | SUCCESS | None | 2018-05-01 13:20:40 | 2018-05-01 13:20:41 |
| 836751fd-03c8-4cfd-9729-e096a421ad27 | 9a9fab1e-00d3-4e81-b3c7-09508448c4e8 | tripleo.baremetal.v1.set_node_state | | sub-workflow execution | 3d811172-2b93-49a7-b06c-f936419130d0 | SUCCESS | None | 2018-05-01 13:20:40 | 2018-05-01 13:20:41 |
| b4990f9d-dbe8-4182-87c3-07351f2d3080 | 9a9fab1e-00d3-4e81-b3c7-09508448c4e8 | tripleo.baremetal.v1.set_node_state | | sub-workflow execution | 3d811172-2b93-49a7-b06c-f936419130d0 | SUCCESS | None | 2018-05-01 13:20:40 | 2018-05-01 13:20:41 |
| b5794849-5c18-46e0-8b3a-3f8a20c043e0 | 294458bb-ebc4-4181-9378-ced70eed8f2b | tripleo.baremetal.v1.provide | | | <none> | SUCCESS | None | 2018-05-01 13:20:40 | 2018-05-01 13:20:58 |
| 30a84084-bf97-4dd2-9420-a52354f8ac25 | 060249b4-8055-4e47-ae53-d1ca4f8daf27 | tripleo.baremetal.v1.cellv2_discovery | | sub-workflow execution | 26b8495d-19eb-43a8-bc5f-61651cbeef97 | SUCCESS | None | 2018-05-01 13:20:42 | 2018-05-01 13:20:53 |
| 09e0e97c-e28d-4ce0-8689-11e074d3b200 | 7c6e8b24-689c-44b3-a35a-72b3d4c6259d | tripleo.baremetal.v1.set_power_state | | sub-workflow execution | 117e278f-f092-4b0f-84b1-bd341a516240 | SUCCESS | None | 2018-05-01 13:20:54 | 2018-05-01 13:20:55 |
| 9aff7425-6a01-4fea-88fd-d767e4e118da | 7c6e8b24-689c-44b3-a35a-72b3d4c6259d | tripleo.baremetal.v1.set_power_state | | sub-workflow execution | 117e278f-f092-4b0f-84b1-bd341a516240 | SUCCESS | None | 2018-05-01 13:20:54 | 2018-05-01 13:20:55 |
| eb77d861-1921-4240-83cd-360ae55fe184 | 7c6e8b24-689c-44b3-a35a-72b3d4c6259d | tripleo.baremetal.v1.set_power_state | | sub-workflow execution | 117e278f-f092-4b0f-84b1-bd341a516240 | SUCCESS | None | 2018-05-01 13:20:54 | 2018-05-01 13:20:55 |
| fcacea22-9eeb-4a48-bcc1-5920cdc20b56 | 7c6e8b24-689c-44b3-a35a-72b3d4c6259d | tripleo.baremetal.v1.set_power_state | | sub-workflow execution | 117e278f-f092-4b0f-84b1-bd341a516240 | SUCCESS | None | 2018-05-01 13:20:54 | 2018-05-01 13:20:55 |
| 2f696819-8340-4c13-90ec-edd6d74f0fe2 | a258162f-550b-47b9-9811-73907d5b832f | tripleo.plan_management.v1.publish_ui_logs_to_swift | | {"triggered_by": {"type": "cron_trigger", "id": "09186f88-d080-442a-9ce7-58672cdbd622", "name": "publish-ui-logs-hourly"}, "description": "Workflow execution created by cron trigger '(09186f88-d080-442a-9ce7-58672cdbd622)'."} | <none> | SUCCESS | None | 2018-05-01 14:09:02 | 2018-05-01 14:09:05 |
| 81f127d7-c060-4041-901d-a53d6bf2ac04 | a258162f-550b-47b9-9811-73907d5b832f | tripleo.plan_management.v1.publish_ui_logs_to_swift | | {"triggered_by": {"type": "cron_trigger", "id": "09186f88-d080-442a-9ce7-58672cdbd622", "name": "publish-ui-logs-hourly"}, "description": "Workflow execution created by cron trigger '(09186f88-d080-442a-9ce7-58672cdbd622)'."} | <none> | SUCCESS | None | 2018-05-01 15:00:35 | 2018-05-01 15:00:38 |
+--------------------------------------+--------------------------------------+-----------------------------------------------------+--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+---------+------------+---------------------+---------------------+
FYI, I have updated our ovirt staging drivers to use RPM instead of pip installed.
I seem to get this consistently now
openstack overcloud node import /home/stack/instackenv.json --provide
Started Mistral Workflow tripleo.baremetal.v1.register_or_update. Execution ID: b56983fb-98e4-422a-a4e3-d2a6c05286d6
Waiting for messages on queue 'tripleo' with no timeout.
4 node(s) successfully moved to the "manageable" state.
Successfully registered node UUID 2ee83985-b252-4210-bffe-c962cea150ec
Successfully registered node UUID 9fefef00-67bd-4ee3-bf97-e95819746d67
Successfully registered node UUID 80c6e914-5be8-42ab-ac55-bbe132f2d5e7
Successfully registered node UUID dbe9b853-a49e-4f1e-9476-a6744a38355f
Started Mistral Workflow tripleo.baremetal.v1.provide. Execution ID: b6558879-1887-4360-ae32-5d970cf0c3bd
Waiting for messages on queue 'tripleo' with no timeout.
[u"Failed to run action [action_ex_id=92ad8728-caa4-4cbe-9473-e993e87d3dc7, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'2ee83985-b252-4210-bffe-c962cea150ec'}']\n NovaAction.hypervisors.find failed: No Hypervisor matching {u'hypervisor_hostname': u'2ee83985-b252-4210-bffe-c962cea150ec'}. (HTTP 404)", u"Failed to run action [action_ex_id=49fd9725-a262-4fe4-a67d-3ab2d835c42f, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'9fefef00-67bd-4ee3-bf97-e95819746d67'}']\n NovaAction.hypervisors.find failed: No Hypervisor matching {u'hypervisor_hostname': u'9fefef00-67bd-4ee3-bf97-e95819746d67'}. (HTTP 404)", u"Failed to run action [action_ex_id=d68d7e28-6506-48e0-81b7-1335a0167ae8, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'80c6e914-5be8-42ab-ac55-bbe132f2d5e7'}']\n NovaAction.hypervisors.find failed: No Hypervisor matching {u'hypervisor_hostname': u'80c6e914-5be8-42ab-ac55-bbe132f2d5e7'}. (HTTP 404)", u"Failed to run action [action_ex_id=10a6175f-3f87-47f6-a65c-6db8997b4c69, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'dbe9b853-a49e-4f1e-9476-a6744a38355f'}']\n NovaAction.hypervisors.find failed: No Hypervisor matching {u'hypervisor_hostname': u'dbe9b853-a49e-4f1e-9476-a6744a38355f'}. (HTTP 404)"]
4 node(s) successfully moved to the "available" state.
Here is the import with the (--debug) but not seem any traceback Using python-ovirt-engine-sdk4-4.2.6-1.el7ev.x86_64.rpm for staging driver and seeing error each import. (undercloud) [stack@undercloud ~]$ openstack overcloud node import /home/stack/instackenv.json --provide --debug START with options: [u'overcloud', u'node', u'import', u'/home/stack/instackenv.json', u'--provide', u'--debug'] options: Namespace(access_key='', access_secret='***', access_token='***', access_token_endpoint='', access_token_type='', aodh_endpoint='', application_credential_id='', application_credential_name='', application_credential_secret='***', auth_type='password', auth_url='http://172.16.0.1:5000/', cacert=None, cert='', client_id='', client_secret='***', cloud='', code='', consumer_key='', consumer_secret='***', debug=True, default_domain='default', default_domain_id='', default_domain_name='', deferred_help=False, discovery_endpoint='', domain_id='', domain_name='', endpoint='', identity_provider='', identity_provider_url='', insecure=None, inspector_api_version='1', inspector_url=None, interface='', key='', log_file=None, openid_scope='', os_alarming_api_version='2', os_baremetal_api_version='1.34', os_beta_command=False, os_compute_api_version='', os_container_infra_api_version='1', os_data_processing_api_version='1.1', os_data_processing_url='', os_database_api_version='1', os_dns_api_version='2', os_identity_api_version='3', os_image_api_version='', os_key_manager_api_version='1', os_loadbalancer_api_version='2.0', os_metrics_api_version='1', os_network_api_version='', os_object_api_version='', os_orchestration_api_version='1', os_project_id=None, os_project_name=None, os_queues_api_version='2', os_tripleoclient_api_version='1', os_volume_api_version='', os_workflow_api_version='2', passcode='', password='***', profile='', project_domain_id='', project_domain_name='Default', project_id='', project_name='admin', protocol='', redirect_uri='', region_name='', remote_project_domain_id='', remote_project_domain_name='', remote_project_id='', remote_project_name='', roles='', service_provider='', service_provider_endpoint='', service_provider_entity_id='', system_scope='', timing=False, token='***', trust_id='', url='', user='', user_domain_id='', user_domain_name='Default', user_id='', username='admin', verbose_level=3, verify=None) Auth plugin password selected auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://172.16.0.1:5000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'loadbalancer_api_version': '2.0', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.34', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}} defaults: {u'auth_type': 'password', u'status': u'active', u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', 'api_timeout': None, u'baremetal_api_version': u'1', u'image_api_version': u'2', u'container_infra_api_version': u'1', u'metering_api_version': u'2', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', 'cacert': None, u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', u'application_catalog_api_version': u'1', u'key_manager_api_version': u'v1', u'workflow_api_version': u'2', 'verify': True, u'identity_api_version': u'2.0', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'container_api_version': u'1', u'dns_api_version': u'2', u'object_store_api_version': u'1', u'interface': None, u'disable_vendor_agent': {}} cloud cfg: {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://172.16.0.1:5000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'loadbalancer_api_version': '2.0', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.34', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}} compute API version 2, cmd group openstack.compute.v2 network API version 2, cmd group openstack.network.v2 image API version 2, cmd group openstack.image.v2 volume API version 2, cmd group openstack.volume.v2 identity API version 3, cmd group openstack.identity.v3 object_store API version 1, cmd group openstack.object_store.v1 neutronclient API version 2, cmd group openstack.neutronclient.v2 orchestration API version 1, cmd group openstack.orchestration.v1 messaging API version 2, cmd group openstack.messaging.v2 baremetal API version 1.34, cmd group openstack.baremetal.v1 baremetal_introspection API version 1, cmd group openstack.baremetal_introspection.v1 workflow_engine API version 2, cmd group openstack.workflow_engine.v2 tripleoclient API version 1, cmd group openstack.tripleoclient.v1 dns API version 2, cmd group openstack.dns.v2 load_balancer API version 2.0, cmd group openstack.load_balancer.v2 database API version 1, cmd group openstack.database.v1 key_manager API version 1, cmd group openstack.key_manager.v1 alarming API version 2, cmd group openstack.alarming.v2 container_infra API version 1, cmd group openstack.container_infra.v1 data_processing API version 1.1, cmd group openstack.data_processing.v1 metric API version 1, cmd group openstack.metric.v1 Auth plugin password selected auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://172.16.0.1:5000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'loadbalancer_api_version': '2.0', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.34', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}} Auth plugin password selected auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://172.16.0.1:5000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'loadbalancer_api_version': '2.0', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.34', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}} command: overcloud node import -> tripleoclient.v1.overcloud_node.ImportNode (auth=True) Auth plugin password selected auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://172.16.0.1:5000/', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'loadbalancer_api_version': '2.0', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.34', 'queues_api_version': '2', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'alarming_api_version': '2', 'debug': True, u'interface': None, u'disable_vendor_agent': {}} Using auth plugin: password Using parameters {'username': 'admin', 'project_name': 'admin', 'user_domain_name': 'Default', 'auth_url': 'http://172.16.0.1:5000/', 'password': '***', 'project_domain_name': 'Default'} Get auth_ref REQ: curl -g -i -X GET http://172.16.0.1:5000/ -H "Accept: application/json" -H "User-Agent: osc-lib/1.9.0 keystoneauth1/3.4.0 python-requests/2.14.2 CPython/2.7.5" Starting new HTTP connection (1): 172.16.0.1 http://172.16.0.1:5000 "GET / HTTP/1.1" 300 595 RESP: [300] Date: Tue, 01 May 2018 18:44:56 GMT Server: Apache Vary: X-Auth-Token Content-Length: 595 Keep-Alive: timeout=15, max=100 Connection: Keep-Alive Content-Type: application/json RESP BODY: {"versions": {"values": [{"status": "stable", "updated": "2018-02-28T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.10", "links": [{"href": "http://172.16.0.1:5000/v3/", "rel": "self"}]}, {"status": "deprecated", "updated": "2016-08-04T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://172.16.0.1:5000/v2.0/", "rel": "self"}, {"href": "https://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}]}} Making authentication request to http://172.16.0.1:5000/v3/auth/tokens http://172.16.0.1:5000 "POST /v3/auth/tokens HTTP/1.1" 201 8032 {"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "1bc1d1ec75bb47b6a7a90985c40cf85d", "name": "admin"}], "expires_at": "2018-05-01T22:44:56.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "6aeec69946724398b297c03373b82a20", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://172.16.0.1:5050", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "543ff698a0e841a78c3eda1fe30455f2"}, {"url": "http://172.16.0.1:5050", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "c78115215c4047fd91592a8fee609049"}, {"url": "http://172.16.0.1:5050", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "ea2aec627d954ae587415644eb5ec871"}], "type": "baremetal-introspection", "id": "0bfc501a6ae04df0b586047348aaefd6", "name": "ironic-inspector"}, {"endpoints": [{"url": "http://172.16.0.1:8989/v2", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "63ab02f8cbaf4f46a3eec7bc5e75c094"}, {"url": "http://172.16.0.1:8989/v2", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "6425b42a43c04d7fbfd6730b443ddf95"}, {"url": "http://172.16.0.1:8989/v2", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "7c5ed023934d4da2a7db0bb546844934"}], "type": "workflowv2", "id": "3b91c312fee24808a971b951948a2ec0", "name": "mistral"}, {"endpoints": [{"url": "http://172.16.0.1:8000/v1/6aeec69946724398b297c03373b82a20", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "2a9dc1a1e3c8413abb140d040f3456b8"}, {"url": "http://172.16.0.1:8000/v1/6aeec69946724398b297c03373b82a20", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "b1cb6f7ea49e4cbf9a4e4f1aeeaed917"}, {"url": "http://172.16.0.1:8000/v1/6aeec69946724398b297c03373b82a20", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "c00c48b4c0a34e079d0c84381d4aac88"}], "type": "cloudformation", "id": "3f80b635fe18447384008b035b78c1de", "name": "heat-cfn"}, {"endpoints": [{"url": "http://172.16.0.1:5000", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "28de66398fd24e5485ef900196339b6f"}, {"url": "http://172.16.0.1:5000", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "84b57d30caed49e89c07dea4bd7ba91c"}, {"url": "http://172.16.0.1:35357", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "e971a3175dcd48358889c29cbfee53ec"}], "type": "identity", "id": "406ff4958f7a4ec48255ce7caa8eddf9", "name": "keystone"}, {"endpoints": [{"url": "http://172.16.0.1:8774/v2.1", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "3792eeedcf97418981c9352bcc6f2e1c"}, {"url": "http://172.16.0.1:8774/v2.1", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "63e40c28bac04df5b574dd10b0b919ed"}, {"url": "http://172.16.0.1:8774/v2.1", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "82636d1e23514e3f9545633ee3b22cee"}], "type": "compute", "id": "6161dca5ce47423fba47c3b37db0d714", "name": "nova"}, {"endpoints": [{"url": "http://172.16.0.1:8004/v1/6aeec69946724398b297c03373b82a20", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "413e05ec7dc346c7ae86713d81aa5f24"}, {"url": "http://172.16.0.1:8004/v1/6aeec69946724398b297c03373b82a20", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "b44fdd6e7ea34f6cb29155c10c07f515"}, {"url": "http://172.16.0.1:8004/v1/6aeec69946724398b297c03373b82a20", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "cf13a092b5134b1dbff68cccadca40cc"}], "type": "orchestration", "id": "ab2f972c1fd648b8a3119b9c811f24f0", "name": "heat"}, {"endpoints": [{"url": "ws://172.16.0.1:9000", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "280f9de09d9b48aca407a57896875b1e"}, {"url": "ws://172.16.0.1:9000", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "5bf9a8125d594e8582d40d9c9ff74c30"}, {"url": "ws://172.16.0.1:9000", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "a428df9c4bfd4e69935652fbc3426861"}], "type": "messaging-websocket", "id": "c7b9308512bb46be9f5278338680ca2b", "name": "zaqar-websocket"}, {"endpoints": [{"url": "http://172.16.0.1:9292", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "05d57b914b2642a2a3e748187c1fb554"}, {"url": "http://172.16.0.1:9292", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "1e886ecd03a844cd805f4885cf2690a5"}, {"url": "http://172.16.0.1:9292", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "75ac030c2d754ae49559bc0f8745ce3c"}], "type": "image", "id": "d0e94690e3364930960f8ad25f3a97d5", "name": "glance"}, {"endpoints": [{"url": "http://172.16.0.1:8888", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "0d72026835834028b72ef04a3efc41f2"}, {"url": "http://172.16.0.1:8888", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "21dbbfe453884ad09394c49a7b2916f7"}, {"url": "http://172.16.0.1:8888", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "6cbec0d1d1fb4b46a886c2b1a5e49731"}], "type": "messaging", "id": "d9c68cd8b61040158ecd6726a29a64a4", "name": "zaqar"}, {"endpoints": [{"url": "http://172.16.0.1:8778/placement", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "065d9d6375d04df7b2d5b9121ab6d168"}, {"url": "http://172.16.0.1:8778/placement", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "345a88ca6a494d4894a02c555d92db4a"}, {"url": "http://172.16.0.1:8778/placement", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "3665432921e14486935ccb54f4a74f78"}], "type": "placement", "id": "e4f7989d7f804b5aa634cb6da125039b", "name": "placement"}, {"endpoints": [{"url": "http://172.16.0.1:9696", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "6327c072fe44450a99ba768ec962afed"}, {"url": "http://172.16.0.1:9696", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "bf164f4e1e384250a7286a4e2e746b09"}, {"url": "http://172.16.0.1:9696", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "d11ce5393ce5426e8250d940c46d1a3e"}], "type": "network", "id": "e7b5c052cdd0453b9547ead0a1c45605", "name": "neutron"}, {"endpoints": [{"url": "http://172.16.0.1:8080/v1/AUTH_6aeec69946724398b297c03373b82a20", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "6adbb2dbee194d4a94aeef758e0bb925"}, {"url": "http://172.16.0.1:8080/v1/AUTH_6aeec69946724398b297c03373b82a20", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "df1a52bc7e8d4851b2d26018e137cedb"}, {"url": "http://172.16.0.1:8080", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "f64940e346ff46e68a020c3b449d92bb"}], "type": "object-store", "id": "f344feac93d94ee1933c021d545c3bf8", "name": "swift"}, {"endpoints": [{"url": "http://172.16.0.1:6385", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "1229230391464b75b76b507bbdc3056c"}, {"url": "http://172.16.0.1:6385", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "502d4f63d11c4a6ca230c6797283b02b"}, {"url": "http://172.16.0.1:6385", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "910b528d3a1d4318a384c55d4d39534a"}], "type": "baremetal", "id": "fbe5c11766e94e46b277ac0ba65f3a92", "name": "ironic"}], "user": {"domain": {"id": "default", "name": "Default"}, "password_expires_at": null, "name": "admin", "id": "946ce4bcc65c483b8995a2629d9b87ab"}, "audit_ids": ["_Q6RJj7zR9W86oooYPJVrg"], "issued_at": "2018-05-01T18:44:56.000000Z"}} run(Namespace(env_file=<open file u'/home/stack/instackenv.json', mode 'r' at 0x7f66a3b6d5d0>, instance_boot_option='local', introspect=False, no_deploy_image=False, provide=True, run_validations=False)) take_action(Namespace(env_file=<open file u'/home/stack/instackenv.json', mode 'r' at 0x7f66a3b6d5d0>, instance_boot_option='local', introspect=False, no_deploy_image=False, provide=True, run_validations=False)) Instantiating workflow engine client: <class 'mistralclient.api.v2.client.Client'> Making authentication request to http://172.16.0.1:5000/v3/auth/tokens http://172.16.0.1:5000 "POST /v3/auth/tokens HTTP/1.1" 201 8032 {"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "1bc1d1ec75bb47b6a7a90985c40cf85d", "name": "admin"}], "expires_at": "2018-05-01T22:44:57.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "6aeec69946724398b297c03373b82a20", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://172.16.0.1:5050", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "543ff698a0e841a78c3eda1fe30455f2"}, {"url": "http://172.16.0.1:5050", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "c78115215c4047fd91592a8fee609049"}, {"url": "http://172.16.0.1:5050", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "ea2aec627d954ae587415644eb5ec871"}], "type": "baremetal-introspection", "id": "0bfc501a6ae04df0b586047348aaefd6", "name": "ironic-inspector"}, {"endpoints": [{"url": "http://172.16.0.1:8989/v2", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "63ab02f8cbaf4f46a3eec7bc5e75c094"}, {"url": "http://172.16.0.1:8989/v2", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "6425b42a43c04d7fbfd6730b443ddf95"}, {"url": "http://172.16.0.1:8989/v2", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "7c5ed023934d4da2a7db0bb546844934"}], "type": "workflowv2", "id": "3b91c312fee24808a971b951948a2ec0", "name": "mistral"}, {"endpoints": [{"url": "http://172.16.0.1:8000/v1/6aeec69946724398b297c03373b82a20", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "2a9dc1a1e3c8413abb140d040f3456b8"}, {"url": "http://172.16.0.1:8000/v1/6aeec69946724398b297c03373b82a20", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "b1cb6f7ea49e4cbf9a4e4f1aeeaed917"}, {"url": "http://172.16.0.1:8000/v1/6aeec69946724398b297c03373b82a20", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "c00c48b4c0a34e079d0c84381d4aac88"}], "type": "cloudformation", "id": "3f80b635fe18447384008b035b78c1de", "name": "heat-cfn"}, {"endpoints": [{"url": "http://172.16.0.1:5000", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "28de66398fd24e5485ef900196339b6f"}, {"url": "http://172.16.0.1:5000", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "84b57d30caed49e89c07dea4bd7ba91c"}, {"url": "http://172.16.0.1:35357", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "e971a3175dcd48358889c29cbfee53ec"}], "type": "identity", "id": "406ff4958f7a4ec48255ce7caa8eddf9", "name": "keystone"}, {"endpoints": [{"url": "http://172.16.0.1:8774/v2.1", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "3792eeedcf97418981c9352bcc6f2e1c"}, {"url": "http://172.16.0.1:8774/v2.1", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "63e40c28bac04df5b574dd10b0b919ed"}, {"url": "http://172.16.0.1:8774/v2.1", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "82636d1e23514e3f9545633ee3b22cee"}], "type": "compute", "id": "6161dca5ce47423fba47c3b37db0d714", "name": "nova"}, {"endpoints": [{"url": "http://172.16.0.1:8004/v1/6aeec69946724398b297c03373b82a20", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "413e05ec7dc346c7ae86713d81aa5f24"}, {"url": "http://172.16.0.1:8004/v1/6aeec69946724398b297c03373b82a20", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "b44fdd6e7ea34f6cb29155c10c07f515"}, {"url": "http://172.16.0.1:8004/v1/6aeec69946724398b297c03373b82a20", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "cf13a092b5134b1dbff68cccadca40cc"}], "type": "orchestration", "id": "ab2f972c1fd648b8a3119b9c811f24f0", "name": "heat"}, {"endpoints": [{"url": "ws://172.16.0.1:9000", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "280f9de09d9b48aca407a57896875b1e"}, {"url": "ws://172.16.0.1:9000", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "5bf9a8125d594e8582d40d9c9ff74c30"}, {"url": "ws://172.16.0.1:9000", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "a428df9c4bfd4e69935652fbc3426861"}], "type": "messaging-websocket", "id": "c7b9308512bb46be9f5278338680ca2b", "name": "zaqar-websocket"}, {"endpoints": [{"url": "http://172.16.0.1:9292", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "05d57b914b2642a2a3e748187c1fb554"}, {"url": "http://172.16.0.1:9292", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "1e886ecd03a844cd805f4885cf2690a5"}, {"url": "http://172.16.0.1:9292", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "75ac030c2d754ae49559bc0f8745ce3c"}], "type": "image", "id": "d0e94690e3364930960f8ad25f3a97d5", "name": "glance"}, {"endpoints": [{"url": "http://172.16.0.1:8888", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "0d72026835834028b72ef04a3efc41f2"}, {"url": "http://172.16.0.1:8888", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "21dbbfe453884ad09394c49a7b2916f7"}, {"url": "http://172.16.0.1:8888", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "6cbec0d1d1fb4b46a886c2b1a5e49731"}], "type": "messaging", "id": "d9c68cd8b61040158ecd6726a29a64a4", "name": "zaqar"}, {"endpoints": [{"url": "http://172.16.0.1:8778/placement", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "065d9d6375d04df7b2d5b9121ab6d168"}, {"url": "http://172.16.0.1:8778/placement", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "345a88ca6a494d4894a02c555d92db4a"}, {"url": "http://172.16.0.1:8778/placement", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "3665432921e14486935ccb54f4a74f78"}], "type": "placement", "id": "e4f7989d7f804b5aa634cb6da125039b", "name": "placement"}, {"endpoints": [{"url": "http://172.16.0.1:9696", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "6327c072fe44450a99ba768ec962afed"}, {"url": "http://172.16.0.1:9696", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "bf164f4e1e384250a7286a4e2e746b09"}, {"url": "http://172.16.0.1:9696", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "d11ce5393ce5426e8250d940c46d1a3e"}], "type": "network", "id": "e7b5c052cdd0453b9547ead0a1c45605", "name": "neutron"}, {"endpoints": [{"url": "http://172.16.0.1:8080/v1/AUTH_6aeec69946724398b297c03373b82a20", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "6adbb2dbee194d4a94aeef758e0bb925"}, {"url": "http://172.16.0.1:8080/v1/AUTH_6aeec69946724398b297c03373b82a20", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "df1a52bc7e8d4851b2d26018e137cedb"}, {"url": "http://172.16.0.1:8080", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "f64940e346ff46e68a020c3b449d92bb"}], "type": "object-store", "id": "f344feac93d94ee1933c021d545c3bf8", "name": "swift"}, {"endpoints": [{"url": "http://172.16.0.1:6385", "interface": "internal", "region": "regionOne", "region_id": "regionOne", "id": "1229230391464b75b76b507bbdc3056c"}, {"url": "http://172.16.0.1:6385", "interface": "public", "region": "regionOne", "region_id": "regionOne", "id": "502d4f63d11c4a6ca230c6797283b02b"}, {"url": "http://172.16.0.1:6385", "interface": "admin", "region": "regionOne", "region_id": "regionOne", "id": "910b528d3a1d4318a384c55d4d39534a"}], "type": "baremetal", "id": "fbe5c11766e94e46b277ac0ba65f3a92", "name": "ironic"}], "user": {"domain": {"id": "default", "name": "Default"}, "password_expires_at": null, "name": "admin", "id": "946ce4bcc65c483b8995a2629d9b87ab"}, "audit_ids": ["w1QHF1v2RXiN3gysigxS2A"], "issued_at": "2018-05-01T18:44:57.000000Z"}} Instantiating messaging websocket client: ws://172.16.0.1:9000 REQ: curl -g -i -X POST http://172.16.0.1:8989/v2/executions -H "User-Agent: osc-lib/1.9.0 keystoneauth1/3.4.0 python-requests/2.14.2 CPython/2.7.5" -H "content-type: application/json" -H "X-Auth-Token: {SHA1}982098c0c7678062eb57f7e5b961819cc9dc23c8" -d '{"input": "{\"instance_boot_option\": \"local\", \"nodes_json\": [{\"pm_password\": \"123456\", \"name\": \"osp13-controller-1\", \"capabilities\": \"profile:control,boot_option:local\", \"pm_type\": \"staging-ovirt\", \"pm_addr\": \"rhvm.field.lab.eng.rdu.redhat.com\", \"ovirt_vm_name\": \"osp13-controller-1\", \"mac\": [\"00:1a:4a:16:01:39\"], \"pm_vm_name\": \"osp13-controller-1\", \"memory\": \"4096\", \"disk\": \"40\", \"arch\": \"x86_64\", \"cpu\": \"2\", \"pm_user\": \"moran@internal\"}, {\"pm_password\": \"123456\", \"name\": \"osp13-controller-2\", \"capabilities\": \"profile:control,boot_option:local\", \"pm_type\": \"staging-ovirt\", \"pm_addr\": \"rhvm.field.lab.eng.rdu.redhat.com\", \"ovirt_vm_name\": \"osp13-controller-2\", \"mac\": [\"00:1a:4a:16:01:3a\"], \"pm_vm_name\": \"osp13-controller-2\", \"memory\": \"4096\", \"disk\": \"40\", \"arch\": \"x86_64\", \"cpu\": \"2\", \"pm_user\": \"moran@internal\"}, {\"pm_password\": \"123456\", \"name\": \"osp13-controller-3\", \"capabilities\": \"profile:control,boot_option:local\", \"pm_type\": \"staging-ovirt\", \"pm_addr\": \"rhvm.field.lab.eng.rdu.redhat.com\", \"ovirt_vm_name\": \"osp13-controller-3\", \"mac\": [\"00:1a:4a:16:01:3b\"], \"pm_vm_name\": \"osp13-controller-3\", \"memory\": \"4096\", \"disk\": \"40\", \"arch\": \"x86_64\", \"cpu\": \"2\", \"pm_user\": \"moran@internal\"}, {\"pm_password\": \"123456\", \"name\": \"osp13-compute-1\", \"capabilities\": \"profile:compute,boot_option:local\", \"pm_type\": \"staging-ovirt\", \"pm_addr\": \"rhvm.field.lab.eng.rdu.redhat.com\", \"ovirt_vm_name\": \"osp13-compute-1\", \"mac\": [\"00:1a:4a:16:01:3c\"], \"pm_vm_name\": \"osp13-compute-1\", \"memory\": \"4096\", \"disk\": \"40\", \"arch\": \"x86_64\", \"cpu\": \"2\", \"pm_user\": \"moran@internal\"}], \"ramdisk_name\": \"bm-deploy-ramdisk\", \"kernel_name\": \"bm-deploy-kernel\"}", "workflow_name": "tripleo.baremetal.v1.register_or_update", "description": ""}' Starting new HTTP connection (1): 172.16.0.1 http://172.16.0.1:8989 "POST /v2/executions HTTP/1.1" 201 2433 RESP: [201] Content-Length: 2433 Content-Type: application/json Date: Tue, 01 May 2018 18:44:58 GMT Connection: keep-alive RESP BODY: {"root_execution_id": null, "state_info": null, "description": "", "state": "RUNNING", "workflow_name": "tripleo.baremetal.v1.register_or_update", "task_execution_id": null, "updated_at": "2018-05-01 18:44:58", "workflow_id": "8324764f-ae0a-483e-a26a-25a348359583", "params": "{\"namespace\": \"\"}", "workflow_namespace": "", "output": "{}", "input": "{\"queue_name\": \"tripleo\", \"remove\": false, \"ramdisk_name\": \"bm-deploy-ramdisk\", \"kernel_name\": \"bm-deploy-kernel\", \"nodes_json\": [{\"pm_password\": \"123456\", \"name\": \"osp13-controller-1\", \"memory\": \"4096\", \"pm_addr\": \"rhvm.field.lab.eng.rdu.redhat.com\", \"cpu\": \"2\", \"mac\": [\"00:1a:4a:16:01:39\"], \"capabilities\": \"profile:control,boot_option:local\", \"pm_type\": \"staging-ovirt\", \"disk\": \"40\", \"arch\": \"x86_64\", \"ovirt_vm_name\": \"osp13-controller-1\", \"pm_user\": \"moran@internal\", \"pm_vm_name\": \"osp13-controller-1\"}, {\"pm_password\": \"123456\", \"name\": \"osp13-controller-2\", \"memory\": \"4096\", \"pm_addr\": \"rhvm.field.lab.eng.rdu.redhat.com\", \"cpu\": \"2\", \"mac\": [\"00:1a:4a:16:01:3a\"], \"capabilities\": \"profile:control,boot_option:local\", \"pm_type\": \"staging-ovirt\", \"disk\": \"40\", \"arch\": \"x86_64\", \"ovirt_vm_name\": \"osp13-controller-2\", \"pm_user\": \"moran@internal\", \"pm_vm_name\": \"osp13-controller-2\"}, {\"pm_password\": \"123456\", \"name\": \"osp13-controller-3\", \"memory\": \"4096\", \"pm_addr\": \"rhvm.field.lab.eng.rdu.redhat.com\", \"cpu\": \"2\", \"mac\": [\"00:1a:4a:16:01:3b\"], \"capabilities\": \"profile:control,boot_option:local\", \"pm_type\": \"staging-ovirt\", \"disk\": \"40\", \"arch\": \"x86_64\", \"ovirt_vm_name\": \"osp13-controller-3\", \"pm_user\": \"moran@internal\", \"pm_vm_name\": \"osp13-controller-3\"}, {\"pm_password\": \"123456\", \"name\": \"osp13-compute-1\", \"memory\": \"4096\", \"pm_addr\": \"rhvm.field.lab.eng.rdu.redhat.com\", \"cpu\": \"2\", \"mac\": [\"00:1a:4a:16:01:3c\"], \"capabilities\": \"profile:compute,boot_option:local\", \"pm_type\": \"staging-ovirt\", \"disk\": \"40\", \"arch\": \"x86_64\", \"ovirt_vm_name\": \"osp13-compute-1\", \"pm_user\": \"moran@internal\", \"pm_vm_name\": \"osp13-compute-1\"}], \"initial_state\": \"manageable\", \"instance_boot_option\": \"local\"}", "created_at": "2018-05-01 18:44:58", "project_id": "6aeec69946724398b297c03373b82a20", "id": "652b3ec5-0b1b-46a0-a388-dc1974a499d1"} HTTP POST http://172.16.0.1:8989/v2/executions 201 Started Mistral Workflow tripleo.baremetal.v1.register_or_update. Execution ID: 652b3ec5-0b1b-46a0-a388-dc1974a499d1 Waiting for messages on queue 'tripleo' with no timeout. 4 node(s) successfully moved to the "manageable" state. Successfully registered node UUID 1c2ed87a-5aeb-464a-8bef-3d9d9548b1b4 Successfully registered node UUID e602f940-33da-4939-8442-095dbe0435e1 Successfully registered node UUID dee9680c-2a95-4a49-871e-592d4c880b14 Successfully registered node UUID bacd39fc-f96c-459a-84e3-1f684fc2f4e8 Instantiating messaging websocket client: ws://172.16.0.1:9000 REQ: curl -g -i -X POST http://172.16.0.1:8989/v2/executions -H "User-Agent: osc-lib/1.9.0 keystoneauth1/3.4.0 python-requests/2.14.2 CPython/2.7.5" -H "content-type: application/json" -H "X-Auth-Token: {SHA1}982098c0c7678062eb57f7e5b961819cc9dc23c8" -d '{"input": "{\"node_uuids\": [\"1c2ed87a-5aeb-464a-8bef-3d9d9548b1b4\", \"e602f940-33da-4939-8442-095dbe0435e1\", \"dee9680c-2a95-4a49-871e-592d4c880b14\", \"bacd39fc-f96c-459a-84e3-1f684fc2f4e8\"]}", "workflow_name": "tripleo.baremetal.v1.provide", "description": ""}' http://172.16.0.1:8989 "POST /v2/executions HTTP/1.1" 201 692 RESP: [201] Content-Length: 692 Content-Type: application/json Date: Tue, 01 May 2018 18:45:14 GMT Connection: keep-alive RESP BODY: {"root_execution_id": null, "state_info": null, "description": "", "state": "RUNNING", "workflow_name": "tripleo.baremetal.v1.provide", "task_execution_id": null, "updated_at": "2018-05-01 18:45:14", "workflow_id": "294458bb-ebc4-4181-9378-ced70eed8f2b", "params": "{\"namespace\": \"\"}", "workflow_namespace": "", "output": "{}", "input": "{\"queue_name\": \"tripleo\", \"node_uuids\": [\"1c2ed87a-5aeb-464a-8bef-3d9d9548b1b4\", \"e602f940-33da-4939-8442-095dbe0435e1\", \"dee9680c-2a95-4a49-871e-592d4c880b14\", \"bacd39fc-f96c-459a-84e3-1f684fc2f4e8\"]}", "created_at": "2018-05-01 18:45:14", "project_id": "6aeec69946724398b297c03373b82a20", "id": "2f54e044-ba51-493d-a1e9-67ba24dbbcb2"} HTTP POST http://172.16.0.1:8989/v2/executions 201 Started Mistral Workflow tripleo.baremetal.v1.provide. Execution ID: 2f54e044-ba51-493d-a1e9-67ba24dbbcb2 Waiting for messages on queue 'tripleo' with no timeout. [u"Failed to run action [action_ex_id=153e5fb9-eb72-4687-9e54-d378e10a9a40, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'1c2ed87a-5aeb-464a-8bef-3d9d9548b1b4'}']\n NovaAction.hypervisors.find failed: No Hypervisor matching {u'hypervisor_hostname': u'1c2ed87a-5aeb-464a-8bef-3d9d9548b1b4'}. (HTTP 404)", u"Failed to run action [action_ex_id=26302869-1fcc-4148-8c53-03afaf5b4932, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'e602f940-33da-4939-8442-095dbe0435e1'}']\n NovaAction.hypervisors.find failed: No Hypervisor matching {u'hypervisor_hostname': u'e602f940-33da-4939-8442-095dbe0435e1'}. (HTTP 404)", u"Failed to run action [action_ex_id=ce2146d7-c985-4698-8a10-1fce0816a6ab, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'dee9680c-2a95-4a49-871e-592d4c880b14'}']\n NovaAction.hypervisors.find failed: No Hypervisor matching {u'hypervisor_hostname': u'dee9680c-2a95-4a49-871e-592d4c880b14'}. (HTTP 404)", u"Failed to run action [action_ex_id=8ef704cf-f9b1-49ce-93fa-65eab1b5bc06, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'bacd39fc-f96c-459a-84e3-1f684fc2f4e8'}']\n NovaAction.hypervisors.find failed: No Hypervisor matching {u'hypervisor_hostname': u'bacd39fc-f96c-459a-84e3-1f684fc2f4e8'}. (HTTP 404)"] 4 node(s) successfully moved to the "available" state. clean_up ImportNode: END return value: None Mike could you give the output of: openstack hypervisor show bacd39fc-f96c-459a-84e3-1f684fc2f4e8 After working with Mike on this issue, the problem seems to stem from a false positive output from the message bus. We ran the node registration
and were able to complete the registration but received the described output but upon further investigation the process completed correctly.
Will check with Dmitry to confirm these findings from the ironic point of view.
(undercloud) [stack@undercloud ~]$ openstack overcloud node import /home/stack/instackenv.json --provide
Started Mistral Workflow tripleo.baremetal.v1.register_or_update. Execution ID: ad3f2576-3dd5-423e-bb7c-d3990be5d742
Waiting for messages on queue 'tripleo' with no timeout.
4 node(s) successfully moved to the "manageable" state.
Successfully registered node UUID ad7983b0-e229-4243-9c4f-10ba13f7f906
Successfully registered node UUID 618deb78-ccfd-4ae6-8c80-82d436a1b418
Successfully registered node UUID ab6ed0d6-4cb7-4df4-a675-cd8a906bd7d8
Successfully registered node UUID 7ced5cc5-f070-4b4e-997f-cf4a510bd641
Started Mistral Workflow tripleo.baremetal.v1.provide. Execution ID: 291ced5b-971c-4da5-9baf-7d83ee168a7a
Waiting for messages on queue 'tripleo' with no timeout.
[u"Failed to run action [action_ex_id=e431ec27-c1bb-48e9-9934-429ec6a1a99d, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervis
'}', params='{u'hypervisor_hostname': u'ad7983b0-e229-4243-9c4f-10ba13f7f906'}']\n NovaAction.hypervisors.find failed: No Hypervisor matching {u'hypervisor_hostname': u'ad7983b0-e229-4243-
a13f7f906'}. (HTTP 404)", u"Failed to run action [action_ex_id=2b485fcc-c2de-492a-b93f-7038ff7872c0, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client
name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'618deb78-ccfd-4ae6-8c80-82d436a1b418'}']\n NovaAction.hypervisors.find failed: No Hypervisor matching {u'hypervisor_hostnam
8deb78-ccfd-4ae6-8c80-82d436a1b418'}. (HTTP 404)", u"Failed to run action [action_ex_id=9d385d08-128a-4aff-b6ea-2c6ae7c6f264, action_cls='<class 'mistral.actions.action_factory.NovaAction'
ibutes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'ab6ed0d6-4cb7-4df4-a675-cd8a906bd7d8'}']\n NovaAction.hypervisors.find failed: No Hypervisor match
ypervisor_hostname': u'ab6ed0d6-4cb7-4df4-a675-cd8a906bd7d8'}. (HTTP 404)", u"Failed to run action [action_ex_id=6e2191e8-9375-425b-8c21-6fc2c7162763, action_cls='<class 'mistral.actions.a
ctory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'7ced5cc5-f070-4b4e-997f-cf4a510bd641'}']\n NovaAction.hypervisors.find fa
Hypervisor matching {u'hypervisor_hostname': u'7ced5cc5-f070-4b4e-997f-cf4a510bd641'}. (HTTP 404)"]
4 node(s) successfully moved to the "available" state.
(undercloud) [stack@undercloud ~]$ mistral execution-get 291ced5b-971c-4da5-9baf-7d83ee168a7a
+--------------------+--------------------------------------+
| Field | Value |
+--------------------+--------------------------------------+
| ID | 291ced5b-971c-4da5-9baf-7d83ee168a7a |
| Workflow ID | 294458bb-ebc4-4181-9378-ced70eed8f2b |
| Workflow name | tripleo.baremetal.v1.provide |
| Workflow namespace | |
| Description | |
| Task Execution ID | <none> |
| State | SUCCESS |
| State info | None |
| Created at | 2018-05-02 20:28:59 |
| Updated at | 2018-05-02 20:29:46 |
+--------------------+--------------------------------------+
(undercloud) [stack@undercloud ~]$ mistral execution-get ad3f2576-3dd5-423e-bb7c-d3990be5d742
+--------------------+-----------------------------------------+
| Field | Value |
+--------------------+-----------------------------------------+
| ID | ad3f2576-3dd5-423e-bb7c-d3990be5d742 |
| Workflow ID | 8324764f-ae0a-483e-a26a-25a348359583 |
| Workflow name | tripleo.baremetal.v1.register_or_update |
| Workflow namespace | |
| Description | |
| Task Execution ID | <none> |
| State | SUCCESS |
| State info | None |
| Created at | 2018-05-02 20:28:44 |
| Updated at | 2018-05-02 20:28:59 |
+--------------------+-----------------------------------------+
(undercloud) [stack@undercloud ~]$
I cannot really comment from Ironic perspective since Ironic is not involved here, strictly speaking. But I suspect we should just remove logging inside of some retry loop. It winally worked for me. Error was caused by another bug, that was causing a duplicate hypervisor problem, and then mistral was parsing that and showing that error. Now the bug was fixed so i don't have this problem anymore The newly attached proposed patch cleans up the mistral defined workflow to prevent erroneous messages from cluttering the command line. (In reply to Yolanda Robla from comment #28) > It winally worked for me. Error was caused by another bug, that was causing > a duplicate hypervisor problem, and then mistral was parsing that and > showing that error. Now the bug was fixed so i don't have this problem > anymore I noticed this too, I am struggling with it right now: >>> nova.hypervisors.findall(hypervisor_hostname='3d4a448b-4410-421a-af5c-4fc220a8abd2') [<Hypervisor: 104>, <Hypervisor: 104>] Do you have any details on which particular bug was fixed that solved this? According to our records, this should be resolved by openstack-tripleo-common-8.6.3-13.el7ost. This build is available now. Hi,
I have Cust who hit a similar issue with openstack-tripleo-common - 8.6.3-13.
They managed to introspect the node but are stuck with making it available for deployment:
The node provide operation results with: NovaAction.hypervisors.find failed:
$ openstack overcloud node provide 16ac74c7-803b-4958-a1b1-ebed29e82e7e
Started Mistral Workflow tripleo.baremetal.v1.provide. Execution ID: bc681531-352f-43f5-913d-228a291879af
Waiting for messages on queue 'tripleo' with no timeout.
[u"Failed to run action [action_ex_id=1259ef07-24e1-4cc7-a20f-9f007e73c7f9, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'16ac74c7-803b-4958-a1b1-ebed29e82e7e'}']\n NovaAction.hypervisors.find failed: "]
[u"Failed to run action [action_ex_id=4f740fcb-98d7-4305-ae8f-c691e844e76c, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'16ac74c7-803b-4958-a1b1-ebed29e82e7e'}']\n NovaAction.hypervisors.find failed: "]
(In reply to Yolanda Robla from comment #28) > It winally worked for me. Error was caused by another bug, that was causing > a duplicate hypervisor problem, and then mistral was parsing that and > showing that error. Now the bug was fixed so i don't have this problem > anymore Hey Yolanda - Could You point me to that another bug that was causing a duplicate hypervisor problem? It seems my Customer hit the same issue. Every Hypervisor is doubled in the output of: (undercloud) [stack@pndh-ostkdir02 environments]$ openstack hypervisor list --long That is the problem i was hitting, with duplicated entries: https://bugzilla.redhat.com/show_bug.cgi?id=1576953 But this was resolved for me long time ago, you should need to check versions of the packages. Dariusz - note that the fix for this bug is available in openstack-tripleo-common-8.6.1-4.el7ost which will be in the upcoming release of OSP-13z4. You might want to do the hot fix route to get openstack-tripleo-common-8.6.1-4.el7ost if needed for your customer before the z4 release. (In reply to Bob Fournier from comment #38) > Dariusz - note that the fix for this bug is available in > openstack-tripleo-common-8.6.1-4.el7ost which will be in the upcoming > release of OSP-13z4. > > You might want to do the hot fix route to get > openstack-tripleo-common-8.6.1-4.el7ost if needed for your customer before > the z4 release. Actually the above mentioned version of tripleo-common is already released. Latest released build is openstack-tripleo-common-8.6.3-13.el7ost and it should have the relevent fixes in it. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:3587 |
On the introspection step, on queens, an error message is constantly shown: $ openstack overcloud node import --provide instackenv.json Started Mistral Workflow tripleo.baremetal.v1.register_or_update. Execution ID: b49129a9-18ad-443d-b326-7a1b3d3d53b4 Waiting for messages on queue '1c7ccb01-462b-40fb-b824-19f61a82b920' with no timeout. Nodes set to managed. Successfully registered node UUID 3cc25cf3-064a-4bb7-ae89-848f03b4dec2 Successfully registered node UUID 0040bc4d-f601-403d-933a-87473a4cf6f0 Started Mistral Workflow tripleo.baremetal.v1.provide. Execution ID: 263d89dd-cab8-4a84-b5c4-c466dd3d28f1 Waiting for messages on queue '1c7ccb01-462b-40fb-b824-19f61a82b920' with no timeout. [{u'cpu_info': u'', u'manager': {u'api': {u'server_groups': None, u'keypairs': None, u'servers': None, u'server_external_events': None, u'server_migrations': None, u'agents': None, u'instance_action': None, u'glance': None, u'hypervisor_stats': None, u'virtual_interfaces': None, u'flavors': None, u'availability_zones': None, u'user_id': None, u'cloudpipe': None, u'os_cache': False, u'quotas': None, u'migrations': None, u'usage': None, u'logger': None, u'project_id': None, u'neutron': None, u'quota_classes': None, u'project_name': None, u'aggregates': None, u'flavor_access': None, u'services': None, u'list_extensions': None, u'limits': None, u'hypervisors': None, u'cells': None, u'versions': None, u'client': None, u'hosts': None, u'volumes': None, u'assisted_volume_snapshots': None, u'certs': None}}, u'free_disk_gb': 0, u'id': 1, u'service': {u'host': u'undercloud-test.localdomain', u'disabled_reason': None, u'id': 6}, u'local_gb_used': 0, u'memory_mb_used': 0, u'current_workload': 0, u'state': u'up', u'status': u'enabled', u'host_ip': u'12.0.0.3', u'hypervisor_hostname': u'3cc25cf3-064a-4bb7-ae89-848f03b4dec2', u'hypervisor_version': 1, u'disk_available_least': 0, u'local_gb': 0, u'free_ram_mb': 0, u'vcpus_used': 0, u'hypervisor_type': u'ironic', u'x_openstack_request_ids': [u'req-56331e62-bab4-4a0b-8c3c-12fe27e4d1a9'], u'memory_mb': 0, u'vcpus': 0, u'running_vms': 0, u'_info': {u'status': u'enabled', u'service': {u'host': u'undercloud-test.localdomain', u'disabled_reason': None, u'id': 6}, u'vcpus_used': 0, u'hypervisor_type': u'ironic', u'local_gb_used': 0, u'host_ip': u'12.0.0.3', u'hypervisor_hostname': u'3cc25cf3-064a-4bb7-ae89-848f03b4dec2', u'memory_mb_used': 0, u'memory_mb': 0, u'current_workload': 0, u'vcpus': 0, u'state': u'up', u'cpu_info': u'', u'running_vms': 0, u'free_disk_gb': 0, u'hypervisor_version': 1, u'disk_available_least': 0, u'local_gb': 0, u'free_ram_mb': 0, u'id': 1}, u'_loaded': True}, u"Failed to run action [action_ex_id=4b0a56dc-478c-4106-a333-7f6c82286d18, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'0040bc4d-f601-403d-933a-87473a4cf6f0'}']\n NovaAction.hypervisors.find failed: No Hypervisor matching {u'hypervisor_hostname': u'0040bc4d-f601-403d-933a-87473a4cf6f0'}. (HTTP 404)"] Initially it looked just as an ugly output, but in my tests, i have to stop the provide command, because it just hangs, and also it leaves mistral with lots of ERROR tasks: openstack overcloud node provide --all-manageable [u"Failed to run action [action_ex_id=cdbec835-efc5-4e80-9590-992ace0deb31, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'8e24a698-3956-4ad8-8723-f0bd7d083237'}']\n NovaAction.hypervisors.find failed: "] [u"Failed to run action [action_ex_id=8f0de763-3644-4c48-8ae0-cddde09b2ed4, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'8e24a698-3956-4ad8-8723-f0bd7d083237'}']\n NovaAction.hypervisors.find failed: "] {u'status': u'FAILED', u'message': [u"Failed to run action [action_ex_id=8f0de763-3644-4c48-8ae0-cddde09b2ed4, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'8e24a698-3956-4ad8-8723-f0bd7d083237'}']\n NovaAction.hypervisors.find failed: "], u'result': None} [u"Failed to run action [action_ex_id=8034ad13-563c-46b4-ab6c-e43fe883794a, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'8e24a698-3956-4ad8-8723-f0bd7d083237'}']\n NovaAction.hypervisors.find failed: "] And also you can see on mistral: mistral execution-list | grep "ERROR"; | ba40a872-9170-4b83-ae5b-c78d465051f3 | 87e20d75-4b86-438b-bb0d-7b574c41a4de | tripleo.validations.v1.copy_ssh_key | | <none> | ERROR | Failure caused by error i... | 2018-01-18 14:35:34 | 2018-01-18 14:35:36 | | 1c0f01f1-0c73-4b4d-8779-c659dd8b9e1f | 17915115-e0b4-44ea-9b9b-c05e5dad5dfc | tripleo.baremetal.v1.provide_manageable_nodes | | <none> | ERROR | Failure caused by error i... | 2018-01-18 15:42:05 | 2018-01-18 15:58:21 | | 41fc80fa-a6da-44e3-a66f-96325ee37755 | 2a29ba5b-eb15-4cae-9fad-da7149bbf54c | tripleo.baremetal.v1.provide | sub-workflow execution | f3e1bf76-ea9c-4d6a-a090-865f1eba9a58 | ERROR | None | 2018-01-18 15:42:06 | 2018-01-18 15:58:21 | | b8aeb67f-6874-4721-a56b-56df59e17227 | b75ece3c-c0c9-40b4-98ec-dba1a19b8fe5 | tripleo.baremetal.v1.cellv2_discovery | sub-workflow execution | 04fcd7dc-cbf2-490a-975e-6b79b741cf0d | ERROR | None | 2018-01-18 15:42:09 | 2018-01-18 15:42:15 | | 80bd4c22-9d4e-4fc2-b276-cee11876cdfa | b75ece3c-c0c9-40b4-98ec-dba1a19b8fe5 | tripleo.baremetal.v1.cellv2_discovery | sub-workflow execution | 04fcd7dc-cbf2-490a-975e-6b79b741cf0d | ERROR | None | 2018-01-18 15:42:44 | 2018-01-18 15:42:50 |