Bug 1660866 - Exception when executed source /home/stack/stackrc;openstack overcloud node import --provide /tmp/cert_instackenv.json command
Summary: Exception when executed source /home/stack/stackrc;openstack overcloud node i...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-common
Version: 14.0 (Rocky)
Hardware: x86_64
OS: Linux
medium
high
Target Milestone: z3
: 14.0 (Rocky)
Assignee: Bob Fournier
QA Contact: Alexander Chuzhoy
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-12-19 12:47 UTC by viksss
Modified: 2019-09-12 20:21 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-09-12 20:21:32 UTC
Target Upstream Version:


Attachments (Terms of Use)
exception (109.96 KB, image/png)
2018-12-19 12:47 UTC, viksss
no flags Details


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 644802 0 None None None 2019-05-22 19:32:05 UTC

Description viksss 2018-12-19 12:47:54 UTC
Created attachment 1515593 [details]
exception

Description of problem:
Exception when executed source /home/stack/stackrc;openstack overcloud node import --provide /tmp/cert_instackenv.json command

[u"The action raised an exception [action_ex_id=1f67bc16-4127-46c9-b4a1-1c1c0a3fa70c, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', 
params='{u'hypervisor_hostname': u'29e8f9cc-6f53-4632-9c11-4c11689176c1'}']\n NovaAction.hypervisors.find failed: No Hypervisor matching {u'hypervisor_hostname': u'29e8f9cc-6f53-4632-9c11-4c11689176c1'}. (HTTP 4
04)"]
[u"The action raised an exception [action_ex_id=38cd3dab-a09a-4637-89ec-76cbf76dc325, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', 
params='{u'hypervisor_hostname': u'29e8f9cc-6f53-4632-9c11-4c11689176c1'}']\n NovaAction.hypervisors.find failed: No Hypervisor matching {u'hypervisor_hostname': u'29e8f9cc-6f53-4632-9c11-4c11689176c1'}. (HTTP 4
04)"]


Version-Release number of selected component (if applicable):
ansible-tripleo-ipsec-9.0.1-0.20181012162415.8b37e93.el7ost.noarch
openstack-tripleo-heat-templates-9.0.1-0.20181013060905.el7ost.noarch
python-tripleoclient-10.6.1-0.20181010222412.8c8f259.el7ost.noarch
puppet-tripleo-9.3.1-0.20181010034754.157eaab.el7ost.noarch
openstack-tripleo-image-elements-9.0.1-0.20181007200835.el7ost.noarch
openstack-tripleo-puppet-elements-9.0.0-0.20181007201103.daf9069.el7ost.noarch
openstack-tripleo-common-containers-9.4.1-0.20181012010884.el7ost.noarch
ansible-role-tripleo-modify-image-1.0.1-0.20181011160036.48a56c1.el7ost.noarch
openstack-tripleo-validations-9.3.1-0.20181008110758.4064fb7.el7ost.noarch
python2-tripleo-common-9.4.1-0.20181012010884.el7ost.noarch
openstack-tripleo-common-9.4.1-0.20181012010884.el7ost.noarch
python-tripleoclient-heat-installer-10.6.1-0.20181010222412.8c8f259.el7ost.noarch


How reproducible:


Steps to Reproduce:
1. Execute above Command
2.
3.

Actual results:
Exception occured after the command execution

Expected results:
It should not give any exception

Additional info:

Comment 1 Cédric Jeanneret 2018-12-19 16:19:14 UTC
Hello,

It would be good to get the following information:
- content of the baremetal.json
- output of openstack baremetal node list before and after the command is launched
- sosreport

Thank you !

C.

Comment 2 Steve Baker 2018-12-19 20:17:27 UTC
Assigning to HardProv since it looks like the mistral layer is behaving as expected when receiving an error from Ironic

Comment 3 Bob Fournier 2018-12-20 18:07:12 UTC
Looks like same issue reported here https://bugzilla.redhat.com/show_bug.cgi?id=1536521.

We can look when we get the requested info in Comment 1 but, as with 1536521, its unlikely that this is an Ironic issue.

Comment 4 viksss 2018-12-21 09:31:57 UTC
- content of the baremetal.json

(undercloud) [stack@undercloud-0 ~]$ cat instackenv.json 
{
    "nodes": [
    {
      "name": "compute-0",
      "mac": ["52:54:00:11:b8:d5"],
      "cpu": "2",
      "memory": "6144",
      "disk": "20",
            "arch": "x86_64",
              "pm_addr": "172.16.0.1",
        "pm_type": "ipmi",
        "pm_user": "admin",
        "pm_password": "password",
        "pm_port": "6231"
                },    {
      "name": "controller-0",
      "mac": ["52:54:00:69:ac:9d"],
      "cpu": "8",
      "memory": "32768",
      "disk": "35",
            "arch": "x86_64",
              "pm_addr": "172.16.0.1",
        "pm_type": "ipmi",
        "pm_user": "admin",
        "pm_password": "password",
        "pm_port": "6230"
                }    ]
}

- output of openstack baremetal node list before and after the command is launched

Before:
(undercloud) [stack@undercloud-0 ~]$ openstack baremetal node list

(undercloud) [stack@undercloud-0 ~]$ 

After:
(undercloud) [stack@undercloud-0 ~]$ openstack baremetal node list
+--------------------------------------+--------------+---------------+-------------+--------------------+-------------+
| UUID                                 | Name         | Instance UUID | Power State | Provisioning State | Maintenance |
+--------------------------------------+--------------+---------------+-------------+--------------------+-------------+
| 5d982e99-093e-4df2-8588-07c315d9b2dd | compute-0    | None          | power off   | available          | False       |
| 6c5c7077-b0b2-4f92-891c-7dd748fc1afd | controller-0 | None          | power off   | available          | False       |
+--------------------------------------+--------------+---------------+-------------+--------------------+-------------+




- sosreport

http://file.pnq.redhat.com/vmulaje/6.4/rhosp14/1/baremetal/sosreport/sosreport-undercloud-0-2018-12-18-ngbfabh.tar.xz


Complete log of the reproducer scenario
http://file.pnq.redhat.com/vmulaje/6.4/rhosp14/1/baremetal/baremetal/output.log

instaenv:
http://file.pnq.redhat.com/vmulaje/6.4/rhosp14/1/baremetal/baremetal/instackenv.json


Let me know if you need any other information

Thanks

Comment 5 Dmitry Tantsur 2019-04-11 10:28:26 UTC
I'm getting HTTP Forbidden when accessing the sosreport. Could you update it and maybe fetch the nova-compute logs?

Comment 6 viksss 2019-04-11 10:36:07 UTC
Updated the permissions for sosreport. Please check

Comment 7 Bob Fournier 2019-05-22 19:32:05 UTC
It looks like patch https://review.opendev.org/#/c/644802 that fixes this issue wasn't backported to Rocky until 3/25/19 so it will be in the latest 14z2 release.  

The sosreport is showing the same mistral logs
containers/mistral/engine.log:2018-12-18 07:40:29.610 1 DEBUG mistral.expressions.yaql_expression [req-fa933867-b768-4950-9fc9-b445b7b5918b 7cbb209356dc4267a0f39b2accd1d307 3ff1e3de83954620b56eb01b9aecfe9f - default default] Finished evaluation. [expression='<% $.get('message', '') %>', result: [u"The action raised an exception [action_ex_id=38cd3dab-a09a-4637-89ec-76cbf76dc325, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'29e8f9cc-6f53-4632-9c11-4c11689176c1'}']\n NovaAction.hypervisors.find failed: No Hypervisor matching {u'hypervisor_hostname': u'29e8f9cc-6f53-4632-9c11-4c11689176c1'}. (HTTP 404)"]] evaluate /usr/lib/python2.7/site-packages/mistral/expressions/yaql_expression.py:168
containers/mistral/engine.log:2018-12-18 07:40:29.684 1 DEBUG mistral.expressions.yaql_expression [req-fa933867-b768-4950-9fc9-b445b7b5918b 7cbb209356dc4267a0f39b2accd1d307 3ff1e3de83954620b56eb01b9aecfe9f - default default] Finished evaluation. [expression='<% {status => $.status, message => $.message, execution => $.execution, execution_id => $.execution.id, plan_name => $.plan_name, deployment_status => $.deployment_status} + $.payload %>', result: {u'status': u'FAILED', u'message': [u"The action raised an exception [action_ex_id=38cd3dab-a09a-4637-89ec-76cbf76dc325, action_cls='<class 'mistral.actions.action_factory.NovaAction'>', attributes='{u'client_method_name': u'hypervisors.find'}', params='{u'hypervisor_hostname': u'29e8f9cc-6f53-4632-9c11-4c11689176c1'}']\n NovaAction.hypervisors.find failed: No Hypervisor matching {u'hypervisor_hostname': u'29e8f9cc-6f53-4632-9c11-4c11689176c1'}. (HTTP 404)"], u'execution': {u'name': u'tripleo.baremetal.v1.cellv2_discovery', u'created_at': u'2018-12-18 12:40:26', u'updated_at': u'2018-12-18 12:40:26', u'id': u'c8c809bd-021b-46ec-a0b3-3b42669923ec', u'params': {u'root_execution_id': u'6b58c359-29fa-4259-b064-e06600a3a688', u'index': 0, u'namespace': u'', u'task_execution_id': u'4e90c19b-79d3-4412-972b-11122635afab', u'env': {}}, u'input': {u'queue_name': u'tripleo', u'node_uuids': [u'29e8f9cc-6f53-4632-9c11-4c11689176c1']}, u'spec': {u'tasks': {u'cell_v2_discover_hosts_failed': {u'version': u'2.0', u'type': u'direct', u'name': u'cell_v2_discover_hosts_failed', u'publish': {u'status': u'FAILED', u'message': u'<% task(cell_v2_discover_hosts).result %>'}, u'on-success': u'send_message'}, u'cell_v2_discover_hosts': {u'name': u'cell_v2_discover_hosts', u'on-error': u'cell_v2_discover_hosts_failed', u'on-success': u'wait_for_nova_resources', u'version': u'2.0', u'action': u'tripleo.baremetal.cell_v2_discover_hosts', u'type': u'direct'}, u'send_message': {u'input': {u'status': u"<% $.get('status', 'SUCCESS') %>", u'message': u"<% $.get('message', '') %>", u'queue_name': u'<% $.queue_name %>', u'type': u'<% execution().name %>', u'execution': u'<% execution() %>'}, u'version': u'2.0', u'type': u'direct', u'name': u'send_message', u'workflow': u'tripleo.messaging.v1.send'}, u'wait_for_nova_resources_failed': {u'version': u'2.0', u'type': u'direct', u'name': u'wait_for_nova_resources_failed', u'publish': {u'status': u'FAILED', u'message': u'<% task(wait_for_nova_resources).result %>'}, u'on-success': u'send_message'}, u'wait_for_nova_resources': {u'with-items': u'node_uuid in <% $.node_uuids %>', u'name': u'wait_for_nova_resources', u'on-error': u'wait_for_nova_resources_failed', u'on-success': u'send_message', u'version': u'2.0', u'action': u'nova.hypervisors_find hypervisor_hostname=<% $.node_uuid %>', u'type': u'direct'}}, u'description': u'Run cell_v2 host discovery', u'tags': [u'tripleo-common-managed'], u'version': u'2.0', u'input': [u'node_uuids', {u'queue_name': u'tripleo'}], u'name': u'cellv2_discovery'}}, u'plan_name': None, u'execution_id': u'c8c809bd-021b-46ec-a0b3-3b42669923ec', u'deployment_status': None}] evaluate /usr/lib/python2.7/site-packages/mistral/expressions/yaql_expression.py:168


Note You need to log in before you can comment on or make changes to this bug.