Bug 1540590
| Summary: | Introspection fails: Invalid input [name=baremetal_introspection.introspect, class=NoneType, unexpected=[u\'uuid\']] | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Filip Hubík <fhubik> |
| Component: | openstack-ironic-inspector | Assignee: | RHOS Maint <rhos-maint> |
| Status: | CLOSED ERRATA | QA Contact: | mlammon |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 10.0 (Newton) | CC: | bfournie, dbecker, derekh, emacchi, jjoyce, mburns, morazi, rhel-osp-director-maint, slinaber |
| Target Milestone: | z7 | Keywords: | Automation, AutomationBlocker, Triaged, ZStream |
| Target Release: | 10.0 (Newton) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-ironic-inspector-4.2.2-4.el7ost | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-02-27 16:17:56 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1502724 | ||
The problem here is that ironic-inspector is (on startup) running iptables to removed some rules. By design these may not exist and an exception is caught.
Due to an update in eventlet the Exception catching doesn't work. We can see this here (if using python-eventlet-0.20.1-2.el7ost.noarch )
#import eventlet
#eventlet.monkey_patch()
import subprocess
def _iptables():
cmd = ('ls', 'sdfsdf')
try:
subprocess.check_output(cmd)
except subprocess.CalledProcessError as exc:
print 'Caught'
print 'Done'
_iptables()
in the above example the exception will be caught, uncomment the eventlet lines and it wont.
eventlet has be upgraded in osp
python-eventlet-0.18.4-2.el7ost -> python-eventlet-0.20.1-2.el7ost
So the options would be to either
1. don't upgrade eventlet, is there a reason its been upgraded? OR
2. backport this patch (there may be others, I've yet to look)
https://review.openstack.org/#/c/449743/
If there is no specific reason to upgrade eventlet, option one is probably safer as other projects may be effected also.
If we do need to do option 2, it looks like that patch (https://review.openstack.org/#/c/449743/) is sufficient. I made the change to firewall.py from the patch and it fixed the problem. 18d17 < from eventlet.green import subprocess Great job investigating the issue! I agree, https://review.openstack.org/#/c/449743/ should be backported. This was found during CI phase1 validation testing. With the backported fix the tests got past this failure. Marking as Verified. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:0355 |
Description of problem: Introspection failing to pass in latest OSPd10 deployment with error: $ openstack baremetal introspection bulk start Setting nodes for introspection to manageable... Starting introspection of manageable nodes Started Mistral Workflow. Execution ID: 65354c28-33eb-4b67-afa3-7f112e565d36 Waiting for introspection to finish... stderr: Exception introspecting nodes: {u'result': u'Failed to run task [error=Invalid input [name=baremetal_introspection.introspect, class=NoneType, unexpected=[u\'uuid\']], wf=WorkflowExecution {\'state_info\': None, \'accepted\': False, \'output\': {}, \'name\': u\'tripleo.baremetal.v1.introspect\', \'workflow_name\': u\'tripleo.baremetal.v1.introspect\', \'state\': \'RUNNING\', \'created_at\': \'2018-01-30 11:53:33.706883\', \'runtime_context\': {\'index\': 0}, \'updated_at\': \'2018-01-30 11:53:33.711091\', \'spec\': {u\'input\': [u\'node_uuids\', {u\'queue_name\': u\'tripleo\'}], u\'tasks\': {u\'send_message\': {u\'retry\': u\'count=5 delay=1\', u\'name\': u\'send_message\', u\'version\': \'2.0\', u\'action\': u\'zaqar.queue_post\', u\'input\': {u\'queue_name\': u\'<% $.queue_name %>\', u\'messages\': {u\'body\': {u\'type\': u\'tripleo.baremetal.v1.introspect\', u\'payload\': {u\'status\': u"<% $.get(\'status\', \'SUCCESS\') %>", u\'message\': u"<% $.get(\'message\', \'\') %>", u\'execution\': u\'<% execution() %>\', u\'introspected_nodes\': u"<% $.get(\'introspected_nodes\', []) %>"}}}}, u\'type\': \'direct\'}, u\'set_status_failed_start_introspection\': {u\'version\': \'2.0\', u\'type\': \'direct\', u\'name\': u\'set_status_failed_start_introspection\', u\'publish\': {u\'status\': u\'FAILED\', u\'message\': u\'<% task(start_introspection).result %>\', u\'introspected_nodes\': []}, u\'on-success\': u\'send_message\'}, u\'wait_for_introspection_to_finish\': {u\'name\': u\'wait_for_introspection_to_finish\', u\'on-success\': u\'send_message\', u\'publish\': {u\'status\': u\'SUCCESS\', u\'message\': u\'Successfully introspected nodes.\', u\'introspected_nodes\': u\'<% task(wait_for_introspection_to_finish).result %>\'}, u\'version\': \'2.0\', u\'action\': u\'baremetal_introspection.wait_for_finish\', u\'input\': {u\'uuids\': u\'<% $.node_uuids %>\'}, u\'type\': \'direct\'}, u\'start_introspection\': {u\'with-items\': u\'uuid in <% $.node_uuids %>\', u\'name\': u\'start_introspection\', u\'on-error\': u\'set_status_failed_start_introspection\', u\'on-success\': u\'wait_for_introspection_to_finish\', u\'version\': \'2.0\', u\'action\': u\'baremetal_introspection.introspect uuid=<% $.uuid %>\', u\'type\': \'direct\'}}, u\'description\': u\'Take a list of nodes and move them through introspection.\', u\'version\': u\'2.0\', u\'name\': u\'introspect\'}, \'workflow_id\': u\'74ad5de2-c991-41c7-8bbb-6206a203cef4\', \'params\': {\'index\': 0, \'task_execution_id\': u\'688b876e-ed59-4e49-9a6a-2120a087496b\'}, \'context\': {\'openstack\': {u\'project_name\': u\'admin\', u\'user_id\': u\'3e329de943ea43a6a2375a1c9b83527a\', u\'roles\': [u\'admin\'], u\'auth_uri\': u\'http://192.168.24.1:5000/v3\', u\'auth_cacert\': None, u\'auth_token\': u\'f64396dae4f345e2ad2a3310060eb6a4\', u\'expires_at\': u\'2018-01-30T15:53:31.000000Z\', u\'is_trust_scoped\': False, u\'service_catalog\': u\'[{"endpoints": [{"adminURL": "http://192.168.24.1:8080", "region": "regionOne", "internalURL": "http://192.168.24.1:8080/v1/AUTH_176cc2884fcb4b6d9c7b73d828937094", "publicURL": "http://192.168.24.1:8080/v1/AUTH_176cc2884fcb4b6d9c7b73d828937094"}], "type": "object-store", "name": "swift"}, {"endpoints": [{"adminURL": "http://192.168.24.1:6385", "region": "regionOne", "internalURL": "http://192.168.24.1:6385", "publicURL": "http://192.168.24.1:6385"}], "type": "baremetal", "name": "ironic"}, {"endpoints": [{"adminURL": "ws://192.168.24.1:9000", "region": "regionOne", "internalURL": "ws://192.168.24.1:9000", "publicURL": "ws://192.168.24.1:9000"}], "type": "messaging-websocket", "name": "zaqar-websocket"}, {"endpoints": [{"adminURL": "http://192.168.24.1:8774/v2.1", "region": "regionOne", "internalURL": "http://192.168.24.1:8774/v2.1", "publicURL": "http://192.168.24.1:8774/v2.1"}], "type": "compute", "name": "nova"}, {"endpoints": [{"adminURL": "http://192.168.24.1:9696", "region": "regionOne", "internalURL": "http://192.168.24.1:9696", "publicURL": "http://192.168.24.1:9696"}], "type": "network", "name": "neutron"}, {"endpoints": [{"adminURL": "http://192.168.24.1:9292", "region": "regionOne", "internalURL": "http://192.168.24.1:9292", "publicURL": "http://192.168.24.1:9292"}], "type": "image", "name": "glance"}, {"endpoints": [{"adminURL": "http://192.168.24.1:8042", "region": "regionOne", "internalURL": "http://192.168.24.1:8042", "publicURL": "http://192.168.24.1:8042"}], "type": "alarming", "name": "aodh"}, {"endpoints": [{"adminURL": "http://192.168.24.1:8777", "region": "regionOne", "internalURL": "http://192.168.24.1:8777", "publicURL": "http://192.168.24.1:8777"}], "type": "metering", "name": "ceilometer"}, {"endpoints": [{"adminURL": "http://192.168.24.1:35357/v2.0", "region": "regionOne", "internalURL": "http://192.168.24.1:5000/v2.0", "publicURL": "http://192.168.24.1:5000/v2.0"}], "type": "identity", "name": "keystone"}, {"endpoints": [{"adminURL": "http://192.168.24.1:8888", "region": "regionOne", "internalURL": "http://192.168.24.1:8888", "publicURL": "http://192.168.24.1:8888"}], "type": "messaging", "name": "zaqar"}, {"endpoints": [{"adminURL": "http://192.168.24.1:8989/v2", "region": "regionOne", "internalURL": "http://192.168.24.1:8989/v2", "publicURL": "http://192.168.24.1:8989/v2"}], "type": "workflowv2", "name": "mistral"}, {"endpoints": [{"adminURL": "http://192.168.24.1:5050", "region": "regionOne", "internalURL": "http://192.168.24.1:5050", "publicURL": "http://192.168.24.1:5050"}], "type": "baremetal-introspection", "name": "ironic-inspector"}, {"endpoints": [{"adminURL": "http://192.168.24.1:8004/v1/176cc2884fcb4b6d9c7b73d828937094", "region": "regionOne", "internalURL": "http://192.168.24.1:8004/v1/176cc2884fcb4b6d9c7b73d828937094", "publicURL": "http://192.168.24.1:8004/v1/176cc2884fcb4b6d9c7b73d828937094"}], "type": "orchestration", "name": "heat"}]\', u\'project_id\': u\'176cc2884fcb4b6d9c7b73d828937094\', u\'user_name\': u\'admin\'}, \'__execution\': {\'id\': u\'dc072fa2-40e9-4e04-be0d-8f7d0e2d7a82\'}}, \'input\': {u\'queue_name\': u\'8830e877-98c9-4dad-a1a1-803379006cbe\', u\'node_uuids\': [u\'3825e110-8969-4ce7-b161-dfa0a696a6b4\', u\'4e31032c-51b3-4a5a-9b3d-85f2080f87bb\', u\'9f4b722d-6dca-4dab-8390-58ffa1b5bcd7\']}, \'scope\': \'private\', \'project_id\': u\'176cc2884fcb4b6d9c7b73d828937094\', \'task_execution_id\': u\'688b876e-ed59-4e49-9a6a-2120a087496b\', \'id\': u\'dc072fa2-40e9-4e04-be0d-8f7d0e2d7a82\', \'description\': \'sub-workflow execution\'}, task=start_introspection]: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/mistral/engine/task_handler.py", line 57, in run_task task.run() File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 154, in wrapper return f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/mistral/engine/tasks.py", line 268, in run self._run_new() File "/usr/lib/python2.7/site-packages/mistral/engine/tasks.py", line 291, in _run_new self._schedule_actions() File "/usr/lib/python2.7/site-packages/mistral/engine/tasks.py", line 460, in _schedule_actions action.validate_input(input_dict) File "/usr/lib/python2.7/site-packages/mistral/engine/actions.py", line 288, in validate_input e_utils.validate_input(self.action_def, input_dict) File "/usr/lib/python2.7/site-packages/mistral/engine/utils.py", line 58, in validate_input msg % tuple(msg_props) InputException: Invalid input [name=baremetal_introspection.introspect, class=NoneType, unexpected=[u\'uuid\']] Version-Release number of selected component (if applicable): OSPd10 Expected results: Introspection should pass without error. Additional info: started since puddle 2018-01-29.2 reproducible in 100% cases $ rpm -qa | grep -E '(mistral|ironic|tripleo)' openstack-ironic-api-6.2.4-2.el7ost.noarch puppet-ironic-9.5.0-3.el7ost.noarch openstack-mistral-executor-3.0.2-12.el7ost.noarch openstack-tripleo-ui-1.2.1-1.el7ost.noarch openstack-tripleo-puppet-elements-5.3.3-1.el7ost.noarch openstack-mistral-engine-3.0.2-12.el7ost.noarch python-openstack-mistral-3.0.2-12.el7ost.noarch openstack-tripleo-heat-templates-5.3.8-1.el7ost.noarch openstack-tripleo-validations-5.1.3-1.el7ost.noarch openstack-tripleo-common-5.4.7-1.el7ost.noarch openstack-tripleo-0.0.8-0.3.4de13b3git.el7ost.noarch puppet-mistral-9.5.0-2.el7ost.noarch openstack-tripleo-image-elements-5.3.1-3.el7ost.noarch python-ironic-inspector-client-1.10.0-1.el7ost.noarch openstack-mistral-common-3.0.2-12.el7ost.noarch python-ironic-lib-2.1.3-1.el7ost.noarch puppet-tripleo-5.6.4-3.el7ost.noarch openstack-ironic-conductor-6.2.4-2.el7ost.noarch python-mistralclient-2.1.2-1.el7ost.noarch python-ironicclient-1.7.1-1.el7ost.noarch python-tripleoclient-5.4.5-1.el7ost.noarch openstack-ironic-common-6.2.4-2.el7ost.noarch openstack-ironic-inspector-4.2.2-3.el7ost.noarch openstack-mistral-api-3.0.2-12.el7ost.noarch