Bug 1645571

Summary: ironic-inspector introspection fails when updating port with missing switch_id
Product: Red Hat OpenStack Reporter: Bob Fournier <bfournie>
Component: openstack-ironic-inspectorAssignee: Bob Fournier <bfournie>
Status: CLOSED ERRATA QA Contact: mlammon
Severity: high Docs Contact:
Priority: high    
Version: 13.0 (Queens)CC: bfournie, dwojewod, jschluet, mariel, mburns, mlehmann, slinaber
Target Milestone: z4Keywords: Triaged, ZStream
Target Release: 13.0 (Queens)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-ironic-inspector-7.2.1-5.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1655076 (view as bug list) Environment:
Last Closed: 2019-01-16 17:56:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1655076    

Description Bob Fournier 2018-11-02 14:42:23 UTC
Description of problem:

When processing lldp data its possible that the switch_id is not set, for example if the switch either doesn't include this file or its not a mac address. When updating the port this can result in a validation error with "Missing mandatory keys".

f00e5ba0f state error] Committing fields: {'finished_at': datetime.datetime(2018, 10, 30, 8, 7, 3, 670716), 'error': "Missing mandatory keys: set(['switch_id']) (HTTP 400)"} _commit /usr/lib/python2.7/site-packages/ironic_inspector/node_cache.py:167
2018-10-30 09:07:03.699 1551 ERROR ironic_inspector.process [req-2c8d2817-af01-4e32-a692-20c17ca6f73e - - - - -] Unexpected exception during processing: BadRequest: Missing mandatory keys: set(['switch_id']) (HTTP 400)
2018-10-30 09:07:03.699 1551 ERROR ironic_inspector.process Traceback (most recent call last):
2018-10-30 09:07:03.699 1551 ERROR ironic_inspector.process File "/usr/lib/python2.7/site-packages/ironic_inspector/process.py", line 235, in process
2018-10-30 09:07:03.699 1551 ERROR ironic_inspector.process result = process_node(node_info, node, introspection_data)
2018-10-30 09:07:03.699 1551 ERROR ironic_inspector.process File "/usr/lib/python2.7/site-packages/ironic_inspector/node_cache.py", line 579, in inner
2018-10-30 09:07:03.699 1551 ERROR ironic_inspector.process return func(node_info, *args, **kwargs)
2018-10-30 09:07:03.699 1551 ERROR ironic_inspector.process File "/usr/lib/python2.7/site-packages/ironic_inspector/node_cache.py", line 557, in inner
2018-10-30 09:07:03.699 1551 ERROR ironic_inspector.process node_info.finished(istate.Events.error, error=str(exc))
2018-10-30 09:07:03.699 1551 ERROR ironic_inspector.process File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in exit
2018-10-30 09:07:03.699 1551 ERROR ironic_inspector.process self.force_reraise()
2018-10-30 09:07:03.699 1551 ERROR ironic_inspector.process File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-10-30 09:07:03.699 1551 ERROR ironic_inspector.process six.reraise(self.type, self.value, self.tb)
2018-10-30 09:07:03.699 1551 ERROR ironic_inspector.process File "/usr/lib/python2.7/site-packages/ironic_inspector/node_cache.py", line 541, in inner
2018-10-30 09:07:03.699 1551 ERROR ironic_inspector.process ret = func(node_info, *args, **kwargs)
2018-10-30 09:07:03.699 1551 ERROR ironic_inspector.process File "/usr/lib/python2.7/site-packages/ironic_inspector/process.py", line 268, in _process_node
2018-10-30 09:07:03.699 1551 ERROR ironic_inspector.process _run_post_hooks(node_info, introspection_data)
2018-10-30 09:07:03.699 1551 ERROR ironic_inspector.process File "/usr/lib/python2.7/site-packages/ironic_inspector/process.py", line 261, in _run_post_hooks
2018-10-30 09:07:03.699 1551 ERROR ironic_inspector.process hook_ext.obj.before_update(introspection_data, node_info)
2018-10-30 09:07:03.699 1551 ERROR ironic_inspector.process File "/usr/lib/python2.7/site-packages/ironic_inspector/plugins/local_link_connection.py", line 160, in before_update
2018-10-30 09:07:03.699 1551 ERROR ironic_inspector.process node_info.patch_port(port, patches)
2018-10-30 09:07:03.699 1551 ERROR ironic_inspector.process File "/usr/lib/python2.7/site-packages/ironic_inspector/node_cache.py", line 438, in patch_port
2018-10-30 09:07:03.699 1551 ERROR ironic_inspector.process new_port = ironic.port.update(port.uuid, patches)
2018-10-30 09:07:03.699 1551 ERROR ironic_inspector.process File "/usr/lib/python2.7/site-packages/ironicclient/v1/port.py", line 125, in update
2018-10-30 09:07:03.699 1551 ERROR ironic_inspector.process return self._update(resource_id=port_id, patch=patch)
2018-10-30 09:07:03.699 1551 ERROR ironic_inspector.process File "/usr/lib/python2.7/site-packages/ironicclient/common/base.py", line 200, in _update
2018-10-30 09:07:03.699 1551 ERROR ironic_inspector.process resp, body = self.api.json_request(method, url, **kwargs)
2018-10-30 09:07:03.699 1551 ERROR ironic_inspector.process File "/usr/lib/python2.7/site-packages/ironicclient/common/http.py", line 645, in json_request
2018-10-30 09:07:03.699 1551 ERROR ironic_inspector.process resp = self._http_request(url, method, **kwargs)
2018-10-30 09:07:03.699 1551 ERROR ironic_inspector.process File "/usr/lib/python2.7/site-packages/ironicclient/common/http.py", line 247, in wrapper
2018-10-30 09:07:03.699 1551 ERROR ironic_inspector.process return func(self, url, method, **kwargs)
2018-10-30 09:07:03.699 1551 ERROR ironic_inspector.process File "/usr/lib/python2.7/site-packages/ironicclient/common/http.py", line 627, in _http_request
2018-10-30 09:07:03.699 1551 ERROR ironic_inspector.process error_json.get('debuginfo'), method, url)

Version-Release number of selected component (if applicable):

OSP-13

openstack-ironic-api-10.1.3-5.el7ost.noarch                 Tue Oct 23 18:14:12 2018
openstack-ironic-common-10.1.3-5.el7ost.noarch              Tue Oct 23 18:13:57 2018
openstack-ironic-conductor-10.1.3-5.el7ost.noarch           Tue Oct 23 18:14:12 2018
openstack-ironic-inspector-7.2.1-2.el7ost.noarch            Tue Oct 23 18:14:13 2018

How reproducible:

Can occur if switch is sending a LLDP packet with non-mac address for switch_id


Steps to Reproduce:
1.  run introspection when switch is sending LLDP packet with non-mac address for switch_id

Actual results:

Introspection fails

Expected results:

Introspection succeeds

Comment 1 Bob Fournier 2018-11-02 16:47:39 UTC
Regarding the attached case, as a workaround the customer should edit /etc/ironic-inspector/inspector.conf and remove the "local_link_connection" field from this line:

processing_hooks=$default_processing_hooks,extra_hardware,lldp_basic,local_link_connection

then restart ironic-inspector

Comment 12 mlammon 2019-01-02 16:03:55 UTC
13   -p 2018-12-13.4

Env:
openstack-ironic-inspector-7.2.1-5.el7ost.noarch

We see the new code in place. This should no longer be an issue. Please re-open or file new BZ if issue still seen
from ironicclient import exceptions

           try:
                node_info.patch_port(port, patches)
            except exceptions.BadRequest as e:
                LOG.warning("Failed to update port %(uuid)s: %(error)s",
                            {'uuid': port.uuid, 'error': e},
                            node_info=node_info)

Comment 14 errata-xmlrpc 2019-01-16 17:56:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0092