Bug 1393974 - LLDP failure causes Introspection to fail
Summary: LLDP failure causes Introspection to fail
Keywords:
Status: CLOSED DUPLICATE of bug 1393008
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-ironic-inspector
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Dmitry Tantsur
QA Contact: Raviv Bar-Tal
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-11-10 18:15 UTC by Kiran Thyagaraja
Modified: 2016-11-11 11:16 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-11-11 11:16:59 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Kiran Thyagaraja 2016-11-10 18:15:29 UTC
Description of problem:
While deploying OpenStack using RH OSP10, introspection goes into a loop and never completes. Peeking into the journal files reveals that LLDP probing of interfaces fails. This problem could be hardware specific. The hardware being used is Dell PowerEdge M1000e chassis with PowerEdge M630 blades. The blades are interconnected with two MXL 10/40Gb switches and two PowerConnect M6348 GigE switches.

Setting ipa-collect-lldp=0 in /httpboot/inspector.ipxe is the currently used workaround for this problem. After disabling LLDP, introspection completes without error.

Version-Release number of selected component (if applicable):
Red Hat OpenStack Platform 10 beta. Using this puddle:
http://download-node-02.eng.bos.redhat.com/rcm-guest/puddles/OpenStack/10.0-RHEL-7/2016-11-08.2/

How reproducible: Always


Steps to Reproduce:
1. Deploy OSP10 undercloud
2. Begin baremetal introspection
3. 

Actual results: Introspection never completes and times out


Expected results: Successful introspection


Additional info:
The error witnessed during introspection is: 
Nov 09 17:42:42 localhost.localdomain ironic-python-agent[782]: 2016-11-09 17:42:42.773 782 INFO ironic_python_agent.netutils [-] Found LLDP info for interface: p2p1
Nov 09 17:42:42 localhost.localdomain ironic-python-agent[782]: 2016-11-09 17:42:42.773 782 ERROR ironic_python_agent.netutils [-] Error while getting LLDP info: unpack requires a string argument of length 2
Nov 09 17:42:42 localhost.localdomain ironic-python-agent[782]: 2016-11-09 17:42:42.773 782 ERROR ironic_python_agent.netutils Traceback (most recent call last):
Nov 09 17:42:42 localhost.localdomain ironic-python-agent[782]: 2016-11-09 17:42:42.773 782 ERROR ironic_python_agent.netutils   File "/usr/lib/python2.7/site-packages/ironic_python_agent/netutils.py", line 118, in get_lldp_info
Nov 09 17:42:42 localhost.localdomain ironic-python-agent[782]: 2016-11-09 17:42:42.773 782 ERROR ironic_python_agent.netutils     return _get_lldp_info(interfaces)
Nov 09 17:42:42 localhost.localdomain ironic-python-agent[782]: 2016-11-09 17:42:42.773 782 ERROR ironic_python_agent.netutils   File "/usr/lib/python2.7/site-packages/ironic_python_agent/netutils.py", line 187, in _get_lldp_info
Nov 09 17:42:42 localhost.localdomain ironic-python-agent[782]: 2016-11-09 17:42:42.773 782 ERROR ironic_python_agent.netutils     _receive_lldp_packets(s))
Nov 09 17:42:42 localhost.localdomain ironic-python-agent[782]: 2016-11-09 17:42:42.773 782 ERROR ironic_python_agent.netutils   File "/usr/lib/python2.7/site-packages/ironic_python_agent/netutils.py", line 154, in _receive_lldp_packets
Nov 09 17:42:42 localhost.localdomain ironic-python-agent[782]: 2016-11-09 17:42:42.773 782 ERROR ironic_python_agent.netutils     return _parse_tlv(pkt)
Nov 09 17:42:42 localhost.localdomain ironic-python-agent[782]: 2016-11-09 17:42:42.773 782 ERROR ironic_python_agent.netutils   File "/usr/lib/python2.7/site-packages/ironic_python_agent/netutils.py", line 133, in _parse_tlv
Nov 09 17:42:42 localhost.localdomain ironic-python-agent[782]: 2016-11-09 17:42:42.773 782 ERROR ironic_python_agent.netutils     tlvhdr = struct.unpack('!H', buff[:2])[0]
Nov 09 17:42:42 localhost.localdomain ironic-python-agent[782]: 2016-11-09 17:42:42.773 782 ERROR ironic_python_agent.netutils error: unpack requires a string argument of length 2
Nov 09 17:42:42 localhost.localdomain ironic-python-agent[782]: 2016-11-09 17:42:42.773 782 ERROR ironic_python_agent.netutils
Nov 09 17:42:42 localhost.localdomain kernel: device em4 left promiscuous mode
Nov 09 17:42:42 localhost.localdomain ironic-python-agent[782]: 2016-11-09 17:42:42.776 782 ERROR ironic_python_agent.netutils [-] Error while using raw socket: <class 'struct.error'>: unpack requires a string argument of length 2
Nov 09 17:42:42 localhost.localdomain ironic-python-agent[782]: 2016-11-09 17:42:42.776 782 ERROR ironic_python_agent.netutils Traceback (most recent call last):
Nov 09 17:42:42 localhost.localdomain ironic-python-agent[782]: 2016-11-09 17:42:42.776 782 ERROR ironic_python_agent.netutils   File "/usr/lib/python2.7/site-packages/ironic_python_agent/netutils.py", line 118, in get_lldp_info
Nov 09 17:42:42 localhost.localdomain ironic-python-agent[782]: 2016-11-09 17:42:42.776 782 ERROR ironic_python_agent.netutils     return _get_lldp_info(interfaces)
Nov 09 17:42:42 localhost.localdomain ironic-python-agent[782]: 2016-11-09 17:42:42.776 782 ERROR ironic_python_agent.netutils   File "/usr/lib/python2.7/site-packages/ironic_python_agent/netutils.py", line 187, in _get_lldp_info
Nov 09 17:42:42 localhost.localdomain ironic-python-agent[782]: 2016-11-09 17:42:42.776 782 ERROR ironic_python_agent.netutils     _receive_lldp_packets(s))
Nov 09 17:42:42 localhost.localdomain ironic-python-agent[782]: 2016-11-09 17:42:42.776 782 ERROR ironic_python_agent.netutils   File "/usr/lib/python2.7/site-packages/ironic_python_agent/netutils.py", line 154, in _receive_lldp_packets
Nov 09 17:42:42 localhost.localdomain ironic-python-agent[782]: 2016-11-09 17:42:42.776 782 ERROR ironic_python_agent.netutils     return _parse_tlv(pkt)
Nov 09 17:42:42 localhost.localdomain ironic-python-agent[782]: 2016-11-09 17:42:42.776 782 ERROR ironic_python_agent.netutils   File "/usr/lib/python2.7/site-packages/ironic_python_agent/netutils.py", line 133, in _parse_tlv
Nov 09 17:42:42 localhost.localdomain ironic-python-agent[782]: 2016-11-09 17:42:42.776 782 ERROR ironic_python_agent.netutils     tlvhdr = struct.unpack('!H', buff[:2])[0]
Nov 09 17:42:42 localhost.localdomain ironic-python-agent[782]: 2016-11-09 17:42:42.776 782 ERROR ironic_python_agent.netutils error: unpack requires a string argument of length 2
Nov 09 17:42:42 localhost.localdomain ironic-python-agent[782]: 2016-11-09 17:42:42.776 782 ERROR ironic_python_agent.netutils
Nov 09 17:42:42 localhost.localdomain kernel: device em2 left promiscuous mode
Nov 09 17:42:42 localhost.localdomain kernel: device em1 left promiscuous mode
Nov 09 17:42:43 localhost.localdomain kernel: device em3 left promiscuous mode
Nov 09 17:42:43 localhost.localdomain kernel: device p2p4 left promiscuous mode
Nov 09 17:42:43 localhost.localdomain kernel: device p2p1 left promiscuous mode
Nov 09 17:42:43 localhost.localdomain kernel: device p2p3 left promiscuous mode
Nov 09 17:42:43 localhost.localdomain kernel: device p2p2 left promiscuous mode
Nov 09 17:42:43 localhost.localdomain ironic-python-agent[782]: 2016-11-09 17:42:43.247 782 ERROR root [-] Unexpected error dispatching list_network_interfaces to manager <ironic_python_agent.hardware.GenericHardwareManager object at 0x1cceb90>: 'NoneType' object has no attribute 'items'
Nov 09 17:42:43 localhost.localdomain ironic-python-agent[782]: 2016-11-09 17:42:43.247 782 ERROR root Traceback (most recent call last):
Nov 09 17:42:43 localhost.localdomain ironic-python-agent[782]: 2016-11-09 17:42:43.247 782 ERROR root   File "/usr/lib/python2.7/site-packages/ironic_python_agent/hardware.py", line 1062, in dispatch_to_managers
Nov 09 17:42:43 localhost.localdomain ironic-python-agent[782]: 2016-11-09 17:42:43.247 782 ERROR root     return getattr(manager, method)(*args, **kwargs)
Nov 09 17:42:43 localhost.localdomain ironic-python-agent[782]: 2016-11-09 17:42:43.247 782 ERROR root   File "/usr/lib/python2.7/site-packages/ironic_python_agent/hardware.py", line 543, in list_network_interfaces
Nov 09 17:42:43 localhost.localdomain ironic-python-agent[782]: 2016-11-09 17:42:43.247 782 ERROR root     self._cache_lldp_data(iface_names)
Nov 09 17:42:43 localhost.localdomain ironic-python-agent[782]: 2016-11-09 17:42:43.247 782 ERROR root   File "/usr/lib/python2.7/site-packages/ironic_python_agent/hardware.py", line 483, in _cache_lldp_data
Nov 09 17:42:43 localhost.localdomain ironic-python-agent[782]: 2016-11-09 17:42:43.247 782 ERROR root     for ifname, tlvs in raw_lldp_data.items():
Nov 09 17:42:43 localhost.localdomain ironic-python-agent[782]: 2016-11-09 17:42:43.247 782 ERROR root AttributeError: 'NoneType' object has no attribute 'items'
Nov 09 17:42:43 localhost.localdomain ironic-python-agent[782]: 2016-11-09 17:42:43.247 782 ERROR root
Nov 09 17:42:43 localhost.localdomain ironic-python-agent[782]: 2016-11-09 17:42:43.248 782 ERROR ironic_python_agent.utils [-] collector default failed: 'NoneType' object has no attribute 'items'
Nov 09 17:42:43 localhost.localdomain ironic-python-agent[782]: 2016-11-09 17:42:43.248 782 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): hardware-detect execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:344

Comment 1 Dmitry Tantsur 2016-11-11 11:16:59 UTC

*** This bug has been marked as a duplicate of bug 1393008 ***


Note You need to log in before you can comment on or make changes to this bug.