Bug 1266597 - [RFE] Utility to Extract Network Introspection Data
[RFE] Utility to Extract Network Introspection Data
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-ironic-inspector-client (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
high Severity medium
: Upstream M1
: 12.0 (Pike)
Assigned To: Bob Fournier
mlammon
: FutureFeature, Triaged
Depends On: 1427608
Blocks: 1214284 1442136 1223054 1474127
  Show dependency treegraph
 
Reported: 2015-09-25 15:04 EDT by Dan Sneddon
Modified: 2018-02-05 14:02 EST (History)
26 users (show)

See Also:
Fixed In Version: python-ironic-inspector-client-1.12.0-0.20170331023022.045fd9e.el7ost
Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-12-13 15:33:46 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Example of extra_hardware blob from Swift (34.54 KB, text/plain)
2015-09-25 15:04 EDT, Dan Sneddon
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
OpenStack gerrit 406496 None None None 2017-02-17 14:09 EST
OpenStack gerrit 419141 None None None 2017-02-17 14:11 EST

  None (edit)
Description Dan Sneddon 2015-09-25 15:04:38 EDT
Created attachment 1077220 [details]
Example of extra_hardware blob from Swift

Description of problem:
Ironic introspection gathers data about the node to be deployed. Several features on the roadmap involve using this data (network validation, MAC-to-IP mapping, network automation, etc.)

Version-Release number of selected component (if applicable):
OSP 7.0.1, OSP-D 7.1

How reproducible:
100%

Steps to Reproduce:
1. Run baremetal introspection
2. 
3.

Actual results:
Data is stored in swift extra_hardware blob.

Expected results:
The introspection results should be extracted and written out to a formatted file (YAML?) for use by other utilities.

Additional info:
This will help greatly with network troubleshooting, since LACP will detect which VLANs are trunked on a particular interface (for many makes/models of switches).

This will also help with pre-allocating IP addresses, since we need a list of the MAC addresses on each host.

I have attached an example extra_hardware blob from Swift. As you can see, this is a rather unwieldy JSON that is hard to read and not easy to parse by other utilities. This needs to be both human and machine-readable (or at the very least, we need a human-readable extract of the most important troubleshooting data).

The things that I would like to see parsed:

NIC names
MAC addresses by NIC
LLDP data
Hardware offload status (by offload capability)
CPU info
Disk info
IPMI info

Even just separating these things by category and presenting them in YAML rather than JSON would be a big step in the right direction.
Comment 3 Dan Sneddon 2015-10-15 12:18:42 EDT
This should also support determining the mapping of nic numbers (used by os-net-config) to real nic names. So something like:

NICs:
nic1: em1
nic2: em2
nic3: p1p1
nic4: p1p2

This information will need to be gathered with a script that runs during introspection. Here is the logic to determine the mapping in Python:

def ordered_active_nics():
    embedded_nics = []
    nics = []
    for name in glob.iglob(_SYS_CLASS_NET + '/*'):
        nic = name[(len(_SYS_CLASS_NET) + 1):]
        if _is_active_nic(nic):
            if nic.startswith('em') or nic.startswith('eth') or \
                    nic.startswith('eno'):
                embedded_nics.append(nic)
            else:
                nics.append(nic)
    return sorted(embedded_nics) + sorted(nics)
Comment 7 Mike Burns 2016-04-07 16:50:54 EDT
This bug did not make the OSP 8.0 release.  It is being deferred to OSP 10.
Comment 10 Bob Fournier 2016-06-17 16:41:29 EDT
In a baremetal deployment using OSP8, I've extracted ironic data from swift using the command:
for node in $(ironic node-list | grep -v UUID| awk '{print $2}'); do swift -U service:ironic -K $IRONIC_DISCOVERD_PASSWORD download ironic-inspector inspector_data-$node; done

We've verified that the LLDP is enabled for all interfaces on the Juniper switch in the config.  However, in the extra network data I don't see any lldp data as I would have expected.

The inspector data for one of the nodes is here: http://pastebin.test.redhat.com/385135
Comment 11 Dmitry Tantsur 2016-08-05 09:06:11 EDT
Hi folks! Starting with the Newton release I can actually get the LLDP data by setting "ipa-collect-lldp=1" in /httpboot/inspector.ipxe and then using:

 openstack baremetal introspection data save 66e887df-c868-4b00-b6f0-adbeb31bb7d8 | jq .inventory.interfaces

(you can omit jq if you want more data, this command outputs everything we collect about network by default upstream).

Now, I'm not sure if this RFE actually belongs to inspector. If so, we can create a new inspector processing hook that will take the network data, convert it to whatever form we prefer, then store on Ironic node extra or in Swift. More about inspector plugins: http://docs.openstack.org/developer/ironic-inspector/contributing.html#writing-a-plugin

We need to hurry up if we want to hit Newton, the feature freeze is around end of August.

Or we can make it a separate utility, then we're not affected by any upstream procedures and deadlines.
Comment 13 Dmitry Tantsur 2016-08-05 09:09:29 EDT
> Hardware offload status (by offload capability)

I'm not sure we collect this one, we might need to extend IPA.

> IPMI info

We only collect IPMI address right now. Again, we might have to extend IPA.
Comment 15 Dmitry Tantsur 2016-08-17 05:03:06 EDT
I assume Bob is working on it, please correct me if I'm wrong. I'm also moving this bug to a more generic component, as it seems like we're not going to implement it in ironic-inspector itself. Please put a correct component as soon as you know it.
Comment 16 Bob Fournier 2016-08-17 10:03:00 EDT
Yes, this is currently being implemented in tripleo-common so I changed the component.  We don't plan on adding a ironic plugin to process the lldp stored by IPA.
Comment 17 Dmitry Tantsur 2016-10-10 10:33:10 EDT
We've changed our plans again, and decided to move this to the inspector client.
Comment 23 Ramon Acedo 2017-01-25 05:00:49 EST
After discussing the progress with Dan Sneddon he mentions that we no longer expect this to make the official Ocata cutoff, so if TripleO cuts on time we won't make the cutoff.

Moving to OSP 12.
Comment 24 Bob Fournier 2017-02-17 14:06:25 EST
Fix for plugin went into OSP-11:
https://review.openstack.org/#/c/406496/

Fix for python-ironic-inspector-client UI went into OSP-12:
https://review.openstack.org/#/c/419141/
Comment 25 Bob Fournier 2017-02-17 15:48:49 EST
Moving back to ON_DEV until patch that automatically enables plugin is
merged.
https://review.openstack.org/#/c/434356/
Comment 26 Ramon Acedo 2017-02-20 12:08:14 EST
Patch is merged. Setting to ON_QA.
Comment 33 mlammon 2017-07-18 15:16:43 EDT
This feature has now been tested and verified with latest OSP 12 7-13-2017

Environment:
python-ironic-inspector-client-1.12.0-0.20170331023022.045fd9e.el7ost.noarch

(undercloud) [stack@undercloud ~]$ ironic node-list
+--------------------------------------+------+---------------+-------------+--------------------+-------------+
| UUID                                 | Name | Instance UUID | Power State | Provisioning State | Maintenance |
+--------------------------------------+------+---------------+-------------+--------------------+-------------+
| efd37c61-5ca2-416e-9229-e9cc46fdb22c | None | None          | power off   | manageable         | False       |
| a8c04d3c-e2e5-4ba0-905a-afbbd1a60621 | None | None          | power off   | manageable         | False       |
| ffe25496-ab6c-4f83-a91e-a651880ea7b6 | None | None          | power off   | manageable         | False       |
| 3a6077c2-d7ea-4cc1-bebb-55ef6ee863b1 | None | None          | power off   | manageable         | False       |
| 225bceeb-e84f-42ae-829b-4128ac20b929 | None | None          | power off   | manageable         | False       |
| 8b917eba-839e-4c1f-9e05-208a286024f6 | None | None          | power off   | manageable         | False       |
| 9b68461f-3704-4b0d-a9f6-cbde7965ebcb | None | None          | power off   | manageable         | False       |
| c7f54398-237e-4382-8e3b-fe8533a52c5b | None | None          | power off   | manageable         | False       |
+--------------------------------------+------+---------------+-------------+--------------------+-------------+


(undercloud) [stack@undercloud ~]$ openstack baremetal introspection interface list efd37c61-5ca2-416e-9229-e9cc46fdb22c
+-----------+-------------------+------------------------+-------------------+----------------+
| Interface | MAC Address       | Switch Port VLAN IDs   | Switch Chassis ID | Switch Port ID |
+-----------+-------------------+------------------------+-------------------+----------------+
| p2p2      | 00:0a:f7:79:93:19 | [103, 102, 18, 20, 42] | 64:64:9b:31:12:00 | 510            |
| p2p1      | 00:0a:f7:79:93:18 | [101]                  | 64:64:9b:31:12:00 | 507            |
| em1       | c8:1f:66:c7:e8:2f | [162]                  | 08:81:f4:a6:b3:80 | 515            |
| em2       | c8:1f:66:c7:e8:30 | [182, 183]             | 08:81:f4:a6:b3:80 | 559            |
+-----------+-------------------+------------------------+-------------------+----------------+


(undercloud) [stack@undercloud ~]$ openstack baremetal introspection interface show efd37c61-5ca2-416e-9229-e9cc46fdb22c p2p1
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------+
| Field                                | Value                                                                                                                  |
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------+
| interface                            | p2p1                                                                                                                   |
| mac                                  | 00:0a:f7:79:93:18                                                                                                      |
| node_ident                           | efd37c61-5ca2-416e-9229-e9cc46fdb22c                                                                                   |
| switch_capabilities_enabled          | [u'Bridge', u'Router']                                                                                                 |
| switch_capabilities_support          | [u'Bridge', u'Router']                                                                                                 |
| switch_chassis_id                    | 64:64:9b:31:12:00                                                                                                      |
| switch_port_autonegotiation_enabled  | True                                                                                                                   |
| switch_port_autonegotiation_support  | True                                                                                                                   |
| switch_port_description              | ge-0/0/2.0                                                                                                             |
| switch_port_id                       | 507                                                                                                                    |
| switch_port_link_aggregation_enabled | False                                                                                                                  |
| switch_port_link_aggregation_id      | 0                                                                                                                      |
| switch_port_link_aggregation_support | True                                                                                                                   |
| switch_port_management_vlan_id       | None                                                                                                                   |
| switch_port_mau_type                 | Unknown                                                                                                                |
| switch_port_mtu                      | 1514                                                                                                                   |
| switch_port_physical_capabilities    | [u'1000BASE-T fdx', u'100BASE-TX fdx', u'100BASE-TX hdx', u'10BASE-T fdx', u'10BASE-T hdx', u'Asym and Sym PAUSE fdx'] |
| switch_port_protocol_vlan_enabled    | None                                                                                                                   |
| switch_port_protocol_vlan_ids        | None                                                                                                                   |
| switch_port_protocol_vlan_support    | None                                                                                                                   |
| switch_port_untagged_vlan_id         | 101                                                                                                                    |
| switch_port_vlan_ids                 | [101]                                                                                                                  |
| switch_port_vlans                    | [{u'name': u'RHOS13-PXE', u'id': 101}]                                                                                 |
| switch_protocol_identities           | None                                                                                                                   |
| switch_system_name                   | rhos-compute-node-sw1                                                                                                  |
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------+
Comment 37 errata-xmlrpc 2017-12-13 15:33:46 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462

Note You need to log in before you can comment on or make changes to this bug.