Bug 1649688 - [osp15] nova diagnostics command is not working with all instances
Summary: [osp15] nova diagnostics command is not working with all instances
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova
Version: 15.0 (Stein)
Hardware: All
OS: All
high
high
Target Milestone: rc
: 15.0 (Stein)
Assignee: Francois Palin
QA Contact: nova-maint
URL:
Whiteboard:
Depends On:
Blocks: 1722253 1722273
TreeView+ depends on / blocked
 
Reported: 2018-11-14 09:01 UTC by Nilesh
Modified: 2019-10-22 13:11 UTC (History)
12 users (show)

Fixed In Version: openstack-nova-19.0.2-0.20190701170413.b01bc2f.el8ost
Doc Type: Bug Fix
Doc Text:
In earlier Red Hat OpenStack Platform versions, the RHOSP Compute service (nova) diagnostics command returned an "IndexError" on compute instances that used VFIO interfaces. In RHOSP 15, this problem has been addressed. The diagnostics command now retrieves interface data directly from the guest XML, and appropriately adds NICs to the diagnostics object.
Clone Of:
: 1722253 (view as bug list)
Environment:
Last Closed: 2019-09-21 11:19:23 UTC
Target Upstream Version:
fpalin: needinfo+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1821798 0 None None None 2019-03-28 15:51:02 UTC
OpenStack gerrit 648123 0 None MERGED Include all network devices in nova diagnostics 2020-04-22 17:26:56 UTC
Red Hat Product Errata RHEA-2019:2811 0 None None None 2019-09-21 11:19:49 UTC

Comment 2 smooney 2018-11-16 14:11:33 UTC
This is a valid bug that will need to be fix upstream first for osp 15 and then back-ported.

Comment 16 Francois Palin 2019-03-28 15:59:52 UTC
Hi Mark,

We have started working on a fix upstream, see Launchpad bug 1821798.

We will work on the backport to OSP 13 as soon as the upstream change is ready.

Thank you,

François

Comment 25 Francois Palin 2019-06-19 19:38:09 UTC
Change merged upstream.

master: commit ab7c968b6f66404c032f62a952e353f94d3be165

Stein:  commit 1d4f64b190afc60b0c2a56de718209869c41cfb3

Comment 26 Matthew Booth 2019-07-03 16:21:46 UTC
Targetting OSP14 for initial backport, OSP13 to follow.

Comment 31 Joe H. Rahme 2019-08-07 08:03:12 UTC
On a SR-IOV enabled deployment:

(overcloud) [stack@undercloud-0 os-smoke]$ openstack network create sriov --provider-network-type vlan \
> --provider-physical-network sriov-1 --disable-port-security
+---------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                     | Value                                                                                                                                                                                      |
+---------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up            | UP                                                                                                                                                                                         |
| availability_zone_hints   |                                                                                                                                                                                            |
| availability_zones        |                                                                                                                                                                                            |
| created_at                | 2019-08-06T17:36:39Z                                                                                                                                                                       |
| description               |                                                                                                                                                                                            |
| dns_domain                | None                                                                                                                                                                                       |
| id                        | a7a949bb-e680-4fb2-adcd-f76c7eaddec4                                                                                                                                                       |
| ipv4_address_scope        | None                                                                                                                                                                                       |
| ipv6_address_scope        | None                                                                                                                                                                                       |
| is_default                | False                                                                                                                                                                                      |
| is_vlan_transparent       | None                                                                                                                                                                                       |
| location                  | Munch({'cloud': '', 'region_name': 'regionOne', 'zone': None, 'project': Munch({'id': '53084bdcf4cb4bd09b43f3b4b50a7cc3', 'name': 'admin', 'domain_id': None, 'domain_name': 'Default'})}) |
| mtu                       | 9000                                                                                                                                                                                       |
| name                      | sriov                                                                                                                                                                                      |
| port_security_enabled     | False                                                                                                                                                                                      |
| project_id                | 53084bdcf4cb4bd09b43f3b4b50a7cc3                                                                                                                                                           |
| provider:network_type     | vlan                                                                                                                                                                                       |
| provider:physical_network | sriov-1                                                                                                                                                                                    |
| provider:segmentation_id  | 600                                                                                                                                                                                        |
| qos_policy_id             | None                                                                                                                                                                                       |
| revision_number           | 2                                                                                                                                                                                          |
| router:external           | Internal                                                                                                                                                                                   |
| segments                  | None                                                                                                                                                                                       |
| shared                    | False                                                                                                                                                                                      |
| status                    | ACTIVE                                                                                                                                                                                     |
| subnets                   |                                                                                                                                                                                            |
| tags                      |                                                                                                                                                                                            |
| updated_at                | 2019-08-06T17:36:39Z                                                                                                                                                                       |
+---------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
 
 
(overcloud) [stack@undercloud-0 os-smoke]$ openstack subnet create --network sriov --ip-version 4 --subnet-range 192.168.111.0/24 --no-dhcp subnet-sriov
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field             | Value                                                                                                                                                                                      |
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| allocation_pools  | 192.168.111.2-192.168.111.254                                                                                                                                                              |
| cidr              | 192.168.111.0/24                                                                                                                                                                           |
| created_at        | 2019-08-06T17:37:12Z                                                                                                                                                                       |
| description       |                                                                                                                                                                                            |
| dns_nameservers   |                                                                                                                                                                                            |
| enable_dhcp       | False                                                                                                                                                                                      |
| gateway_ip        | 192.168.111.1                                                                                                                                                                              |
| host_routes       |                                                                                                                                                                                            |
| id                | a368e50e-3bc5-44d1-8c4e-07ea9cb7bf69                                                                                                                                                       |
| ip_version        | 4                                                                                                                                                                                          |
| ipv6_address_mode | None                                                                                                                                                                                       |
| ipv6_ra_mode      | None                                                                                                                                                                                       |
| location          | Munch({'cloud': '', 'region_name': 'regionOne', 'zone': None, 'project': Munch({'id': '53084bdcf4cb4bd09b43f3b4b50a7cc3', 'name': 'admin', 'domain_id': None, 'domain_name': 'Default'})}) |
| name              | subnet-sriov                                                                                                                                                                               |
| network_id        | a7a949bb-e680-4fb2-adcd-f76c7eaddec4                                                                                                                                                       |
| prefix_length     | None                                                                                                                                                                                       |
| project_id        | 53084bdcf4cb4bd09b43f3b4b50a7cc3                                                                                                                                                           |
| revision_number   | 0                                                                                                                                                                                          |
| segment_id        | None                                                                                                                                                                                       |
| service_types     |                                                                                                                                                                                            |
| subnetpool_id     | None                                                                                                                                                                                       |
| tags              |                                                                                                                                                                                            |
| updated_at        | 2019-08-06T17:37:12Z                                                                                                                                                                       |
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
 
(overcloud) [stack@undercloud-0 os-smoke]$ openstack port create --network sriov --vnic-type direct sriov_vf
+-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                   | Value                                                                                                                                                                                      |
+-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up          | UP                                                                                                                                                                                         |
| allowed_address_pairs   |                                                                                                                                                                                            |
| binding_host_id         |                                                                                                                                                                                            |
| binding_profile         |                                                                                                                                                                                            |
| binding_vif_details     |                                                                                                                                                                                            |
| binding_vif_type        | unbound                                                                                                                                                                                    |
| binding_vnic_type       | direct                                                                                                                                                                                     |
| created_at              | 2019-08-06T17:41:24Z                                                                                                                                                                       |
| data_plane_status       | None                                                                                                                                                                                       |
| description             |                                                                                                                                                                                            |
| device_id               |                                                                                                                                                                                            |
| device_owner            |                                                                                                                                                                                            |
| dns_assignment          | None                                                                                                                                                                                       |
| dns_domain              | None                                                                                                                                                                                       |
| dns_name                | None                                                                                                                                                                                       |
| extra_dhcp_opts         |                                                                                                                                                                                            |
| fixed_ips               | ip_address='192.168.111.41', subnet_id='a368e50e-3bc5-44d1-8c4e-07ea9cb7bf69'                                                                                                              |
| id                      | e40056c2-5ce0-49e0-b277-8fc5749d5130                                                                                                                                                       |
| location                | Munch({'cloud': '', 'region_name': 'regionOne', 'zone': None, 'project': Munch({'id': '53084bdcf4cb4bd09b43f3b4b50a7cc3', 'name': 'admin', 'domain_id': None, 'domain_name': 'Default'})}) |
| mac_address             | fa:16:3e:09:36:a5                                                                                                                                                                          |
| name                    | sriov_vf                                                                                                                                                                                   |
| network_id              | a7a949bb-e680-4fb2-adcd-f76c7eaddec4                                                                                                                                                       |
| port_security_enabled   | False                                                                                                                                                                                      |
| project_id              | 53084bdcf4cb4bd09b43f3b4b50a7cc3                                                                                                                                                           |
| propagate_uplink_status | None                                                                                                                                                                                       |
| qos_policy_id           | None                                                                                                                                                                                       |
| resource_request        | None                                                                                                                                                                                       |
| revision_number         | 2                                                                                                                                                                                          |
| security_group_ids      |                                                                                                                                                                                            |
| status                  | DOWN                                                                                                                                                                                       |
| tags                    |                                                                                                                                                                                            |
| trunk_details           | None                                                                                                                                                                                       |
| updated_at              | 2019-08-06T17:41:24Z                                                                                                                                                                       |
+-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


(overcloud) [stack@undercloud-0 ~]$ openstack server create --flavor m1.medium.huge_pages_cpu_pinning_numa_node-1 --image rhel-guest-image-7-6-210-x86-64-qcow2 --port sriov_vf --config-drive True test_instance
+-------------------------------------+-------------------------------------------------------------------------------------+
| Field                               | Value                                                                               |
+-------------------------------------+-------------------------------------------------------------------------------------+
| OS-DCF:diskConfig                   | MANUAL                                                                              |
| OS-EXT-AZ:availability_zone         |                                                                                     |
| OS-EXT-SRV-ATTR:host                | None                                                                                |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None                                                                                |
| OS-EXT-SRV-ATTR:instance_name       |                                                                                     |
| OS-EXT-STS:power_state              | NOSTATE                                                                             |
| OS-EXT-STS:task_state               | scheduling                                                                          |
| OS-EXT-STS:vm_state                 | building                                                                            |
| OS-SRV-USG:launched_at              | None                                                                                |
| OS-SRV-USG:terminated_at            | None                                                                                |
| accessIPv4                          |                                                                                     |
| accessIPv6                          |                                                                                     |
| addresses                           |                                                                                     |
| adminPass                           | eqBYtTJ582HU                                                                        |
| config_drive                        | True                                                                                |
| created                             | 2019-08-07T06:29:15Z                                                                |
| flavor                              | m1.medium.huge_pages_cpu_pinning_numa_node-1 (26b593c8-9612-4ed4-91fb-c5c4380e3086) |
| hostId                              |                                                                                     |
| id                                  | 76086693-477f-451e-8232-76b2daeb723e                                                |
| image                               | rhel-guest-image-7-6-210-x86-64-qcow2 (9b28f1bd-9fd7-49d8-a8ee-52333d22867c)        |
| key_name                            | None                                                                                |
| name                                | test_instance                                                                       |
| progress                            | 0                                                                                   |
| project_id                          | 53084bdcf4cb4bd09b43f3b4b50a7cc3                                                    |
| properties                          |                                                                                     |
| security_groups                     | name='default'                                                                      |
| status                              | BUILD                                                                               |
| updated                             | 2019-08-07T06:29:17Z                                                                |
| user_id                             | db5afe7915aa45798eef71e4b4faf82e                                                    |
| volumes_attached                    |                                                                                     |
+-------------------------------------+-------------------------------------------------------------------------------------+

(overcloud) [stack@undercloud-0 ~]$ openstack server list
+--------------------------------------+---------------+--------+-------------------------------------------------+---------------------------------------+----------------------------------------------+
| ID                                   | Name          | Status | Networks                                        | Image                                 | Flavor                                       |
+--------------------------------------+---------------+--------+-------------------------------------------------+---------------------------------------+----------------------------------------------+
| 76086693-477f-451e-8232-76b2daeb723e | test_instance | ACTIVE | sriov=192.168.111.41                            | rhel-guest-image-7-6-210-x86-64-qcow2 | m1.medium.huge_pages_cpu_pinning_numa_node-1 |
+--------------------------------------+---------------+--------+-------------------------------------------------+---------------------------------------+----------------------------------------------+

(overcloud) [stack@undercloud-0 ~]$ nova diagnostics test_instance
+----------------+----------------------------------------------------------------------------------+
| Property       | Value                                                                            |
+----------------+----------------------------------------------------------------------------------+
| config_drive   | True                                                                             |
| cpu_details    | [{"id": 0, "time": 15500000000, "utilisation": null}, {"id": 1, "time":          |
|                | 9700000000, "utilisation": null}, {"id": 2, "time": 8510000000, "utilisation":   |
|                | null}, {"id": 3, "time": 10500000000, "utilisation": null}, {"id": 4, "time":    |
|                | 10260000000, "utilisation": null}, {"id": 5, "time": 9600000000, "utilisation":  |
|                | null}]                                                                           |
| disk_details   | [{"read_bytes": 227602432, "read_requests": 10932, "write_bytes": 39107072,      |
|                | "write_requests": 1123, "errors_count": -1}, {"read_bytes": 174364,              |
|                | "read_requests": 59, "write_bytes": 0, "write_requests": 0, "errors_count": -1}] |
| driver         | libvirt                                                                          |
| hypervisor     | kvm                                                                              |
| hypervisor_os  | linux                                                                            |
| memory_details | {"maximum": 8, "used": 8}                                                        |
| nic_details    | [{"mac_address": "fa:16:3e:09:36:a5", "rx_octets": null, "rx_errors": null,      |
|                | "rx_drop": null, "rx_packets": null, "rx_rate": null, "tx_octets": null,         |
|                | "tx_errors": null, "tx_drop": null, "tx_packets": null, "tx_rate": null}]        |
| num_cpus       | 6                                                                                |
| num_disks      | 2                                                                                |
| num_nics       | 1                                                                                |
| state          | running                                                                          |
| uptime         | 5479                                                                             |
+----------------+----------------------------------------------------------------------------------+

Comment 36 errata-xmlrpc 2019-09-21 11:19:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2811


Note You need to log in before you can comment on or make changes to this bug.