Bug 1527345 - Detaching a ceph volume results error: nodeLibvirtConfigGuestDeviceAddressPCI missing format_dom method
Summary: Detaching a ceph volume results error: nodeLibvirtConfigGuestDeviceAddressPCI...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova
Version: 10.0 (Newton)
Hardware: x86_64
OS: Linux
high
high
Target Milestone: z7
: 10.0 (Newton)
Assignee: Sahid Ferdjaoui
QA Contact: Joe H. Rahme
URL:
Whiteboard:
Depends On:
Blocks: 1537027 1537028
TreeView+ depends on / blocked
 
Reported: 2017-12-19 09:37 UTC by Pablo Iranzo Gómez
Modified: 2022-07-09 10:00 UTC (History)
14 users (show)

Fixed In Version: openstack-nova-14.1.0-2.el7ost
Doc Type: Bug Fix
Doc Text:
Previously, when detaching a Ceph volume connected to guest as a PCI device, the volume was not detached and libvirt reported an invalid PCI address. This arose because the PCI address of the device was not properly formatted due to a missing method in the guest XML formatter. To address this issue, this update adds a proper `format_dom` method for `LibvirtConfigGuestDeviceAddressPCI`. As a result, it is now possible to detach a Ceph volume.
Clone Of:
: 1537027 (view as bug list)
Environment:
Last Closed: 2018-02-27 16:24:48 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1709319 0 None None None 2017-12-19 09:37:49 UTC
Red Hat Issue Tracker OSP-4799 0 None None None 2022-07-09 10:00:15 UTC
Red Hat Product Errata RHSA-2018:0369 0 normal SHIPPED_LIVE Moderate: openstack-nova and python-novaclient security, bug fix, and enhancement update 2018-02-27 21:24:56 UTC

Description Pablo Iranzo Gómez 2017-12-19 09:37:49 UTC 
Description of problem:

2017-12-14 09:30:15.372 1 TRACE nova.compute.manager [instance: 72a5bc2a-40ac-4d0b-bf72-359883e76069]  >>>>>Traceback (most recent call last):
2017-12-14 09:30:15.372 1 TRACE nova.compute.manager [instance: 72a5bc2a-40ac-4d0b-bf72-359883e76069]  >>>>>  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4743, in _driver_detach_volume
2017-12-14 09:30:15.372 1 TRACE nova.compute.manager [instance: 72a5bc2a-40ac-4d0b-bf72-359883e76069]  >>>>>    encryption=encryption)
2017-12-14 09:30:15.372 1 TRACE nova.compute.manager [instance: 72a5bc2a-40ac-4d0b-bf72-359883e76069]  >>>>>  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1321, in detach_volume
2017-12-14 09:30:15.372 1 TRACE nova.compute.manager [instance: 72a5bc2a-40ac-4d0b-bf72-359883e76069]  >>>>>    live=live)
2017-12-14 09:30:15.372 1 TRACE nova.compute.manager [instance: 72a5bc2a-40ac-4d0b-bf72-359883e76069]  >>>>>  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 421, in detach_device_with_retry
2017-12-14 09:30:15.372 1 TRACE nova.compute.manager [instance: 72a5bc2a-40ac-4d0b-bf72-359883e76069]  >>>>>    _try_detach_device(conf, persistent, live)
2017-12-14 09:30:15.372 1 TRACE nova.compute.manager [instance: 72a5bc2a-40ac-4d0b-bf72-359883e76069]  >>>>>  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 410, in _try_detach_device
2017-12-14 09:30:15.372 1 TRACE nova.compute.manager [instance: 72a5bc2a-40ac-4d0b-bf72-359883e76069]  >>>>>    device=alternative_device_name)
2017-12-14 09:30:15.372 1 TRACE nova.compute.manager [instance: 72a5bc2a-40ac-4d0b-bf72-359883e76069]  >>>>>  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-12-14 09:30:15.372 1 TRACE nova.compute.manager [instance: 72a5bc2a-40ac-4d0b-bf72-359883e76069]  >>>>>    self.force_reraise()
2017-12-14 09:30:15.372 1 TRACE nova.compute.manager [instance: 72a5bc2a-40ac-4d0b-bf72-359883e76069]  >>>>>  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-12-14 09:30:15.372 1 TRACE nova.compute.manager [instance: 72a5bc2a-40ac-4d0b-bf72-359883e76069]  >>>>>    six.reraise(self.type_, self.value, self.tb)
2017-12-14 09:30:15.372 1 TRACE nova.compute.manager [instance: 72a5bc2a-40ac-4d0b-bf72-359883e76069]  >>>>>  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 390, in _try_detach_device
2017-12-14 09:30:15.372 1 TRACE nova.compute.manager [instance: 72a5bc2a-40ac-4d0b-bf72-359883e76069]  >>>>>    self.detach_device(conf, persistent=persistent, live=live)
2017-12-14 09:30:15.372 1 TRACE nova.compute.manager [instance: 72a5bc2a-40ac-4d0b-bf72-359883e76069]  >>>>>  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 467, in detach_device
2017-12-14 09:30:15.372 1 TRACE nova.compute.manager [instance: 72a5bc2a-40ac-4d0b-bf72-359883e76069]  >>>>>    self._domain.detachDeviceFlags(device_xml, flags=flags)
2017-12-14 09:30:15.372 1 TRACE nova.compute.manager [instance: 72a5bc2a-40ac-4d0b-bf72-359883e76069]  >>>>>  File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2017-12-14 09:30:15.372 1 TRACE nova.compute.manager [instance: 72a5bc2a-40ac-4d0b-bf72-359883e76069]  >>>>>    result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-12-14 09:30:15.372 1 TRACE nova.compute.manager [instance: 72a5bc2a-40ac-4d0b-bf72-359883e76069]  >>>>>  File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2017-12-14 09:30:15.372 1 TRACE nova.compute.manager [instance: 72a5bc2a-40ac-4d0b-bf72-359883e76069]  >>>>>    rv = execute(f, *args, **kwargs)
2017-12-14 09:30:15.372 1 TRACE nova.compute.manager [instance: 72a5bc2a-40ac-4d0b-bf72-359883e76069]  >>>>>  File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2017-12-14 09:30:15.372 1 TRACE nova.compute.manager [instance: 72a5bc2a-40ac-4d0b-bf72-359883e76069]  >>>>>    six.reraise(c, e, tb)
2017-12-14 09:30:15.372 1 TRACE nova.compute.manager [instance: 72a5bc2a-40ac-4d0b-bf72-359883e76069]  >>>>>  File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2017-12-14 09:30:15.372 1 TRACE nova.compute.manager [instance: 72a5bc2a-40ac-4d0b-bf72-359883e76069]  >>>>>    rv = meth(*args, **kwargs)
2017-12-14 09:30:15.372 1 TRACE nova.compute.manager [instance: 72a5bc2a-40ac-4d0b-bf72-359883e76069]  >>>>>  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1194, in detachDeviceFlags
2017-12-14 09:30:15.372 1 TRACE nova.compute.manager [instance: 72a5bc2a-40ac-4d0b-bf72-359883e76069]  >>>>>    if ret == -1: raise libvirtError ('virDomainDetachDeviceFlags() failed', dom=self)
2017-12-14 09:30:15.372 1 TRACE nova.compute.manager [instance: 72a5bc2a-40ac-4d0b-bf72-359883e76069]  >>>>>libvirtError: XML error: Invalid PCI address 0000:00:00, at least one of domain, bus, or slot must be > 0
2017-12-14 09:30:15.372 1 TRACE nova.compute.manager [instance: 72a5bc2a-40ac-4d0b-bf72-359883e76069]  >>>>>


Version-Release number of selected component (if applicable):

openstack-nova-14.0.8-5.el7ost.noarch



Seems to be fixed in https://bugs.launchpad.net/nova/+bug/1709319
Comment 2 Sahid Ferdjaoui 2017-12-19 09:45:17 UTC 
Can you provide reproducer steps and a bit more context on when this error is happening?
Comment 4 Pablo Iranzo Gómez 2017-12-19 10:42:34 UTC 
Problem appears when detaching cinder volumes
Comment 5 Sahid Ferdjaoui 2017-12-20 14:39:04 UTC 
Based on error message and context it seems you right, [0] is going to fix the issue.

  https://review.openstack.org/#/c/491822/
Comment 14 Joe H. Rahme 2018-02-15 17:07:39 UTC 
Verified that Ceph volumes can be detached.

[stack@undercloud-0 testingdir]$ nova boot --flavor m1.micro --image cirros-0.3.4-x86_64-disk.img vm1
+--------------------------------------+---------------------------------------------------------------------+
| Property                             | Value                                                               |
+--------------------------------------+---------------------------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                                              |
| OS-EXT-AZ:availability_zone          |                                                                     |
| OS-EXT-SRV-ATTR:host                 | -                                                                   |
| OS-EXT-SRV-ATTR:hostname             | vm1                                                                 |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | -                                                                   |
| OS-EXT-SRV-ATTR:instance_name        |                                                                     |
| OS-EXT-SRV-ATTR:kernel_id            |                                                                     |
| OS-EXT-SRV-ATTR:launch_index         | 0                                                                   |
| OS-EXT-SRV-ATTR:ramdisk_id           |                                                                     |
| OS-EXT-SRV-ATTR:reservation_id       | r-9s233uhz                                                          |
| OS-EXT-SRV-ATTR:root_device_name     | -                                                                   |
| OS-EXT-SRV-ATTR:user_data            | -                                                                   |
| OS-EXT-STS:power_state               | 0                                                                   |
| OS-EXT-STS:task_state                | scheduling                                                          |
| OS-EXT-STS:vm_state                  | building                                                            |
| OS-SRV-USG:launched_at               | -                                                                   |
| OS-SRV-USG:terminated_at             | -                                                                   |
| accessIPv4                           |                                                                     |
| accessIPv6                           |                                                                     |
| adminPass                            | rzDaq3TPAEat                                                        |
| config_drive                         |                                                                     |
| created                              | 2018-02-15T17:00:00Z                                                |
| description                          | -                                                                   |
| flavor                               | m1.micro (72b4a6a5-c256-48c0-a25e-b2dd6dbcfd21)                     |
| hostId                               |                                                                     |
| host_status                          |                                                                     |
| id                                   | f61eac88-d42e-4c22-8377-59e035d7f8c0                                |
| image                                | cirros-0.3.4-x86_64-disk.img (7eef97bb-e6b0-4435-ad27-795e3e6560e3) |
| key_name                             | -                                                                   |
| locked                               | False                                                               |
| metadata                             | {}                                                                  |
| name                                 | vm1                                                                 |
| os-extended-volumes:volumes_attached | []                                                                  |
| progress                             | 0                                                                   |
| security_groups                      | default                                                             |
| status                               | BUILD                                                               |
| tags                                 | []                                                                  |
| tenant_id                            | eb02bd117ea147aebf826bf3cae381d7                                    |
| updated                              | 2018-02-15T17:00:00Z                                                |
| user_id                              | 211460986bd0491fa8e3535699db448b                                    |
+--------------------------------------+---------------------------------------------------------------------+

[stack@undercloud-0 testingdir]$ cinder create --name vol1 1
+--------------------------------+--------------------------------------+
| Property                       | Value                                |
+--------------------------------+--------------------------------------+
| attachments                    | []                                   |
| availability_zone              | nova                                 |
| bootable                       | false                                |
| consistencygroup_id            | None                                 |
| created_at                     | 2018-02-15T17:00:14.000000           |
| description                    | None                                 |
| encrypted                      | False                                |
| id                             | 0d9f5abf-9674-4e36-a02f-3a2e9a66c77c |
| metadata                       | {}                                   |
| migration_status               | None                                 |
| multiattach                    | False                                |
| name                           | vol1                                 |
| os-vol-host-attr:host          | None                                 |
| os-vol-mig-status-attr:migstat | None                                 |
| os-vol-mig-status-attr:name_id | None                                 |
| os-vol-tenant-attr:tenant_id   | eb02bd117ea147aebf826bf3cae381d7     |
| replication_status             | disabled                             |
| size                           | 1                                    |
| snapshot_id                    | None                                 |
| source_volid                   | None                                 |
| status                         | creating                             |
| updated_at                     | None                                 |
| user_id                        | 211460986bd0491fa8e3535699db448b     |
| volume_type                    | None                                 |
+--------------------------------+--------------------------------------+

[stack@undercloud-0 testingdir]$ nova list
+--------------------------------------+------+--------+------------+-------------+-------------------+
| ID                                   | Name | Status | Task State | Power State | Networks          |
+--------------------------------------+------+--------+------------+-------------+-------------------+
| f61eac88-d42e-4c22-8377-59e035d7f8c0 | vm1  | ACTIVE | -          | Running     | public=10.0.0.210 |
+--------------------------------------+------+--------+------------+-------------+-------------------+

[stack@undercloud-0 testingdir]$ nova volume-attach vm1 0d9f5abf-9674-4e36-a02f-3a2e9a66c77c
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/vdb                             |
| id       | 0d9f5abf-9674-4e36-a02f-3a2e9a66c77c |
| serverId | f61eac88-d42e-4c22-8377-59e035d7f8c0 |
| volumeId | 0d9f5abf-9674-4e36-a02f-3a2e9a66c77c |
+----------+--------------------------------------+


[stack@undercloud-0 testingdir]$ nova volume-detach vm1 0d9f5abf-9674-4e36-a02f-3a2e9a66c77c
[stack@undercloud-0 testingdir]$ nova list
+--------------------------------------+------+--------+------------+-------------+-------------------+
| ID                                   | Name | Status | Task State | Power State | Networks          |
+--------------------------------------+------+--------+------------+-------------+-------------------+
| f61eac88-d42e-4c22-8377-59e035d7f8c0 | vm1  | ACTIVE | -          | Running     | public=10.0.0.210 |
+--------------------------------------+------+--------+------------+-------------+-------------------+
[stack@undercloud-0 testingdir]$ cinder list
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| ID                                   | Status    | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| 0d9f5abf-9674-4e36-a02f-3a2e9a66c77c | available | vol1 | 1    | -           | false    |             |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
Comment 17 errata-xmlrpc 2018-02-27 16:24:48 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:0369
Note You need to log in before you can comment on or make changes to this bug.