1838786 – Impossible to attach any volume to an instance

Bug 1838786 - Impossible to attach any volume to an instance

Summary: Impossible to attach any volume to an instance

Keywords:
Status:	CLOSED INSUFFICIENT_DATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-nova
Sub Component:
Version:	13.0 (Queens)
Hardware:	x86_64
OS:	Linux
Priority:	medium
Severity:	medium
Target Milestone:	---
Target Release:	---
Assignee:	Lee Yarwood
QA Contact:	OSP DFG:Compute
Docs Contact:
URL:
Whiteboard:
Depends On:	1849488
Blocks:
TreeView+	depends on / blocked

Reported:	2020-05-21 19:17 UTC by Madhur Gupta
Modified:	2023-12-15 17:59 UTC (History)
CC List:	11 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2020-07-16 12:23:27 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	OSP-11918	0	None	None	None	2021-12-22 06:58:06 UTC

Internal Links: 1981571

Description Madhur Gupta 2020-05-21 19:17:38 UTC

Description of problem:
While attaching any volume to an instance, operation fails with error:
libvirtError: Requested operation is not valid: target vdf already exists


Version-Release number of selected component (if applicable):
libgudev1-219-62.el7_6.9.x86_64                             Mon Oct 21 15:40:22 2019
iscsi-initiator-utils-6.2.0.874-11.el7.x86_64               Tue Dec 10 12:41:50 2019

docker exec -it nova_libvirt rpm -qa|egrep -i "qemu|libvirt"
libvirt-bash-completion-4.5.0-23.el7_7.1.x86_64
libvirt-python-4.5.0-1.el7.x86_64

python2-os-brick-2.3.8-1.el7ost.noarch                      Mon Dec  9 11:43:07 2019
openstack-nova-compute-17.0.12-1.el7ost.noarch              Mon Dec  9 11:43:39 2019
openstack-cinder-12.0.8-3.el7ost.noarch                     Mon Dec  9 11:43:45 2019


Steps to reproduce the issue on the specific instance:

$ openstack server add volume dc2-dev-node-3-19 panunvolume1
$ openstack server show dc2-dev-node-3-19
+-----------------------------+-------------------------------------------------------------------+
| Field                       | Value                                                             |
+-----------------------------+-------------------------------------------------------------------+
| OS-DCF:diskConfig           | MANUAL                                                            |
| OS-EXT-AZ:availability_zone | zone-3                                                            |
| OS-EXT-STS:power_state      | Running                                                           |
| OS-EXT-STS:task_state       | None                                                              |
| OS-EXT-STS:vm_state         | active                                                            |
| OS-SRV-USG:launched_at      | 2019-10-21T10:41:04.000000                                        |
| OS-SRV-USG:terminated_at    | None                                                              |
| accessIPv4                  |                                                                   |
| accessIPv6                  |                                                                   |
| addresses                   | dc2-dev-net=192.168.0.69                                          |
| config_drive                |                                                                   |
| created                     | 2019-10-21T10:40:44Z                                              |
| flavor                      | openshift.default (455720a5-de0d-4a96-873d-5c8586d22c3c)          |
| hostId                      | 6a213b8f4a5ef45d04ab2a798f2cfc829f108b0a3a05c6a68913c367          |
| id                          | b491b576-ed38-42e7-a26e-569ec7a3eb2c                              |
| image                       | rhel-server-7.6-x86_64-kvm (57fbc216-f01b-442f-afb7-9ec6e6f7fb51) |
| key_name                    | dc2-dev                                                           |
| name                        | dc2-dev-node-3-19                                                 |
| progress                    | 0                                                                 |
| project_id                  | d12e0698785f498386146459fd6db64d                                  |
| properties                  |                                                                   |
| security_groups             | name='dc2-dev-security-group-node'                                |
| status                      | ACTIVE                                                            |
| updated                     | 2019-12-12T08:01:30Z                                              |
| user_id                     | 7533626a8a020e9d2102e2fd243fd3bab885ff45747c5e5639c2bebdca74d588  |
| volumes_attached            | id='03a42d6a-e1f9-491d-980a-1c8bd45a333e'                         |
|                             | id='4546da68-7b54-4ecb-81f0-3de147ecc9da'                         |
|                             | id='ee9d8306-579f-42af-8189-6d66430ad8c6'                         |
+-----------------------------+-------------------------------------------------------------------+
$ openstack volume show panunvolume1
+------------------------------+------------------------------------------------------------------+
| Field                        | Value                                                            |
+------------------------------+------------------------------------------------------------------+
| attachments                  | []                                                               |
| availability_zone            | nova                                                             |
| bootable                     | false                                                            |
| consistencygroup_id          | None                                                             |
| created_at                   | 2020-05-04T13:43:58.000000                                       |
| description                  | None                                                             |
| encrypted                    | False                                                            |
| id                           | c8135c3c-5181-4346-92db-6e76f5024bb2                             |
| multiattach                  | False                                                            |
| name                         | panunvolume1                                                     |
| os-vol-tenant-attr:tenant_id | d12e0698785f498386146459fd6db64d                                 |
| properties                   |                                                                  |
| replication_status           | None                                                             |
| size                         | 5                                                                |
| snapshot_id                  | None                                                             |
| source_volid                 | None                                                             |
| status                       | available                                                        |
| type                         | solidfire                                                        |
| updated_at                   | 2020-05-06T04:51:56.000000                                       |
| user_id                      | 7533626a8a020e9d2102e2fd243fd3bab885ff45747c5e5639c2bebdca74d588 |
+------------------------------+------------------------------------------------------------------+

$ openstack server add volume dc2-dev-node-3-19 panunvolume1 --device /dev/vdf
$ openstack volume show panunvolume1
+------------------------------+------------------------------------------------------------------+
| Field                        | Value                                                            |
+------------------------------+------------------------------------------------------------------+
| attachments                  | []                                                               |
| availability_zone            | nova                                                             |
| bootable                     | false                                                            |
| consistencygroup_id          | None                                                             |
| created_at                   | 2020-05-04T13:43:58.000000                                       |
| description                  | None                                                             |
| encrypted                    | False                                                            |
| id                           | c8135c3c-5181-4346-92db-6e76f5024bb2                             |
| multiattach                  | False                                                            |
| name                         | panunvolume1                                                     |
| os-vol-tenant-attr:tenant_id | d12e0698785f498386146459fd6db64d                                 |
| properties                   |                                                                  |
| replication_status           | None                                                             |
| size                         | 5                                                                |
| snapshot_id                  | None                                                             |
| source_volid                 | None                                                             |
| status                       | available                                                        |
| type                         | solidfire                                                        |
| updated_at                   | 2020-05-06T04:53:08.000000                                       |
| user_id                      | 7533626a8a020e9d2102e2fd243fd3bab885ff45747c5e5639c2bebdca74d588 |
+------------------------------+------------------------------------------------------------------+
$ openstack server add volume dc2-dev-node-3-19 panunvolume1 --device /dev/vdg
$ openstack volume show panunvolume1
+------------------------------+------------------------------------------------------------------+
| Field                        | Value                                                            |
+------------------------------+------------------------------------------------------------------+
| attachments                  | []                                                               |
| availability_zone            | nova                                                             |
| bootable                     | false                                                            |
| consistencygroup_id          | None                                                             |
| created_at                   | 2020-05-04T13:43:58.000000                                       |
| description                  | None                                                             |
| encrypted                    | False                                                            |
| id                           | c8135c3c-5181-4346-92db-6e76f5024bb2                             |
| multiattach                  | False                                                            |
| name                         | panunvolume1                                                     |
| os-vol-tenant-attr:tenant_id | d12e0698785f498386146459fd6db64d                                 |
| properties                   |                                                                  |
| replication_status           | None                                                             |
| size                         | 5                                                                |
| snapshot_id                  | None                                                             |
| source_volid                 | None                                                             |
| status                       | available                                                        |
| type                         | solidfire                                                        |
| updated_at                   | 2020-05-06T04:53:41.000000                                       |
| user_id                      | 7533626a8a020e9d2102e2fd243fd3bab885ff45747c5e5639c2bebdca74d588 |
+------------------------------+------------------------------------------------------------------+

$ openstack server add volume dc2-dev-node-3-19 panunvolume1 --device /dev/vdq
$ openstack volume show panunvolume1
+------------------------------+------------------------------------------------------------------+
| Field                        | Value                                                            |
+------------------------------+------------------------------------------------------------------+
| attachments                  | []                                                               |
| availability_zone            | nova                                                             |
| bootable                     | false                                                            |
| consistencygroup_id          | None                                                             |
| created_at                   | 2020-05-04T13:43:58.000000                                       |
| description                  | None                                                             |
| encrypted                    | False                                                            |
| id                           | c8135c3c-5181-4346-92db-6e76f5024bb2                             |
| multiattach                  | False                                                            |
| name                         | panunvolume1                                                     |
| os-vol-tenant-attr:tenant_id | d12e0698785f498386146459fd6db64d                                 |
| properties                   |                                                                  |
| replication_status           | None                                                             |
| size                         | 5                                                                |
| snapshot_id                  | None                                                             |
| source_volid                 | None                                                             |
| status                       | available                                                        |
| type                         | solidfire                                                        |
| updated_at                   | 2020-05-06T04:57:12.000000                                       |
| user_id                      | 7533626a8a020e9d2102e2fd243fd3bab885ff45747c5e5639c2bebdca74d588 |
+------------------------------+------------------------------------------------------------------+
$ ssh 192.168.0.69 sudo mount | grep cinder/mounts
Warning: Permanently added '192.168.0.69' (ECDSA) to the list of known hosts.
/dev/vdc on /data/origin/openshift.local.volumes/plugins/kubernetes.io/cinder/mounts/4546da68-7b54-4ecb-81f0-3de147ecc9da type ext4 (rw,relatime,seclabel,data=ordered)
/dev/vdd on /data/origin/openshift.local.volumes/plugins/kubernetes.io/cinder/mounts/ee9d8306-579f-42af-8189-6d66430ad8c6 type ext4 (rw,relatime,seclabel,data=ordered)
/dev/vde on /data/origin/openshift.local.volumes/plugins/kubernetes.io/cinder/mounts/03a42d6a-e1f9-491d-980a-1c8bd45a333e type ext4 (rw,relatime,seclabel,data=ordered)
$ ssh 192.168.0.69 lsblk | grep ^vd.
Warning: Permanently added '192.168.0.69' (ECDSA) to the list of known hosts.
vda                                                                                            252:0    0   10G  0 disk 
vdb                                                                                            252:16   0   90G  0 disk 
vdc                                                                                            252:32   0    4G  0 disk /data/origin/openshift.local.volumes/pods/c7859076-83b6-11ea-a859-fa163e7e3641/volumes/kubernetes.io~cinder/pvc-3355e378-35e7-11ea-9446-fa163e7e3641
vdd                                                                                            252:48   0   20G  0 disk /data/origin/openshift.local.volumes/pods/ceb78046-83b6-11ea-a859-fa163e7e3641/volumes/kubernetes.io~cinder/pvc-a6d31f03-0b9f-11ea-a97c-fa163ed73ca3
vde                                                                                            252:64   0    5G  0 disk /data/origin/openshift.local.volumes/pods/e385babd-83bc-11ea-a859-fa163e7e3641/volumes/kubernetes.io~cinder/pvc-21e75fe6-7319-11ea-8cef-fa163e6ae5da





Note: this issue random and is with specific instance only, not all instances.

Comment 1 Madhur Gupta 2020-05-21 19:24:11 UTC

2020-05-06 07:51:51.926 9 INFO nova.compute.manager [req-1fefc1ea-79cd-4466-9e48-f4da6e0dd462 7533626a8a020e9d2102e2fd243fd3bab885ff45747c5e5639c2bebdca74d588 d12e0698785f498386146459fd6db64d - e62707285b7045589371c85cfeb735cb e62707285b7045589371c85cfeb735cb] [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c] Attaching volume c8135c3c-5181-4346-92db-6e76f5024bb2 to /dev/vdf
2020-05-06 07:51:52.963 9 INFO os_brick.initiator.connectors.iscsi [req-1fefc1ea-79cd-4466-9e48-f4da6e0dd462 7533626a8a020e9d2102e2fd243fd3bab885ff45747c5e5639c2bebdca74d588 d12e0698785f498386146459fd6db64d - e62707285b7045589371c85cfeb735cb e62707285b7045589371c85cfeb735cb] Trying to connect to iSCSI portal 10.222.37.193:3260
2020-05-06 07:51:55.227 9 ERROR nova.virt.libvirt.driver [req-1fefc1ea-79cd-4466-9e48-f4da6e0dd462 7533626a8a020e9d2102e2fd243fd3bab885ff45747c5e5639c2bebdca74d588 d12e0698785f498386146459fd6db64d - e62707285b7045589371c85cfeb735cb e62707285b7045589371c85cfeb735cb] [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c] Failed to attach volume at mountpoint: /dev/vdf: libvirtError: Requested operation is not valid: target vdf already exists
2020-05-06 07:51:55.227 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c] Traceback (most recent call last):
2020-05-06 07:51:55.227 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1493, in attach_volume
2020-05-06 07:51:55.227 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     guest.attach_device(conf, persistent=True, live=live)
2020-05-06 07:51:55.227 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 306, in attach_device
2020-05-06 07:51:55.227 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     self._domain.attachDeviceFlags(device_xml, flags=flags)
2020-05-06 07:51:55.227 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2020-05-06 07:51:55.227 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2020-05-06 07:51:55.227 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2020-05-06 07:51:55.227 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     rv = execute(f, *args, **kwargs)
2020-05-06 07:51:55.227 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2020-05-06 07:51:55.227 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     six.reraise(c, e, tb)
2020-05-06 07:51:55.227 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2020-05-06 07:51:55.227 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     rv = meth(*args, **kwargs)
2020-05-06 07:51:55.227 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 605, in attachDeviceFlags
2020-05-06 07:51:55.227 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2020-05-06 07:51:55.227 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c] libvirtError: Requested operation is not valid: target vdf already exists
2020-05-06 07:51:55.227 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]



2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [req-1fefc1ea-79cd-4466-9e48-f4da6e0dd462 7533626a8a020e9d2102e2fd243fd3bab885ff45747c5e5639c2bebdca74d588 d12e0698785f498386146459fd6db64d - e62707285b7045589371c85cfeb735cb e62707285b7045589371c85cfeb735cb] [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c] Driver failed to attach volume c8135c3c-5181-4346-92db-6e76f5024bb2 at /dev/vdf: libvirtError: Requested operation is not valid: target vdf already exists
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c] Traceback (most recent call last):
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 540, in _volume_attach
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     device_type=self['device_type'], encryption=encryption)
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1520, in attach_volume
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     encryption=encryption)
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     self.force_reraise()
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     six.reraise(self.type_, self.value, self.tb)
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1493, in attach_volume
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     guest.attach_device(conf, persistent=True, live=live)
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 306, in attach_device
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     self._domain.attachDeviceFlags(device_xml, flags=flags)
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     rv = execute(f, *args, **kwargs)
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     six.reraise(c, e, tb)
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     rv = meth(*args, **kwargs)
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 605, in attachDeviceFlags
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c] libvirtError: Requested operation is not valid: target vdf already exists
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]




2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [req-1fefc1ea-79cd-4466-9e48-f4da6e0dd462 7533626a8a020e9d2102e2fd243fd3bab885ff45747c5e5639c2bebdca74d588 d12e0698785f498386146459fd6db64d - e62707285b7045589371c85cfeb735cb e62707285b7045589371c85cfeb735cb] [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c] Failed to attach c8135c3c-5181-4346-92db-6e76f5024bb2 at /dev/vdf: libvirtError: Requested operation is not valid: target vdf already exists
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c] Traceback (most recent call last):
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5433, in _attach_volume
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     do_driver_attach=True)
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 46, in wrapped
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     ret_val = method(obj, context, *args, **kwargs)
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 624, in attach
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     virt_driver, do_driver_attach)
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 602, in _do_attach
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     do_driver_attach)
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 549, in _volume_attach
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     attachment_id)
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     self.force_reraise()
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     six.reraise(self.type_, self.value, self.tb)
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 540, in _volume_attach
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     device_type=self['device_type'], encryption=encryption)
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1520, in attach_volume
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     encryption=encryption)
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     self.force_reraise()
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     six.reraise(self.type_, self.value, self.tb)
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1493, in attach_volume
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     guest.attach_device(conf, persistent=True, live=live)
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 306, in attach_device
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     self._domain.attachDeviceFlags(device_xml, flags=flags)
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     rv = execute(f, *args, **kwargs)
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     six.reraise(c, e, tb)
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     rv = meth(*args, **kwargs)
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 605, in attachDeviceFlags
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c] libvirtError: Requested operation is not valid: target vdf already exists



-> When using --device option to attach it on node /dev/vdg:

2020-05-06 07:53:35.637 9 WARNING nova.virt.libvirt.driver [req-6825c6cc-3b0c-4e67-86e9-f68356034272 7533626a8a020e9d2102e2fd243fd3bab885ff45747c5e5639c2bebdca74d588 d12e0698785f498386146459fd6db64d - e62707285b7045589371c85cfeb735cb e62707285b7045589371c85cfeb735cb] [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c] Ignoring supplied device name: /dev/vdg
2020-05-06 07:53:36.474 9 INFO nova.compute.manager [req-6825c6cc-3b0c-4e67-86e9-f68356034272 7533626a8a020e9d2102e2fd243fd3bab885ff45747c5e5639c2bebdca74d588 d12e0698785f498386146459fd6db64d - e62707285b7045589371c85cfeb735cb e62707285b7045589371c85cfeb735cb] [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c] Attaching volume c8135c3c-5181-4346-92db-6e76f5024bb2 to /dev/vdf
2020-05-06 07:53:37.833 9 INFO os_brick.initiator.connectors.iscsi [req-6825c6cc-3b0c-4e67-86e9-f68356034272 7533626a8a020e9d2102e2fd243fd3bab885ff45747c5e5639c2bebdca74d588 d12e0698785f498386146459fd6db64d - e62707285b7045589371c85cfeb735cb e62707285b7045589371c85cfeb735cb] Trying to connect to iSCSI portal 10.222.37.193:3260
2020-05-06 07:53:40.126 9 ERROR nova.virt.libvirt.driver [req-6825c6cc-3b0c-4e67-86e9-f68356034272 7533626a8a020e9d2102e2fd243fd3bab885ff45747c5e5639c2bebdca74d588 d12e0698785f498386146459fd6db64d - e62707285b7045589371c85cfeb735cb e62707285b7045589371c85cfeb735cb] [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c] Failed to attach volume at mountpoint: /dev/vdf: libvirtError: Requested operation is not valid: target vdf already exists
2020-05-06 07:53:40.126 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c] Traceback (most recent call last):
2020-05-06 07:53:40.126 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1493, in attach_volume
2020-05-06 07:53:40.126 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     guest.attach_device(conf, persistent=True, live=live)
2020-05-06 07:53:40.126 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 306, in attach_device
2020-05-06 07:53:40.126 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     self._domain.attachDeviceFlags(device_xml, flags=flags)
2020-05-06 07:53:40.126 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2020-05-06 07:53:40.126 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2020-05-06 07:53:40.126 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2020-05-06 07:53:40.126 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     rv = execute(f, *args, **kwargs)
2020-05-06 07:53:40.126 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2020-05-06 07:53:40.126 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     six.reraise(c, e, tb)
2020-05-06 07:53:40.126 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2020-05-06 07:53:40.126 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     rv = meth(*args, **kwargs)
2020-05-06 07:53:40.126 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 605, in attachDeviceFlags
2020-05-06 07:53:40.126 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2020-05-06 07:53:40.126 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c] libvirtError: Requested operation is not valid: target vdf already exists
2020-05-06 07:53:40.126 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]

Comment 10 smooney 2020-06-26 02:25:11 UTC

upstream we have a similar issue with detach too.
https://bugs.launchpad.net/cinder/+bug/1882521
im make the description and first comment public as there is no sensitive info in them but it will be needed for other to understand this bug.

Note You need to log in before you can comment on or make changes to this bug.