Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1838786

Summary:	Impossible to attach any volume to an instance
Product:	Red Hat OpenStack	Reporter:	Madhur Gupta <madgupta>
Component:	openstack-nova	Assignee:	Lee Yarwood <lyarwood>
Status:	CLOSED INSUFFICIENT_DATA	QA Contact:	OSP DFG:Compute <osp-dfg-compute>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	13.0 (Queens)	CC:	dasmith, eglynn, jhakimra, kchamart, lyarwood, pkopec, sbauza, sgordon, smooney, stephenfin, vromanso
Target Milestone:	---	Keywords:	ZStream
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-07-16 12:23:27 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1849488
Bug Blocks:

Description Madhur Gupta 2020-05-21 19:17:38 UTC

Description of problem:
While attaching any volume to an instance, operation fails with error:
libvirtError: Requested operation is not valid: target vdf already exists


Version-Release number of selected component (if applicable):
libgudev1-219-62.el7_6.9.x86_64                             Mon Oct 21 15:40:22 2019
iscsi-initiator-utils-6.2.0.874-11.el7.x86_64               Tue Dec 10 12:41:50 2019

docker exec -it nova_libvirt rpm -qa|egrep -i "qemu|libvirt"
libvirt-bash-completion-4.5.0-23.el7_7.1.x86_64
libvirt-python-4.5.0-1.el7.x86_64

python2-os-brick-2.3.8-1.el7ost.noarch                      Mon Dec  9 11:43:07 2019
openstack-nova-compute-17.0.12-1.el7ost.noarch              Mon Dec  9 11:43:39 2019
openstack-cinder-12.0.8-3.el7ost.noarch                     Mon Dec  9 11:43:45 2019


Steps to reproduce the issue on the specific instance:

$ openstack server add volume dc2-dev-node-3-19 panunvolume1
$ openstack server show dc2-dev-node-3-19
+-----------------------------+-------------------------------------------------------------------+
| Field                       | Value                                                             |
+-----------------------------+-------------------------------------------------------------------+
| OS-DCF:diskConfig           | MANUAL                                                            |
| OS-EXT-AZ:availability_zone | zone-3                                                            |
| OS-EXT-STS:power_state      | Running                                                           |
| OS-EXT-STS:task_state       | None                                                              |
| OS-EXT-STS:vm_state         | active                                                            |
| OS-SRV-USG:launched_at      | 2019-10-21T10:41:04.000000                                        |
| OS-SRV-USG:terminated_at    | None                                                              |
| accessIPv4                  |                                                                   |
| accessIPv6                  |                                                                   |
| addresses                   | dc2-dev-net=192.168.0.69                                          |
| config_drive                |                                                                   |
| created                     | 2019-10-21T10:40:44Z                                              |
| flavor                      | openshift.default (455720a5-de0d-4a96-873d-5c8586d22c3c)          |
| hostId                      | 6a213b8f4a5ef45d04ab2a798f2cfc829f108b0a3a05c6a68913c367          |
| id                          | b491b576-ed38-42e7-a26e-569ec7a3eb2c                              |
| image                       | rhel-server-7.6-x86_64-kvm (57fbc216-f01b-442f-afb7-9ec6e6f7fb51) |
| key_name                    | dc2-dev                                                           |
| name                        | dc2-dev-node-3-19                                                 |
| progress                    | 0                                                                 |
| project_id                  | d12e0698785f498386146459fd6db64d                                  |
| properties                  |                                                                   |
| security_groups             | name='dc2-dev-security-group-node'                                |
| status                      | ACTIVE                                                            |
| updated                     | 2019-12-12T08:01:30Z                                              |
| user_id                     | 7533626a8a020e9d2102e2fd243fd3bab885ff45747c5e5639c2bebdca74d588  |
| volumes_attached            | id='03a42d6a-e1f9-491d-980a-1c8bd45a333e'                         |
|                             | id='4546da68-7b54-4ecb-81f0-3de147ecc9da'                         |
|                             | id='ee9d8306-579f-42af-8189-6d66430ad8c6'                         |
+-----------------------------+-------------------------------------------------------------------+
$ openstack volume show panunvolume1
+------------------------------+------------------------------------------------------------------+
| Field                        | Value                                                            |
+------------------------------+------------------------------------------------------------------+
| attachments                  | []                                                               |
| availability_zone            | nova                                                             |
| bootable                     | false                                                            |
| consistencygroup_id          | None                                                             |
| created_at                   | 2020-05-04T13:43:58.000000                                       |
| description                  | None                                                             |
| encrypted                    | False                                                            |
| id                           | c8135c3c-5181-4346-92db-6e76f5024bb2                             |
| multiattach                  | False                                                            |
| name                         | panunvolume1                                                     |
| os-vol-tenant-attr:tenant_id | d12e0698785f498386146459fd6db64d                                 |
| properties                   |                                                                  |
| replication_status           | None                                                             |
| size                         | 5                                                                |
| snapshot_id                  | None                                                             |
| source_volid                 | None                                                             |
| status                       | available                                                        |
| type                         | solidfire                                                        |
| updated_at                   | 2020-05-06T04:51:56.000000                                       |
| user_id                      | 7533626a8a020e9d2102e2fd243fd3bab885ff45747c5e5639c2bebdca74d588 |
+------------------------------+------------------------------------------------------------------+

$ openstack server add volume dc2-dev-node-3-19 panunvolume1 --device /dev/vdf
$ openstack volume show panunvolume1
+------------------------------+------------------------------------------------------------------+
| Field                        | Value                                                            |
+------------------------------+------------------------------------------------------------------+
| attachments                  | []                                                               |
| availability_zone            | nova                                                             |
| bootable                     | false                                                            |
| consistencygroup_id          | None                                                             |
| created_at                   | 2020-05-04T13:43:58.000000                                       |
| description                  | None                                                             |
| encrypted                    | False                                                            |
| id                           | c8135c3c-5181-4346-92db-6e76f5024bb2                             |
| multiattach                  | False                                                            |
| name                         | panunvolume1                                                     |
| os-vol-tenant-attr:tenant_id | d12e0698785f498386146459fd6db64d                                 |
| properties                   |                                                                  |
| replication_status           | None                                                             |
| size                         | 5                                                                |
| snapshot_id                  | None                                                             |
| source_volid                 | None                                                             |
| status                       | available                                                        |
| type                         | solidfire                                                        |
| updated_at                   | 2020-05-06T04:53:08.000000                                       |
| user_id                      | 7533626a8a020e9d2102e2fd243fd3bab885ff45747c5e5639c2bebdca74d588 |
+------------------------------+------------------------------------------------------------------+
$ openstack server add volume dc2-dev-node-3-19 panunvolume1 --device /dev/vdg
$ openstack volume show panunvolume1
+------------------------------+------------------------------------------------------------------+
| Field                        | Value                                                            |
+------------------------------+------------------------------------------------------------------+
| attachments                  | []                                                               |
| availability_zone            | nova                                                             |
| bootable                     | false                                                            |
| consistencygroup_id          | None                                                             |
| created_at                   | 2020-05-04T13:43:58.000000                                       |
| description                  | None                                                             |
| encrypted                    | False                                                            |
| id                           | c8135c3c-5181-4346-92db-6e76f5024bb2                             |
| multiattach                  | False                                                            |
| name                         | panunvolume1                                                     |
| os-vol-tenant-attr:tenant_id | d12e0698785f498386146459fd6db64d                                 |
| properties                   |                                                                  |
| replication_status           | None                                                             |
| size                         | 5                                                                |
| snapshot_id                  | None                                                             |
| source_volid                 | None                                                             |
| status                       | available                                                        |
| type                         | solidfire                                                        |
| updated_at                   | 2020-05-06T04:53:41.000000                                       |
| user_id                      | 7533626a8a020e9d2102e2fd243fd3bab885ff45747c5e5639c2bebdca74d588 |
+------------------------------+------------------------------------------------------------------+

$ openstack server add volume dc2-dev-node-3-19 panunvolume1 --device /dev/vdq
$ openstack volume show panunvolume1
+------------------------------+------------------------------------------------------------------+
| Field                        | Value                                                            |
+------------------------------+------------------------------------------------------------------+
| attachments                  | []                                                               |
| availability_zone            | nova                                                             |
| bootable                     | false                                                            |
| consistencygroup_id          | None                                                             |
| created_at                   | 2020-05-04T13:43:58.000000                                       |
| description                  | None                                                             |
| encrypted                    | False                                                            |
| id                           | c8135c3c-5181-4346-92db-6e76f5024bb2                             |
| multiattach                  | False                                                            |
| name                         | panunvolume1                                                     |
| os-vol-tenant-attr:tenant_id | d12e0698785f498386146459fd6db64d                                 |
| properties                   |                                                                  |
| replication_status           | None                                                             |
| size                         | 5                                                                |
| snapshot_id                  | None                                                             |
| source_volid                 | None                                                             |
| status                       | available                                                        |
| type                         | solidfire                                                        |
| updated_at                   | 2020-05-06T04:57:12.000000                                       |
| user_id                      | 7533626a8a020e9d2102e2fd243fd3bab885ff45747c5e5639c2bebdca74d588 |
+------------------------------+------------------------------------------------------------------+
$ ssh 192.168.0.69 sudo mount | grep cinder/mounts
Warning: Permanently added '192.168.0.69' (ECDSA) to the list of known hosts.
/dev/vdc on /data/origin/openshift.local.volumes/plugins/kubernetes.io/cinder/mounts/4546da68-7b54-4ecb-81f0-3de147ecc9da type ext4 (rw,relatime,seclabel,data=ordered)
/dev/vdd on /data/origin/openshift.local.volumes/plugins/kubernetes.io/cinder/mounts/ee9d8306-579f-42af-8189-6d66430ad8c6 type ext4 (rw,relatime,seclabel,data=ordered)
/dev/vde on /data/origin/openshift.local.volumes/plugins/kubernetes.io/cinder/mounts/03a42d6a-e1f9-491d-980a-1c8bd45a333e type ext4 (rw,relatime,seclabel,data=ordered)
$ ssh 192.168.0.69 lsblk | grep ^vd.
Warning: Permanently added '192.168.0.69' (ECDSA) to the list of known hosts.
vda                                                                                            252:0    0   10G  0 disk 
vdb                                                                                            252:16   0   90G  0 disk 
vdc                                                                                            252:32   0    4G  0 disk /data/origin/openshift.local.volumes/pods/c7859076-83b6-11ea-a859-fa163e7e3641/volumes/kubernetes.io~cinder/pvc-3355e378-35e7-11ea-9446-fa163e7e3641
vdd                                                                                            252:48   0   20G  0 disk /data/origin/openshift.local.volumes/pods/ceb78046-83b6-11ea-a859-fa163e7e3641/volumes/kubernetes.io~cinder/pvc-a6d31f03-0b9f-11ea-a97c-fa163ed73ca3
vde                                                                                            252:64   0    5G  0 disk /data/origin/openshift.local.volumes/pods/e385babd-83bc-11ea-a859-fa163e7e3641/volumes/kubernetes.io~cinder/pvc-21e75fe6-7319-11ea-8cef-fa163e6ae5da





Note: this issue random and is with specific instance only, not all instances.

Comment 1 Madhur Gupta 2020-05-21 19:24:11 UTC

2020-05-06 07:51:51.926 9 INFO nova.compute.manager [req-1fefc1ea-79cd-4466-9e48-f4da6e0dd462 7533626a8a020e9d2102e2fd243fd3bab885ff45747c5e5639c2bebdca74d588 d12e0698785f498386146459fd6db64d - e62707285b7045589371c85cfeb735cb e62707285b7045589371c85cfeb735cb] [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c] Attaching volume c8135c3c-5181-4346-92db-6e76f5024bb2 to /dev/vdf
2020-05-06 07:51:52.963 9 INFO os_brick.initiator.connectors.iscsi [req-1fefc1ea-79cd-4466-9e48-f4da6e0dd462 7533626a8a020e9d2102e2fd243fd3bab885ff45747c5e5639c2bebdca74d588 d12e0698785f498386146459fd6db64d - e62707285b7045589371c85cfeb735cb e62707285b7045589371c85cfeb735cb] Trying to connect to iSCSI portal 10.222.37.193:3260
2020-05-06 07:51:55.227 9 ERROR nova.virt.libvirt.driver [req-1fefc1ea-79cd-4466-9e48-f4da6e0dd462 7533626a8a020e9d2102e2fd243fd3bab885ff45747c5e5639c2bebdca74d588 d12e0698785f498386146459fd6db64d - e62707285b7045589371c85cfeb735cb e62707285b7045589371c85cfeb735cb] [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c] Failed to attach volume at mountpoint: /dev/vdf: libvirtError: Requested operation is not valid: target vdf already exists
2020-05-06 07:51:55.227 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c] Traceback (most recent call last):
2020-05-06 07:51:55.227 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1493, in attach_volume
2020-05-06 07:51:55.227 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     guest.attach_device(conf, persistent=True, live=live)
2020-05-06 07:51:55.227 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 306, in attach_device
2020-05-06 07:51:55.227 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     self._domain.attachDeviceFlags(device_xml, flags=flags)
2020-05-06 07:51:55.227 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2020-05-06 07:51:55.227 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2020-05-06 07:51:55.227 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2020-05-06 07:51:55.227 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     rv = execute(f, *args, **kwargs)
2020-05-06 07:51:55.227 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2020-05-06 07:51:55.227 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     six.reraise(c, e, tb)
2020-05-06 07:51:55.227 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2020-05-06 07:51:55.227 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     rv = meth(*args, **kwargs)
2020-05-06 07:51:55.227 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 605, in attachDeviceFlags
2020-05-06 07:51:55.227 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2020-05-06 07:51:55.227 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c] libvirtError: Requested operation is not valid: target vdf already exists
2020-05-06 07:51:55.227 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]



2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [req-1fefc1ea-79cd-4466-9e48-f4da6e0dd462 7533626a8a020e9d2102e2fd243fd3bab885ff45747c5e5639c2bebdca74d588 d12e0698785f498386146459fd6db64d - e62707285b7045589371c85cfeb735cb e62707285b7045589371c85cfeb735cb] [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c] Driver failed to attach volume c8135c3c-5181-4346-92db-6e76f5024bb2 at /dev/vdf: libvirtError: Requested operation is not valid: target vdf already exists
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c] Traceback (most recent call last):
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 540, in _volume_attach
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     device_type=self['device_type'], encryption=encryption)
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1520, in attach_volume
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     encryption=encryption)
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     self.force_reraise()
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     six.reraise(self.type_, self.value, self.tb)
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1493, in attach_volume
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     guest.attach_device(conf, persistent=True, live=live)
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 306, in attach_device
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     self._domain.attachDeviceFlags(device_xml, flags=flags)
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     rv = execute(f, *args, **kwargs)
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     six.reraise(c, e, tb)
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     rv = meth(*args, **kwargs)
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 605, in attachDeviceFlags
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c] libvirtError: Requested operation is not valid: target vdf already exists
2020-05-06 07:51:55.717 9 ERROR nova.virt.block_device [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]




2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [req-1fefc1ea-79cd-4466-9e48-f4da6e0dd462 7533626a8a020e9d2102e2fd243fd3bab885ff45747c5e5639c2bebdca74d588 d12e0698785f498386146459fd6db64d - e62707285b7045589371c85cfeb735cb e62707285b7045589371c85cfeb735cb] [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c] Failed to attach c8135c3c-5181-4346-92db-6e76f5024bb2 at /dev/vdf: libvirtError: Requested operation is not valid: target vdf already exists
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c] Traceback (most recent call last):
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5433, in _attach_volume
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     do_driver_attach=True)
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 46, in wrapped
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     ret_val = method(obj, context, *args, **kwargs)
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 624, in attach
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     virt_driver, do_driver_attach)
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 602, in _do_attach
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     do_driver_attach)
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 549, in _volume_attach
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     attachment_id)
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     self.force_reraise()
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     six.reraise(self.type_, self.value, self.tb)
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 540, in _volume_attach
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     device_type=self['device_type'], encryption=encryption)
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1520, in attach_volume
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     encryption=encryption)
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     self.force_reraise()
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     six.reraise(self.type_, self.value, self.tb)
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1493, in attach_volume
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     guest.attach_device(conf, persistent=True, live=live)
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 306, in attach_device
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     self._domain.attachDeviceFlags(device_xml, flags=flags)
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     rv = execute(f, *args, **kwargs)
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     six.reraise(c, e, tb)
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     rv = meth(*args, **kwargs)
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 605, in attachDeviceFlags
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2020-05-06 07:51:56.444 9 ERROR nova.compute.manager [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c] libvirtError: Requested operation is not valid: target vdf already exists



-> When using --device option to attach it on node /dev/vdg:

2020-05-06 07:53:35.637 9 WARNING nova.virt.libvirt.driver [req-6825c6cc-3b0c-4e67-86e9-f68356034272 7533626a8a020e9d2102e2fd243fd3bab885ff45747c5e5639c2bebdca74d588 d12e0698785f498386146459fd6db64d - e62707285b7045589371c85cfeb735cb e62707285b7045589371c85cfeb735cb] [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c] Ignoring supplied device name: /dev/vdg
2020-05-06 07:53:36.474 9 INFO nova.compute.manager [req-6825c6cc-3b0c-4e67-86e9-f68356034272 7533626a8a020e9d2102e2fd243fd3bab885ff45747c5e5639c2bebdca74d588 d12e0698785f498386146459fd6db64d - e62707285b7045589371c85cfeb735cb e62707285b7045589371c85cfeb735cb] [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c] Attaching volume c8135c3c-5181-4346-92db-6e76f5024bb2 to /dev/vdf
2020-05-06 07:53:37.833 9 INFO os_brick.initiator.connectors.iscsi [req-6825c6cc-3b0c-4e67-86e9-f68356034272 7533626a8a020e9d2102e2fd243fd3bab885ff45747c5e5639c2bebdca74d588 d12e0698785f498386146459fd6db64d - e62707285b7045589371c85cfeb735cb e62707285b7045589371c85cfeb735cb] Trying to connect to iSCSI portal 10.222.37.193:3260
2020-05-06 07:53:40.126 9 ERROR nova.virt.libvirt.driver [req-6825c6cc-3b0c-4e67-86e9-f68356034272 7533626a8a020e9d2102e2fd243fd3bab885ff45747c5e5639c2bebdca74d588 d12e0698785f498386146459fd6db64d - e62707285b7045589371c85cfeb735cb e62707285b7045589371c85cfeb735cb] [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c] Failed to attach volume at mountpoint: /dev/vdf: libvirtError: Requested operation is not valid: target vdf already exists
2020-05-06 07:53:40.126 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c] Traceback (most recent call last):
2020-05-06 07:53:40.126 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1493, in attach_volume
2020-05-06 07:53:40.126 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     guest.attach_device(conf, persistent=True, live=live)
2020-05-06 07:53:40.126 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 306, in attach_device
2020-05-06 07:53:40.126 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     self._domain.attachDeviceFlags(device_xml, flags=flags)
2020-05-06 07:53:40.126 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2020-05-06 07:53:40.126 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2020-05-06 07:53:40.126 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2020-05-06 07:53:40.126 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     rv = execute(f, *args, **kwargs)
2020-05-06 07:53:40.126 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2020-05-06 07:53:40.126 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     six.reraise(c, e, tb)
2020-05-06 07:53:40.126 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2020-05-06 07:53:40.126 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     rv = meth(*args, **kwargs)
2020-05-06 07:53:40.126 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 605, in attachDeviceFlags
2020-05-06 07:53:40.126 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2020-05-06 07:53:40.126 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c] libvirtError: Requested operation is not valid: target vdf already exists
2020-05-06 07:53:40.126 9 ERROR nova.virt.libvirt.driver [instance: b491b576-ed38-42e7-a26e-569ec7a3eb2c]

Comment 10 smooney 2020-06-26 02:25:11 UTC

upstream we have a similar issue with detach too.
https://bugs.launchpad.net/cinder/+bug/1882521
im make the description and first comment public as there is no sensitive info in them but it will be needed for other to understand this bug.