Bug 1096327

+++ This bug was initially created as a clone of Bug #1084046 +++

Description of problem:

on lvm with rhel7 I fail to delete a volume with ISCSITargetNotFoundForVolume

Version-Release number of selected component (if applicable):

[root@orange-vdsf ~(keystone_admin)]# rpm -qa |grep cinder
python-cinderclient-1.0.7-2.el7.noarch
openstack-cinder-2014.1-0.6.b3.el7.noarch
python-cinder-2014.1-0.6.b3.el7.noarch

How reproducible:

100%

Steps to Reproduce:
1. install AIO with lvm backend for cinder 
2. create a volume (cinder create 1)
3.  delete the volume (cinder delete <volume>)

Actual results:

we fail to delete the volume with raise exception.ISCSITargetNotFoundForVolume

Expected results:

we should be able to delete the volume 

Additional info: full log attached 


2014-04-03 16:32:20.140 2616 DEBUG cinder.volume.manager [req-6aecfae7-5977-41af-b73b-31f2e2f931f6 None] volume b3fcb999-eb36-422c-ae35-da3276794665: removing export delete_volume /usr/lib/python2.7/site-packages/cinder/volume/manager.py:396
2014-04-03 16:32:20.177 2616 DEBUG cinder.openstack.common.lockutils [req-6aecfae7-5977-41af-b73b-31f2e2f931f6 None] Released file lock "b3fcb999-eb36-422c-ae35-da3276794665-delete_volume" at /var/lib/cinder/tmp/cinder-b3fcb999-eb36-422c-ae35-da3276794665-delete_volume for method "lvo_inner2"... inner /usr/lib/python2.7/site-packages/cinder/openstack/common/lockutils.py:239
2014-04-03 16:32:20.177 2616 ERROR cinder.openstack.common.rpc.amqp [req-6aecfae7-5977-41af-b73b-31f2e2f931f6 None] Exception during message handling
2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last):
2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/openstack/common/rpc/amqp.py", line 462, in _process_data
2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp     **args)
2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 143, in lvo_inner1
2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp     return lvo_inner2(inst, context, volume_id, **kwargs)
2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/openstack/common/lockutils.py", line 233, in inner
2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp     retval = f(*args, **kwargs)
2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 142, in lvo_inner2
2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp     return f(*_args, **_kwargs)
2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 413, in delete_volume
2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp     {'status': 'error_deleting'})
2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/openstack/common/excutils.py", line 68, in __exit__
2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp     six.reraise(self.type_, self.value, self.tb)
2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 397, in delete_volume
2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp     self.driver.remove_export(context, volume_ref)
2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 540, in remove_export
2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp     self.target_helper.remove_export(context, volume)
2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/volume/iscsi.py", line 232, in remove_export
2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp     volume['id'])
2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/db/api.py", line 232, in volume_get_iscsi_target_num
2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp     return IMPL.volume_get_iscsi_target_num(context, volume_id)
2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 117, in wrapper
2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp     return f(*args, **kwargs)
2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 1221, in volume_get_iscsi_target_num
2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp     raise exception.ISCSITargetNotFoundForVolume(volume_id=volume_id)
2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp ISCSITargetNotFoundForVolume: No target id found for volume b3fcb999-eb36-422c-ae35-da3276794665.
2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp 
2014-04-03 16:32:57.363 2616 DEBUG cinder.openstack.common.periodic_task [-] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/cinder/openstack/common/periodic_task.py:178



[root@orange-vdsf ~(keystone_admin)]# cinder create --display-name dafna1 1
+---------------------+--------------------------------------+
|       Property      |                Value                 |
+---------------------+--------------------------------------+
|     attachments     |                  []                  |
|  availability_zone  |                 nova                 |
|       bootable      |                false                 |
|      created_at     |      2014-04-03T13:32:07.576476      |
| display_description |                 None                 |
|     display_name    |                dafna1                |
|      encrypted      |                False                 |
|          id         | b3fcb999-eb36-422c-ae35-da3276794665 |
|       metadata      |                  {}                  |
|         size        |                  1                   |
|     snapshot_id     |                 None                 |
|     source_volid    |                 None                 |
|        status       |               creating               |
|     volume_type     |                 None                 |
+---------------------+--------------------------------------+
[root@orange-vdsf ~(keystone_admin)]# cinder list 
+--------------------------------------+----------------+--------------+------+-------------+----------+-------------+
|                  ID                  |     Status     | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+----------------+--------------+------+-------------+----------+-------------+
| 51c841ec-24a5-49fe-8441-01593b39b2f7 | error_deleting |    dafna     |  10  |     None    |  false   |             |
| 54d28f61-2f0f-4be4-8e27-855a50a50c33 |   available    |   emptyVol   |  1   |     None    |  false   |             |
| 82a7a825-6106-4139-9bc8-4334ccc38e85 |   available    | volFromImage |  1   |     None    |   true   |             |
| b3fcb999-eb36-422c-ae35-da3276794665 |   available    |    dafna1    |  1   |     None    |  false   |             |
+--------------------------------------+----------------+--------------+------+-------------+----------+-------------+
[root@orange-vdsf ~(keystone_admin)]# cinder delete b3fcb999-eb36-422c-ae35-da3276794665
[root@orange-vdsf ~(keystone_admin)]# cinder list 
+--------------------------------------+----------------+--------------+------+-------------+----------+-------------+
|                  ID                  |     Status     | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+----------------+--------------+------+-------------+----------+-------------+
| 51c841ec-24a5-49fe-8441-01593b39b2f7 | error_deleting |    dafna     |  10  |     None    |  false   |             |
| 54d28f61-2f0f-4be4-8e27-855a50a50c33 |   available    |   emptyVol   |  1   |     None    |  false   |             |
| 82a7a825-6106-4139-9bc8-4334ccc38e85 |   available    | volFromImage |  1   |     None    |   true   |             |
| b3fcb999-eb36-422c-ae35-da3276794665 | error_deleting |    dafna1    |  1   |     None    |  false   |             |
+--------------------------------------+----------------+--------------+------+-------------+----------+-------------+

--- Additional comment from Eric Harney on 2014-05-06 15:28:46 EDT ---

The problem here is that when using LIO, the iscsi_targets table does not get an entry until an attach is attempted (which is when the LIO target is created for that volume).

The delete path expects there to always be a target id in that table tied to that volume, but that is not guaranteed to the be the case.

LioAdm's remove_export tries to handle this via "except exception.NotFound:" but that is referencing cinder.brick.exception.NotFound rather than cinder.exception.NotFound so the error is not skipped.