Created attachment 882276 [details] cinder volume log Description of problem: on lvm with rhel7 I fail to delete a volume with ISCSITargetNotFoundForVolume Version-Release number of selected component (if applicable): [root@orange-vdsf ~(keystone_admin)]# rpm -qa |grep cinder python-cinderclient-1.0.7-2.el7.noarch openstack-cinder-2014.1-0.6.b3.el7.noarch python-cinder-2014.1-0.6.b3.el7.noarch How reproducible: 100% Steps to Reproduce: 1. install AIO with lvm backend for cinder 2. create a volume (cinder create 1) 3. delete the volume (cinder delete <volume>) Actual results: we fail to delete the volume with raise exception.ISCSITargetNotFoundForVolume Expected results: we should be able to delete the volume Additional info: full log attached 2014-04-03 16:32:20.140 2616 DEBUG cinder.volume.manager [req-6aecfae7-5977-41af-b73b-31f2e2f931f6 None] volume b3fcb999-eb36-422c-ae35-da3276794665: removing export delete_volume /usr/lib/python2.7/site-packages/cinder/volume/manager.py:396 2014-04-03 16:32:20.177 2616 DEBUG cinder.openstack.common.lockutils [req-6aecfae7-5977-41af-b73b-31f2e2f931f6 None] Released file lock "b3fcb999-eb36-422c-ae35-da3276794665-delete_volume" at /var/lib/cinder/tmp/cinder-b3fcb999-eb36-422c-ae35-da3276794665-delete_volume for method "lvo_inner2"... inner /usr/lib/python2.7/site-packages/cinder/openstack/common/lockutils.py:239 2014-04-03 16:32:20.177 2616 ERROR cinder.openstack.common.rpc.amqp [req-6aecfae7-5977-41af-b73b-31f2e2f931f6 None] Exception during message handling 2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last): 2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/cinder/openstack/common/rpc/amqp.py", line 462, in _process_data 2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp **args) 2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 172, in dispatch 2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs) 2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 143, in lvo_inner1 2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp return lvo_inner2(inst, context, volume_id, **kwargs) 2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/cinder/openstack/common/lockutils.py", line 233, in inner 2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp retval = f(*args, **kwargs) 2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 142, in lvo_inner2 2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp return f(*_args, **_kwargs) 2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 413, in delete_volume 2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp {'status': 'error_deleting'}) 2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/cinder/openstack/common/excutils.py", line 68, in __exit__ 2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp six.reraise(self.type_, self.value, self.tb) 2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 397, in delete_volume 2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp self.driver.remove_export(context, volume_ref) 2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 540, in remove_export 2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp self.target_helper.remove_export(context, volume) 2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/cinder/volume/iscsi.py", line 232, in remove_export 2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp volume['id']) 2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/cinder/db/api.py", line 232, in volume_get_iscsi_target_num 2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp return IMPL.volume_get_iscsi_target_num(context, volume_id) 2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 117, in wrapper 2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp return f(*args, **kwargs) 2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 1221, in volume_get_iscsi_target_num 2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp raise exception.ISCSITargetNotFoundForVolume(volume_id=volume_id) 2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp ISCSITargetNotFoundForVolume: No target id found for volume b3fcb999-eb36-422c-ae35-da3276794665. 2014-04-03 16:32:20.177 2616 TRACE cinder.openstack.common.rpc.amqp 2014-04-03 16:32:57.363 2616 DEBUG cinder.openstack.common.periodic_task [-] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/site-packages/cinder/openstack/common/periodic_task.py:178 [root@orange-vdsf ~(keystone_admin)]# cinder create --display-name dafna1 1 +---------------------+--------------------------------------+ | Property | Value | +---------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | created_at | 2014-04-03T13:32:07.576476 | | display_description | None | | display_name | dafna1 | | encrypted | False | | id | b3fcb999-eb36-422c-ae35-da3276794665 | | metadata | {} | | size | 1 | | snapshot_id | None | | source_volid | None | | status | creating | | volume_type | None | +---------------------+--------------------------------------+ [root@orange-vdsf ~(keystone_admin)]# cinder list +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ | 51c841ec-24a5-49fe-8441-01593b39b2f7 | error_deleting | dafna | 10 | None | false | | | 54d28f61-2f0f-4be4-8e27-855a50a50c33 | available | emptyVol | 1 | None | false | | | 82a7a825-6106-4139-9bc8-4334ccc38e85 | available | volFromImage | 1 | None | true | | | b3fcb999-eb36-422c-ae35-da3276794665 | available | dafna1 | 1 | None | false | | +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ [root@orange-vdsf ~(keystone_admin)]# cinder delete b3fcb999-eb36-422c-ae35-da3276794665 [root@orange-vdsf ~(keystone_admin)]# cinder list +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+ | 51c841ec-24a5-49fe-8441-01593b39b2f7 | error_deleting | dafna | 10 | None | false | | | 54d28f61-2f0f-4be4-8e27-855a50a50c33 | available | emptyVol | 1 | None | false | | | 82a7a825-6106-4139-9bc8-4334ccc38e85 | available | volFromImage | 1 | None | true | | | b3fcb999-eb36-422c-ae35-da3276794665 | error_deleting | dafna1 | 1 | None | false | | +--------------------------------------+----------------+--------------+------+-------------+----------+-------------+
The problem here is that when using LIO, the iscsi_targets table does not get an entry until an attach is attempted (which is when the LIO target is created for that volume). The delete path expects there to always be a target id in that table tied to that volume, but that is not guaranteed to the be the case. LioAdm's remove_export tries to handle this via "except exception.NotFound:" but that is referencing cinder.brick.exception.NotFound rather than cinder.exception.NotFound so the error is not skipped.