Description of problem: lioadm target creation limited by iscsi_num_targets. Constant attaching and detaching of volumes can quickly saturate the iscsi_targets table. By default this table has 100 rows for each cinder-volume host listing the target_num for each volume. This information is not however used by the lioadm target helper. This results in the following trace with the patch from BZ#1244446 in place to allow for create_export exceptions to be correctly handled : 2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager Traceback (most recent call last): 2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 794, in initialize_connection 2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager volume) 2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 525, in create_export 2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager return self._create_export(context, volume) 2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 547, in _create_export 2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager data = self.target_helper.create_export(context, volume, volume_path) 2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/iscsi.py", line 42, in create_export 2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager iscsi_target, lun = self._get_target_and_lun(context, volume) 2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/iscsi.py", line 143, in _get_target_and_lun 2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager volume['host']) 2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/db/api.py", line 163, in volume_allocate_iscsi_target 2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager return IMPL.volume_allocate_iscsi_target(context, volume_id, host) 2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 121, in wrapper 2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager return f(*args, **kwargs) 2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 177, in wrapped 2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager return f(*args, **kwargs) 2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 994, in volume_allocate_iscsi_target 2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager raise exception.NoMoreTargets() 2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager NoMoreTargets: An unknown exception occurred. Downstream in RHEL OSP 5 the following patch can workaround this however it is likely missing additional calls to the DB by the lioadm helper : # diff -u /usr/lib/python2.7/site-packages/cinder/volume/iscsi.py.backup /usr/lib/python2.7/site-packages/cinder/volume/iscsi.py --- /usr/lib/python2.7/site-packages/cinder/volume/iscsi.py.backup 2015-07-19 10:26:16.977000000 -0400 +++ /usr/lib/python2.7/site-packages/cinder/volume/iscsi.py 2015-07-19 12:11:41.214000000 -0400 @@ -232,15 +232,7 @@ class LioAdm(_ExportMixin, iscsi.LioAdm): def remove_export(self, context, volume): - try: - iscsi_target = self.db.volume_get_iscsi_target_num(context, - volume['id']) - except exception.NotFound: - LOG.info(_("Skipping remove_export. No iscsi_target " - "provisioned for volume: %s"), volume['id']) - return - - self.remove_iscsi_target(iscsi_target, 0, volume['id'], volume['name']) + self.remove_iscsi_target(0, 0, volume['id'], volume['name']) def ensure_export(self, context, volume_id, iscsi_name, volume_path, old_name=None): @@ -262,6 +254,11 @@ self.create_iscsi_target(iscsi_name, iscsi_target, 0, volume_path, chap_auth, check_exit_code=False) + def _get_target_and_lun(self, context, volume): + lun = 0 # For lio, the lun starts at lun 0. + iscsi_target = 0 # NOTE: Not used by lio. + return iscsi_target, lun + Version-Release number of selected component (if applicable): openstack-cinder-2014.1.4-6.el7ost How reproducible: Always. Steps to Reproduce: 1. Attach and detach LVMSCSI hosted volumes more than 100 times. Actual results: NoMoreTargets exception thrown. Expected results: The lioadm helper does not use the information stored in the iscsi_targets table and thus should not populate this table with anything when attaching a volume. Additional info: https://launchpad.net/bugs/1410566 https://review.openstack.org/#/c/148038/
Since we can't just backport a Liberty fix as is to Kilo/Juno/Icehouse, we can take a code proposed by Lee, but probably just put it in the parent class ExportMixin. ExportMixin introduces a default implementation of the "_get_target_and_lun" which is already overridden in the same way in 2 out of 4 sub-classes. Lee's code proposes to do the same fix in the LioAdm the third sub-class, so it's probably a good opportunity to just change a default behavior in the ExportMixin.
This bug is already fixed in 6.0 and later, so it only affects 5.0. To keep it in line with how it's done in all other versions we should just do: + def _get_iscsi_target(self, context, vol_id): + return 0 + + def _get_target_and_lun(self, context, volume): + lun = 0 # For lio, the lun starts at lun 0. + iscsi_target = 0 # NOTE: Not used by lio. + return iscsi_target, lun
verified on openstack-cinder-2014.1.5-1.el7ost.noarch
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2015-1759.html