Bug 1244548

Summary:	lioadm target creation artificially limited by iscsi_num_targets
Product:	Red Hat OpenStack	Reporter:	Lee Yarwood <lyarwood>
Component:	openstack-cinder	Assignee:	Gorka Eguileor <geguileo>
Status:	CLOSED ERRATA	QA Contact:	lkuchlan <lkuchlan>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	5.0 (RHEL 7)	CC:	ddomingo, eharney, jschluet, scohen, sgotliv, yeylon, yrabl
Target Milestone:	z5	Keywords:	ZStream
Target Release:	5.0 (RHEL 7)
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	openstack-cinder-2014.1.4-7.el7ost	Doc Type:	Bug Fix
Doc Text:	In previous releases, LIO used the iSCSI targets table, which was unnecessary. As a result, quick attach/detach sequences incorrectly filled the table and resulted in an attachment failure. This, in turn, meant that lioadm target creation was artificially limited by iscsi_num_targets. With this release, LIO no longer uses the target table. As a result, LIO is no longer limited by iscsi_num_targets.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-09-10 11:47:15 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Lee Yarwood 2015-07-19 22:37:01 UTC

Description of problem:

lioadm target creation limited by iscsi_num_targets. Constant attaching and detaching of volumes can quickly saturate the iscsi_targets table.

By default this table has 100 rows for each cinder-volume host listing the target_num for each volume. This information is not however used by the lioadm target helper.

This results in the following trace with the patch from BZ#1244446 in place to allow for create_export exceptions to be correctly handled :

2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager Traceback (most recent call last):
2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 794, in initialize_connection
2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager     volume)
2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 525, in create_export
2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager     return self._create_export(context, volume)
2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 547, in _create_export
2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager     data = self.target_helper.create_export(context, volume, volume_path)
2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/iscsi.py", line 42, in create_export
2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager     iscsi_target, lun = self._get_target_and_lun(context, volume)
2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/iscsi.py", line 143, in _get_target_and_lun
2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager     volume['host'])
2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/db/api.py", line 163, in volume_allocate_iscsi_target
2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager     return IMPL.volume_allocate_iscsi_target(context, volume_id, host)
2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 121, in wrapper
2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager     return f(*args, **kwargs)
2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 177, in wrapped
2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager     return f(*args, **kwargs)
2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 994, in volume_allocate_iscsi_target
2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager     raise exception.NoMoreTargets()
2015-07-19 03:14:26.670 79549 TRACE cinder.volume.manager NoMoreTargets: An unknown exception occurred.

Downstream in RHEL OSP 5 the following patch can workaround this however it is likely missing additional calls to the DB by the lioadm helper :

# diff -u /usr/lib/python2.7/site-packages/cinder/volume/iscsi.py.backup /usr/lib/python2.7/site-packages/cinder/volume/iscsi.py
--- /usr/lib/python2.7/site-packages/cinder/volume/iscsi.py.backup	2015-07-19 10:26:16.977000000 -0400
+++ /usr/lib/python2.7/site-packages/cinder/volume/iscsi.py	2015-07-19 12:11:41.214000000 -0400
@@ -232,15 +232,7 @@
 class LioAdm(_ExportMixin, iscsi.LioAdm):
 
     def remove_export(self, context, volume):
-        try:
-            iscsi_target = self.db.volume_get_iscsi_target_num(context,
-                                                               volume['id'])
-        except exception.NotFound:
-            LOG.info(_("Skipping remove_export. No iscsi_target "
-                       "provisioned for volume: %s"), volume['id'])
-            return
-
-        self.remove_iscsi_target(iscsi_target, 0, volume['id'], volume['name'])
+        self.remove_iscsi_target(0, 0, volume['id'], volume['name'])
 
     def ensure_export(self, context, volume_id, iscsi_name, volume_path,
                       old_name=None):
@@ -262,6 +254,11 @@
         self.create_iscsi_target(iscsi_name, iscsi_target, 0, volume_path,
                                  chap_auth, check_exit_code=False)
 
+    def _get_target_and_lun(self, context, volume):
+        lun = 0  # For lio, the lun starts at lun 0.
+        iscsi_target = 0  # NOTE: Not used by lio.
+        return iscsi_target, lun
+


Version-Release number of selected component (if applicable):
openstack-cinder-2014.1.4-6.el7ost

How reproducible:
Always.

Steps to Reproduce:
1. Attach and detach LVMSCSI hosted volumes more than 100 times.

Actual results:
NoMoreTargets exception thrown.

Expected results:
The lioadm helper does not use the information stored in the iscsi_targets table and thus should not populate this table with anything when attaching a volume.

Additional info:
https://launchpad.net/bugs/1410566
https://review.openstack.org/#/c/148038/

Comment 3 Sergey Gotliv 2015-07-20 08:52:33 UTC

Since we can't just backport a Liberty fix as is to Kilo/Juno/Icehouse, we can take a code proposed by Lee, but probably just put it in the parent class ExportMixin.

ExportMixin introduces a default implementation of the "_get_target_and_lun" which is already overridden in the same way in 2 out of 4 sub-classes. Lee's code proposes to do the same fix in the LioAdm the third sub-class, so it's probably a good opportunity to just change a default behavior in the ExportMixin.

Comment 4 Gorka Eguileor 2015-08-03 10:41:10 UTC

This bug is already fixed in 6.0 and later, so it only affects 5.0.

To keep it in line with how it's done in all other versions we should just do:

+    def _get_iscsi_target(self, context, vol_id):
+        return 0
+
+    def _get_target_and_lun(self, context, volume):
+        lun = 0  # For lio, the lun starts at lun 0.
+        iscsi_target = 0  # NOTE: Not used by lio.
+        return iscsi_target, lun

Comment 8 Yogev Rabl 2015-09-09 09:12:50 UTC

verified on openstack-cinder-2014.1.5-1.el7ost.noarch

Comment 10 errata-xmlrpc 2015-09-10 11:47:15 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-1759.html