Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1221614

Summary: Cinder LVM iSCSI driver with LIO helper fails to handle multiple concurrent requests to attach/detach volumes
Product: Red Hat OpenStack Reporter: Eduard Barrera <ebarrera>
Component: openstack-cinderAssignee: Gorka Eguileor <geguileo>
Status: CLOSED ERRATA QA Contact: Yogev Rabl <yrabl>
Severity: unspecified Docs Contact:
Priority: high    
Version: 5.0 (RHEL 7)CC: ddomingo, dmaley, dshetty, ebarrera, eharney, geguileo, jbuchta, jschluet, jwaterwo, lyarwood, pcaruana, scohen, sgotliv, yeylon
Target Milestone: z5Keywords: ZStream
Target Release: 5.0 (RHEL 7)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-cinder-2014.1.4-3.el7ost Doc Type: Bug Fix
Doc Text:
Previous releases did not have synchronized access to configfs in rtslib. This made it possible for rtstool or rtslib to attempt access to an element that had just been removed mid-loop by another Block Storage request (and was, therefore, non-existent). This, in turn, could result in any of the following exceptions: - .dump() - KeyError - IOError - RTSLibError on storage_object This release implements synchronized access to all rtstool calls that access or modify configfs. With this, concurrent access from multiple threads and processes now work consistently.
Story Points: ---
Clone Of:
: 1227404 1227405 1254755 1289526 (view as bug list) Environment:
Last Closed: 2015-09-10 11:49:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1227404, 1227405, 1254755, 1289526    

Description Eduard Barrera 2015-05-14 12:46:31 UTC
Description of problem:

I opening this bugzilla requesting help for 

01445048 - None of the local cinder volumes are attaching

Customer reports they have  found that the provider_host dose not get populated and when one attempts to attach it tried to read it from the database and it is null and an exception is thrown. This is causing that compute nodes can't attach volumes.





Version-Release number of selected component (if applicable):

openstack 5 
openstack-cinder-2014.1.2-2.el7ost.noarch
python-cinder-2014.1.2-2.el7ost.noarch
python-cinderclient-1.0.9-1.el7ost.noarch



How reproducible:
dont' know

Steps to Reproduce:
1.
2.
3.

Actual results:
Not able to mount any cinder volume

Comment 18 Jack Waterworth 2015-05-18 22:07:56 UTC
there appears to be 2 issues being covered in via 2 separate support cases.

    1. Customer is unable to attach volumes sporadically.  This appears to occur when the add-initiator is called to update the acl in LIO. This is failing for an unknown reason. An error has been identified in comment #9. After a reboot, the issue seems to have vanished.

    2. Customer is unable to detach volumes sporadically.  This is covered in comment #16.  Apparently the delete against the target is being run twice for some reason.  On the second run, the target does not exist and the script returns 1.

Should these issues be split into different BZs? They do not appear to be related.

Comment 21 Sergey Gotliv 2015-05-20 22:58:14 UTC
Guys,

I suggest to open another BZ to track a deletion issue. Let's separate between
attach and delete.

Comment 24 Sergey Gotliv 2015-05-25 09:01:14 UTC
We have 2 separate issues here:

1. Attach existing volume backed by LIO LVM driver. 

According to the logs it seems like a customer is trying to perform stress tests of driver that we use for POC or as a reference implementation, I don't recommend to use it on production workloads. Now about the problem, attaching of the volumes fails because the driver is trying to add node's iSCSI initiator name to the ACL of the non-existing target. Unfortunately, it's hard to say at this moment whether the target is really doesn't exist (most probably that what happens in this case) or cinder-rtstool just fails to add an initiator due to CHAP credentials mismatch, because the flow in the log looks different between successful attempts to attach a volume and failures (attaching the same volume sometimes succeeds and sometimes it fails). 

When it succeeds, the logs clearly shows that the target is created before the attempt to add an initiator. When it fails, it's missing a create target step in the log. We are currently trying to reproduce the problem in-house and in parallel understand why sometimes we miss target creating step in the log!


2. Detach volume issue.

Please, open a separate case for this issue. Jack raised a concern that the user has a script trying to delete the same volume more than once. If it's true then this is not a Cinder bug. Customer has to fix the script. Please, open another case with the relevant logs and provide a customer script if possible.

Comment 25 Sergey Gotliv 2015-05-25 09:47:48 UTC
Jack/Eduard,

Please, ask a customer to provide the script he is running to perform this stress testing. It will help us to reproduce the problem in-house.

Comment 27 Gorka Eguileor 2015-05-28 20:08:57 UTC
To reproduce this issue we created a good number of concurrent continuous loops of create/attach/detach/delete against the same VM and let it run for quite some time until we could see any issues.

Apparently errors come from rtstool use of rtslib and an issue inside rtslib itself, and in both cases these errors are very rare and hard to see. They affect attaching, detaching and deleting volumes.

The source of problems in both cases (rtstool and rtslib) is the same, another request deleted an LIO target while we were getting information about it.

Since it is needed to go through all targets in order to find the one we are interested in, we can get this error when accessing a complete different target from the one we want to work on; and when this happens errors in logs will display a different volume (misleading us), because that's the one we were trying to get information from.

This is a very specific problem affecting LIO.

Now that the issue has been identified we'll have a solution shortly.

Comment 28 Dave Maley 2015-05-29 16:16:48 UTC
Pablo/Eduard - Engineering is still interested in obtaining the script in use by the customer.  Please ping them again w/ the request for them to provide it.  Thanks!

Comment 30 Gorka Eguileor 2015-06-02 14:49:04 UTC
We believe we have addressed all issues that were appearing in the logs in a "rtslib concurrency" patch we have submitted to upstream Liberty, which has already been merged, and we have fixed this in RHOS-5 as well; so it will be included in the next package release.

Backports to Kilo and Juno stable branches will follow.

Comment 45 Sergey Gotliv 2015-06-18 07:03:27 UTC
Jan/Dave,

Do you know if the customer already installed our test package?

Comment 46 Dave Maley 2015-06-18 16:39:35 UTC
(In reply to Sergey Gotliv from comment #45)
> Jan/Dave,
> 
> Do you know if the customer already installed our test package?

It was just provided to them this morning and we have yet to receive any feedback.  Once we do hear back we'll update here w/ results.

Comment 47 Dave Maley 2015-06-22 16:50:08 UTC
(In reply to Dave Maley from comment #46)
> (In reply to Sergey Gotliv from comment #45)
> > Jan/Dave,
> > 
> > Do you know if the customer already installed our test package?
> 
> It was just provided to them this morning and we have yet to receive any
> feedback.  Once we do hear back we'll update here w/ results.

accidentally cleared needinfo

Comment 51 Sergey Gotliv 2015-06-27 21:53:57 UTC
It seems like the user is trying to attach volume managed by Cinder on osp1-compute03 to the VM running on osp1-compute01. 

This volume is exposed through the target iqn.2010-10.org.openstack:volume-9dea2617-2290-446f-a147-cf5b069b87db which doesn't contain osp1-compute01 iSCSI initiator name in the ACL therefore this initiator can't access that volume.

According to the Cinder log on osp1-compute03, it's getting a request from Nova to initialize connection, as a result Cinder is creating the iSCSI target to expose the volume:

2015-06-26 15:47:55.441 15846 DEBUG cinder.volume.manager [req-41163346-bcea-4166-9d3f-ba7d87135158 50a742bdfe4e4a8ca0ddc150fc648904 648ffe15b9fe4a1bba71e4643d4dbbf2 - - -] Volume 9dea2617-2290-446f-a147-cf5b069b87db: creating export initialize_connection /usr/lib/python2.7/site-packages/cinder/volume/manager.py:792
2015-06-26 15:47:55.510 15846 INFO cinder.brick.iscsi.iscsi [req-41163346-bcea-4166-9d3f-ba7d87135158 50a742bdfe4e4a8ca0ddc150fc648904 648ffe15b9fe4a1bba71e4643d4dbbf2 - - -] Creating iscsi_target for volume: volume-9dea2617-2290-446f-a147-cf5b069b87db

Cinder is running "cinder-rtstool create", which is creating the target and adding LOCAL(osp1-compute03) iSCSI initiator name to the target's ACL:

2015-06-26 15:47:55.565 15846 DEBUG cinder.openstack.common.processutils [req-41163346-bcea-4166-9d3f-ba7d87135158 50a742bdfe4e4a8ca0ddc150fc648904 648ffe15b9fe4a1bba71e4643d4dbbf2 - - -] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf cinder-rtstool create /dev/cinder-volumes/volume-9dea2617-2290-446f-a147-cf5b069b87db iqn.2010-10.org.openstack:volume-9dea2617-2290-446f-a147-cf5b069b87db uU43kJW7ydw849ortxXy 5E3uBj9dKUZHucZHoFMb execute /usr/lib/python2.7/site-packages/cinder/openstack/common/processutils.py:147

Cinder is retrieving a newly created target to verify it really exists:

2015-06-26 15:47:55.800 15846 DEBUG cinder.openstack.common.processutils [req-41163346-bcea-4166-9d3f-ba7d87135158 50a742bdfe4e4a8ca0ddc150fc648904 648ffe15b9fe4a1bba71e4643d4dbbf2 - - -] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf cinder-rtstool get-targets execute /usr/lib/python2.7/site-packages/cinder/openstack/common/processutils.py:147

And after a few other manipulations Cinder is finally saving the target configuration and returning the connection info back to Nova:

2015-06-26 15:47:56.020 15846 DEBUG cinder.openstack.common.processutils [req-41163346-bcea-4166-9d3f-ba7d87135158 50a742bdfe4e4a8ca0ddc150fc648904 648ffe15b9fe4a1bba71e4643d4dbbf2 - - -] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf cinder-rtstool save execute /usr/lib/python2.7/site-packages/cinder/openstack/common/processutils.py:147

The problem is that one step is missing in the middle, Cinder must add Nova host iSCSI initiator name to the target ACL or in other words, it must run

cinder-rtstool add-initiator .... 

which is missing in this particular run. The code is there, we have to understand why it didn't run.

Comment 52 Gorka Eguileor 2015-06-29 11:29:33 UTC
We have detected a misconfiguration in their systems:
 iscsi_helper                    = lioadm
 lvm-osp1-compute03.iscsi_helper = tgtadm

Both should be set to lioadm.

They need to change the lvm driver configuration in /etc/cinder/cinder.conf and add the right iscsi_helper:

 Search for:
     [lvm-osp1-compute03]
 And add this line right below:
     iscsi_helper = lioadm

Comment 53 Gorka Eguileor 2015-06-29 12:13:59 UTC
In previous comment I specifically mentioned compute03's configuration, but this actually needs to be checked, and changed if incorrect, in *ALL* nodes running Cinder with LVM to ensure the right target is set consistently.

Comment 59 Sergey Gotliv 2015-07-11 21:16:09 UTC
We provided a hotfix on July 1. Any feedback from the customer?

Comment 60 Dave Maley 2015-07-14 22:46:47 UTC
We received an update and testing feedback from the customer on 9 July indicating their initial results were good however On 7 July started seeing some errors.  It was not clear if they were still actually having problems or if they were just reporting the errors to us and so we asked for clarification.  The customer contact has been out of the office and so we have yet to hear back.  I'll push the full update from the customer momentarily.

Comment 65 Yogev Rabl 2015-09-07 08:24:11 UTC
verified on
openstack-cinder-2014.1.5-1.el7ost.noarch

Comment 67 errata-xmlrpc 2015-09-10 11:49:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-1758.html