Bug 1221614
| Summary: | Cinder LVM iSCSI driver with LIO helper fails to handle multiple concurrent requests to attach/detach volumes | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Eduard Barrera <ebarrera> | |
| Component: | openstack-cinder | Assignee: | Gorka Eguileor <geguileo> | |
| Status: | CLOSED ERRATA | QA Contact: | Yogev Rabl <yrabl> | |
| Severity: | unspecified | Docs Contact: | ||
| Priority: | high | |||
| Version: | 5.0 (RHEL 7) | CC: | ddomingo, dmaley, dshetty, ebarrera, eharney, geguileo, jbuchta, jschluet, jwaterwo, lyarwood, pcaruana, scohen, sgotliv, yeylon | |
| Target Milestone: | z5 | Keywords: | ZStream | |
| Target Release: | 5.0 (RHEL 7) | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | openstack-cinder-2014.1.4-3.el7ost | Doc Type: | Bug Fix | |
| Doc Text: |
Previous releases did not have synchronized access to configfs in rtslib. This made it possible for rtstool or rtslib to attempt access to an element that had just been removed mid-loop by another Block Storage request (and was, therefore, non-existent). This, in turn, could result in any of the following exceptions:
- .dump()
- KeyError
- IOError
- RTSLibError on storage_object
This release implements synchronized access to all rtstool calls that access or modify configfs. With this, concurrent access from multiple threads and processes now work consistently.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1227404 1227405 1254755 1289526 (view as bug list) | Environment: | ||
| Last Closed: | 2015-09-10 11:49:08 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1227404, 1227405, 1254755, 1289526 | |||
|
Description
Eduard Barrera
2015-05-14 12:46:31 UTC
there appears to be 2 issues being covered in via 2 separate support cases.
1. Customer is unable to attach volumes sporadically. This appears to occur when the add-initiator is called to update the acl in LIO. This is failing for an unknown reason. An error has been identified in comment #9. After a reboot, the issue seems to have vanished.
2. Customer is unable to detach volumes sporadically. This is covered in comment #16. Apparently the delete against the target is being run twice for some reason. On the second run, the target does not exist and the script returns 1.
Should these issues be split into different BZs? They do not appear to be related.
Guys, I suggest to open another BZ to track a deletion issue. Let's separate between attach and delete. We have 2 separate issues here: 1. Attach existing volume backed by LIO LVM driver. According to the logs it seems like a customer is trying to perform stress tests of driver that we use for POC or as a reference implementation, I don't recommend to use it on production workloads. Now about the problem, attaching of the volumes fails because the driver is trying to add node's iSCSI initiator name to the ACL of the non-existing target. Unfortunately, it's hard to say at this moment whether the target is really doesn't exist (most probably that what happens in this case) or cinder-rtstool just fails to add an initiator due to CHAP credentials mismatch, because the flow in the log looks different between successful attempts to attach a volume and failures (attaching the same volume sometimes succeeds and sometimes it fails). When it succeeds, the logs clearly shows that the target is created before the attempt to add an initiator. When it fails, it's missing a create target step in the log. We are currently trying to reproduce the problem in-house and in parallel understand why sometimes we miss target creating step in the log! 2. Detach volume issue. Please, open a separate case for this issue. Jack raised a concern that the user has a script trying to delete the same volume more than once. If it's true then this is not a Cinder bug. Customer has to fix the script. Please, open another case with the relevant logs and provide a customer script if possible. Jack/Eduard, Please, ask a customer to provide the script he is running to perform this stress testing. It will help us to reproduce the problem in-house. To reproduce this issue we created a good number of concurrent continuous loops of create/attach/detach/delete against the same VM and let it run for quite some time until we could see any issues. Apparently errors come from rtstool use of rtslib and an issue inside rtslib itself, and in both cases these errors are very rare and hard to see. They affect attaching, detaching and deleting volumes. The source of problems in both cases (rtstool and rtslib) is the same, another request deleted an LIO target while we were getting information about it. Since it is needed to go through all targets in order to find the one we are interested in, we can get this error when accessing a complete different target from the one we want to work on; and when this happens errors in logs will display a different volume (misleading us), because that's the one we were trying to get information from. This is a very specific problem affecting LIO. Now that the issue has been identified we'll have a solution shortly. Pablo/Eduard - Engineering is still interested in obtaining the script in use by the customer. Please ping them again w/ the request for them to provide it. Thanks! We believe we have addressed all issues that were appearing in the logs in a "rtslib concurrency" patch we have submitted to upstream Liberty, which has already been merged, and we have fixed this in RHOS-5 as well; so it will be included in the next package release. Backports to Kilo and Juno stable branches will follow. Jan/Dave, Do you know if the customer already installed our test package? (In reply to Sergey Gotliv from comment #45) > Jan/Dave, > > Do you know if the customer already installed our test package? It was just provided to them this morning and we have yet to receive any feedback. Once we do hear back we'll update here w/ results. (In reply to Dave Maley from comment #46) > (In reply to Sergey Gotliv from comment #45) > > Jan/Dave, > > > > Do you know if the customer already installed our test package? > > It was just provided to them this morning and we have yet to receive any > feedback. Once we do hear back we'll update here w/ results. accidentally cleared needinfo It seems like the user is trying to attach volume managed by Cinder on osp1-compute03 to the VM running on osp1-compute01. This volume is exposed through the target iqn.2010-10.org.openstack:volume-9dea2617-2290-446f-a147-cf5b069b87db which doesn't contain osp1-compute01 iSCSI initiator name in the ACL therefore this initiator can't access that volume. According to the Cinder log on osp1-compute03, it's getting a request from Nova to initialize connection, as a result Cinder is creating the iSCSI target to expose the volume: 2015-06-26 15:47:55.441 15846 DEBUG cinder.volume.manager [req-41163346-bcea-4166-9d3f-ba7d87135158 50a742bdfe4e4a8ca0ddc150fc648904 648ffe15b9fe4a1bba71e4643d4dbbf2 - - -] Volume 9dea2617-2290-446f-a147-cf5b069b87db: creating export initialize_connection /usr/lib/python2.7/site-packages/cinder/volume/manager.py:792 2015-06-26 15:47:55.510 15846 INFO cinder.brick.iscsi.iscsi [req-41163346-bcea-4166-9d3f-ba7d87135158 50a742bdfe4e4a8ca0ddc150fc648904 648ffe15b9fe4a1bba71e4643d4dbbf2 - - -] Creating iscsi_target for volume: volume-9dea2617-2290-446f-a147-cf5b069b87db Cinder is running "cinder-rtstool create", which is creating the target and adding LOCAL(osp1-compute03) iSCSI initiator name to the target's ACL: 2015-06-26 15:47:55.565 15846 DEBUG cinder.openstack.common.processutils [req-41163346-bcea-4166-9d3f-ba7d87135158 50a742bdfe4e4a8ca0ddc150fc648904 648ffe15b9fe4a1bba71e4643d4dbbf2 - - -] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf cinder-rtstool create /dev/cinder-volumes/volume-9dea2617-2290-446f-a147-cf5b069b87db iqn.2010-10.org.openstack:volume-9dea2617-2290-446f-a147-cf5b069b87db uU43kJW7ydw849ortxXy 5E3uBj9dKUZHucZHoFMb execute /usr/lib/python2.7/site-packages/cinder/openstack/common/processutils.py:147 Cinder is retrieving a newly created target to verify it really exists: 2015-06-26 15:47:55.800 15846 DEBUG cinder.openstack.common.processutils [req-41163346-bcea-4166-9d3f-ba7d87135158 50a742bdfe4e4a8ca0ddc150fc648904 648ffe15b9fe4a1bba71e4643d4dbbf2 - - -] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf cinder-rtstool get-targets execute /usr/lib/python2.7/site-packages/cinder/openstack/common/processutils.py:147 And after a few other manipulations Cinder is finally saving the target configuration and returning the connection info back to Nova: 2015-06-26 15:47:56.020 15846 DEBUG cinder.openstack.common.processutils [req-41163346-bcea-4166-9d3f-ba7d87135158 50a742bdfe4e4a8ca0ddc150fc648904 648ffe15b9fe4a1bba71e4643d4dbbf2 - - -] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf cinder-rtstool save execute /usr/lib/python2.7/site-packages/cinder/openstack/common/processutils.py:147 The problem is that one step is missing in the middle, Cinder must add Nova host iSCSI initiator name to the target ACL or in other words, it must run cinder-rtstool add-initiator .... which is missing in this particular run. The code is there, we have to understand why it didn't run. We have detected a misconfiguration in their systems:
iscsi_helper = lioadm
lvm-osp1-compute03.iscsi_helper = tgtadm
Both should be set to lioadm.
They need to change the lvm driver configuration in /etc/cinder/cinder.conf and add the right iscsi_helper:
Search for:
[lvm-osp1-compute03]
And add this line right below:
iscsi_helper = lioadm
In previous comment I specifically mentioned compute03's configuration, but this actually needs to be checked, and changed if incorrect, in *ALL* nodes running Cinder with LVM to ensure the right target is set consistently. We provided a hotfix on July 1. Any feedback from the customer? We received an update and testing feedback from the customer on 9 July indicating their initial results were good however On 7 July started seeing some errors. It was not clear if they were still actually having problems or if they were just reporting the errors to us and so we asked for clarification. The customer contact has been out of the office and so we have yet to hear back. I'll push the full update from the customer momentarily. verified on openstack-cinder-2014.1.5-1.el7ost.noarch Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2015-1758.html |