Bug 1854210 - [OSP13] Failing to detach a multi-attached netapp iscsi volume
Summary: [OSP13] Failing to detach a multi-attached netapp iscsi volume
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: z13
: 13.0 (Queens)
Assignee: Alan Bishop
QA Contact: Tzach Shefi
Chuck Copello
URL:
Whiteboard:
Depends On: 1729898
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-07-06 18:49 UTC by Alan Bishop
Modified: 2020-12-21 19:23 UTC (History)
15 users (show)

Fixed In Version: openstack-cinder-12.0.10-16.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1729898
Environment:
Last Closed: 2020-10-05 13:31:27 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1839384 0 None None None 2020-07-06 21:49:02 UTC
OpenStack gerrit 725135 0 None MERGED NetApp ONTAP: Fix iSCSI multiattach volume terminates connection 2021-01-19 20:51:30 UTC

Description Alan Bishop 2020-07-06 18:49:27 UTC
+++ This bug was initially created as a clone of Bug #1729898 +++

Description of problem: While testing Cinder multi-attach volume, I'd attached a volume to three instance, while detaching volume worked fine from 2 instances the third instance failed to detach, noticed some  multipath errors on compute's nova.log see end of bz and attached log. 


Version-Release number of selected component (if applicable):
puppet-nova-14.4.1-0.20190605170411.17663a5.el8ost.noarch
openstack-nova-common-19.0.2-0.20190616040418.acd2daa.el8ost.noarch
openstack-nova-compute-19.0.2-0.20190616040418.acd2daa.el8ost.noarch
python3-nova-19.0.2-0.20190616040418.acd2daa.el8ost.noarch
python3-novaclient-13.0.1-0.20190617080642.ef842ca.el8ost.noarch
openstack-nova-migration-19.0.2-0.20190616040418.acd2daa.el8ost.noarch

python3-os-brick-2.8.1-0.20190617142514.05b6c9f.el8ost.noarch

puppet-cinder-14.4.1-0.20190420083336.1cf0604.el8ost.noarch
openstack-cinder-14.0.1-0.20190607000407.23d1a72.el8ost.noarch
python3-cinderclient-4.2.0-0.20190520060354.953243d.el8ost.noarch
python3-cinder-14.0.1-0.20190607000407.23d1a72.el8ost.noarch


How reproducible:
Unsure first time I hit this, 
if I try to detach volume again it fails over and over the same why. 

Steps to Reproduce:
1. I'd created a multi-attached netapp iscsi backed volume:

(overcloud) [stack@undercloud-0 ~]$ cinder show 5e1eed64-b107-4737-b5a4-ed37265fbb7e
+--------------------------------+------------------------------------------+
| Property                       | Value                                    |
+--------------------------------+------------------------------------------+
| attached_servers               | ['a4946e59-08e2-41a9-b00a-d052a6a33d5e'] |
| attachment_ids                 | ['f59237c1-b3d2-4c20-b826-8ddf695fae12'] |
| availability_zone              | nova                                     |
| bootable                       | false                                    |
| consistencygroup_id            | None                                     |
| created_at                     | 2019-07-14T08:30:30.000000               |
| description                    | None                                     |
| encrypted                      | False                                    |
| id                             | 5e1eed64-b107-4737-b5a4-ed37265fbb7e     |
| metadata                       | readonly : True                          |
| migration_status               | None                                     |
| multiattach                    | True                                     |
| name                           | MA-netapp-vol-RW                         |
| os-vol-host-attr:host          | hostgroup@netapp#rhos_infra_tlv2         |
| os-vol-mig-status-attr:migstat | None                                     |
| os-vol-mig-status-attr:name_id | None                                     |
| os-vol-tenant-attr:tenant_id   | 5850830f52774e0dab7b7b8e508b6a56         |
| readonly                       | True                                     |
| replication_status             | None                                     |
| size                           | 1                                        |
| snapshot_id                    | None                                     |
| source_volid                   | None                                     |
| status                         | in-use                                   |
| updated_at                     | 2019-07-15T08:15:08.000000               |
| user_id                        | 8fe58395864a43158a847d1a9ffd4e9d         |
| volume_type                    | netapp-ma                                |
+--------------------------------+------------------------------------------+

2. Attached it to three instances:

| 5e1eed64-b107-4737-b5a4-ed37265fbb7e | in-use    | MA-netapp-vol-RW | 1    | netapp-ma   | false    | 6e80a37d-a052-4a8b-b041-d0e7c46ab6eb,a4946e59-08e2-41a9-b00a-d052a6a33d5e,cae1a6bd-1199-4010-8e04-e46aa50c27a3 |



3.  When I tried to detach it detached fine from 
inst2 and isnt3, however from inst1  it fails to detach:
nova volume-detach a4946e59-08e2-41a9-b00a-d052a6a33d5e 5e1eed64-b107-4737-b5a4-ed37265fbb7e

The volume reaches detaching state but after timeout as it fails to detach it returns to in-use state. 


| 5e1eed64-b107-4737-b5a4-ed37265fbb7e | detaching | MA-netapp-vol-RW | 1    | netapp-ma   | false    | a4946e59-08e2-41a9-b00a-d052a6a33d5e |



| 5e1eed64-b107-4737-b5a4-ed37265fbb7e | in-use    | MA-netapp-vol-RW | 1    | netapp-ma   | false    | a4946e59-08e2-41a9-b00a-d052a6a33d5e |



Actual results:
Volume fails to detach

Expected results:
Volume should detach

Comment 9 Tzach Shefi 2020-10-04 14:18:49 UTC
Alan, 
Are we sure netapp/OSP13 queens support multi-attach? 
I recall the MA RFE only landed on osp15
https://bugzilla.redhat.com/show_bug.cgi?id=1661022


Here is my cinder vol type

| a6ed1692-83cb-418a-9208-f743d56626e6 | netappiscsi | {'volume_backend_name': 'netapp', 'multiattach': '<is> True'} |


And when I try to create a MA volume

Filtering removed all hosts for the request with volume ID '776c5fa5-6201-48ef-8c47-629ba55d2b4c'. Filter results: AvailabilityZoneFilter: (start: 1, end: 1), CapacityFilter: (start: 1, end: 1), CapabilitiesFilter: (start: 1, end: 0)
/var/log/containers/cinder/cinder-scheduler.log:2406:2020-10-04 09:14:59.918 8 DEBUG cinder.volume.flows.common [req-39dd88f3-1c2c-4c54-9c52-b530ab51f910 01f4435ad92f4a6d8e1cea9651ec4229 ed9eea377e1644d4985647c1204bd696 - default default] Setting Volume 776c5fa5-6201-48ef-8c47-629ba55d2b4c to error due to: No valid backend was found. No weighed backends available error_out /usr/lib/python2.7/site-packages/cinder/volume/flows/common.py:83

I suspect that is why CapabilitiesFilter: (start: 1, end: 0) 
Unless I had messed up some config which is totally possible.
A netapp iscsi volume, pre setting the MA key, worked fine so basic vol was created OK.

Comment 10 Alan Bishop 2020-10-05 13:31:27 UTC
Ouch, you're absolutely correct Tzach. I think I accidentally confused this with another backend that -does- support multiattach in OSP-13.

I reviewed this patch and am confident it will not cause a regression in our OSP-13 netapp driver (no need to revert the patch). I'm closing this BZ "NOTABUG" and dropping the release note.


Note You need to log in before you can comment on or make changes to this bug.