Bug 1854210

Summary:	[OSP13] Failing to detach a multi-attached netapp iscsi volume
Product:	Red Hat OpenStack	Reporter:	Alan Bishop <abishop>
Component:	openstack-cinder	Assignee:	Alan Bishop <abishop>
Status:	CLOSED NOTABUG	QA Contact:	Tzach Shefi <tshefi>
Severity:	medium	Docs Contact:	Chuck Copello <ccopello>
Priority:	medium
Version:	13.0 (Queens)	CC:	abishop, akarlsso, apevec, dasmith, eglynn, eharney, gcase, jhakimra, jschluet, kchamart, lhh, sbauza, sgordon, tshefi, vromanso
Target Milestone:	z13	Keywords:	Triaged, ZStream
Target Release:	13.0 (Queens)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-cinder-12.0.10-16.el7ost	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1729898	Environment:
Last Closed:	2020-10-05 13:31:27 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1729898
Bug Blocks:

Description Alan Bishop 2020-07-06 18:49:27 UTC

+++ This bug was initially created as a clone of Bug #1729898 +++

Description of problem: While testing Cinder multi-attach volume, I'd attached a volume to three instance, while detaching volume worked fine from 2 instances the third instance failed to detach, noticed some  multipath errors on compute's nova.log see end of bz and attached log. 


Version-Release number of selected component (if applicable):
puppet-nova-14.4.1-0.20190605170411.17663a5.el8ost.noarch
openstack-nova-common-19.0.2-0.20190616040418.acd2daa.el8ost.noarch
openstack-nova-compute-19.0.2-0.20190616040418.acd2daa.el8ost.noarch
python3-nova-19.0.2-0.20190616040418.acd2daa.el8ost.noarch
python3-novaclient-13.0.1-0.20190617080642.ef842ca.el8ost.noarch
openstack-nova-migration-19.0.2-0.20190616040418.acd2daa.el8ost.noarch

python3-os-brick-2.8.1-0.20190617142514.05b6c9f.el8ost.noarch

puppet-cinder-14.4.1-0.20190420083336.1cf0604.el8ost.noarch
openstack-cinder-14.0.1-0.20190607000407.23d1a72.el8ost.noarch
python3-cinderclient-4.2.0-0.20190520060354.953243d.el8ost.noarch
python3-cinder-14.0.1-0.20190607000407.23d1a72.el8ost.noarch


How reproducible:
Unsure first time I hit this, 
if I try to detach volume again it fails over and over the same why. 

Steps to Reproduce:
1. I'd created a multi-attached netapp iscsi backed volume:

(overcloud) [stack@undercloud-0 ~]$ cinder show 5e1eed64-b107-4737-b5a4-ed37265fbb7e
+--------------------------------+------------------------------------------+
| Property                       | Value                                    |
+--------------------------------+------------------------------------------+
| attached_servers               | ['a4946e59-08e2-41a9-b00a-d052a6a33d5e'] |
| attachment_ids                 | ['f59237c1-b3d2-4c20-b826-8ddf695fae12'] |
| availability_zone              | nova                                     |
| bootable                       | false                                    |
| consistencygroup_id            | None                                     |
| created_at                     | 2019-07-14T08:30:30.000000               |
| description                    | None                                     |
| encrypted                      | False                                    |
| id                             | 5e1eed64-b107-4737-b5a4-ed37265fbb7e     |
| metadata                       | readonly : True                          |
| migration_status               | None                                     |
| multiattach                    | True                                     |
| name                           | MA-netapp-vol-RW                         |
| os-vol-host-attr:host          | hostgroup@netapp#rhos_infra_tlv2         |
| os-vol-mig-status-attr:migstat | None                                     |
| os-vol-mig-status-attr:name_id | None                                     |
| os-vol-tenant-attr:tenant_id   | 5850830f52774e0dab7b7b8e508b6a56         |
| readonly                       | True                                     |
| replication_status             | None                                     |
| size                           | 1                                        |
| snapshot_id                    | None                                     |
| source_volid                   | None                                     |
| status                         | in-use                                   |
| updated_at                     | 2019-07-15T08:15:08.000000               |
| user_id                        | 8fe58395864a43158a847d1a9ffd4e9d         |
| volume_type                    | netapp-ma                                |
+--------------------------------+------------------------------------------+

2. Attached it to three instances:

| 5e1eed64-b107-4737-b5a4-ed37265fbb7e | in-use    | MA-netapp-vol-RW | 1    | netapp-ma   | false    | 6e80a37d-a052-4a8b-b041-d0e7c46ab6eb,a4946e59-08e2-41a9-b00a-d052a6a33d5e,cae1a6bd-1199-4010-8e04-e46aa50c27a3 |



3.  When I tried to detach it detached fine from 
inst2 and isnt3, however from inst1  it fails to detach:
nova volume-detach a4946e59-08e2-41a9-b00a-d052a6a33d5e 5e1eed64-b107-4737-b5a4-ed37265fbb7e

The volume reaches detaching state but after timeout as it fails to detach it returns to in-use state. 


| 5e1eed64-b107-4737-b5a4-ed37265fbb7e | detaching | MA-netapp-vol-RW | 1    | netapp-ma   | false    | a4946e59-08e2-41a9-b00a-d052a6a33d5e |



| 5e1eed64-b107-4737-b5a4-ed37265fbb7e | in-use    | MA-netapp-vol-RW | 1    | netapp-ma   | false    | a4946e59-08e2-41a9-b00a-d052a6a33d5e |



Actual results:
Volume fails to detach

Expected results:
Volume should detach

Comment 9 Tzach Shefi 2020-10-04 14:18:49 UTC

Alan, 
Are we sure netapp/OSP13 queens support multi-attach? 
I recall the MA RFE only landed on osp15
https://bugzilla.redhat.com/show_bug.cgi?id=1661022


Here is my cinder vol type

| a6ed1692-83cb-418a-9208-f743d56626e6 | netappiscsi | {'volume_backend_name': 'netapp', 'multiattach': '<is> True'} |


And when I try to create a MA volume

Filtering removed all hosts for the request with volume ID '776c5fa5-6201-48ef-8c47-629ba55d2b4c'. Filter results: AvailabilityZoneFilter: (start: 1, end: 1), CapacityFilter: (start: 1, end: 1), CapabilitiesFilter: (start: 1, end: 0)
/var/log/containers/cinder/cinder-scheduler.log:2406:2020-10-04 09:14:59.918 8 DEBUG cinder.volume.flows.common [req-39dd88f3-1c2c-4c54-9c52-b530ab51f910 01f4435ad92f4a6d8e1cea9651ec4229 ed9eea377e1644d4985647c1204bd696 - default default] Setting Volume 776c5fa5-6201-48ef-8c47-629ba55d2b4c to error due to: No valid backend was found. No weighed backends available error_out /usr/lib/python2.7/site-packages/cinder/volume/flows/common.py:83

I suspect that is why CapabilitiesFilter: (start: 1, end: 0) 
Unless I had messed up some config which is totally possible.
A netapp iscsi volume, pre setting the MA key, worked fine so basic vol was created OK.

Comment 10 Alan Bishop 2020-10-05 13:31:27 UTC

Ouch, you're absolutely correct Tzach. I think I accidentally confused this with another backend that -does- support multiattach in OSP-13.

I reviewed this patch and am confident it will not cause a regression in our OSP-13 netapp driver (no need to revert the patch). I'm closing this BZ "NOTABUG" and dropping the release note.