Bug 1731276 - [OSP14] Detaching second instance from a multiattached 3PAR volume leaves volume in detaching state
Summary: [OSP14] Detaching second instance from a multiattached 3PAR volume leaves vol...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder
Version: 14.0 (Rocky)
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: z4
: 14.0 (Rocky)
Assignee: Alan Bishop
QA Contact: Tzach Shefi
Chuck Copello
URL:
Whiteboard:
Depends On: 1731275
Blocks: 1713669 1708705 1725012
TreeView+ depends on / blocked
 
Reported: 2019-07-18 20:48 UTC by Alan Bishop
Modified: 2019-09-03 15:22 UTC (History)
10 users (show)

Fixed In Version: openstack-cinder-13.0.5-5.el7ost
Doc Type: Bug Fix
Doc Text:
This patch fixes the 3PAR driver's attach/detach code that under certain circumstances, would fail to maintain a volume's correct connection status when multiple instances are attached.
Clone Of: 1731275
Environment:
Last Closed: 2019-09-03 15:22:23 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Launchpad 1834660 None None None 2019-07-18 20:48:43 UTC
OpenStack gerrit 668793 None None None 2019-07-18 20:48:43 UTC
Red Hat Product Errata RHBA-2019:2618 None None None 2019-09-03 15:22:29 UTC

Description Alan Bishop 2019-07-18 20:48:44 UTC
+++ This bug was initially created as a clone of Bug #1731275 +++

+++ This bug was initially created as a clone of Bug #1725012 +++

Description of problem:
While verifying backport of Cinder multi attach support to OSP13 detach of isnt1 from a mulitattach volume succeeded, however detaching second instance (isnt2) from same volume fails, volume remains in "detaching" status.

Version-Release number of selected component (if applicable):


How reproducible:
Unsure

Steps to Reproduce:
1. Create a multi attached volume

cinder type-create multiattach
cinder type-key multiattach set volume_backend_name=3pariscsi_1
cinder type-key multiattach set multiattach="<is> True"
openstack volume create --size 1 --type multiattach vol1 
vol id - >  c5d00d17-5eaa-49d1-aac6-5ecc2c429f2d 

2. Boot two instances
nova list
+--------------------------------------+-------+--------+------------+-------------+--------------------+
| ID                                   | Name  | Status | Task State | Power State | Networks           |
+--------------------------------------+-------+--------+------------+-------------+--------------------+
| 438c5fcd-bd81-44c8-ad88-1f8ef0e1b50d | inst1 | ACTIVE | -          | Running     | sneha=10.50.9.122  |
| fd2acc67-25ab-4a56-bc6b-d8bdf5732f6b | inst2 | ACTIVE | -          | Running     | default=172.20.1.6 |
+--------------------------------------+-------+--------+------------+-------------+--------------------+           |


3. Attach volume to both instances

cinder list
+--------------------------------------+-----------+------------------------+------+-------------+----------+---------------------------------------------------------------------------+
| ID                                   | Status    | Name                   | Size | Volume Type | Bootable | Attached to                                                               |
+--------------------------------------+-----------+------------------------+------+-------------+----------+---------------------------------------------------------------------------+
| c5d00d17-5eaa-49d1-aac6-5ecc2c429f2d | in-use| vol1                   | 1    | multiattach | false    | 438c5fcd-bd81-44c8-ad88-1f8ef0e1b50d, fd2acc67-25ab-4a56-bc6b-d8bdf5732f6b |
+--------------------------------------+-----------+------------------------+------+-------------+----------+---------------------------------------------------------------------------+





4. Detach inst2 instance

openstack server remove volume inst2 vol1

cinder list
+--------------------------------------+-----------+------------------------+------+-------------+----------+---------------------------------------------------------------------------+
| ID                                   | Status    | Name                   | Size | Volume Type | Bootable | Attached to                                                               |
+--------------------------------------+-----------+------------------------+------+-------------+----------+---------------------------------------------------------------------------+
| c5d00d17-5eaa-49d1-aac6-5ecc2c429f2d | in-use| vol1                   | 1    | multiattach | false    | 438c5fcd-bd81-44c8-ad88-1f8ef0e1b50d |
+--------------------------------------+-----------+------------------------+------+-------------+----------+---------------------------------------------------------------------------+

Volume detached successfully from inst2, as we see above it only remains attached to isnt1 (status in-use).  


5. Detach inst1 instance 
openstack server remove volume inst1 vol1

cinder list
+--------------------------------------+-----------+------------------------+------+-------------+----------+---------------------------------------------------------------------------+
| ID                                   | Status    | Name                   | Size | Volume Type | Bootable | Attached to                                                               |
+--------------------------------------+-----------+------------------------+------+-------------+----------+---------------------------------------------------------------------------+
| c5d00d17-5eaa-49d1-aac6-5ecc2c429f2d | detaching | vol1                   | 1    | multiattach | false    | 438c5fcd-bd81-44c8-ad88-1f8ef0e1b50d                                      |
+--------------------------------------+-----------+------------------------+------+-------------+----------+---------------------------------------------------------------------------+

Even after a few minutes volume status remains in detaching state. 

what I noticed in the cinder-volume.log is, when user sends a detach instance call for the first time it removes the host entry from 3PAR. Now since there is no LUN connectivity with 3PAR, further detach of instances fails. For detailed logs please check attached log file.

Actual results:
Nova volume detach fails to complete due to no host found error in cinder-volume.log
Volume remains in detaching state. 

Expected results:
Volume should successfully detach from second instance as it did from the first one. 

Additional info:

Comment 8 Tzach Shefi 2019-08-20 08:49:37 UTC
Verified on:
openstack-cinder-13.0.5-5.el7ost.noarch

Configure 3par iscsi as Cinder's backend:
(overcloud) [stack@undercloud-0 ~]$ cinder service-list
+------------------+-------------------------+------+---------+-------+----------------------------+-----------------+
| Binary           | Host                    | Zone | Status  | State | Updated_at                 | Disabled Reason |
+------------------+-------------------------+------+---------+-------+----------------------------+-----------------+
| cinder-backup    | controller-1            | nova | enabled | up    | 2019-08-20T07:36:14.000000 | -               |
| cinder-scheduler | controller-0            | nova | enabled | up    | 2019-08-20T07:36:14.000000 | -               |
| cinder-scheduler | controller-1            | nova | enabled | up    | 2019-08-20T07:36:12.000000 | -               |
| cinder-scheduler | controller-2            | nova | enabled | up    | 2019-08-20T07:36:08.000000 | -               |
| cinder-volume    | controller-0@3par       | nova | enabled | up    | 2019-08-20T07:36:13.000000 | -               |




(overcloud) [stack@undercloud-0 ~]$ cinder type-create multiattach                                                
+--------------------------------------+-------------+-------------+-----------+                                  
| ID                                   | Name        | Description | Is_Public |                                  
+--------------------------------------+-------------+-------------+-----------+                                  
| 785fb68c-7b2e-453c-aca4-68d16dc28fab | multiattach | -           | True      |                                  
+--------------------------------------+-------------+-------------+-----------+                                  
(overcloud) [stack@undercloud-0 ~]$ cinder type-key multiattach set multiattach="<is> True"                                           
(overcloud) [stack@undercloud-0 ~]$ cinder type-key multiattach set volume_backend_name=3par                      
(overcloud) [stack@undercloud-0 ~]$ cinder create 1 --volume-type multiattach
+--------------------------------+--------------------------------------+
| Property                       | Value                                |
+--------------------------------+--------------------------------------+
| attachments                    | []                                   |
| availability_zone              | nova                                 |
| bootable                       | false                                |
| consistencygroup_id            | None                                 |
| created_at                     | 2019-08-20T07:35:12.000000           |
| description                    | None                                 |
| encrypted                      | False                                |
| id                             | 5cbc4920-6bbe-4045-981c-e298dfb6a85b |
| metadata                       | {}                                   |
| migration_status               | None                                 |
| multiattach                    | True                                 |
| name                           | None                                 |
| os-vol-host-attr:host          | controller-0@3par#SSD_r5             |
| os-vol-mig-status-attr:migstat | None                                 |
| os-vol-mig-status-attr:name_id | None                                 |
| os-vol-tenant-attr:tenant_id   | 5d709c2af9044a9d9fbee83ee81157a2     |
| replication_status             | None                                 |
| size                           | 1                                    |
| snapshot_id                    | None                                 |
| source_volid                   | None                                 |
| status                         | available                            |
| updated_at                     | 2019-08-20T07:35:13.000000           |
| user_id                        | 6f3cf8f602e64e4dbf94c4a9ffc0c122     |
| volume_type                    | multiattach                          |
+--------------------------------+--------------------------------------+


Boot two instances:
(overcloud) [stack@undercloud-0 ~]$ nova list
+--------------------------------------+-------+--------+------------+-------------+-----------------------------------+
| ID                                   | Name  | Status | Task State | Power State | Networks                          |
+--------------------------------------+-------+--------+------------+-------------+-----------------------------------+
| d21f8b47-54a2-4948-8ece-793797bf64ab | inst1 | ACTIVE | -          | Running     | internal=192.168.0.16, 10.0.0.211 |
| 975aea99-d6a6-4f25-ad81-c60192e1dc32 | inst2 | ACTIVE | -          | Running     | internal=192.168.0.20, 10.0.0.215 |
+--------------------------------------+-------+--------+------------+-------------+-----------------------------------+


Attach said volume to both instances:
(overcloud) [stack@undercloud-0 ~]$ nova volume-attach inst1 5cbc4920-6bbe-4045-981c-e298dfb6a85b auto
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/vdb                             |
| id       | 5cbc4920-6bbe-4045-981c-e298dfb6a85b |
| serverId | d21f8b47-54a2-4948-8ece-793797bf64ab |
| volumeId | 5cbc4920-6bbe-4045-981c-e298dfb6a85b |
+----------+--------------------------------------+

(overcloud) [stack@undercloud-0 ~]$ nova volume-attach inst2 5cbc4920-6bbe-4045-981c-e298dfb6a85b auto
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/vdb                             |
| id       | 5cbc4920-6bbe-4045-981c-e298dfb6a85b |
| serverId | 975aea99-d6a6-4f25-ad81-c60192e1dc32 |
| volumeId | 5cbc4920-6bbe-4045-981c-e298dfb6a85b |
+----------+--------------------------------------+

Both instances attached to same volume:
(overcloud) [stack@undercloud-0 ~]$ cinder list
+--------------------------------------+--------+------+------+-------------+----------+---------------------------------------------------------------------------+
| ID                                   | Status | Name | Size | Volume Type | Bootable | Attached to                                                               |
+--------------------------------------+--------+------+------+-------------+----------+---------------------------------------------------------------------------+
| 5cbc4920-6bbe-4045-981c-e298dfb6a85b | in-use | -    | 1    | multiattach | false    | 975aea99-d6a6-4f25-ad81-c60192e1dc32,d21f8b47-54a2-4948-8ece-793797bf64ab |
+--------------------------------------+--------+------+------+-------------+----------+---------------------------------------------------------------------------+



Detach inst2 from volume first:
(overcloud) [stack@undercloud-0 ~]$ nova volume-detach inst2 5cbc4920-6bbe-4045-981c-e298dfb6a85b

inst2 detached, only isnt1 remains attached
(overcloud) [stack@undercloud-0 ~]$ cinder list
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| ID                                   | Status | Name | Size | Volume Type | Bootable | Attached to                          |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| 5cbc4920-6bbe-4045-981c-e298dfb6a85b | in-use | -    | 1    | multiattach | false    | d21f8b47-54a2-4948-8ece-793797bf64ab |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+


Now detach inst1

(overcloud) [stack@undercloud-0 ~]$ nova volume-detach inst1 5cbc4920-6bbe-4045-981c-e298dfb6a85b 
(overcloud) [stack@undercloud-0 ~]$ cinder list
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| ID                                   | Status    | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| 5cbc4920-6bbe-4045-981c-e298dfb6a85b | available | -    | 1    | multiattach | false    |             |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+

Successfully detach both instances from multiattach volume 3par iscsi volume. 

Good to verify.

Comment 10 errata-xmlrpc 2019-09-03 15:22:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2618


Note You need to log in before you can comment on or make changes to this bug.