Bug 1725012
| Summary: | [OSP13] Detaching second instance from a multiattached 3PAR volume leaves volume in detaching state | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Sneha Rai <sneha.rai> | ||||
| Component: | openstack-cinder | Assignee: | Alan Bishop <abishop> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Tzach Shefi <tshefi> | ||||
| Severity: | urgent | Docs Contact: | Chuck Copello <ccopello> | ||||
| Priority: | high | ||||||
| Version: | 13.0 (Queens) | CC: | abishop, cschwede, cswanson, eharney, ndeevy, pgrist, shdunne | ||||
| Target Milestone: | z8 | Keywords: | Triaged, ZStream | ||||
| Target Release: | 13.0 (Queens) | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | openstack-cinder-12.0.7-4.el7ost | Doc Type: | Bug Fix | ||||
| Doc Text: |
Previously, the Cinder HPE 3PAR driver did not correctly maintain the connection status of a volume when multiple instances were attached to the same volume. This caused a volume status to remain in the detaching state when a second instance was detached from a volume with multiple connections.
With this update, the multi-attach behavior of the 3PAR driver functions correctly.
|
Story Points: | --- | ||||
| Clone Of: | |||||||
| : | 1731275 (view as bug list) | Environment: | |||||
| Last Closed: | 2019-09-03 16:54:21 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | 1731275, 1731276 | ||||||
| Bug Blocks: | 1708705, 1713669 | ||||||
| Attachments: |
|
||||||
Host Entry in 3PAR which gets deleted after first detach call is made: showhost Id Name Persona ------------WWN/iSCSI_Name------------ Port 147 overcloud-compute-0 Generic-ALUA iqn.1994-05.com.redhat:adddf076872c 2:2:1 rpm -qa| grep openstack-cinder* openstack-cinder-12.0.6-3.el7ost.noarch Not sure if it's the same root cause but sounds similar to LVM detach issue: https://bugzilla.redhat.com/show_bug.cgi?id=1701172 (In reply to Tzach Shefi from comment #5) > Not sure if it's the same root cause but sounds similar to LVM detach issue: > https://bugzilla.redhat.com/show_bug.cgi?id=1701172 It's the same test scenario, but not the same root cause. That bug was fixed in the LVM driver. I think this bug will need to be fixed in the 3par driver. Fix https://review.opendev.org/#/c/668793/ merged in openstack master. Verified on: openstack-cinder-12.0.7-4.0.bz1705563.el7ost.noarch On 3par iscsi, create a cinder multi-attach vol (overcloud) [stack@undercloud-0 ~]$ cinder create 1 --volume-type multiattach --name multiattach +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | consistencygroup_id | None | | created_at | 2019-08-15T14:21:09.000000 | | description | None | | encrypted | False | | id | c50e5862-1382-4a53-9982-076def9d5c5c | | metadata | {} | | migration_status | None | | multiattach | True | | name | multiattach | | os-vol-host-attr:host | controller-0@3par#SSD_r5 | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | 67844cb7ae4a4d29ad599e53cdeec3f9 | | replication_status | None | | size | 1 | | snapshot_id | None | | source_volid | None | | status | creating | | updated_at | 2019-08-15T14:21:09.000000 | | user_id | 767dfd54ba6d49aebe01b7f4edb9725c | | volume_type | multiattach | +--------------------------------+--------------------------------------+ Attach to both instances: (overcloud) [stack@undercloud-0 ~]$ nova volume-attach inst1 c50e5862-1382-4a53-9982-076def9d5c5c auto +----------+--------------------------------------+ | Property | Value | +----------+--------------------------------------+ | device | /dev/vdb | | id | c50e5862-1382-4a53-9982-076def9d5c5c | | serverId | b3b27bbe-87f4-44ad-a846-a1f9363ef0cb | | volumeId | c50e5862-1382-4a53-9982-076def9d5c5c | +----------+--------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ nova volume-attach inst2 c50e5862-1382-4a53-9982-076def9d5c5c auto +----------+--------------------------------------+ | Property | Value | +----------+--------------------------------------+ | device | /dev/vdb | | id | c50e5862-1382-4a53-9982-076def9d5c5c | | serverId | b3b27bbe-87f4-44ad-a846-a1f9363ef0cb | | volumeId | c50e5862-1382-4a53-9982-076def9d5c5c | +----------+--------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ cinder list +--------------------------------------+-----------+-------------+------+-------------+----------+---------------------------------------------------------------------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+-------------+------+-------------+----------+---------------------------------------------------------------------------+ | 838922ce-0792-4a77-910d-70e7a49ecb8c | available | - | 1 | tripleo | false | | | c50e5862-1382-4a53-9982-076def9d5c5c | in-use | multiattach | 1 | multiattach | false | bf4b22f3-fd9d-4066-ad8e-f8e2dac3422e,b3b27bbe-87f4-44ad-a846-a1f9363ef0cb | +--------------------------------------+-----------+-------------+------+-------------+----------+---------------------------------------------------------------------------+ Detach vol from inst1 (overcloud) [stack@undercloud-0 ~]$ openstack server remove volume inst1 multiattach (overcloud) [stack@undercloud-0 ~]$ cinder list +--------------------------------------+-----------+-------------+------+-------------+----------+--------------------------------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+-------------+------+-------------+----------+--------------------------------------+ | 838922ce-0792-4a77-910d-70e7a49ecb8c | available | - | 1 | tripleo | false | | | c50e5862-1382-4a53-9982-076def9d5c5c | in-use | multiattach | 1 | multiattach | false | bf4b22f3-fd9d-4066-ad8e-f8e2dac3422e | +--------------------------------------+-----------+-------------+------+-------------+----------+--------------------------------------+ Detach vol from inst2 (overcloud) [stack@undercloud-0 ~]$ openstack server remove volume inst2 multiattach (overcloud) [stack@undercloud-0 ~]$ cinder list +--------------------------------------+-----------+-------------+------+-------------+----------+-------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+-------------+------+-------------+----------+-------------+ | 838922ce-0792-4a77-910d-70e7a49ecb8c | available | - | 1 | tripleo | false | | | c50e5862-1382-4a53-9982-076def9d5c5c | available | multiattach | 1 | multiattach | false | | +--------------------------------------+-----------+-------------+------+-------------+----------+-------------+ This worked, now I attached and inst1 and isnt2 back to same volume. This time I detached isnt2 first and then inst1 (overcloud) [stack@undercloud-0 ~]$ nova volume-attach inst1 c50e5862-1382-4a53-9982-076def9d5c5c auto +----------+--------------------------------------+ | Property | Value | +----------+--------------------------------------+ | device | /dev/vdb | | id | c50e5862-1382-4a53-9982-076def9d5c5c | | serverId | b3b27bbe-87f4-44ad-a846-a1f9363ef0cb | | volumeId | c50e5862-1382-4a53-9982-076def9d5c5c | +----------+--------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ nova volume-attach inst2 c50e5862-1382-4a53-9982-076def9d5c5c auto +----------+--------------------------------------+ | Property | Value | +----------+--------------------------------------+ | device | /dev/vdb | | id | c50e5862-1382-4a53-9982-076def9d5c5c | | serverId | bf4b22f3-fd9d-4066-ad8e-f8e2dac3422e | | volumeId | c50e5862-1382-4a53-9982-076def9d5c5c | +----------+--------------------------------------+ Both instances are attached: (overcloud) [stack@undercloud-0 ~]$ cinder list +--------------------------------------+-----------+-------------+------+-------------+----------+---------------------------------------------------------------------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+-------------+------+-------------+----------+---------------------------------------------------------------------------+ | 838922ce-0792-4a77-910d-70e7a49ecb8c | available | - | 1 | tripleo | false | | | c50e5862-1382-4a53-9982-076def9d5c5c | in-use | multiattach | 1 | multiattach | false | b3b27bbe-87f4-44ad-a846-a1f9363ef0cb,bf4b22f3-fd9d-4066-ad8e-f8e2dac3422e | +--------------------------------------+-----------+-------------+------+-------------+----------+---------------------------------------------------------------------------+ Now detach inst2 first (overcloud) [stack@undercloud-0 ~]$ openstack server remove volume inst2 multiattach (overcloud) [stack@undercloud-0 ~]$ cinder list +--------------------------------------+-----------+-------------+------+-------------+----------+--------------------------------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+-------------+------+-------------+----------+--------------------------------------+ | 838922ce-0792-4a77-910d-70e7a49ecb8c | available | - | 1 | tripleo | false | | | c50e5862-1382-4a53-9982-076def9d5c5c | in-use | multiattach | 1 | multiattach | false | b3b27bbe-87f4-44ad-a846-a1f9363ef0cb | +--------------------------------------+-----------+-------------+------+-------------+----------+--------------------------------------+ Now detach vol from isnt1 (overcloud) [stack@undercloud-0 ~]$ openstack server remove volume inst1 multiattach (overcloud) [stack@undercloud-0 ~]$ cinder list +--------------------------------------+-----------+-------------+------+-------------+----------+-------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+-------------+------+-------------+----------+-------------+ | 838922ce-0792-4a77-910d-70e7a49ecb8c | available | - | 1 | tripleo | false | | | c50e5862-1382-4a53-9982-076def9d5c5c | available | multiattach | 1 | multiattach | false | | +--------------------------------------+-----------+-------------+------+-------------+----------+-------------+ As can be seen above volume was detached from both instances, on both tests. Looks good to verify. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2627 |
Created attachment 1585501 [details] cinder-volume.log Description of problem: While verifying backport of Cinder multi attach support to OSP13 detach of isnt1 from a mulitattach volume succeeded, however detaching second instance (isnt2) from same volume fails, volume remains in "detaching" status. Version-Release number of selected component (if applicable): How reproducible: Unsure Steps to Reproduce: 1. Create a multi attached volume cinder type-create multiattach cinder type-key multiattach set volume_backend_name=3pariscsi_1 cinder type-key multiattach set multiattach="<is> True" openstack volume create --size 1 --type multiattach vol1 vol id - > c5d00d17-5eaa-49d1-aac6-5ecc2c429f2d 2. Boot two instances nova list +--------------------------------------+-------+--------+------------+-------------+--------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-------+--------+------------+-------------+--------------------+ | 438c5fcd-bd81-44c8-ad88-1f8ef0e1b50d | inst1 | ACTIVE | - | Running | sneha=10.50.9.122 | | fd2acc67-25ab-4a56-bc6b-d8bdf5732f6b | inst2 | ACTIVE | - | Running | default=172.20.1.6 | +--------------------------------------+-------+--------+------------+-------------+--------------------+ | 3. Attach volume to both instances cinder list +--------------------------------------+-----------+------------------------+------+-------------+----------+---------------------------------------------------------------------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+------------------------+------+-------------+----------+---------------------------------------------------------------------------+ | c5d00d17-5eaa-49d1-aac6-5ecc2c429f2d | in-use| vol1 | 1 | multiattach | false | 438c5fcd-bd81-44c8-ad88-1f8ef0e1b50d, fd2acc67-25ab-4a56-bc6b-d8bdf5732f6b | +--------------------------------------+-----------+------------------------+------+-------------+----------+---------------------------------------------------------------------------+ 4. Detach inst2 instance openstack server remove volume inst2 vol1 cinder list +--------------------------------------+-----------+------------------------+------+-------------+----------+---------------------------------------------------------------------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+------------------------+------+-------------+----------+---------------------------------------------------------------------------+ | c5d00d17-5eaa-49d1-aac6-5ecc2c429f2d | in-use| vol1 | 1 | multiattach | false | 438c5fcd-bd81-44c8-ad88-1f8ef0e1b50d | +--------------------------------------+-----------+------------------------+------+-------------+----------+---------------------------------------------------------------------------+ Volume detached successfully from inst2, as we see above it only remains attached to isnt1 (status in-use). 5. Detach inst1 instance openstack server remove volume inst1 vol1 cinder list +--------------------------------------+-----------+------------------------+------+-------------+----------+---------------------------------------------------------------------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+------------------------+------+-------------+----------+---------------------------------------------------------------------------+ | c5d00d17-5eaa-49d1-aac6-5ecc2c429f2d | detaching | vol1 | 1 | multiattach | false | 438c5fcd-bd81-44c8-ad88-1f8ef0e1b50d | +--------------------------------------+-----------+------------------------+------+-------------+----------+---------------------------------------------------------------------------+ Even after a few minutes volume status remains in detaching state. what I noticed in the cinder-volume.log is, when user sends a detach instance call for the first time it removes the host entry from 3PAR. Now since there is no LUN connectivity with 3PAR, further detach of instances fails. For detailed logs please check attached log file. Actual results: Nova volume detach fails to complete due to no host found error in cinder-volume.log Volume remains in detaching state. Expected results: Volume should successfully detach from second instance as it did from the first one. Additional info: