Created attachment 1585501 [details] cinder-volume.log Description of problem: While verifying backport of Cinder multi attach support to OSP13 detach of isnt1 from a mulitattach volume succeeded, however detaching second instance (isnt2) from same volume fails, volume remains in "detaching" status. Version-Release number of selected component (if applicable): How reproducible: Unsure Steps to Reproduce: 1. Create a multi attached volume cinder type-create multiattach cinder type-key multiattach set volume_backend_name=3pariscsi_1 cinder type-key multiattach set multiattach="<is> True" openstack volume create --size 1 --type multiattach vol1 vol id - > c5d00d17-5eaa-49d1-aac6-5ecc2c429f2d 2. Boot two instances nova list +--------------------------------------+-------+--------+------------+-------------+--------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-------+--------+------------+-------------+--------------------+ | 438c5fcd-bd81-44c8-ad88-1f8ef0e1b50d | inst1 | ACTIVE | - | Running | sneha=10.50.9.122 | | fd2acc67-25ab-4a56-bc6b-d8bdf5732f6b | inst2 | ACTIVE | - | Running | default=172.20.1.6 | +--------------------------------------+-------+--------+------------+-------------+--------------------+ | 3. Attach volume to both instances cinder list +--------------------------------------+-----------+------------------------+------+-------------+----------+---------------------------------------------------------------------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+------------------------+------+-------------+----------+---------------------------------------------------------------------------+ | c5d00d17-5eaa-49d1-aac6-5ecc2c429f2d | in-use| vol1 | 1 | multiattach | false | 438c5fcd-bd81-44c8-ad88-1f8ef0e1b50d, fd2acc67-25ab-4a56-bc6b-d8bdf5732f6b | +--------------------------------------+-----------+------------------------+------+-------------+----------+---------------------------------------------------------------------------+ 4. Detach inst2 instance openstack server remove volume inst2 vol1 cinder list +--------------------------------------+-----------+------------------------+------+-------------+----------+---------------------------------------------------------------------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+------------------------+------+-------------+----------+---------------------------------------------------------------------------+ | c5d00d17-5eaa-49d1-aac6-5ecc2c429f2d | in-use| vol1 | 1 | multiattach | false | 438c5fcd-bd81-44c8-ad88-1f8ef0e1b50d | +--------------------------------------+-----------+------------------------+------+-------------+----------+---------------------------------------------------------------------------+ Volume detached successfully from inst2, as we see above it only remains attached to isnt1 (status in-use). 5. Detach inst1 instance openstack server remove volume inst1 vol1 cinder list +--------------------------------------+-----------+------------------------+------+-------------+----------+---------------------------------------------------------------------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+------------------------+------+-------------+----------+---------------------------------------------------------------------------+ | c5d00d17-5eaa-49d1-aac6-5ecc2c429f2d | detaching | vol1 | 1 | multiattach | false | 438c5fcd-bd81-44c8-ad88-1f8ef0e1b50d | +--------------------------------------+-----------+------------------------+------+-------------+----------+---------------------------------------------------------------------------+ Even after a few minutes volume status remains in detaching state. what I noticed in the cinder-volume.log is, when user sends a detach instance call for the first time it removes the host entry from 3PAR. Now since there is no LUN connectivity with 3PAR, further detach of instances fails. For detailed logs please check attached log file. Actual results: Nova volume detach fails to complete due to no host found error in cinder-volume.log Volume remains in detaching state. Expected results: Volume should successfully detach from second instance as it did from the first one. Additional info:
Host Entry in 3PAR which gets deleted after first detach call is made: showhost Id Name Persona ------------WWN/iSCSI_Name------------ Port 147 overcloud-compute-0 Generic-ALUA iqn.1994-05.com.redhat:adddf076872c 2:2:1
rpm -qa| grep openstack-cinder* openstack-cinder-12.0.6-3.el7ost.noarch
Not sure if it's the same root cause but sounds similar to LVM detach issue: https://bugzilla.redhat.com/show_bug.cgi?id=1701172
(In reply to Tzach Shefi from comment #5) > Not sure if it's the same root cause but sounds similar to LVM detach issue: > https://bugzilla.redhat.com/show_bug.cgi?id=1701172 It's the same test scenario, but not the same root cause. That bug was fixed in the LVM driver. I think this bug will need to be fixed in the 3par driver.
Fix in https://review.opendev.org/#/c/668793/
Fix https://review.opendev.org/#/c/668793/ merged in openstack master.
Verified on: openstack-cinder-12.0.7-4.0.bz1705563.el7ost.noarch On 3par iscsi, create a cinder multi-attach vol (overcloud) [stack@undercloud-0 ~]$ cinder create 1 --volume-type multiattach --name multiattach +--------------------------------+--------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | consistencygroup_id | None | | created_at | 2019-08-15T14:21:09.000000 | | description | None | | encrypted | False | | id | c50e5862-1382-4a53-9982-076def9d5c5c | | metadata | {} | | migration_status | None | | multiattach | True | | name | multiattach | | os-vol-host-attr:host | controller-0@3par#SSD_r5 | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | 67844cb7ae4a4d29ad599e53cdeec3f9 | | replication_status | None | | size | 1 | | snapshot_id | None | | source_volid | None | | status | creating | | updated_at | 2019-08-15T14:21:09.000000 | | user_id | 767dfd54ba6d49aebe01b7f4edb9725c | | volume_type | multiattach | +--------------------------------+--------------------------------------+ Attach to both instances: (overcloud) [stack@undercloud-0 ~]$ nova volume-attach inst1 c50e5862-1382-4a53-9982-076def9d5c5c auto +----------+--------------------------------------+ | Property | Value | +----------+--------------------------------------+ | device | /dev/vdb | | id | c50e5862-1382-4a53-9982-076def9d5c5c | | serverId | b3b27bbe-87f4-44ad-a846-a1f9363ef0cb | | volumeId | c50e5862-1382-4a53-9982-076def9d5c5c | +----------+--------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ nova volume-attach inst2 c50e5862-1382-4a53-9982-076def9d5c5c auto +----------+--------------------------------------+ | Property | Value | +----------+--------------------------------------+ | device | /dev/vdb | | id | c50e5862-1382-4a53-9982-076def9d5c5c | | serverId | b3b27bbe-87f4-44ad-a846-a1f9363ef0cb | | volumeId | c50e5862-1382-4a53-9982-076def9d5c5c | +----------+--------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ cinder list +--------------------------------------+-----------+-------------+------+-------------+----------+---------------------------------------------------------------------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+-------------+------+-------------+----------+---------------------------------------------------------------------------+ | 838922ce-0792-4a77-910d-70e7a49ecb8c | available | - | 1 | tripleo | false | | | c50e5862-1382-4a53-9982-076def9d5c5c | in-use | multiattach | 1 | multiattach | false | bf4b22f3-fd9d-4066-ad8e-f8e2dac3422e,b3b27bbe-87f4-44ad-a846-a1f9363ef0cb | +--------------------------------------+-----------+-------------+------+-------------+----------+---------------------------------------------------------------------------+ Detach vol from inst1 (overcloud) [stack@undercloud-0 ~]$ openstack server remove volume inst1 multiattach (overcloud) [stack@undercloud-0 ~]$ cinder list +--------------------------------------+-----------+-------------+------+-------------+----------+--------------------------------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+-------------+------+-------------+----------+--------------------------------------+ | 838922ce-0792-4a77-910d-70e7a49ecb8c | available | - | 1 | tripleo | false | | | c50e5862-1382-4a53-9982-076def9d5c5c | in-use | multiattach | 1 | multiattach | false | bf4b22f3-fd9d-4066-ad8e-f8e2dac3422e | +--------------------------------------+-----------+-------------+------+-------------+----------+--------------------------------------+ Detach vol from inst2 (overcloud) [stack@undercloud-0 ~]$ openstack server remove volume inst2 multiattach (overcloud) [stack@undercloud-0 ~]$ cinder list +--------------------------------------+-----------+-------------+------+-------------+----------+-------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+-------------+------+-------------+----------+-------------+ | 838922ce-0792-4a77-910d-70e7a49ecb8c | available | - | 1 | tripleo | false | | | c50e5862-1382-4a53-9982-076def9d5c5c | available | multiattach | 1 | multiattach | false | | +--------------------------------------+-----------+-------------+------+-------------+----------+-------------+ This worked, now I attached and inst1 and isnt2 back to same volume. This time I detached isnt2 first and then inst1 (overcloud) [stack@undercloud-0 ~]$ nova volume-attach inst1 c50e5862-1382-4a53-9982-076def9d5c5c auto +----------+--------------------------------------+ | Property | Value | +----------+--------------------------------------+ | device | /dev/vdb | | id | c50e5862-1382-4a53-9982-076def9d5c5c | | serverId | b3b27bbe-87f4-44ad-a846-a1f9363ef0cb | | volumeId | c50e5862-1382-4a53-9982-076def9d5c5c | +----------+--------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ nova volume-attach inst2 c50e5862-1382-4a53-9982-076def9d5c5c auto +----------+--------------------------------------+ | Property | Value | +----------+--------------------------------------+ | device | /dev/vdb | | id | c50e5862-1382-4a53-9982-076def9d5c5c | | serverId | bf4b22f3-fd9d-4066-ad8e-f8e2dac3422e | | volumeId | c50e5862-1382-4a53-9982-076def9d5c5c | +----------+--------------------------------------+ Both instances are attached: (overcloud) [stack@undercloud-0 ~]$ cinder list +--------------------------------------+-----------+-------------+------+-------------+----------+---------------------------------------------------------------------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+-------------+------+-------------+----------+---------------------------------------------------------------------------+ | 838922ce-0792-4a77-910d-70e7a49ecb8c | available | - | 1 | tripleo | false | | | c50e5862-1382-4a53-9982-076def9d5c5c | in-use | multiattach | 1 | multiattach | false | b3b27bbe-87f4-44ad-a846-a1f9363ef0cb,bf4b22f3-fd9d-4066-ad8e-f8e2dac3422e | +--------------------------------------+-----------+-------------+------+-------------+----------+---------------------------------------------------------------------------+ Now detach inst2 first (overcloud) [stack@undercloud-0 ~]$ openstack server remove volume inst2 multiattach (overcloud) [stack@undercloud-0 ~]$ cinder list +--------------------------------------+-----------+-------------+------+-------------+----------+--------------------------------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+-------------+------+-------------+----------+--------------------------------------+ | 838922ce-0792-4a77-910d-70e7a49ecb8c | available | - | 1 | tripleo | false | | | c50e5862-1382-4a53-9982-076def9d5c5c | in-use | multiattach | 1 | multiattach | false | b3b27bbe-87f4-44ad-a846-a1f9363ef0cb | +--------------------------------------+-----------+-------------+------+-------------+----------+--------------------------------------+ Now detach vol from isnt1 (overcloud) [stack@undercloud-0 ~]$ openstack server remove volume inst1 multiattach (overcloud) [stack@undercloud-0 ~]$ cinder list +--------------------------------------+-----------+-------------+------+-------------+----------+-------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+-------------+------+-------------+----------+-------------+ | 838922ce-0792-4a77-910d-70e7a49ecb8c | available | - | 1 | tripleo | false | | | c50e5862-1382-4a53-9982-076def9d5c5c | available | multiattach | 1 | multiattach | false | | +--------------------------------------+-----------+-------------+------+-------------+----------+-------------+ As can be seen above volume was detached from both instances, on both tests. Looks good to verify.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2627