Bug 1710373
| Summary: | [OSP14] live-migration fails with NoFibreChannelVolumeDeviceFound due to FC case sensitive scanning | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Alan Bishop <abishop> | |
| Component: | python-os-brick | Assignee: | Alan Bishop <abishop> | |
| Status: | CLOSED ERRATA | QA Contact: | Tzach Shefi <tshefi> | |
| Severity: | urgent | Docs Contact: | ||
| Priority: | urgent | |||
| Version: | 14.0 (Rocky) | CC: | abishop, akaris, apevec, cinder-bugs, dasmith, dhill, eglynn, eharney, geguileo, gkumar, jfindysz, jhakimra, jkalliya, jobernar, jschluet, kbhaskar, kchamart, lhh, lyarwood, madgupta, nlevinki, pcaruana, pgrist, rheslop, sbandyop, sbauza, sgordon, supadhya, tenobreg, tshefi, vromanso | |
| Target Milestone: | z3 | Keywords: | Triaged, ZStream | |
| Target Release: | 14.0 (Rocky) | |||
| Hardware: | x86_64 | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | python-os-brick-2.5.5-2.el7ost | Doc Type: | Bug Fix | |
| Doc Text: |
Previously, the OpenStack os-brick library discovered fiber channel connections by scanning for lower case WWNs. This would lead to live-migration errors with storage back ends that report their WWN in upper case. Now, os-brick recognizes both upper and lower case WWNs.
|
Story Points: | --- | |
| Clone Of: | 1709988 | |||
| : | 1710375 (view as bug list) | Environment: | ||
| Last Closed: | 2019-07-02 20:08:47 UTC | Type: | --- | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1709988, 1710375 | |||
|
Description
Alan Bishop
2019-05-15 12:49:24 UTC
Verified on:
python2-os-brick-2.5.6-1.el7ost.noarch
On a system with 3par FC Cinder backend, I'd booted an instance on compute-1:
(overcloud) [stack@puma52 ~]$ nova show 069bd786-bd41-40ce-952f-3765cc049c20
+--------------------------------------+----------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | compute-1.localdomain
Created a volume, from an image so as to have some data in it.
(overcloud) [stack@puma52 ~]$ cinder create 1 --image cirros --name 3parFCVol
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2019-06-08T03:37:29.000000 |
| description | None |
| encrypted | False |
| id | a9cab76d-69d4-45c5-a321-bee45f89d2f6 |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | 3parFCVol |
| os-vol-host-attr:host | controller-0@3parfc#SSD_r5 |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | aa14945a4e6e4dc8ba7471d2c7d73c06 |
| replication_status | None |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| updated_at | 2019-06-08T03:37:30.000000 |
| user_id | b57c5d2f15b142439df17f09ef157dcd |
| volume_type | tripleo |
+--------------------------------+--------------------------------------+
(overcloud) [stack@puma52 ~]$ cinder list
+--------------------------------------+-----------+-----------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+-----------+------+-------------+----------+-------------+
| a9cab76d-69d4-45c5-a321-bee45f89d2f6 | available | 3parFCVol | 1 | tripleo | true | |
Attach volume to instance
(overcloud) [stack@puma52 ~]$ nova volume-attach 069bd786-bd41-40ce-952f-3765cc049c20 a9cab76d-69d4-45c5-a321-bee45f89d2f6 auto
+----------+--------------------------------------+
| Property | Value |
+----------+--------------------------------------+
| device | /dev/vdb |
| id | a9cab76d-69d4-45c5-a321-bee45f89d2f6 |
| serverId | 069bd786-bd41-40ce-952f-3765cc049c20 |
| volumeId | a9cab76d-69d4-45c5-a321-bee45f89d2f6 |
+----------+--------------------------------------+
(overcloud) [stack@puma52 ~]$ cinder list
+--------------------------------------+-----------+-----------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+-----------+------+-------------+----------+--------------------------------------+
| 2d5d071a-0acf-4d1b-9e57-8b6cab762201 | available | - | 1 | tripleo | false | |
| a9cab76d-69d4-45c5-a321-bee45f89d2f6 | in-use | 3parFCVol | 1 | tripleo | true | 069bd786-bd41-40ce-952f-3765cc049c20 |
+--------------------------------------+-----------+-----------+------+-------------+----------+--------------------------------------+
Inside the instance we see the attach volume
login as 'cirros' user. default password: 'gocubsgo'. use 'sudo' for root.
inst1 login: cirros
Password:
$ sudo -i
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 253:0 0 1G 0 disk
|-vda1 253:1 0 1015M 0 part /
`-vda15 253:15 0 8M 0 part
vdb 253:16 0 1G 0 disk
|-vdb1 253:17 0 35M 0 part
`-vdb15 253:31 0 8M 0 part
Now lets check multipath on compute-1 before we notice this one:
360002ac0000000000000068f00021f6b dm-4 3PARdata,VV
size=1.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
|- 6:0:1:2 sdf 8:80 active ready running
`- 6:0:0:2 sdg 8:96 active ready running
Now we migrate the instance
(overcloud) [stack@puma52 ~]$ nova migrate 069bd786-bd41-40ce-952f-3765cc049c20 --poll
Server migrating... 100% complete
Finished
Instance now on compute-0->
(overcloud) [stack@puma52 ~]$ nova show 069bd786-bd41-40ce-952f-3765cc049c20
+--------------------------------------+----------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | compute-0.localdomain |
| OS-EXT-SRV-ATTR:hostname | inst1 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | compute-0.localdomain |
| OS-EXT-SRV-ATTR:instance_name | instance-00000001 |
inside instance we still have device connected as should be->
[root@compute-0 ~]# virsh list
Id Name State
----------------------------------------------------
1 instance-00000001 running
[root@compute-0 ~]# virsh console instance-00000001
Connected to domain instance-00000001
Escape character is ^]
login as 'cirros' user. default password: 'gocubsgo'. use 'sudo' for root.
inst1 login: cirros
Password:
$ sudo -i
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 253:0 0 1G 0 disk
|-vda1 253:1 0 1015M 0 part /
`-vda15 253:15 0 8M 0 part
vdb 253:16 0 1G 0 disk
|-vdb1 253:17 0 35M 0 part
`-vdb15 253:31 0 8M 0 part
And we notice a new added device on compute-0
360002ac0000000000000068f00021f6b dm-3 3PARdata,VV
size=1.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
|- 7:0:1:3 sdh 8:112 active ready running
`- 7:0:0:3 sdi 8:128 active ready running
Successfully migrated an instance+attached FC 3par vol from compute-1 to compute-0
Good to verify as is.
Adding another test, this time I'll migrate an instance booted from off an FC volume created from a cirros image.
(overcloud) [stack@puma52 ~]$ nova boot --flavor tiny --block-device source=image,id=4c09cae4-93ca-48d6-99ac-47dc8ce96713,dest=volume,size=2,shutdown=preserve,bootindex=0 isnt2 --nic net-id=5e72a3a9-47bd-4ba5-a94e-68823a6fa729
+--------------------------------------+-------------------------------------------------+
| Property | Value |
+--------------------------------------+-------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-SRV-ATTR:host | - |
| OS-EXT-SRV-ATTR:hostname | isnt2 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | - |
| OS-EXT-SRV-ATTR:instance_name | |
| OS-EXT-SRV-ATTR:kernel_id | |
| OS-EXT-SRV-ATTR:launch_index | 0 |
| OS-EXT-SRV-ATTR:ramdisk_id | |
| OS-EXT-SRV-ATTR:reservation_id | r-k2wcg3c7 |
| OS-EXT-SRV-ATTR:root_device_name | - |
| OS-EXT-SRV-ATTR:user_data | - |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | pd6m3Q36z4ac |
| config_drive | |
| created | 2019-06-08T04:08:03Z |
| description | - |
| flavor:disk | 1 |
| flavor:ephemeral | 0 |
| flavor:extra_specs | {} |
| flavor:original_name | tiny |
| flavor:ram | 512 |
| flavor:swap | 0 |
| flavor:vcpus | 1 |
| hostId | |
| host_status | |
| id | 2b4c0b39-3125-4609-9a3b-e07998924ce9 |
| image | Attempt to boot from volume - no image supplied |
| key_name | - |
| locked | False |
| metadata | {} |
| name | isnt2 |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| status | BUILD |
| tags | [] |
| tenant_id | aa14945a4e6e4dc8ba7471d2c7d73c06 |
| trusted_image_certificates | - |
| updated | 2019-06-08T04:08:04Z |
| user_id | b57c5d2f15b142439df17f09ef157dcd |
+--------------------------------------+-------------------------------------------------+
we see instance was booted and is live on compute-1
(overcloud) [stack@puma52 ~]$ nova show isnt2
+--------------------------------------+----------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | compute-1.localdomain |
| OS-EXT-SRV-ATTR:hostname | isnt2 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | compute-1.localdomain |
| OS-EXT-SRV-ATTR:instance_name | instance-00000003
we also see a new Cinder FC backed vol created and attached to said instance.
(overcloud) [stack@puma52 ~]$ cinder list
+--------------------------------------+-----------+-----------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+-----------+------+-------------+----------+--------------------------------------+
| a98aa925-a224-4ebb-bf2b-d6c4675683e6 | in-use | | 2 | tripleo | true | 2b4c0b39-3125-4609-9a3b-e07998924ce9 |
Now lets migrate inst2 to compute-0
(overcloud) [stack@puma52 ~]$ nova migrate isnt2 --poll
Server migrating... 100% complete
Finished
while migrating again we noticed FC multipath device vanish from compute-1 and popup on compute-0
Instance is now running on compoute-0 plus it's attached disk.
(overcloud) [stack@puma52 ~]$ nova show isnt2
+--------------------------------------+----------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | compute-0.localdomain |
| OS-EXT-SRV-ATTR:hostname | isnt2 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | compute-0.localdomain |
| OS-EXT-SRV-ATTR:instance_name | instance-00000003
(overcloud) [stack@puma52 ~]$ cinder list
+--------------------------------------+-----------+-----------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+-----------+------+-------------+----------+--------------------------------------+
| a98aa925-a224-4ebb-bf2b-d6c4675683e6 | in-use | | 2 | tripleo | true | 2b4c0b39-3125-4609-9a3b-e07998924ce9 |
Good to verify.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:1672 |