Bug 1752560
| Summary: | Cinder retype from non-type to new type on NFS deletes the Volume | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Alan Bishop <abishop> | ||||
| Component: | openstack-cinder | Assignee: | Alan Bishop <abishop> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Tzach Shefi <tshefi> | ||||
| Severity: | high | Docs Contact: | Chuck Copello <ccopello> | ||||
| Priority: | high | ||||||
| Version: | 10.0 (Newton) | CC: | abishop, ccopello, gfidente, j.beisiegel, lmarsh, tshefi | ||||
| Target Milestone: | z14 | Keywords: | Triaged, ZStream | ||||
| Target Release: | 10.0 (Newton) | ||||||
| Hardware: | All | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | openstack-cinder-9.1.4-54.el7ost | Doc Type: | Bug Fix | ||||
| Doc Text: |
In previous releases, the Block Storage NFS volume migration code renamed the volume file on the destination NFS server to match the name on the source NFS server. The volume migration process finished by deleting the source volume. However, when retyping an NFS volume using migration, the source and destination might be the same. When you retyped an NFS volume using migration, the NFS volume file would be inadvertently deleted if the source and destination back ends were on the same NFS server.
With this release, retyping an NFS volume using migration works correctly. When you migrate an NFS volume, the volume file is renamed only when the source and destination files reside on different NFS servers.
|
Story Points: | --- | ||||
| Clone Of: | 1740560 | Environment: | |||||
| Last Closed: | 2020-04-02 10:26:00 UTC | Type: | --- | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | 1740560, 1749491, 1749493 | ||||||
| Bug Blocks: | |||||||
| Attachments: |
|
||||||
|
Description
Alan Bishop
2019-09-16 15:56:59 UTC
Alan,
Gist of things retype seams to work, however attach/clone of retyped vol fails.
Tested on:
openstack-cinder-9.1.4-53.el7ost.noarch
On Cinder backed by NFS share
Create a volume:
[stack@undercloud-0 ~]$ cinder create 5 --name EmptyVolType
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2020-03-11T10:30:53.000000 |
| description | None |
| encrypted | False |
| id | 12a2dc46-687c-41c9-a390-49cddd77bb3e |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | EmptyVolType |
| os-vol-host-attr:host | None |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 36d3464abfe0476cad8f668b2261cd8e |
| replication_status | disabled |
| size | 5 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| updated_at | None |
| user_id | bb83a7a259dc444ebf4da26964d011bc |
| volume_type | None | -> Volume type is the default "none" / not set
+--------------------------------+--------------------------------------+
[stack@undercloud-0 ~]$ cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| 12a2dc46-687c-41c9-a390-49cddd77bb3e | available | EmptyVolType | 5 | - | false | |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
Create a Legacy volume type or same nfs
[stack@undercloud-0 ~]$ cinder type-create Legacy
+--------------------------------------+--------+-------------+-----------+
| ID | Name | Description | Is_Public |
+--------------------------------------+--------+-------------+-----------+
| 13867add-bb3f-4eed-b6c2-976ef2dd9ca3 | Legacy | - | True |
+--------------------------------------+--------+-------------+-----------+
[stack@undercloud-0 ~]$ cinder type-key Legacy set volume_backend_name=nfs
[stack@undercloud-0 ~]$ cinder extra-specs-list
+--------------------------------------+--------+----------------------------------+
| ID | Name | extra_specs |
+--------------------------------------+--------+----------------------------------+
| 13867add-bb3f-4eed-b6c2-976ef2dd9ca3 | Legacy | {u'volume_backend_name': u'nfs'} |
+--------------------------------------+--------+----------------------------------+
Now lets retype the volume:
[stack@undercloud-0 ~]$ cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| 12a2dc46-687c-41c9-a390-49cddd77bb3e | available | EmptyVolType | 5 | - | false | |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
[stack@undercloud-0 ~]$ cinder retype 12a2dc46-687c-41c9-a390-49cddd77bb3e Legacy --migration-policy on-demand
[stack@undercloud-0 ~]$ cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| 12a2dc46-687c-41c9-a390-49cddd77bb3e | retyping | EmptyVolType | 5 | - | false | |
| 25977f64-829b-4abc-a60b-7f95fdd857c1 | available | EmptyVolType | 5 | Legacy | false | |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
[stack@undercloud-0 ~]$ cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| 12a2dc46-687c-41c9-a390-49cddd77bb3e | available | EmptyVolType | 5 | Legacy | false | |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
On first glance this looks OK to verify
An NFS volume of type "none"/none set was successfully retyped to Legacy volume type (same NFS server)
[stack@undercloud-0 ~]$ cinder show 12a2dc46-687c-41c9-a390-49cddd77bb3e
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2020-03-11T10:30:53.000000 |
| description | None |
| encrypted | False |
| id | 12a2dc46-687c-41c9-a390-49cddd77bb3e |
| metadata | {} |
| migration_status | success |
| multiattach | False |
| name | EmptyVolType |
| os-vol-host-attr:host | hostgroup@nfs#nfs |
| os-vol-mig-status-attr:migstat | success |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 36d3464abfe0476cad8f668b2261cd8e |
| replication_status | disabled |
| size | 5 |
| snapshot_id | None |
| source_volid | None |
| status | available |
| updated_at | 2020-03-11T10:55:20.000000 |
| user_id | bb83a7a259dc444ebf4da26964d011bc |
| volume_type | Legacy |
+--------------------------------+--------------------------------------+
But that retyped volume fails to attach/get cloned.
If I recall OSP10 didn't support NFS snapshots (added later)
Lets try to attach volume
[stack@undercloud-0 ~]$ nova list
+--------------------------------------+-------+--------+------------+-------------+-----------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------+--------+------------+-------------+-----------------------------------+
| 192ff28f-b592-4b4e-9c42-69d3beedca5c | inst1 | ACTIVE | - | Running | internal=192.168.0.16, 10.0.0.218 |
+--------------------------------------+-------+--------+------------+-------------+-----------------------------------+
[stack@undercloud-0 ~]$ nova volume-attach 192ff28f-b592-4b4e-9c42-69d3beedca5c 12a2dc46-687c-41c9-a390-49cddd77bb3e auto
+----------+--------------------------------------+
| Property | Value |
+----------+--------------------------------------+
| device | /dev/vdb |
| id | 12a2dc46-687c-41c9-a390-49cddd77bb3e |
| serverId | 192ff28f-b592-4b4e-9c42-69d3beedca5c |
| volumeId | 12a2dc46-687c-41c9-a390-49cddd77bb3e |
+----------+--------------------------------------+
So this is odd volume failed to attach
Compute logs
/var/log/nova/nova-compute.log:25098:2020-03-11 11:06:31.639 44544 ERROR nova.virt.libvirt.driver [req-de56387d-a625-4ddd-bff0-031e8f1a09be bb83a7a259dc444ebf4da26964d011bc 36d3464abfe0476cad8f668b2261cd8e - - -] [instance: 192ff28f-b592-4b4e-9c42-69d3beedca5c] Failed to attach volume at mountpoint: /dev/vdb
Lets use original volume as source for a new volume
[stack@undercloud-0 ~]$ cinder create 5 --source-volid 12a2dc46-687c-41c9-a390-49cddd77bb3e --name ClonedVolB
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2020-03-11T11:20:18.000000 |
| description | None |
| encrypted | False |
| id | 7eebc70f-36bc-46c2-8d19-239d05810690 |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | ClonedVolB |
| os-vol-host-attr:host | hostgroup@nfs#nfs |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 36d3464abfe0476cad8f668b2261cd8e |
| replication_status | disabled |
| size | 5 |
| snapshot_id | None |
| source_volid | 12a2dc46-687c-41c9-a390-49cddd77bb3e |
| status | creating |
| updated_at | 2020-03-11T11:20:18.000000 |
| user_id | bb83a7a259dc444ebf4da26964d011bc |
| volume_type | Legacy |
+--------------------------------+--------------------------------------+
This too failed
[stack@undercloud-0 ~]$ cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| 12a2dc46-687c-41c9-a390-49cddd77bb3e | available | EmptyVolType | 5 | Legacy | false | |
| 7eebc70f-36bc-46c2-8d19-239d05810690 | error | ClonedVolB | 5 | Legacy | false | |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
Lets call in the cavalry/dev,
odd I had just verified osp15 clone of ths bz, attach worked fine, so maybe just an osp10 issue?
https://bugzilla.redhat.com/show_bug.cgi?id=1749491#c5
Created attachment 1669237 [details]
cinder and nova logs
Clone fail
c-vol
2020-03-11 11:20:19.481 44251 ERROR cinder.volume.manager Traceback (most recent call last):
2020-03-11 11:20:19.481 44251 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2020-03-11 11:20:19.481 44251 ERROR cinder.volume.manager result = task.execute(**arguments)
2020-03-11 11:20:19.481 44251 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 931, in execute
2020-03-11 11:20:19.481 44251 ERROR cinder.volume.manager context, volume, **volume_spec)
2020-03-11 11:20:19.481 44251 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 493, in _create_from_source_volume
2020-03-11 11:20:19.481 44251 ERROR cinder.volume.manager model_update = self.driver.create_cloned_volume(volume, srcvol_ref)
2020-03-11 11:20:19.481 44251 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/driver.py", line 1535, in create_cloned_volume
2020-03-11 11:20:19.481 44251 ERROR cinder.volume.manager raise NotImplementedError()
2020-03-11 11:20:19.481 44251 ERROR cinder.volume.manager NotImplementedError
2020-03-11 11:20:19.481 44251 ERROR cinder.volume.manager
2020-03-11 11:20:19.493 44251 DEBUG cinder.volume.manager [req-847234a9-b02e-4ad1-ab30-7f5db770a7f6 bb83a7a259dc444ebf4da26964d011bc 36d3464abfe0476cad8f668b2261cd8e - default default] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (4d2faa81-31d8-439c-8a0b-2b73a1342c4a) transitioned into state 'REVERTING' from state 'FAILURE' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:189
Nova compute attach fail
25534 2020-03-11 11:08:31.703 44544 ERROR nova.virt.block_device [req-6d748059-916d-48b6-b831-8067e388fe07 bb83a7a259dc444ebf4da26964d011bc 36d3464abf e0476cad8f668b2261cd8e - - -] [instance: 192ff28f-b592-4b4e-9c42-69d3beedca5c] Driver failed to attach volume 12a2dc46-687c-41c9-a390-49cddd77bb 3e at /dev/vdb
25535 2020-03-11 11:08:31.703 44544 ERROR nova.virt.block_device [instance: 192ff28f-b592-4b4e-9c42-69d3beedca5c] Traceback (most recent call last):
25536 2020-03-11 11:08:31.703 44544 ERROR nova.virt.block_device [instance: 192ff28f-b592-4b4e-9c42-69d3beedca5c] File "/usr/lib/python2.7/site-pack ages/nova/virt/block_device.py", line 274, in attach
25537 2020-03-11 11:08:31.703 44544 ERROR nova.virt.block_device [instance: 192ff28f-b592-4b4e-9c42-69d3beedca5c] device_type=self['device_type'], encryption=encryption)
25538 2020-03-11 11:08:31.703 44544 ERROR nova.virt.block_device [instance: 192ff28f-b592-4b4e-9c42-69d3beedca5c] File "/usr/lib/python2.7/site-pack ages/nova/virt/libvirt/driver.py", line 1240, in attach_volume
25539 2020-03-11 11:08:31.703 44544 ERROR nova.virt.block_device [instance: 192ff28f-b592-4b4e-9c42-69d3beedca5c] self._disconnect_volume(connecti on_info, disk_dev, instance)
25540 2020-03-11 11:08:31.703 44544 ERROR nova.virt.block_device [instance: 192ff28f-b592-4b4e-9c42-69d3beedca5c] File "/usr/lib/python2.7/site-pack ages/oslo_utils/excutils.py", line 220, in __exit__
25541 2020-03-11 11:08:31.703 44544 ERROR nova.virt.block_device [instance: 192ff28f-b592-4b4e-9c42-69d3beedca5c] self.force_reraise()
25542 2020-03-11 11:08:31.703 44544 ERROR nova.virt.block_device [instance: 192ff28f-b592-4b4e-9c42-69d3beedca5c] File "/usr/lib/python2.7/site-pack ages/oslo_utils/excutils.py", line 196, in force_reraise
25543 2020-03-11 11:08:31.703 44544 ERROR nova.virt.block_device [instance: 192ff28f-b592-4b4e-9c42-69d3beedca5c] six.reraise(self.type_, self.val ue, self.tb)
25544 2020-03-11 11:08:31.703 44544 ERROR nova.virt.block_device [instance: 192ff28f-b592-4b4e-9c42-69d3beedca5c] File "/usr/lib/python2.7/site-pack ages/nova/virt/libvirt/driver.py", line 1228, in attach_volume
25545 2020-03-11 11:08:31.703 44544 ERROR nova.virt.block_device [instance: 192ff28f-b592-4b4e-9c42-69d3beedca5c] guest.attach_device(conf, persis tent=True, live=live)
25546 2020-03-11 11:08:31.703 44544 ERROR nova.virt.block_device [instance: 192ff28f-b592-4b4e-9c42-69d3beedca5c] File "/usr/lib/python2.7/site-pack ages/nova/virt/libvirt/guest.py", line 304, in attach_device
25547 2020-03-11 11:08:31.703 44544 ERROR nova.virt.block_device [instance: 192ff28f-b592-4b4e-9c42-69d3beedca5c] self._domain.attachDeviceFlags(d evice_xml, flags=flags)
25548 2020-03-11 11:08:31.703 44544 ERROR nova.virt.block_device [instance: 192ff28f-b592-4b4e-9c42-69d3beedca5c] File "/usr/lib/python2.7/site-pack ages/eventlet/tpool.py", line 186, in doit
25549 2020-03-11 11:08:31.703 44544 ERROR nova.virt.block_device [instance: 192ff28f-b592-4b4e-9c42-69d3beedca5c] result = proxy_call(self._autowr ap, f, *args, **kwargs)
25550 2020-03-11 11:08:31.703 44544 ERROR nova.virt.block_device [instance: 192ff28f-b592-4b4e-9c42-69d3beedca5c] File "/usr/lib/python2.7/site-pack ages/eventlet/tpool.py", line 144, in proxy_call
25551 2020-03-11 11:08:31.703 44544 ERROR nova.virt.block_device [instance: 192ff28f-b592-4b4e-9c42-69d3beedca5c] rv = execute(f, *args, **kwargs)
25552 2020-03-11 11:08:31.703 44544 ERROR nova.virt.block_device [instance: 192ff28f-b592-4b4e-9c42-69d3beedca5c] File "/usr/lib/python2.7/site-pack ages/eventlet/tpool.py", line 125, in execute
25553 2020-03-11 11:08:31.703 44544 ERROR nova.virt.block_device [instance: 192ff28f-b592-4b4e-9c42-69d3beedca5c] six.reraise(c, e, tb)
25554 2020-03-11 11:08:31.703 44544 ERROR nova.virt.block_device [instance: 192ff28f-b592-4b4e-9c42-69d3beedca5c] File "/usr/lib/python2.7/site-pack ages/eventlet/tpool.py", line 83, in tworker
25555 2020-03-11 11:08:31.703 44544 ERROR nova.virt.block_device [instance: 192ff28f-b592-4b4e-9c42-69d3beedca5c] rv = meth(*args, **kwargs)
25556 2020-03-11 11:08:31.703 44544 ERROR nova.virt.block_device [instance: 192ff28f-b592-4b4e-9c42-69d3beedca5c] File "/usr/lib64/python2.7/site-pa ckages/libvirt.py", line 605, in attachDeviceFlags
25557 2020-03-11 11:08:31.703 44544 ERROR nova.virt.block_device [instance: 192ff28f-b592-4b4e-9c42-69d3beedca5c] if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
25558 2020-03-11 11:08:31.703 44544 ERROR nova.virt.block_device [instance: 192ff28f-b592-4b4e-9c42-69d3beedca5c] libvirtError: Cannot access storage file '/var/lib/nova/mnt/47266020eacec99097bdec49f2451d38/volume-12a2dc46-687c-41c9-a390-49cddd77bb3e': No such file or directory
25559 2020-03-11 11:08:31.703 44544 ERROR nova.virt.block_device [instance: 192ff28f-b592-4b4e-9c42-69d3beedca5c]
Hi Tzach, Your post-retype observations are interesting, but should be treated separately from this BZ and its fix. The scope of this BZ is to ensure the retype operation doesn't result in data loss (i.e. the underlying volume being deleted). It looks like the clone failure is due to lack of snapshot support in newton/OSP-10. The nova attach issue is not great, but I don't know if this is a bug in newton or if it's associated with a "feature" in a post-newton release. The core issue is retyping a volume results in it being assigned a new UUID internally, with metadata that maps it to its original UUID. Here, it seems nova is trying to use the original UUID without looking up its mapping. Verified based on #5 and #8 Original bz issue was fixed, works as expected. For that new found Nova problem reported a new bz: https://bugzilla.redhat.com/show_bug.cgi?id=1812541 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:1301 |