Bug 1698159 - [OSP16] Swap volume of multiattached volume will corrupt data
Summary: [OSP16] Swap volume of multiattached volume will corrupt data
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova
Version: 16.0 (Train)
Hardware: x86_64
OS: Linux
high
high
Target Milestone: Upstream M2
: 16.0 (Train on RHEL 8.1)
Assignee: Lee Yarwood
QA Contact: OSP DFG:Compute
URL:
Whiteboard:
Depends On:
Blocks: 1698162 1698167 1698175
TreeView+ depends on / blocked
 
Reported: 2019-04-09 17:41 UTC by Lee Yarwood
Modified: 2023-03-21 19:16 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1698162 (view as bug list)
Environment:
Last Closed: 2019-07-18 14:40:52 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 572790 0 None MERGED Block swap volume on volumes with >1 rw attachment 2020-07-06 15:02:45 UTC
Red Hat Issue Tracker OSP-23431 0 None None None 2023-03-21 19:16:03 UTC

Description Lee Yarwood 2019-04-09 17:41:36 UTC
Description of problem:

https://bugs.launchpad.net/nova/+bug/1775418

We currently permit the following:

Create multiattach volumes a and b
Create servers 1 and 2
Attach volume a to servers 1 and 2
swap_volume(server 1, volume a, volume b)

In fact, we have a tempest test which tests exactly this sequence: api.compute.admin.test_volume_swap.TestMultiAttachVolumeSwap.test_volume_swap_with_multiattach

The problem is that writes from server 2 during the copy operation on server 1 will continue to hit the underlying storage, but as server 1 doesn't know about them they won't be reflected on the copy on volume b. This will lead to an inconsistent copy, and therefore data corruption on volume b.

Also, this whole flow makes no sense for a multiattached volume because even if we managed a consistent copy all we've achieved is forking our data between the 2 volumes. The purpose of this call is to allow the operator to move volumes. We need a fundamentally different approach for multiattached volumes.

In the short term we should at least prevent data corruption by preventing swap volume of a multiattached volume. This would also cause the above tempest test to fail, but as I don't believe it's possible to implement the test safely this would be correct.

Version-Release number of selected component (if applicable):
OpenStack Train

How reproducible:
Always

Steps to Reproduce:
- Create multiattach volumes a and b
- Create servers 1 and 2
- Attach volume a to servers 1 and 2
- swap_volume(server 1, volume a, volume b)

Actual results:
Volume corruption due to multiple active R/W attachments.

Expected results:
Attempt to swap volumes is rejected.

Additional info:
https://review.openstack.org/#/q/topic:bug/1775418+(status:open+OR+status:merged)

Comment 3 Lee Yarwood 2019-07-18 14:40:52 UTC
https://bugzilla.redhat.com/show_bug.cgi?id=1698162 will track this into the OSP 15.0 release.


Note You need to log in before you can comment on or make changes to this bug.