Description of problem: We have two instances mounted with the same volume, but it shouldn't be "multi-attached". How can we make sure that the volume ac0e9404-a363-4d78-bcb0-eea60bbb535a is only mounted with instance 821e8463-c265-452f-8149-4a5a1e3cf2b3? Instance USM_VTR_0429-CFMS-00(19268728-1be9-4543-8330-5c0215b6943f) attach to volume ac0e9404-a363-4d78-bcb0-eea60bbb535a Instance USM_VTR_0429-CFMA-00(821e8463-c265-452f-8149-4a5a1e3cf2b3) attach to volume ac0e9404-a363-4d78-bcb0-eea60bbb535a Server "A" $ openstack server remove volume 19268728-1be9-4543-8330-5c0215b6943f ac0e9404-a363-4d78-bcb0-eea60bbb535a Invalid volume: Invalid input received: Invalid volume: Unable to detach volume. Volume status must be 'in-use' and attach_status must be 'attached' to detach. (HTTP 400) (Request-ID: req-95e597e4-d461-4e30-8b84-f8f40ae59434) (HTTP 400) (Request-ID: req-84d5753f-c561-45a3-9970-cc5e864c2549) Server "B" $ openstack server remove volume 821e8463-c265-452f-8149-4a5a1e3cf2b3 ac0e9404-a363-4d78-bcb0-eea60bbb535a Invalid volume: Invalid input received: Invalid volume: Unable to detach volume. Volume status must be 'in-use' and attach_status must be 'attached' to detach. (HTTP 400) (Request-ID: req-0a685e0c-c947-4c6f-8aa8-dfef81ca4e09) (HTTP 400) (Request-ID: req-801aca85-b9a6-4d2e-8a2b-227c4b2f144f) $ openstack volume show ac0e9404-a363-4d78-bcb0-eea60bbb535a --fit-width +--------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +--------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | attachments | [{u'server_id': u'821e8463-c265-452f-8149-4a5a1e3cf2b3', u'attachment_id': u'f6693bfb-c113-40d7-b8e7-b9011764c92a', u'attached_at': | | | u'2020-12-04T16:42:53.000000', u'host_name': u'vl-mo-nova014.ip.example.com', u'volume_id': u'ac0e9404-a363-4d78-bcb0-eea60bbb535a', | | | u'device': u'/dev/vdb', u'id': u'ac0e9404-a363-4d78-bcb0-eea60bbb535a'}] | | availability_zone | nova | | bootable | true | | consistencygroup_id | None | | created_at | 2020-05-11T14:26:10.000000 | | description | None | | encrypted | False | | id | ac0e9404-a363-4d78-bcb0-eea60bbb535a | | migration_status | None | | multiattach | False Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: We shutdown the compute on which Instance A is running to simulate the outage. In order to do the failover of the service running on "Instance A" to "Instance B", some of our users have to manually run these commands: 1) nova volume-detach [Instance A] [Volume A] --> After running this command the volume A is in "Detaching" state forever in cinder db, so the user goes to step 2. 2) cinder reset-state --attach-status detached --state available [Volume A] --> At this point Volume A is now "Available" in cinder db, but still linked to Instance A in nova db even if the instance is currently down. 3) nova volume-attach [Instance B] [Volume A] --> Now the Volume A is attach to Instance B and linked to it if we do "openstack volume list", or volume show. If we login to instance B we can see that /dev/vdb is mounted 4) We boot the compute node, instance A goes back online. If we login to instance A we can see that /dev/vdb is not on the server. But If we run openstack server show "instance A" we see volume_attached to "Volume A" even if it's not mounted on the VM. If we run openstack server show "instance B" we see volume_attach to "Volume A". So now "instance A" have a "false" volume_attached" to "volume A" in the nova DB after doing these steps. Actual results: Expected results: Additional info:
(In reply to Cristian Muresanu from comment #0) > > Steps to Reproduce: > > We shutdown the compute on which Instance A is running to simulate the > outage. In order to do the failover of the service running on "Instance A" > to "Instance B", some of our users have to manually run these commands: So why are they not placing the instance on shared storage and evacuating the instance in this scenario? https://docs.openstack.org/api-ref/compute/?expanded=evacuate-server-evacuate-action-detail#evacuate-server-evacuate-action FWIW this is the underlying mechanism used by our downstream Instance HA feature that the customer could be using here: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/high_availability_for_compute_instances/index This evacuates the instance to a new host and restarts it if the original compute goes offline. > 1) nova volume-detach [Instance A] [Volume A] --> After running this command > the volume A is in "Detaching" state forever in cinder db, so the user goes > to step 2. Yeah that's due to n-api trying to cast to the down n-cpu, IMHO that's an issue in the detach API that I'm going to try to resolve via the following bug: [OSP 17] n-api should reject user requests to detach volumes from instances hosted on down computes https://bugzilla.redhat.com/1847958 IMHO we should be rejecting this request while the compute is down. > 2) cinder reset-state --attach-status detached --state available [Volume A] > --> At this point Volume A is now "Available" in cinder db, but still linked > to Instance A in nova db even if the instance is currently down. This isn't a valid action by an admin without also updating the Nova DB, reset-state is a cinder only admin API that only updates the cinder DB: https://docs.openstack.org/api-ref/block-storage/v3/index.html?expanded=reset-a-volume-s-statuses-detail#reset-a-volume-s-statuses As you have noted above it doesn't remove the attachment from Nova and at present we don't have a reset-state API for volume attachments in Nova. We could add one in the future but for now this means the only valid thing to do after updating this is to soft delete the associated record from the block_device_mapping table in Nova. Anyway, I've raised the following cinderclient bug to have a warning printed when this API is used as this isn't the first time we've seen it recently: cinder reset-state should print a warning to callers when moving a volume from attached to available https://bugzilla.redhat.com/show_bug.cgi?id=1907391 > 3) nova volume-attach [Instance B] [Volume A] --> Now the Volume A is attach > to Instance B and linked to it if we do "openstack volume list", or volume > show. > > If we login to instance B we can see that /dev/vdb is mounted Regardless of the fact that cinder had the volume listed as available I think this is a valid bug in Nova's attach volume API. Nova should reject the request to attach a non-multiattach volume to a second instance if it finds an active block_device_mapping record in the database for another instance. I've written this up in the following bug: [OSP 17] n-api should reject requests to attach a volume to an instance *if* an active bdm record exists https://bugzilla.redhat.com/show_bug.cgi?id=1907392 > 4) We boot the compute node, instance A goes back online. If we login to > instance A we can see that /dev/vdb is not on the server. But If we run > openstack server show "instance A" we see volume_attached to "Volume A" even > if it's not mounted on the VM. If we run openstack server show "instance B" > we see volume_attach to "Volume A". > > So now "instance A" have a "false" volume_attached" to "volume A" in the > nova DB after doing these steps. So overall this isn't a valid way to failover an instance and/or volume attachment. The customer has a number of alternatives here when the compute goes offline: - Manually orchestrate an evacuation of instance A to another host. - Configure our Instance HA feature downstream in OSP 13 to automatically orchestrate the evacuation of the instance. - Delete and recreate the instance, leading to the volume being detached and then reattached in both Cinder and Nova. Obviously only useful if application data etc persists in the volume. - Use multi-attached volumes attached to multiple ACTIVE instances across different hosts, allowing the application within the instance to handle failover. I'm leaving this open until our triage call on Friday after which I'll be closing this out as NOTABUG.
(In reply to Lee Yarwood from comment #3) > As you have noted above it doesn't remove the attachment from Nova and at > present we don't have a reset-state API for volume attachments in Nova. We > could add one in the future but for now this means the only valid thing to > do after updating this is to soft delete the associated record from the > block_device_mapping table in Nova. Anyway, I've raised the following > cinderclient bug to have a warning printed when this API is used as this > isn't the first time we've seen it recently: I forgot to note that I am looking at introducing an admin only force detach API in 17 that could be useful in this case but obviously that wouldn't be something we would backport. [OSP 17][RFE] Introduce a force detach volume API https://bugzilla.redhat.com/show_bug.cgi?id=1872882
(In reply to Lee Yarwood from comment #4) > (In reply to Lee Yarwood from comment #3) > > As you have noted above it doesn't remove the attachment from Nova and at > > present we don't have a reset-state API for volume attachments in Nova. We > > could add one in the future but for now this means the only valid thing to > > do after updating this is to soft delete the associated record from the > > block_device_mapping table in Nova. Anyway, I've raised the following > > cinderclient bug to have a warning printed when this API is used as this > > isn't the first time we've seen it recently: > > I forgot to note that I am looking at introducing an admin only force detach > API in 17 that could be useful in this case but obviously that wouldn't be > something we would backport. > > [OSP 17][RFE] Introduce a force detach volume API > https://bugzilla.redhat.com/show_bug.cgi?id=1872882 Lee, thank you so much for the detailed answered. Here are additional queries as well ~~~ We have told the user not to user anymore the command "cinder reset-state". But for now, if we want to fix the issue for this case, from what I understand we need to soft delete the associated record from the block_device_mapping table in Nova. Can you help us with this? In the meantime, i'll let our user know the different option in replacement of their failover procedure. ~~~